You can create a plugin and register it on create of a quote product. (pre-operation) Then in the plug-in logic get the quote and loop through the quote products and check if the product in the current quote product is already in the quote. If throw throw
an invalid plugin execution exception.