You mean an N to 1 relationship?
Doing something with multi-selected records sounds like a workflow. The problem is with the 1. You have to present another selection to the user to determine which of the custom entity records is to be blessed with the selected set. I haven't seen that sort of interlaced functionality. This calls for a call to lookup. I'm pretty sure you can't go modal from an asynchronous process.
There was a thread on here recently on how to tie a workflow or plugin to a button. I'm guessing the more actions menu can be augmented via the ISV.config. But I still don't know how you go about stipulating the 1. It has to be a whole 'nother page.