I don't think this is possible. The Required and Optional attendees can only be a combination of Accounts, Contacts, Leads, Facilities/Equipment, and Users. I would recommend customizing the Appointment entity to include lookups to your custom entity which
you can then link to the Appointment. However, you will still need an underlying Contact/Lead/Account etc so set in the Required and Optional fields.
I hope this helps.