Then you'll need to sync all active contacts from CRM to Outlook and not just My Active Contacts.
https://www.microsoft.com/en-US/dynamics/crm-customer-center/choose-the-records-to-synchronize-between-crm-and-outlook-or-exchange.aspx
Regards