How are you getting CRM to communicate with Azure and back to the on-premises Active Directory? Azure Active Directory isn't supported by CRM on-premises. Is the Azure-hosted ADFS a proxy for your local ADFS?
In all, I don't believe that is a supported configuration, or even a logical one without more information.
The postings on this site are solely my own and do not represent or constitute Hitachi Solutions' positions, views, strategies or opinions.