This could be an Exchange problem, or a problem with the Exchange Email router. Either way I'd start by looking at the application event log on the Exchange Server, then try determining where in the following sequence the delay is occuring:
Email comes into queue mailbox
Exchange forwarding rule forwards email to crm mailbox
Exchange email router reads email and creates email activity in CRM
One thing to watch for; the Exchange Email router can suffer massively with large emails (in the size range of 10s of MB), whcih could cause your problem