Check your spn i.e. service principal name and make sure it is there for your pool. Also check the following caching settings on your server. I have a similar deployment and I had the same issues:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings
On the Edit menu, click Add Value, and then add the following registry values:
"BypassSSLNoCacheCheck"=Dword:00000001