locked
CRM 3.0 Issues RRS feed

  • Question

  • Hi Everyone,

     

    I've got a problem with my CRM installation that's driving me NUTS. I deperately need some help.

     

    The system is:

     

    Dual Xeon 3.2ghz, 2GB RAM, RAID 5 (6X 36GB)

     

    SBS 2003 R2 Premium SP1.

    SQL 2005 SP1

    CRM 3.0 SBE R2 with Rollup 2

    ISA 2004 SP2

     

    The server also has WSUS installed - though it's not currently operational as yet though.

     

    When I try to login to CRM or manage users (anything at all) I get an error stating MSCRMServices is unavailable.

     

    I suspect there is a problem with IIS - such as the CRMAppPool is hanging.

     

    I've got a number of IIS debugging programs running including CRM Tracing enabled with Verbos output, I can provide logs from these if they will help. One line in the CRM Trace I've noticed is the following: w3wp-MSCRMServices-xxxxx.log

     

    [2007-09-23 11:00:18.8] Process: w3wp |Thread: 3404 |Category: Platform.Sql |User: 00000000-0000-0000-0000-000000000000 |Level: Info | NotificationManager.NotificationWatcherThreadProc

     

    Notice the User is all zeros, when it's usually something like this:

     

    [2007-09-23 11:00:16.7] Process: w3wp |Thread: 8004 |Category: Platform.Sql |User: 9701b63a-ac45-4402-94a2-a355f5aebc10 |Level: Info | BusinessProcessObject.ExecuteQuery

     

    [2007-09-23 11:00:16.7] Process: w3wp |Thread: 8004 |Category: Platform.Soap |User: 9701b63a-ac45-4402-94a2-a355f5aebc10 |Level: Verbose | SoapContext.DumpInfo

     

     

    The main event log also displays these errors every 6 minutes:

     

    Event - Application Log: Category: None, Event ID: 0 (really helpful)

     

    Source: MSCRMFaxRouterService

     

    Error:

    Microsoft.Crm.Tools.FaxConnectorService.FaxSinkServiceException: Failed to initialize the fax analysis module for folder: [CRMServerUrl:'http://2W001.2way.local:5555' Folder:'C:\Program Files\Microsoft CRM\Server\bin\fax\incoming'] (Microsoft.Crm.Tools.FaxConnectorService.FaxSinkServiceException: Failed to load users for fax recognition purposes for folder: [CRMServerUrl:'http://2W001.2way.local:5555' Folder:'C:\Program Files\Microsoft CRM\Server\bin\fax\incoming'] (System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive.

    at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request)

    at System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(WebRequest request)

    at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)

    at Microsoft.Crm.Platform.Proxy.BizUser.WhoAmI()

    at Microsoft.Crm.Tools.FaxConnectorService.FaxSinkServiceCore.LoadUsers(FaxSinkServiceFolder faxSinkServiceFolder, ArrayList contactInfoArrayList, Guid& setupUserId, Guid& setupUserBusinessId)). ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive.

    at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request)

    at System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(WebRequest request)

    at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)

    at Microsoft.Crm.Platform.Proxy.BizUser.WhoAmI()

    at Microsoft.Crm.Tools.FaxConnectorService.FaxSinkServiceCore.LoadUsers(FaxSinkServiceFolder faxSinkServiceFolder, ArrayList contactInfoArrayList, Guid& setupUserId, Guid& setupUserBusinessId)

    --- End of inner exception stack trace ---

    at Microsoft.Crm.Tools.FaxConnectorService.FaxSinkServiceCore.LoadUsers(FaxSinkServiceFolder faxSinkServiceFolder, ArrayList contactInfoArrayList, Guid& setupUserId, Guid& setupUserBusinessId)

    at Microsoft.Crm.Tools.FaxConnectorService.FaxSinkServiceCore.SetupFaxAnalysisModule(FaxSinkServiceFolder faxSinkServiceFolder)). ---> Microsoft.Crm.Tools.FaxConnectorService.FaxSinkServiceException: Failed to load users for fax recognition purposes for folder: [CRMServerUrl:'http://2W001.2way.local:5555' Folder:'C:\Program Files\Microsoft CRM\Server\bin\fax\incoming'] (System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive.

    at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request)

    at System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(WebRequest request)

    at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)

    at Microsoft.Crm.Platform.Proxy.BizUser.WhoAmI()

    at Microsoft.Crm.Tools.FaxConnectorService.FaxSinkServiceCore.LoadUsers(FaxSinkServiceFolder faxSinkServiceFolder, ArrayList contactInfoArrayList, Guid& setupUserId, Guid& setupUserBusinessId)). ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive.

    at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request)

    at System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(WebRequest request)

    at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)

    at Microsoft.Crm.Platform.Proxy.BizUser.WhoAmI()

    at Microsoft.Crm.Tools.FaxConnectorService.FaxSinkServiceCore.LoadUsers(FaxSinkServiceFolder faxSinkServiceFolder, ArrayList contactInfoArrayList, Guid& setupUserId, Guid& setupUserBusinessId)

    --- End of inner exception stack trace ---

    at Microsoft.Crm.Tools.FaxConnectorService.FaxSinkServiceCore.LoadUsers(FaxSinkServiceFolder faxSinkServiceFolder, ArrayList contactInfoArrayList, Guid& setupUserId, Guid& setupUserBusinessId)

    at Microsoft.Crm.Tools.FaxConnectorService.FaxSinkServiceCore.SetupFaxAnalysisModule(FaxSinkServiceFolder faxSinkServiceFolder)

    --- End of inner exception stack trace ---

    at Microsoft.Crm.Tools.FaxConnectorService.FaxSinkServiceCore.SetupFaxAnalysisModule(FaxSinkServiceFolder faxSinkServiceFolder)

    at Microsoft.Crm.Tools.FaxConnectorService.FaxSinkServiceCore.ExecuteService()

     

     

    Can anyone help me?

     

    I would be most appreciative.

     

    Regards,

    Ben

    Sunday, September 23, 2007 1:08 AM

Answers

  • Ben

     

    As it seems to work for a while this implies there are no fundamental problems with the security configuration. To answer your question about testing SQL connectiivity under Network Service, I normally do this with an ASP .Net app running without impersonation, but that involves some developer knowledge.

     

    From the logs I've seen I'd say the problem is more to do with general IIS or AppPool stability than specifically CRM. To help work out which:

    1. When it fails, does it hang, or do you get an error message ?
    2. When it fails, can you access any other web content on the server ?
    3. When it fails, try just restarting the CRM Application pool, rather than using IISRESET. You can script this - see tip 3 on http://www.andrewconnell.com/blog/archive/2006/08/21/3882.aspx
    4. Anything in the CRMAppPool other than <ROOT> and MSCRMService ? If so, I'd move them to another application pool

    Otherwise:

    1. Any event log messages ?
    2. Anything in the SQL logs - under Management in SQL Server Mgmt Studio. And how many do you have ? A separate log file is created every time SQL is restarted. These may help work out if SQL is crashing

     

    One thing you could try is recycling the CRM application pool frequently (say every few hours) to see if that helps. It'll mess with CRM's caching and performance but may help stability.

     

    Tuesday, September 25, 2007 11:29 AM
    Moderator

All replies

  • It does sound like an IIS related issue, or possibly SQL. It would help to know what happens at IIS with any request - a sample of the IIS log for the CRM web site should show this, also any errors from the CRM trace log. And what form is the erorr message you get when accessing CRM - is it the CRM erorr dialog or a general IIS / IE message

     

    The MSCRMFaxRouterService error message is just a symptom of the underlying problem and won't help the diagnosis, nor (I think) the NotificationManager message in the trace log

     

    Monday, September 24, 2007 12:28 PM
    Moderator
  • Hi David,

     

    Thanks for your time responding, I am very grateful you have taken the time and naturally I'm hopefull you will be able to aid me in finding a resolution for this problem.

     

    The log files are fairly big, as I don't really know which parts you would be interested in or that would be of any help, the only thing I can think of is if I place them into a zip file and make them available for you to download.

     

    You can grab the zip file from here: http://www.2way.net.au/crm/CRM-IIS-Logs.zip (49mb). The Crash Dump file is huge and yet looking through Win Debug it contains nothing. Is there something I need to do or something I should enable in order to view more information from dump files?

     

    Surely these dump files aren't this big and only contain less than an single page of information?

     

    Inside the zip file there is 3 nested zips.

     

    1. The IIS Debug Log files as generated by the IIS Debug Tools (IISDump etc). This includes a Crash Dump file, I have no idea how to debug this. I've been poking around with Win Debug but have no real clue.

     

    2. The CRM Trace verbose logs. When you unzip this, just sort by date last modified to see the latest.

     

    3. The CRM Website IIS Log file, just the standard logs from C:\Windows\System32\Logfiles\W3SVC5 (5 obviously being the ID of the website)

     

     

    I'm really hoping you can help me get this one figured out...

     

    Thanks,

    Ben

     

    Monday, September 24, 2007 12:51 PM
  • Ben

     

    One thing I can see in the CRM trace log is:

     

    [2007-09-22 09:51:52.209] .... <description>Generic SQL error.</description><details>Login failed for user 'NT AUTHORITY\NETWORK SERVICE'. SQL state - 42000, Native error - 18456</details>

     

    Does the Network Service account exist as a login in SQL, with dbo rights in the MSCRM and METABASE databases ? If not, this could be your problem. What account was specified on installation for the CRM services, and have any accounts used for the CrmAppPool or any services been changed after installation.

     

    As to what is useful for logging, it's probably best to limit CRM tracing to a trace level of error, along with the website IIS logs from when you try connecting to CRM. The output from the IIS debug tools are only really useful for Microsoft to wade through.

     

    What form does the error take ? The IIS logs from around 2007-09-23 00:32:10 looks pretty normal to me

    Monday, September 24, 2007 1:48 PM
    Moderator
  • Hi David,

     

    Thanks again for your time and efforts.

     

    Does the Network Service account exist as a login in SQL - Yes.

    with dbo rights in the MSCRM and METABASE databases - Yes.

     

    What account was specified on installation for the CRM services - Network Service Account.

     

    The CRMAppPool was created by the CRM 3.0 installation and also uses the Network Service Account.

     

    The IIS logs from around 2007-09-23 00:32:10 looks pretty normal to me - These logs are probably looking normal because sometimes the error isn't occuring. I get to the point where the only way I can get into the CRM is if I do an IISReset.

     

    The longest I've had CRM function without the hangup occuring is about 2 maybe 3 days.

     

    However, for the past week at least, I've been lucky to get 24 hours.

     

    I have adjusted the Trace Level down to just Error. I'm not sure how to do that for the IIS Log though.

     

    I enable the CRMAppPool logging but can't seem to find the log file...

     

    I'll have to reset IIS again - which means the problem will probably not surface again for another 24-36hours (if I'm lucky). Hopefully the next set of log files will provide more information.

     

     - Question - are you aware of any way I can manually test SQL/DB connectivity in the context of the Network Service Account?

     

    I'm starting to wonder if I should install SQL SP2 - even though I know this will cause massive pain with CRM and Reporting Services, hopefully I can rely on the subsequent patches to address those problems.

     

    I'm reluctant to do anything with it, I'm even holding off on using the CRM and my Exchange Server until it's all sorted out, so as you can imagine, it's causing me much grief...

     

     - Once again, very grateful for your efforts.

     

    Regards,

    Ben

    Tuesday, September 25, 2007 10:51 AM
  • Ben

     

    As it seems to work for a while this implies there are no fundamental problems with the security configuration. To answer your question about testing SQL connectiivity under Network Service, I normally do this with an ASP .Net app running without impersonation, but that involves some developer knowledge.

     

    From the logs I've seen I'd say the problem is more to do with general IIS or AppPool stability than specifically CRM. To help work out which:

    1. When it fails, does it hang, or do you get an error message ?
    2. When it fails, can you access any other web content on the server ?
    3. When it fails, try just restarting the CRM Application pool, rather than using IISRESET. You can script this - see tip 3 on http://www.andrewconnell.com/blog/archive/2006/08/21/3882.aspx
    4. Anything in the CRMAppPool other than <ROOT> and MSCRMService ? If so, I'd move them to another application pool

    Otherwise:

    1. Any event log messages ?
    2. Anything in the SQL logs - under Management in SQL Server Mgmt Studio. And how many do you have ? A separate log file is created every time SQL is restarted. These may help work out if SQL is crashing

     

    One thing you could try is recycling the CRM application pool frequently (say every few hours) to see if that helps. It'll mess with CRM's caching and performance but may help stability.

     

    Tuesday, September 25, 2007 11:29 AM
    Moderator
  • Hi David,

     

    I normally do this with an ASP .Net app running without impersonation, but that involves some developer knowledge. > I've got VS2005 installed so maybe I'll try knocking something quick up when I get a moment.

     

    1. When it fails, does it hang, or do you get an error message. > This is when I receive the CRMFaxRouterService errors in the Application log every 6 minutes. In addition to that, I'm unable to do anything with CRM. The client just hangs and eventually times out. Even the CRM tools (user/deployment etc) are unresponsive.

     

    2. When it fails, can you access any other web content on the server ? > Yes. Other sites work fine - no problem. Companyweb and Default are both operational.

     

    3. When it fails, try just restarting the CRM Application pool, rather than using IISRESET. > Thanks for the tip, I've just ran the refresh now. Doing that seems to have pulled it out of it's locked state. This is definately something to do with the AppPool by the looks of things.

     

    4. Anything in the CRMAppPool other than <ROOT> and MSCRMService > No, just the 2 CRM sites are in the CRMAppPool.

     

     

    Any event log messages ?

     

    I also get these:

     

    A process serving application pool 'CRMAppPool' exceeded time limits during shut down. The process id was '4264'.

     

    A process serving application pool 'CRMAppPool' failed to respond to a ping. The process id was '5880'.

     

     

    Anything in the SQL logs - under Management in SQL Server Mgmt Studio. > I was just poking around in there, the logs appear to only be producing on-going informational logs.

     

     And how many do you have ? > Do you mean Databases? 

     

    A separate log file is created every time SQL is restarted. These may help work out if SQL is crashing > I'll spend some time looking further into the SQL logs. Admittedly I haven't spent a great deal of time with SQL logs, so this will be a bit of a learning curve for me.

     

    One thing you could try is recycling the CRM application pool frequently  > There are some settings in the CRMAppPool for recycling the worker process (in minutes), I might look into that and see if there is a recommended setting.

     

     

    Thanks for your help.

     

    Regards,

    Ben

    Tuesday, September 25, 2007 11:55 AM
  • Hi David,

     

    I thought I would report back on how things have been going.

     

    I think we have definately isolated it down to being an AppPool issue.

     

    Recyling the CRMAppPool using the command line from andrewconnell.com (%windir%\system32\cscript.exe c:\windows\system32\iisapp.vbs /a "[Name of your App Pool]" /r)  seems to fix the problem when it's happening.

     

    Only problem is, I don't want to be having to look over this or be having to run clunky scheduled jobs to essentially keep CRM functioning.

     

    Is there anything or anyone that you know of that could help resolve this issue properly now that we know roughly what needs to be looked at?

     

    I want to start offering clients CRM for their businesses, but how can I confidently do that when I'm facing issues like this myself. Until I've got a reliable, stable and error free functioning CRM, I'm not even mentioning it to a client.

     

    Your assistance has been most appreciated.

     

    Thanks

    Ben

    Thursday, September 27, 2007 11:34 AM
  • Ben

     

    I've been involved in around 40+ CRM implementations and we've never met this problem, so it's a bit unusual.

     

    I can think of 2 avenues for investigation:

    • It could be that Network Service doesn't have sufficient privileges. I've met such a scenario which caused the AppPool to crash (but would then restart OK) periodically, which was fixed by running it as LocalSystem. Note that you may have to change some SQL permissions to get this to work
    • There may be some SQL locking happening. Have a look at current activity within SQL Management Studio when the problem recurs to see if anything's being blocked

    David

    Thursday, September 27, 2007 5:06 PM
    Moderator
  • Hi David,

     

    I'm still not having any joy with this problem.

     

    I have exhausted everything possible from Google and other forums and news groups.

     

    Do you know where or who I can take this problem to for further investigation?

     

     

    Thanks

    Ben

    Saturday, September 29, 2007 11:03 PM
  • Been having the same problem - recycle the app pool would fix, then 2 hours later it would freeze again.

     

    I've just shut off the symantec antivirus autoprotect on the test server and so far it has been running for 6 hours with no glitches at all... think i need to look into the exclusions... early days though. just need to see how long it will keep running for...

    Monday, October 1, 2007 2:17 PM
  • We've suffered from this problem ever since we went live with CRM. We've only got 40 users on a way overspecced server, and I've spent so much time running diagnostics, performance monitoring, etc. CRM looks stable, SQL is fine also and way under-utilised, as is the memory, disk and ethernet resources. Yet, it all slows down almost once a day, and only an IISRESET clears it.

     

    Ultimately, its the IIS application pool that seems to be causing the CRM access to slow eventually to a stand-still - but I still think its due to poor coding in CRM.

     

    We did originally apply the following patch from MS PSS support:

    http://support.microsoft.com/kb/921065

     

    This appeared to help for a while until roll-up 2 went onto the server.

     

    I also spent some time fine-tuning the asp.net threadpool settings as per the Optimizing CRM White Paper. These have helped but again, it can be very irregular - sometimes we go for a few days without having to issue a manual IISRESET, and other times it does it twice in one day. I did script IISRESET to run at 8am and 12pm each day at one point, which alleviates manual IISRESET mostly, but even then sometimes we have to still issue a manual one.

     

    Finally, I've modified the CRM App Pool in IIS to recycle every 4 hours - not wonderful as affects performance/caching but what can ya do?

     

    Steph

     

    Wednesday, October 31, 2007 2:58 PM
  • We are experiencing the same problem on our in-house server and a client's server.  Both are SBS.  We were running IISRESET every time but have switched to just recycling the app pool.  Either reset unfreezes CRM for a while.  It seems to lock up at least daily.  We haven't found anything helpful in logs.

    Same environment in both places:  CRM 3.0 with rollup 2.  Windows 2003 Server for Small Business Server with SP1.

    SQL 2000 with SP4.

    I need to get this fixed.  Have you learned more since this thread ran in October?

     

    Thanks,

    Greg

     

     

    Monday, January 21, 2008 10:05 PM