locked
Sync Files to Azure Blob Storage - RCW error RRS feed

  • Question

  • Hey all,

    I would like to sync the files in file folders on multiple client's computers with Azure blob storage (separate containers for each client).  This also needs to implement user authentication and security, so WCF is needed (so the code sample "Synchronizing Files to Windows Azure Storage" is just not going to work).

    I have created my own KnowledgeSyncProvider to handle the Azure blob storage, and I have created the WCF service (including the proxy for the client).  The Sync metadata store for Azure blob storage needs to be in the cloud, so it is being saved to an Azure drive.  I have tested the whole thing, and everything works great on my computer.  I have also deployed the service to Azure, and it also works in the cloud.

    There is one problem though, and I can find absolutely nothing about it.  Here is the issue: 

    A client synchronizes with Azure Blob Storage and that completes 100% successfully.  The client then turns around and syncs with the cloud again.  If the second sync starts a couple seconds after the first sync finishes, then an error is returned to the client - the error is:

    An attempt has been made to free an RCW that is in use.  The RCW is in use on the active thread or another thread.  Attempting to free an in-use RCW can cause corruption or data loss.

    I know that this error has to do with the fact that the Sync framework relies on COM objects (such as synchronization21.dll), which are deployed to the cloud in a side-by-side assembly.  I think the error is being generated during the SqlMetadataStore.OpenStore call, so that would point to the COM object for metadata storage or one of the sqlce helper dll's.  I have tried in vain to reproduce the error on my own computer with no luck, so I am wondering if anyone on the Sync team knows anything about this error in the metadata store COM object.

    I should also point out that if the client waits for a little while before starting the second sync attempt, then the sync finishes successfully.  The only real workaround I can come up with is to program in a time interval to wait after a sync with the cloud before another sync can be performed.  I have also confirmed the issue is not a concurrency issue.

    Any help would be appreciated - thx.

    Wednesday, October 27, 2010 6:53 PM

Answers

  • Whoo Hooo!!! I have found the exact problem (although I don't understand it).  The problem has something to do with the Sync Framework and WCF.

    In the web.config file where the WCF service is defined, I have configured the wsHttpBinding for the WCF service as follows (a little simplified to highlight the exact problem) - this configuration causes the RCW errors mentioned above:

    <bindings>
    	<wsHttpBinding>
    		<binding name="AzureBlobSyncBinding" closeTimeout="00:01:00" openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00" maxReceivedMessageSize="73400320" messageEncoding="Mtom">
    			<readerQuotas maxArrayLength="73400320"/>
    			<reliableSession enabled="true"/>
    		</binding>
    	</wsHttpBinding>
    </bindings>
    
    

    Now if I disable the reliableSession as follows, then I never get RCW errors:

    <bindings>
    	<wsHttpBinding>
    		<binding name="AzureBlobSyncBinding" closeTimeout="00:01:00" openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00" maxReceivedMessageSize="73400320" messageEncoding="Mtom">
    			<readerQuotas maxArrayLength="73400320"/>
    			<reliableSession enabled="false"/>
    		</binding>
    	</wsHttpBinding>
    </bindings>
    
    

    So the bottom line is that enabling reliableSession for the WCF service over which Sync runs causes a RCW error in the Sync metadata store calls.  Anyone have any idea why?

    Thursday, October 28, 2010 3:58 PM
  • Hi,

    WCF Reliable Session has its own threadpool, and it sometimes try to re-fetch some SyncFx objects in order for its internal retries. At that moment, the particular SyncFx object might be disposed already in the native layer. We have saw this kind of issue with other sync provider with WCF Reliable Session.

    If Reliable Session is not necessary to your scenario. Please don't enable it. Otherwise, please pay attention to two points:

    1. In Knowledge Sync Provider WCF client Proxy, please add below line in ProcessCahngeBatch(...) as the first line to avoid changeBatch be disposed immediately after SyncOrchestrator completes ProcessChangeBatch(...)

              sourceChanges.DisposeAfterProcessing = false;

    2. WCF Reliable Session is running in MTA (multiple-thread apartment), your client app need be MTA as well because SyncFx native ojects cannot be automatically transfered between STA and MTA.

    Thanks,
    Dong


    This posting is provided AS IS with no warranties, and confers no rights.
    Thursday, October 28, 2010 6:17 PM

All replies

  • I have seen that error before, though in different circumstances, I'm not convinced it's directly related to the metadata store. However, you might want to look at the KnowledgeSyncProvider.EndSession Method and make sure the metadata store is disposed properly at the end of the synchronisation.

    As an aside, for anyone that doesn't already know and RCW is a Runtime Callable Wrapper.

    Wednesday, October 27, 2010 7:46 PM
  • Thanks for the response.  I have been able to reproduce the error on my computer now - I am still trying to figure out where the error is occurring.  The RCW error occurs before the EndSession method.  Of course, if I stop debugging the client at that point, then the EndSession method isn't called and the metadata store is left open.  I have got the metadata store dispose method covered - any other ideas?

    I never had this error when going against the registered Sync dll's.  This error cropped up when implementing the side-by-side assembly setup in preparation for Azure deployment -- see the document "How to: Deploy Sync Framework to Windows Azure" for what I am talking about.

    Wednesday, October 27, 2010 8:05 PM
  • The error definitely has to do with the metadata store. 

    I get the above RCW error in the BeginSession method, and there are only two functions that could be responsible -- SqlMetadataStore.OpenStore or the _metadataStore.GetReplicaMetadata. 

    If I go through the RCW error, then I get a second error in the EndSession method.  The second error is:

    COM object that has been separated from its underlying RCW cannot be used.

    I only have two lines of code in the EndSession method -- _metadataStore.Dispose() and then _metadataStore = null.

    If I get the errors and manage to go through them, then the files successfully sync.  So it definitely seems that the error is occuring because of a COM call related to the metadata store.

    Wednesday, October 27, 2010 8:57 PM
  • Hi, Brian.

    a quick search online got me into this article - http://forums.asp.net/p/1130334/2620870.aspx and based on what your workarround is (with waiting some more time) I think what's going on is that the first sync has not finished completely (perhaps the objects are kept in memory) when the second sync request kicks in.

    And based on what I've found online, the error message is displayed when a thread tries to use an object which was created by another thread.

    Let me know if this helps, otherwise we can look further into the issue.

    Thanks.

    Adrian

    Wednesday, October 27, 2010 11:55 PM
  • Hi Brian,

    If the exception is thrown  due to usage of RCW which is in use on the active\main thread, can you try initiating sync operation on a new thread each time.   I did not try this option though. Let me know if this helps.

    Thanks,
    Ramesh V

    Thursday, October 28, 2010 1:48 AM
  • Adrian

    Thanks for the response.  I had not seen the post you mentioned, and it is close to what I describe.  None of the possible solutions apply though (I am not doing threading, and it isn't a multiple client concurrency issue).

    I have some further observations.  First, I completely backed out the Sync side-by-side assembly for Azure deployment and went back to using the standard registered Sync components on my computer.  I get the exact same RCW errors.  Second, I went back to saving the metadata store in a regular folder on my computer rather than using an Azure drive (simulated in development), and I had the same issue.  So I am pretty well convinced that the problem is not related to Azure.

    The metadata store is implemented almost exactly as in the sample "Sync 101 with Remote Change Application over WCF".  In fact, 95% of the code is verbatim from the samples.

    I would appreciate any help you can give - I am stumped.

    Thursday, October 28, 2010 1:48 PM
  • Whoo Hooo!!! I have found the exact problem (although I don't understand it).  The problem has something to do with the Sync Framework and WCF.

    In the web.config file where the WCF service is defined, I have configured the wsHttpBinding for the WCF service as follows (a little simplified to highlight the exact problem) - this configuration causes the RCW errors mentioned above:

    <bindings>
    	<wsHttpBinding>
    		<binding name="AzureBlobSyncBinding" closeTimeout="00:01:00" openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00" maxReceivedMessageSize="73400320" messageEncoding="Mtom">
    			<readerQuotas maxArrayLength="73400320"/>
    			<reliableSession enabled="true"/>
    		</binding>
    	</wsHttpBinding>
    </bindings>
    
    

    Now if I disable the reliableSession as follows, then I never get RCW errors:

    <bindings>
    	<wsHttpBinding>
    		<binding name="AzureBlobSyncBinding" closeTimeout="00:01:00" openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00" maxReceivedMessageSize="73400320" messageEncoding="Mtom">
    			<readerQuotas maxArrayLength="73400320"/>
    			<reliableSession enabled="false"/>
    		</binding>
    	</wsHttpBinding>
    </bindings>
    
    

    So the bottom line is that enabling reliableSession for the WCF service over which Sync runs causes a RCW error in the Sync metadata store calls.  Anyone have any idea why?

    Thursday, October 28, 2010 3:58 PM
  • Hi,

    WCF Reliable Session has its own threadpool, and it sometimes try to re-fetch some SyncFx objects in order for its internal retries. At that moment, the particular SyncFx object might be disposed already in the native layer. We have saw this kind of issue with other sync provider with WCF Reliable Session.

    If Reliable Session is not necessary to your scenario. Please don't enable it. Otherwise, please pay attention to two points:

    1. In Knowledge Sync Provider WCF client Proxy, please add below line in ProcessCahngeBatch(...) as the first line to avoid changeBatch be disposed immediately after SyncOrchestrator completes ProcessChangeBatch(...)

              sourceChanges.DisposeAfterProcessing = false;

    2. WCF Reliable Session is running in MTA (multiple-thread apartment), your client app need be MTA as well because SyncFx native ojects cannot be automatically transfered between STA and MTA.

    Thanks,
    Dong


    This posting is provided AS IS with no warranties, and confers no rights.
    Thursday, October 28, 2010 6:17 PM
  • Dong,

    Thanks for the explanation.  Just finished testing without Reliable Sessions, and all seems to work.  That info should be in the documentation somewhere with the WCF examples.

    Friday, October 29, 2010 7:04 PM