locked
ChangeNeedsKnowledgeException on ItemChange serialization, ItemChange from GetLocalVersions loses knowledge RRS feed

  • Question


  • Background:
    I wrote KnowledgeSyncProvider based providers (very similar to the examples) that communicate over HTTP with a REST based webservice. All logic is retained in the provider the same as in the Sync Framework example but instead of accessing the metadata store directly I preform what could be considered a remote procedure call over the HTTP link to the service which actually houses the metadata store.

    Consider this code from the examples:

    [code]
            // Apply the changes.
            public override void ProcessChangeBatch(ConflictResolutionPolicy resolutionPolicy, ChangeBatch sourceChanges,
                object changeDataRetriever, SyncCallbacks syncCallback, SyncSessionStatistics sessionStatistics)
            {
                BeginUpdates();

                // Get all my local change versions from the metadata store.
                IEnumerable<ItemChange> localChanges = _metadata.GetLocalVersions(sourceChanges);

                // Create a change applier object to make change application easier.
                // This makes the engine call me when it needs data and when I should save data.
                NotifyingChangeApplier changeApplier = new NotifyingChangeApplier(_idFormats);
                changeApplier.ApplyChanges(resolutionPolicy, sourceChanges, changeDataRetriever as IChangeDataRetriever, localChanges, _metadata.GetKnowledge(),
                    _metadata.GetForgottenKnowledge(), this, _currentSessionContext, syncCallback);

                EndUpdates();
            }
    [/code]

    My code is relatively similar but sends the obejcts over HTTP and back:
    [code]
            //Change application!
            public override void ProcessChangeBatch( ConflictResolutionPolicy resolutionPolicy ,
                                                     ChangeBatch sourceChanges                 ,
                                                     object changeDataRetriever                ,
                                                     SyncCallbacks syncCallback                ,
                                                     SyncSessionStatistics sessionStatistics   )
            {
                BeginUpdates();

                // Get all my local change versions from the metadata store
                IEnumerable<ItemChange> localChanges = RetrieveSyncLocalChanges( m_restFeedUri, sourceChanges );
               
                // Retrieve some data from our 'local' metadata store trough the REST interface
                SyncKnowledge localKnowledge = RetrieveSyncKnowledge( m_restFeedUri ); 
                ForgottenKnowledge localForgottenKnowledge = RetrieveSyncForgottenKnowledge( m_restFeedUri );
               
                // Create a changeapplier object to make change application easier (make the engine call me
                // when it needs data and when I should save data). We let the Sync framework drive the decision making
                // while we just provide the communication conduit.
                NotifyingChangeApplier changeApplier = new NotifyingChangeApplier( m_idFormats );
                changeApplier.ApplyChanges( resolutionPolicy                            ,
                                            sourceChanges                               ,
                                            changeDataRetriever as IChangeDataRetriever ,
                                            localChanges                                ,
                                            localKnowledge                              ,
                                            localForgottenKnowledge                     ,
                                            this                                        ,
                                            m_currentSessionContext                     ,
                                            syncCallback                                );

                EndUpdates();
      
      [/code]

    As you can see the code of the sample and my code are very similar with the exception that instead of acecssing the metadata store directly locally I use "Retrieve*" functions to trigger the metadata store call remotely over HTTP and serialize/deserialize the result which is retuned by those functions. This ability is a key feature for my providers to be able to genericly interact with our REST based webservices.

    On the service itself I have the following code:

    [code]
                        ChangeBatch remoteChangeBatch = SyncChangeBatchStreamer.DeserializeStreamToChangeBatch( stream, m_metaDataAccess.IdFormats );
                       
                        if ( null == remoteChangeBatch )
                        {
                            // Fatal: we need the change batch to proceed
                            return new DeserializeResult( DeserializeState.CorruptInput );
                        }
                       
                        // Retrieve a list of resources metadata that has been altered remotely for comparison
                        IEnumerable< ItemChange > localChanges = m_metaDataAccess.GetLocalVersions( remoteChangeBatch );
                       
                        // Serialize the list of local changes into a stream and send it back as the result of the deserialize action
                        Stream localChangesStream = SyncLocalChangesStreamer.SerializeLocalChangesToStream( localChanges );
                        localChangesStream.Seek( 0, SeekOrigin.Begin );
                        return new DeserializeResult( DeserializeState.Succeeded, localChangesStream, Representations.SyncLocalChangesRepresentationType );
    [/code]

    the "localChanges" are obtained without a problem but the ItemChange objects have null for their knowledge references. My streamer internally calls Serialize on the ItemChange objects to serialize them which then triggers a "ChangeNeedsKnowledgeException". The ChangeBatch variable "remoteChangeBatch" Is reconstructed at the service without any problems and contains all the knowledge refrences. I have no idea why a call to GetLocalVersions with a seemingly correct ChangeBatch would cause non-serializable ItemChange objects to be created or why their knowledge refrences are missing in the first place.

    Can anyone shed some light on what might be going on here?

    I've been trying to debug the problem for some time but the fact that most calls go through COM doesn't help matters. I use the binary Serialize/Deserialize methods since the XML ones are broken to begin with. Any idea when those will be fixed?

    • Moved by Max Wang_1983 Thursday, April 21, 2011 5:56 PM forum consolidation (From:SyncFx - Technical Discussion [ReadOnly])
    Friday, January 23, 2009 7:16 PM

Answers

  • 0 is actually replicaKey of local replicaId in local replica knowledge's ReplicaKeyMap but not local replicaId itself. ReplicaIdKeyMap is a container that has pairs of keys(uint) and actual replicaIds(byte[]) in sync community. Local replicaId's key is always 0 as it initialize the local knowledge and different replicas have different ReplicaKeyMaps.

     

    For example:

    At time 0:

    Soure Replica: replicaId = "ReplicaA". Replica's knowledge is {(0:10)}.

    It has Item1 with createVersion = (0:1), currentVersion = (0:10).

    In knowledge's ReplicaKeyMap, 0 is mapped to "ReplicaA".

     

    Destination Replica: replicaId = "ReplicaB". Replica's knowledge is {(0:1)}. It's empty.

     

    At time 1 after destination synced from source replica.

    Destination knowledge becomes {(0:1), (1, 10)}. 

    It has Item1 with createVersion = (1:1), currentVersion = (1:10).

    In knowledge's ReplicaKeyMap, 0 is mapped to "ReplicaB" and 1 is mapped to "ReplicaA".

    Thursday, January 29, 2009 12:28 AM

All replies


  • Workaround:
    I got around this problem by writing my own serialization/deserialization code for LocalChange objects
    . I read the public properties of the objects in my streaming code and turn it into an entry in a XML document per LocalChange. I stream this XML doc which I parse on the other end using my own code and use the public constructor of LocalChange to recreate the object.

    Next problem:
    I now get a different exception at the next step of the process. I'm not sure whether this is unrelated or due to my workaround.
    The exception reads as follows:

    failed: Microsoft.Synchronization.ChangeVersionNotInKnowledgeException : Change version was not contained in knowledge as expected.

    [Output]

      ----> System.Runtime.InteropServices.COMException : Change version was not contained in knowledge as expected.
        at Microsoft.Synchronization.NotifyingChangeApplier.ISynchronousNotifyingChangeApplierImpl.ApplyChanges(CONFLICT_RESOLUTION_POLICY resolutionPolicy, ISyncChangeBatch pSourceChanges, Object pUnkDataRetriever, IEnumSyncChanges pDestinationVersions, ISyncKnowledge pDestinationKnowledge, IForgottenKnowledge pDestinationForgottenKnowledge, ISynchronousNotifyingChangeApplierTarget pChangeApplierTarget, ISyncSessionState pSessionState, ISyncCallback pCallback)
        at Microsoft.Synchronization.NotifyingChangeApplier.ApplyChanges(ConflictResolutionPolicy resolutionPolicy, ChangeBatch sourceChanges, IChangeDataRetriever changeDataRetriever, IEnumerable`1 destinationVersions, SyncKnowledge destinationKnowledge, ForgottenKnowledge destinationForgottenKnowledge, INotifyingChangeApplierTarget changeApplierTarget, SyncSessionContext syncSessionState, SyncCallbacks syncCallback)
        H:\SVNDepot\PII-Vision\Sync\REST\software\Phase1\Refactored\Rest.Sync\SimpleRestSyncProvider.cs(561,0): at Rest.Sync.SimpleRestSyncProvider.ProcessChangeBatch(ConflictResolutionPolicy resolutionPolicy, ChangeBatch sourceChanges, Object changeDataRetriever, SyncCallbacks syncCallback, SyncSessionStatistics sessionStatistics)
        at Microsoft.Synchronization.KnowledgeProviderProxy.ProcessChangeBatch(CONFLICT_RESOLUTION_POLICY resolutionPolicy, ISyncChangeBatch pSourceChangeManager, Object pUnkDataRetriever, ISyncCallback pCallback, _SYNC_SESSION_STATISTICS& pSyncSessionStatistics)
        at Microsoft.Synchronization.CoreInterop.ISyncSession.Start(CONFLICT_RESOLUTION_POLICY resolutionPolicy, _SYNC_SESSION_STATISTICS& pSyncSessionStatistics)
        at Microsoft.Synchronization.KnowledgeSyncOrchestrator.DoOneWaySyncHelper(SyncIdFormatGroup sourceIdFormats, SyncIdFormatGroup destinationIdFormats, KnowledgeSyncProviderConfiguration destinationConfiguration, SyncCallbacks DestinationCallbacks, ISyncProvider sourceProxy, ISyncProvider destinationProxy, Int32& changesApplied, Int32& changesFailed)
        at Microsoft.Synchronization.KnowledgeSyncOrchestrator.DoOneWayKnowledgeSync(SyncProvider sourceProvider, SyncProvider destinationProvider, Int32& changesApplied, Int32& changesFailed)
        at Microsoft.Synchronization.KnowledgeSyncOrchestrator.Synchronize()
        at Microsoft.Synchronization.SyncOrchestrator.Synchronize()
        H:\SVNDepot\PII-Vision\Sync\REST\software\Phase1\Refactored\Rest.Sync.Test\Utils.cs(32,0): at Rest.Sync.Test.SyncTester.Synchronize()
        H:\SVNDepot\PII-Vision\Sync\REST\software\Phase1\Refactored\Rest.Sync.Test\UnitTests.cs(159,0): at Rest.Sync.Test.SyncUnitTests.TestA()
        --COMException
        at Microsoft.Synchronization.CoreInterop.ISynchronousNotifyingChangeApplier.ApplyChanges(CONFLICT_RESOLUTION_POLICY resolutionPolicy, ISyncChangeBatch pSourceChanges, Object pUnkDataRetriever, IEnumSyncChanges pDestinationVersions, ISyncKnowledge pDestinationKnowledge, IForgottenKnowledge pDestinationForgottenKnowledge, ISynchronousNotifyingChangeApplierTarget pChangeApplierTarget, ISyncSessionState pSessionState, ISyncCallback pCallback)
        at Microsoft.Synchronization.NotifyingChangeApplier.ISynchronousNotifyingChangeApplierImpl.ApplyChanges(CONFLICT_RESOLUTION_POLICY resolutionPolicy, ISyncChangeBatch pSourceChanges, Object pUnkDataRetriever, IEnumSyncChanges pDestinationVersions, ISyncKnowledge pDestinationKnowledge, IForgottenKnowledge pDestinationForgottenKnowledge, ISynchronousNotifyingChangeApplierTarget pChangeApplierTarget, ISyncSessionState pSessionState, ISyncCallback pCallback)

    [/Output]

    MSDN states:
    "Remarks

    This exception typically indicates a problem in the source provider. It is thrown when the source provider creates a change batch that contains a change that is not contained in the made-with knowledge for the change batch."


    What would be a solution/workaround for this problem?




    Monday, January 26, 2009 7:45 PM

  • I modified the code in the provider not to use the local changes obtained from the server to see if the local changes passed into the change applier is what is causing the
    ChangeVersionNotInKnowledgeException. See the code below:

    [code]
                // Create a changeapplier object to make change application easier (make the engine call me
                // when it needs data and when I should save data). We let the Sync framework drive the decision making
                // while we just provide the communication conduit.
                NotifyingChangeApplier changeApplier = new NotifyingChangeApplier( m_idFormats );
                changeApplier.ApplyChanges( resolutionPolicy                            ,
                                            sourceChanges                               ,
                                            changeDataRetriever as IChangeDataRetriever ,
                                            //localChanges                                ,
                                            destinationKnowledge                        ,
                                            destinationForgottenKnowledge               ,
                                            this                                        ,
                                            m_currentSessionContext                     ,
                                            syncCallback                                );
    [/code]

    I get the same
    ChangeVersionNotInKnowledgeException whether localChanges is commented out or not.
    I'm still suspecting something that is needed at this point is either lost or incorrectly configured due to the serialization/deserialization of Sync Framework objects.

    Any help would be appreciated
    Tuesday, January 27, 2009 5:02 PM
  • "Destination" changes (localChanges in your example) are not supposed to be serialized and deserialized during a sync operation. You would have to retrieve your "localChanges" through a metadata store instance that is local to your destinantion provider just like the way in code sample instead of going through the REST webservice.

     

    Is there any reason that you must use REST webservice on your destination provider? Source provider and destination provider don't have to share a same metadata store instance. They can have their own metadata store instance.

     

    Hope this helps,

    Tuesday, January 27, 2009 8:27 PM

  • Because we want to handle all CRUD of our data through a single communication channel (the REST based webservice) we moved the metadata store to the service we are sync'ing. That way the service can record all CUD operations in the metadata store as they occur. So we have two services each running on a seperate box, each with a generic API with some sync logic code in front of it which handles CUD updates and dispatching sync objects that would otherwise be retrieved locally from the metadata store. The latter is how we get the data to our provider code which runs on a seperate box as well.

    So your typical example version is:

    Sync logic -> Provider -> metadatastore   (source)
                    -> Provider ->
    metadatastore  (destination)

    What we have is the same except we in essense just have a HTTP connection between the provider and the metadatastore:

    Sync logic -> Provider -> HTTP -> sync adapter -> metadatastore  (source)
                    -> Provider -> HTTP -> sync adapter -> metadatastore (destination)

    So we do have multiple metadata stores and the only difference with most of your examples is that we access the metadatastore(s) from a remote location. This is why we serialize/deserialize the Sync objects based between the provider API and the metadatastore API.

    The local changes are obtained from the destination sync provider over HTTP. This is the exact same as your example code with the only difference being the fact that the data is retrieved from a metadatastore that is not on the same box as the provider logic.

    In terms of deployment we want to run 1 box with the Sync runtime and our 2 REST compatible providers and then have the REST services we are syncing on seperate boxes themselves (which also house the metadata storege for changes relevant to that service).

    As long as serialization/deserialization of the sync objects used by providers is implemented correctly this scenario should work like a charm.

    So am I understanding you correctly that you are saying this configuration is fundementally impossible? Mainly due to non-operational serialization/deserialization functionality?


    Wednesday, January 28, 2009 7:52 PM

  • We made some more progress,.. we added callback handler implementations and all which fixed some problems but the main change that makes or brakes the functionality is the value used for the Replica ID. If the uint used in SyncVersion objects is the string name of the replica turned into a uint or whatever other scheme we come up with it fails. If we hardcode the Replica ID to 0 it works. This is the same as in the examples but it makes no sense to me. Should the ReplicaID uint value not be the key mapped version of the string name we gave the replica instead of always 0???
    Why do you always use 0 for the replicaID in the examples?
    Thursday, January 29, 2009 12:05 AM
  • 0 is actually replicaKey of local replicaId in local replica knowledge's ReplicaKeyMap but not local replicaId itself. ReplicaIdKeyMap is a container that has pairs of keys(uint) and actual replicaIds(byte[]) in sync community. Local replicaId's key is always 0 as it initialize the local knowledge and different replicas have different ReplicaKeyMaps.

     

    For example:

    At time 0:

    Soure Replica: replicaId = "ReplicaA". Replica's knowledge is {(0:10)}.

    It has Item1 with createVersion = (0:1), currentVersion = (0:10).

    In knowledge's ReplicaKeyMap, 0 is mapped to "ReplicaA".

     

    Destination Replica: replicaId = "ReplicaB". Replica's knowledge is {(0:1)}. It's empty.

     

    At time 1 after destination synced from source replica.

    Destination knowledge becomes {(0:1), (1, 10)}. 

    It has Item1 with createVersion = (1:1), currentVersion = (1:10).

    In knowledge's ReplicaKeyMap, 0 is mapped to "ReplicaB" and 1 is mapped to "ReplicaA".

    Thursday, January 29, 2009 12:28 AM