locked
Sync knowledge size is getting out of hand -- every change is being recorded as an 'itemOverride' RRS feed

  • Question

  • I have an application I'm building which syncs notes from one device to another.  I've built a sync provider based on the generic sync provider shipped with the contacts sync sample (CTP2).

    My code uses the FeedSync provider along with my custom sync provider to synchronize notes using feed sync as a protocol.

    The code records the sync knowledge after a sync is completed, and provides it to the remote device along with the sync feed.  On subsequent syncs, the device sends the sync knowledge back -- this is used to create an 'EndPointState' which is used to create an incremental feed, so that each sync doesn't re-retrieve all data.

    Most of this is working great -- I have test cases set up which put the system through a variety of scenarios, and in each case the sync data only contains the differences needed by that endpoint.  I've put breakpoints in the code at places where sync errors might be encountered, detect conflicts, etc... all of that is working well.

    The problem is -- even after the very first sync in the system, starting with NO data, and adding data from one endpoint, the sync knowledge immediately is filled with 'itemOverride' elements.

    Further, even when 3 endpoints are involved, the <clockVector> block always conains one child.
    Shouldn't this contain a child for every endpoint?

    Here's an example of the sync knowledge:

    <idFormatGroup xmlns="http://schemas.microsoft.com/2008/02/sync">
      <replicaIdFormat isVariable="true" maxLength="128" />
      <itemIdFormat isVariable="true" maxLength="128" />
      <changeUnitIdFormat isVariable="false" maxLength="1" />
    </idFormatGroup>
    <replicaKeyMap xmlns="http://schemas.microsoft.com/2008/02/sync">
      <replicaKeyMapEntry replicaId="AHMAeQBuAGMAOgAvAC8AcwBlAHIAdgBlAHIAVQBzAGUAcgAvAEIAaQBqAGkAYgBlAG4AUwBlAHIAdgBlAHI=" replicaKey="0" />
      <replicaKeyMapEntry replicaId="AHMAeQBuAGMAOgAvAC8AcgBlAG0AbwB0AGUALQBjAGwAaQBlAG4AdABBAC8AdABlAHMAdABVAHMAZQByAEE=" replicaKey="1" />
    </replicaKeyMap>
    <clockVector xmlns="http://schemas.microsoft.com/2008/02/sync">
      <clockVectorElement replicaKey="0" tickCount="3" />
    </clockVector>
    <itemOverrides xmlns="http://schemas.microsoft.com/2008/02/sync">
      <itemOverride itemId="AGQAMQA5AGIANwAwADcAOQAtADEAYwA1ADAALQA0ADUANgAxAC0AOQA5ADcANgAtAGEANQBlADQAMwBiADUANQBmADEAYQA4">
        <feedClockVector updateCount="1" hasConflicts="false">
          <feedClockVectorElement replicaKey="0" tickCount="3" flags="0" />
          <feedClockVectorElement replicaKey="1" tickCount="2" flags="3" />
        </feedClockVector>
      </itemOverride>


    What can cause this situation?

    I've basically followed the example with the generic sync provider -- and just changed the pieces for the sync metadata, metadata provider, and code which actually saves/loads changes from the data source.


    thanks!

    • Moved by Max Wang_1983 Thursday, April 21, 2011 10:01 PM forum consolidation (From:SyncFx - Technical Discussion [ReadOnly])
    Wednesday, May 7, 2008 1:00 AM

Answers

  • Unfortunately, due to the way that metadata is represented in a FeedSync feed, any items that come from the feed have to be represented as item overrides.  When a normal sync is done, when the last batch is done, the source provider can pass a knowledge that will collapse the item overrides into a more concise representation.  Because FeedSync feeds don't contain this scope level knowledge, we can't do that last step of collapsing the knowledge.

     

    -Aaron

    Thursday, May 8, 2008 8:38 PM

All replies

  • Unfortunately, due to the way that metadata is represented in a FeedSync feed, any items that come from the feed have to be represented as item overrides.  When a normal sync is done, when the last batch is done, the source provider can pass a knowledge that will collapse the item overrides into a more concise representation.  Because FeedSync feeds don't contain this scope level knowledge, we can't do that last step of collapsing the knowledge.

     

    -Aaron

    Thursday, May 8, 2008 8:38 PM
  • "When a normal sync is done, when the last batch is done, the source provider can pass a knowledge that will collapse the item overrides into a more concise representation."

     

    What do you mean by this?

     

    I have a similar problem.  I managed to reduce the ItemOverrides to RangeOverrides by doing BeginOrderedGroup in the ChangeBatch instead of an unordered group.  However, my knowledge still ends up with 1 range override per change batch.

     

    <syncKnowledge xmlns="http://schemas.microsoft.com/2008/02/sync">

    <idFormatGroup>

    <replicaIdFormat isVariable="false" maxLength="16" />

    <itemIdFormat isVariable="false" maxLength="24" />

    <changeUnitIdFormat isVariable="false" maxLength="1" />

    </idFormatGroup>

    <replicaKeyMap>

    <replicaKeyMapEntry replicaId="0cjdEtPxTmeVZW3XwUkHzQ==" replicaKey="0" />

    <replicaKeyMapEntry replicaId="5qLXPuDhQviTvPlurvu/mw==" replicaKey="1" />

    </replicaKeyMap>

    <clockVector>

    <clockVectorElement replicaKey="0" tickCount="2" />

    </clockVector>

    <rangeOverrides>

    <rangeOverride closedLowerBound="AAAAAAAAAAHmotc+4OFC+JO8+W6u+7+b" closedUpperBound="AAAAAAAAAAXmotc+4OFC+JO8+W6u+7+a">

    <clockVector>

    <clockVectorElement replicaKey="0" tickCount="2" />

    <clockVectorElement replicaKey="1" tickCount="30" />

    </clockVector>

    </rangeOverride>

    <rangeOverride closedLowerBound="AAAAAAAAAAXmotc+4OFC+JO8+W6u+7+b" closedUpperBound="AAAAAAAAACLmotc+4OFC+JO8+W6u+7+b">

    <clockVector>

    <clockVectorElement replicaKey="0" tickCount="2" />

    <clockVectorElement replicaKey="1" tickCount="35" />

    </clockVector>

    </rangeOverride>

    </rangeOverrides>

    </syncKnowledge>

    Thursday, July 17, 2008 6:41 PM
  • Well, I think I just found it myself.

     

    ProcessChangeBatch results in the range overrides, but if I do

     

    destKnowledge.Combine(destKnowledge.MapRemoteToLocal(sourceKnowledge))

     

    the range overrides go away.

     

    Thursday, July 17, 2008 7:13 PM
  • I think the reason you see the ranges not collapsing is because the source provider is bumping up its tick count each time it is enumerating changes in GetChangeBatch.

     

    You will notice the the tickCounts for the source provider in the different ranges are 30 and 35.

     

    This may be desirable if you intend to have some sort of concurrency in your provider where data changes are allowed in between two calls of GetChangeBatch. If the data belonging to the first range has changed when reporting the changes for the second batch, then the range overrides are correct.

     

    However, if you do not allow any changes to occur while a sync session is in progress, you could make sure to not increment the local tick count of the knowledge of the source between two batches. If that happens, you will see the knowledge collapse and range overrides dissapear.

     

    You will also see the knowledge collapse if you do another (this time empty) sync frmo the source to the destination immediately following this sync, since this time there will only be one batch and range from 0 to infinity.

    Thursday, July 17, 2008 7:23 PM
    Moderator
  •  Richard Ponton wrote:

    Well, I think I just found it myself.

     

    ProcessChangeBatch results in the range overrides, but if I do

     

    destKnowledge.Combine(destKnowledge.MapRemoteToLocal(sourceKnowledge))

     

    the range overrides go away.

     

     

    You should not be doing this yourself - the engine will take care of combining the appropriate knowledges and creating item and range overrides when needed and will then give you the knowledge you need to save. By combining it yourself you may be overwriting item/range overrides for failed changes etc.

    Thursday, July 17, 2008 7:25 PM
    Moderator
  • This was the result of two seperate sync sessions with one change batch each.

     

    Thursday, July 17, 2008 7:25 PM
  • How are you beginning and ending the ranges? Are you starting with SyncIdFormat.Zero and ending with SyncIdFormat.Infinity?

     

    From the range overrides in the knowledge it looks like you start with itemid and end with itemid+1

     

    Also, are you making sure you do a changeBatch.SetLastBatch() for the last batch?

    Thursday, July 17, 2008 7:37 PM
    Moderator
  • In the source provider, I'm putting all the ItemChanges in a List<ItemChange>(C#).  Then I'm doing

     

    Code Snippet

    if (localChanges.Count > 0)

    {

    SyncId firstItemId = localChanges[0].ItemId;

    SyncId lastItemId = localChanges[localChanges.Count - 1].ItemId;

    cb.BeginOrderedGroup(firstItemId);

    cb.AddChanges(localChanges);

    cb.EndOrderedGroup(lastItemId, ReplicaKnowledge);

    }

    cb.IsLastBatch = true;

     

     

    And I just realized that if there are no changes, I'm not doing a begin and end group.  Hmmm.
    Thursday, July 17, 2008 7:44 PM
  • You should do a BeginOrderedGroup(myItemIdFormat.Zero) each time and then EndOrderedGroup(myItemIdFormat.Infinity) for the last batch. This should take care of your knowledge getting consolidated.
    Thursday, July 17, 2008 7:49 PM
    Moderator
  • Thanks!

     

    (and I sure hope that gets in the documentation for RTM Wink )

     

    Thursday, July 17, 2008 7:51 PM