none
Standard custom provider - Enumerating changes using destination sync knowledge RRS feed

  • Question

  • Hi,

     I apologize in advance if this question has been asked and answered.

    I am trying to implement sync using a standard custom provider. As one of the first steps, the source provider has to provide the changes in the GetChangeBatch method based on the destination provider's SyncKnowledge. In the samples I have seen and in the documentation, the approach seems to be: iterate through all items in the store and check if each is contained in the destination knowledge. 

    This is ok when the item store is a small collection, but iterating through the item store every time when a sync sessions is initiated, doesn't seem to make sense in a Production environment when the item store may contain a large (and growing) number of items. Can you point me to a more real-world implementation of GetChangeBatch ? Am I getting this completely wrong?

    In one of the help topics, I see this note: Change enumeration can also be handled by using either a direct query on the item store or by using a combination of a query and this algorithm, if it is supported by the item store

    This is what I would like to do, i.e. use the information in the (mapped) destination knowledge to run a query on my item store, to get the changes. Can you point me to an example for this, or provide guidance?

    Thanks and regards,


    Tuesday, June 15, 2010 6:44 AM

All replies

  • have you tried using the FindMinTickCountForReplica and use that value as the lower bound value or starting point for enumerating changes in the store so you dont go thru all items?
    Tuesday, June 15, 2010 3:19 PM
    Moderator
  • If you want to implement a standard sync provider, be sure to take a look at the Metadata store component of Sync framework at http://msdn.microsoft.com/en-us/library/bb902857.aspx. Essentially, you use the API of Metadata store component to store and retrieve the sync metadata, the metadata component actually implements GetChangeBatch for you so you can delegate this task to the metadata store component.

    There is also sample for it at http://code.msdn.microsoft.com/sync with the name "Sync101 with Metadata Storage Service".

    Tuesday, June 15, 2010 5:45 PM
    Answerer
  • Hi,

    Thanks for your replies,

    @JuneT : That was my first thought, I was however put off a little by this note in the remarks for FindMinTickCountForReplica.

    "Using this method can be expensive because it searches every clock vector that is associated with replicaId."

    I will however give it a try, and design my store in to optimize.

    @Jin H: I will take a look at how SqlMetadata store does the enumeration. My impression is that this uses a SQLCE database to store the metadata. I already have a database for my item store, so I wanted to use the same database.

    Regards,

    Tuesday, June 15, 2010 5:59 PM
  • One important thing to note is that people frequently enumerate all items when getting changes because they combine change enumeration with change detection.  If you handle change detection (updating sync versions) when you modify your data, then you can typically get around this.  Otherwise, you'll have to enumerate all items at some point in order to update versions.

    One approach that people frequently take for database-like things is to keep a monotonically increasing tick count stored along-side the sync versions for your items, which you update whenever they are touched, including during sync.  Then you set this tick count as your local tick count in knowledge.  When you see the destination knowledge, you can do a FindMinTickCountForReplica for the source replica id.  This will search every clock vector in the destination knowledge, which is typically only one, but will almost always be less than your number of items (if you don't send items in ranges, the number of vectors will grow with each request and be collapsed at the end of the sync).  The important thing to note is that using this approach will give you a subset of the total items for which you will still have to check the versions.

    The trick to all of this is having a metadata store which you can query efficiently in this manner.  Otherwise, you're not going to do much better than looping through every item.

    Aaron


    SDE, Microsoft Sync Framework
    Tuesday, June 15, 2010 6:40 PM
    Answerer