locked
Any examples of custom MetadataStores? RRS feed

  • Question

  • After inheriting both MetadataStore and ReplicaMetadata, I'm left with about 30 (literally) method stubs and only a rough idea of what each one of them does. Are there any examples of custom metadata stores?

    After diving pretty deep into the Sync Framework the past 4 days (all day), I'm getting the impression the level of abstraction needed to implement it completely obfuscates actually getting it to function (beyond the default providers) and mitigates any utility it would provide.

    I've watched every video I could find on and downloaded all the examples, and they all make the inaccurate assumptions I know what all these new terms mean (ReplicaID, ChangeDataRetriever) and make it all look so easy "all you do is implement ProcessChangeBatch passing in the source's knowledge provider and call Sync, badda bing, badda boom", then in the presentation's code examples they use variables and we're just suppose to just magically know where those came from. Then in sample code, as opposed to actually clarifying to the the reader as to what the code is actually doing, it basically just comments on what the code says ("increment the SyncTick"), well as someone who didn't write the Sync Framework - what's a "SyncTick" and why are you incrementing it. It's painfully obvious the person who wrote the presentations and samples was someone who was extemely close to the actual writing of the Sync Framework.

    They say with synchronization, "the devil is in the details"...Sync Framework seems to have just shifted those details, not removed them.

    -josh
    • Moved by Max Wang_1983 Thursday, April 21, 2011 5:27 PM forum consolidation (From:SyncFx - Technical Discussion [ReadOnly])
    Wednesday, November 26, 2008 4:07 PM

Answers

  • Hi Josh,

     

     Josh Jonte wrote:
    I get the impression an instance of a metadata store is optional?  So is a metadata store "object" not required when writing a Knowledge Sync Provider?  All the examples use i like it's a required piece of the puzzel? Not the case?

     

    That's right, using our metadata store is optional. It's a good idea if you can use it because it implements a lot of functionality common across providers, but for syncing databases when you keep metadata in the same database I wouldn't really advise it. It will be more work because you now have to implement GetChangeBatch and get the destination item versions for the ProcessChangeBatch call yourself, as well as some other smaller things (storing knowledge, doing change detection, etc.). But it's by no means a required component. Some of our samples do use the metadata store but I'm pretty sure we have at lease one generic sample which uses its own in-memory metadata store without using or implementing MSF's metadata store interfaces.

     

    Also, the source code for our metadata store is not available publicly, sorry.

     

    thanks,

    Lukasz

    Friday, December 5, 2008 7:42 PM

All replies

  • Hey Josh,

     

    You are absolutely right that with sync, the devil is in the details. And yes, writing knowledge providers requires a fair bit of familiarity with the sync problem space. I think this: http://msdn.microsoft.com/en-us/sync/bb821992.aspx is a fairly useful overview and explanation of the overall process if you haven't seen it already. But you probably have already done a fair bit of reading.

     

    Having said that I'd also say that part of the problem here is that you're trying to tackle a fairly advanced scenario - re-implementing our metadata store interfaces. It's hard to say without knowing what is your end goal but you shouldn't have to do this for most scenarios. This is probably our fault in not explaining very well what the available pieces are and how you put them together. Let me try to give you a brief overview.

     

    Ultimately you are responsible for putting together a sync provider which plugs into MSF's sync session and talks to the other provider through our interfaces. If you can find a provider that already does what you need, you only need to write a small sync app that glues everything together. If you're trying to sync databases or files, you probably don't need to write a provider yourself. For custom data stores, keep reading. In V1, there's one basic type of provider, a knowledge sync provider. As you know a sync provider's responsibilities are maintaining the sync metadata and communicating with the data store. You can write a knowledge sync provider that stores its sync metadata in a custom store without using any additional MSF components. Alternatively, you can use MSF's metadata store to take the task of storing metadata off your shoulders. Our metadata store provides all basic operations that the provider typically uses when dealing with metadata. It uses a lightweight database to store the metadata. But typically, you wouldn't try to re-implement the metadata store interfaces if you are writing a knowledge sync provider - you either store metadata however you see fit or use our implementation of the metadata store to make it easier.

     

    In V2, which hasn't RTM'ed yet but we have a CTP available we added a new type of providers, simple sync providers. They completely manage the sync metadata for you using the metadata store behind the scenes. With simple sync providers you are responsible for only data management: applying changes (implementing Insert, Update, Delete) and detecting changes. There are two strategies for data management:

    - if the store cannot track changes, you can simply enumerate all the items in your store and we'll determine what changed for you. This is called a full enumeration provider.

    - if the store can track changes (including remembering items that were deleted - we call them tombstones), you can enumerate all changes since the last time you did change detection. This is called an anchor provider. (anchor is a piece of data that tells you since what point in time you need to enumerate changes)

     

    Writing simple sync providers is really quick and probably the best way to get introduced to MSF. In fact, listing the different options in order of difficulty (easiest -> hardest):

     

    - simple sync provider (anchor/full enumeration)

    - knowledge sync provider, delegating most of metadata duties to MSF's metadata store

    - knowledge sync provider, handling metadata itself

     

    Of course the trade off you're making is less flexibility for easier options - simple sync providers will not allow you to tweak the sync metadata for one-off scenarios. However, they are fully functional and quick to write once you're somewhat familiar with them.

     

    Hope that helps. If you tell us what you're trying to accomplish I or someone else should be able to give you more detailed guidance.

     

    Lukasz [MSF dev]

    Wednesday, November 26, 2008 11:39 PM
  • Lukasz,

    Thanks for the quick reply. 

    I read the article you recommended and it did provide some additional insight, thanks.  I actually have a fair amount of sync experience; I wrote a SyncML server a few years ago and have closely followed the sync space since.  So, concepts like maintaining records of deletes (tombstones) and anchoring aren’t completely foreign to me.  It’s a problem that’s easy to understand but difficult to solve, which is why I’m so interested in MSF. 

    Here’s a basic embodiment of my problem:  I have a database in SQL Server 2008.  I’m writing a piece of server/web software, so multiple clients are going to be syncing at the same time, but each client is going to be syncing from a pool of entities that is client-specific.  Using the built-in metadata provider does not appeal to me because it uses SQL Compact Edition and maintains a separate file, which I’d have to backup, worry about, and I want to centralize my data in the already established database.   So I want to keep all my metadata in the SQL Server database.  I control the schema and can freely make changes to the database. 

     I have 3 tables:

     1) Contact  - ID (bigint, seeded, PK), Name (nvarchar(256)), Created (datetime), Modified (datetime), User_ID (bigint, FK)

    2) ContactTombstone - Contact_ID (bigint, FK), DeleteDate (datetime), User_ID  (bigint, FK)

    3) User - ID (bigint, seeded, PK)

    My conflict resolution policy is such; whoever has the greatest Modified/DeleteDate wins, if they’re equal (which is highly unlikely, almost impossible) then the first one wins (completely arbitrary).

    My requirements seem pretty basic, perhaps even fundamental for server-oriented software.  Is a server-based, multiple client synchronization scenario not a supported use-case for Microsoft Sync Framework?

    Are there any examples of a MetadataStore using SQL Server Standard Edition?

    Monday, December 1, 2008 5:55 PM
  • Thanks Josh, that clears it up. The direct answer to your question is no, there aren't any examples of our metadata store implementation using SQL Server. But I'd also say it's not the optimal approach to solve your problem (and yes, your problem does seem pretty standard). Since you're trying to synchronize databases, you should look into using Sync Services for ADO.NET, that's our default recommendation for this scenario.

     

    If you have already looked at Sync Services for ADO.NET and think it is not the right choice for you, we will be curious to find out why. Technically you can solve the problem writing a knowledge/simple sync provider, but with Sync Services the provider is already written for you and getting your scenario working should be an order of magnitude less work.

     

    Just for completeness (but again, not recommended) if I for some reason had to write a new provider to handle this I would choose to write a knowledge sync provider without implementing MSF's metadata store interfaces for the following reasons:

     

    - efficiency - because your data and metadata are in the same place, you don't want to access them separately. Our metadata store is designed mostly for scenarios where you can't really keep the metadata together with the data.

    - consistency - by writing a knowledge provider you can share a transaction between data and metadata operations. This is not currently supported by our metadata store since the primary scenario is a non-transacted data store.

     

    hope that helps

     

    Lukasz

     

    Wednesday, December 3, 2008 12:50 AM
  • Lukasz,

    I had originally not looked at ADO.Net Sync Services, but after looking at them they don't really solve my problem.

    It would work if both sync sources were databases, but unfortunately that's not the case - one endpoint is a database, other is a web-based API.  I would like to synchronize contacts in Gmail, Yahoo Mail, Facebook, and MySpace and photos in Flickr and MySpace using the Sync Framework.

    I feel confident I could write a knowledge sync provider for all these various types of endpoints, some of them already support tombstoning themselves.

    Based on your statement:

    "Just for completeness (but again, not recommended) if I for some reason had to
    write a new provider to handle this I would choose to write a knowledge
    sync provider without implementing MSF's metadata store..."

    I get the impression an instance of a metadata store is optional?  So is a metadata store "object" not required when writing a Knowledge Sync Provider?  All the examples use i like it's a required piece of the puzzel? Not the case?


    If that is an inaccuate impression then:

    Is there any way I could get the source code to the SqlServer Compact Edition one?  I'm sure I could hack it up to solve my problem - which would actually be simpler than the CE MetadataStore because schema doesn't need to "flex" to accommodate variable key types and such.


    thank you!

    -josh jonte
    Friday, December 5, 2008 6:02 PM
  • Hi Josh,

     

     Josh Jonte wrote:
    I get the impression an instance of a metadata store is optional?  So is a metadata store "object" not required when writing a Knowledge Sync Provider?  All the examples use i like it's a required piece of the puzzel? Not the case?

     

    That's right, using our metadata store is optional. It's a good idea if you can use it because it implements a lot of functionality common across providers, but for syncing databases when you keep metadata in the same database I wouldn't really advise it. It will be more work because you now have to implement GetChangeBatch and get the destination item versions for the ProcessChangeBatch call yourself, as well as some other smaller things (storing knowledge, doing change detection, etc.). But it's by no means a required component. Some of our samples do use the metadata store but I'm pretty sure we have at lease one generic sample which uses its own in-memory metadata store without using or implementing MSF's metadata store interfaces.

     

    Also, the source code for our metadata store is not available publicly, sorry.

     

    thanks,

    Lukasz

    Friday, December 5, 2008 7:42 PM
  • Hi Lukasz

    You said:
    "we have a CTP available we added a new type of providers, simple sync providers. They completely manage the sync metadata for you using the metadata store behind the scenes. With simple sync providers you are responsible for only data management: applying changes (implementing Insert, Update, Delete) and detecting changes."

    Can you point guide me further or point to some article on how to implement simple sync provider.

    Thanks in advance.
    -Sushil

    Tuesday, April 21, 2009 2:13 PM