none
Sync Framework architecture for offline app. RRS feed

  • Question

  • Hi,

    I am in the process of designing a new application which consists of a central SQL-Azure database and many (potentially +2000) remote clients [kiosks]. I need to synchronise a simple (10 table) local SQL CE databases with the Azure DB and envisage that the synchronisation will need to take place at least once a day - there may not be a huge number of transactions that need to be applied to the kiosks in each sync i.e. a couple of inserts, a couple of updates (quite frequently no changes will need to be applied); at present I also only envisage it will be master > client updates not bidirectional. I have considered client - client synchronisation to help with scalability on the server however unfortunately my client has indicated that this may not possible since kiosks only have 3G connections and do not participate in a LAN.

    To achieve this I am looking to use the MS Sync Framework which works really well in my simple trials to date, however I have concerns regarding potential contention and also for performance when many clients may simultaneously connect to the central Azure DB at approx the same time. I can see two possible solutions to this:

    1) Have a server side scheduling work queue which calls a WCF service on the client which would initiate the synchronisation from the client. This feels a little over engineered to me and adds an added degree of complexity to the solution. The added benefit of the WCF sync service on the client is that if needed I can initiate a sync on the remote client from a server admin app (providing the service is accessible of course!)

    2) Configure each client or batches of clients to synchronise at a different time of day - this is workable but could become an admin. burden.

    ...or, am I over thinking the problem/is the problem not existent and the scale of transactions manageable.

    Also, how would I go about enabling access to the DB? As I understand it, I will need to open the Azure firewall for each client - this may not be feasible (or even desireable from a security stand point), what are my alternatives? A server side WCF sync service? 

    Thanks

    Saturday, April 30, 2011 12:15 PM

Answers

All replies

  • On re-reading the above i thought it might be worth summising what I am trying to achieve since I'm now not sure if I am taking the correct approach:

    1) I have a network of potentially 2000 kiosks running a WPF application.
    2) I have opted for a multi-tenant Azure-SQL database to centrally manage simple data for groups of kiosks e.g. Product info.
    3) The kiosks have 3g connections and should be treated as occassionally online (although it may well be that they are accessible all the time). I need to sync the tenant specific product data to each kiosk on a daily basis, the volume of data to be sync'd is expected to be minimal. I initially planned on hosting a SQL CE db on each client but am leaning towards odata/atom (following the release of odata support)  since it is pull only, in addition this would open up options for non MS based clients in the future (mobile consumers).
    4) i need to sync resources such as images etc onto the clients, i planned to use the file sync providers for this.

    My concerns are:

    1) Scalabilty of frequent client syncs.
    2) Azure firewall implications when granting each client acess to the master db/blob storage

    I would ideally like to migrate the WPF app to Silverlight, presumably this could simplify things since i could then lever the browser to cache the odata feed thus removing the scalability concerns i have and allowing updates to be applied immediately?

    Apologies for the rather open ended question but I'm fairly new to synfx and therefore am trying to figure out if it is the best fit for my requirements.

     

    Thanks  

    Saturday, April 30, 2011 1:15 PM
  • rather than having each client connect to the Azure DB directly, i would suggest building an Azure-based sync service. The Sync Fx team has sample for doing this at: http://blogs.msdn.com/b/sync/archive/2010/10/07/windows-azure-sync-service-demo-available-for-download.aspx

    now regarding your Atom/OData/Non-MS platform clients, Sync Fx v4 CTP has built functionalities around OData (but not part of the OData itself) and has samples for Silverlight, Windows Phone and other clients. Unfortunately, the release has been postponed but the CTP bits will instead be released as code samples (instead of a product release).

    Another option in CTP is the OData Reference Caching, check it out at: http://blogs.msdn.com/b/astoriateam/archive/2011/04/13/reference-data-caching-walkthrough.aspx

    Using the sample Azure Sync Service, you can easily tailor fit it to a SQL CE - Azure Sync Service - SQL Azure synch. You just need to size the average size of your synch and duration so you can spread out the synching of the 2000 clients.

    another option you may explore is peer-to-peer synching.

    if you have kiosks located and connected to each other, you can have one kiosk sync with Azure and have the other kiosks sync to that kiosk.

    Saturday, April 30, 2011 2:46 PM
    Moderator
  • Thanks for the reply - I had a feeling a service may be the way to go. This article on MSDN covers off migrating to an N-Tier sync architecture http://msdn.microsoft.com/en-us/library/bb902831.aspx.

    I also found this article which looks to me like it offers a similar solution (minus the nice server side job queue) http://omarbesiso.typepad.com/the-archidev/2009/08/sql-2008-sync-framework-over-wcf-with-simplicity.html. It is a little bit more straight forward than the syncfx team example and I cant see any immediate hurdles in migrating this to an Azure solution? I could then implement some retry logic on the client to handle potential time outs.

     

    The non .NET clients aren't an immediate requirement so I will wait until the code samples are released (I've plenty to be getting on with in the meantime!)

     

    I had considered peer to peer synching but my client has advised me that the kiosks will have to work in isolation since there is no guarantee they will be networked which is a shame because it would have significantly reduced the server load and thus improved scalability and reduced Azure bandwidth costs.


    Saturday, April 30, 2011 8:28 PM
  • the samples you linked above are both using the offline providers which out-of-the-box only supports SQL CE (SqlCeClientSyncProvider) and SQL Server (DBServerSyncProvider).

    so you will have to write your own provider for Azure. Moreover, Sql Azure unlike SQL Server doesnt come with an Integrated Change Tracking, so you will have to write your own custom tracking as well (see http://msdn.microsoft.com/en-us/library/cc305973.aspx). In addition, you'll have to write your own batching code just in case you have large amount of rows to sync and you need to batch them.

    I'd recommend you use the SqlCeSyncProvider/SqlSyncProvider combo instead referenced in the Windows Azure Sync Service Demo above as the providers work natively with Sql Azure.

    on another note, you may want to check our the Sql Azure Data Sync service CTP as well: http://social.technet.microsoft.com/wiki/contents/articles/sql-azure-data-sync-overview.aspx

     

    Sunday, May 1, 2011 1:03 AM
    Moderator
  • Ok, fair point - I did wonder if Azure had change tracking but hadn't got round to finding out. 

    I read up a little on SQL Azure Data Sync services but it appears it is only suitable for Azure-Azure/SQL scenarios - I'm presuming that in the case of on premises SQL this refers to a full SQL server installation or would CE be an option?

    From further reading and watching Mike Clarks MIX2010 presentation, I am starting to lean towards the ODATA sync solution since I only plan to sync lightweight data; in addition I could eventually consider migrating the WPF solution to Silverlight* and I could easily target non MS platforms with the same architecture, simplifying maintenance - I was also pretty impressed by the inclusion of the sync scope within the protocol. I've downloaded the CTP after watching Nina Hu's presentation (http://channel9.msdn.com/Events/PDC/PDC10/CS57) and will have a play around. I also like the idea of keeping the clients fairly simple as it will simplify admin significantly.

    * The main barrier to this has been accessing locally cached media however this improved in SL4 and looks to get better in SL5 which appears to offer elevated permissions to access other disk areas.

     

    Thanks



    • Edited by SideBP Sunday, May 1, 2011 10:10 AM Considering ODATA sync
    Sunday, May 1, 2011 9:32 AM