none
Synchronization for 'never online' / always 'offline' / disconnected clients RRS feed

  • Question

  • Hi,

    In our project we need to be able to implement both: file-based and service-based synchronization. It is intended to be used in environments where connectivity is enabled or DISABLED/NOT ALLOWED. When it is disabled we should be able to exchange files (portable media: usb memory, CD etc)

    The synchronization should also be bi-directional, although the data flowing in every direction is independent which means that conflict resolution problem is off the table.

    The client data storage can be XML or SQL Server CE. On the server side we will use MS SQL Server 2008.

    Theoretically thinking, what we need is: formatted document(s) containing changes, (de-)serializers on both sides (client and server) and a delivery service. The (de-)serializers should be able to consume/produce documents and delivery service should either contact our web-based service or prepare files on available media - depending on the scenario.

    I understand we can build a very nice home made wheel and believe that is possible on WCF, but am trying to avoid that.

    Appreciate if you shed some light on how SyncFx could help in case when connectivity is disabled/not allowed.

    Thank you!

    P.S.

    We used sql server merge replication before, but because both, subscriber and distributor have to be online to do a sync that is not an option.


    Everything is possible
    Monday, April 25, 2011 11:49 AM

All replies

  • Hi,

    You may consider using SyncFx database provider to build up this scenario. There are two possible patterns just based on my current understanding to your scenario.

    Setup:

       1. Use Collaboration SqlSyncProvider for Server. Server is SQL Server 2008.

       2. Use Collaboration SqlCeSyncProvider for client. Client uses SQL CE store.

    Pattern one (2 sync replicas):

       1. Always store the SQL CE database in a portable device such as a USB key

       2. If connection is available, sync client with server through your web service. If not, pull the USB  key from your local machine and plug it into your server machine to sync locally.

     

    Pattern two (3 sync replicas):

      1. Store SQL CE database in the client local machine, and have another SQL CE databse in the portable device

      2. If connection is available, sync client with server through your web service. If not, syncing client ce database to the ce database in the portable device.

      3. Connection the portable device to the server machine and sync.  

      4. Connection the portable device to client machine and sync.

    If you haven't try the SyncFx database providers yet, please look at the MSDN (http://msdn.microsoft.com/en-us/library/bb902854(v=SQL.110).aspx) and sample (http://code.msdn.microsoft.com/site/search?f%5B0%5D.Type=SearchText&f%5B0%5D.Value=Microsoft%20Sync%20Framework&f%5B1%5D.Type=Affiliation&f%5B1%5D.Value=Official&f%5B1%5D.Text=Microsoft) links.

    Thanks,

    Dong

     

     

     

     


    This posting is provided AS IS with no warranties, and confers no rights.
    Monday, April 25, 2011 6:37 PM
    Moderator
  • Hello Dong,

    Thank you for the prompt reply.

    In our scenario we hope to have only one main database file on the client. When it comes to synchronization we need to support two options:

    A. CONNECTION IS DISABLED: File exchange between server and client using files containing incremental changes.

    B. CONNECTION IS ENABLED: Service-based synshronization.

    The questions here are:

    1. Is it possible to implement 'A' with MS Sync Fx?

    2. What we have to implement in order to allow resuming synchronization when switching between 'A' and 'B' synchronization scenarios? Say, we synchronizaed using option 'A' and then switched to synchronizing with option 'B' and vice versa.

    Thanks!


    Everything is possible
    Monday, April 25, 2011 7:17 PM
  • Hi,

    The Pattern two in my previous reply is for your scenario 'A'. The exchanged file is a SQL CE database file. You need to use SyncFx database provider to sync this exchanged SQL CE database file to both client and server by directly connecting it to the physical machines. Still, this SQL CE database file contains all records of client storage instead of just incremental changes since the previous service-sync between client and server. Considering that you can have a decent size of portable storage today, I assume it should be ok. If it is not, the SyncFx may not be the solutoin for you scenario.

     

    Thanks,

    Dong


    This posting is provided AS IS with no warranties, and confers no rights.
    Tuesday, April 26, 2011 5:36 PM
    Moderator
  • Hi Dong,

    The size of the full client database can be too large for porting it from a remote site to the central database location. Plus, we have a considerable number of remote sites, which makes an automatic sync at a site-relay a strong requirement. For that we need small files with only incremental updates.

    I implemented in the past and have looked at supported "offline" synchronization scenarios developments and do not see solutions for mixed synchronization file-based(A)/service-based(B).

    It appears that the key part in any synchronization scenario is a session. That's so because subscriber/distributor needs to know if the data pack sent to the other party has been processed. Session is necessary to simplify synchronization framework and for security reasons... but only at first glance. The problem can be solved if the session information is passed along with the incremental changes. When next  packages comes to the receiving party it should check if the previously sent package has been applied. This is a viable solution for scenarios when remote sites are truly disconnected and still have to have some sort of automatic exchange with the central site. There are cases when only e-mail communication is enabled, for example.... or when communication is done over a non-standard application protocol than HTTPS. Regardless of how computer systems would evolve, file is the atom. It all boils down to generating and processing files as digital information on a media.

    We'd be glad to implement exchange and tracking mechanism for true offline synchronization, but because the major part of the synchronization framework is implemented on the kernel level (subject to a change by the provider) we cannot invest in that.

    Does Microsoft have any solution, workaround or approach for the problem at all or have we hit the wall already?

    Waiting for yours.

    Thank you!

     

     


    Everything is possible
    Tuesday, April 26, 2011 6:23 PM
  • Hi,

    I agree with your point that storing incremental change packages as files in a intermediate storage will be a nice solution for your scenario. Unfortunately, SyncFx doesn't implement a sync provider for this kind of usage.

    Since I don't know details of your scenario, my previous proposal may not work well for your requirements, but I would like to add a couple more points in case they may help your evaluation. A CE database cannot be very big (about 2GB?). It means one USB key with 4-8 GB storage will be more than enough from data size perspective. If all of your client databases share similar data from the same server database, with SyncFx collaboration providers that supports peer-to-peer sync, a USB key can sync among different clients and servers with just one SQL CE database on it.

    Thanks,
    Dong


    This posting is provided AS IS with no warranties, and confers no rights.
    Wednesday, April 27, 2011 6:53 AM
    Moderator
  • Hi Dong,

    Probably the only detail I've missed is that our customers share ovelapping partitions of the central database.

    Exchange with USB keys is just one of the options in the stack of available session-less (disconnected) data exchanges. Here is the ordered list:

    1. Mail exchange.

    2. Exchange using all kinds of avilable media and offline delivery services (manned, posted etc)

    If we have a chance to deploy incremental changes via e-mail we are going to use that (option #1). If that's not available then it's option #2.

    I've actually found a DBMS that supports file-based synchronizations: http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.help.sqlanywhere.12.0.0/mlserver/ml-filedownload.html Not only it allows file-based synchronizations, but is also a cross-platform DBMS. My understanding is that Sync Fx only for Windows-devices. Please confirm that. The downside is that our solutions are based on Microsoft technologies. If you looked at the link you'd probably get more insight of what we need?

    FYI: SSCE databases (*.sdf) grow upto 4Gb.

    Waiting for yours.


    Everything is possible
    Wednesday, April 27, 2011 11:44 AM
  • Hi Alexander,

    SyncFx native dlls can only be executed on Windows OSs. Thanks for the details of your offline options. I understand your needs better now. With SyncFx KnowledgeSyncProvider, you may be able to write a file based sync solution for this offline scenario. It is just like a n-tier sync, all changes need to be serialized before sending to the remote side. With SyncFx knowlegeSyncProvider, you need to write custom sync providers for both client and server storages, and write file base proxy providers to sync with client and server separately. The flow will be:

    1. Client provider syncs to client proxy provider, proxy provider stores knowledge, changebatch, dataretriever and data in serialized format. After each sync, the proxy provider's knowledge need to be adjusted to include the synced changes.

    2. Send files to remote side

    3. server proxy provider sync to server provider. Server proxy provider deserializes files in sequence and behave as the client provider, and changes will be applied on server side.

    You can achieve server-> client sync direction in the same way with the same set of proxy providers.

    I hesitate to recommand this solution because you need to write a few new custom sync providers with SyncFx. It will be a lot of code implementation, and you need to know SyncFx very well. There is no similar code sample for you to follow either. If you would like to consider this direction, you can learn SyncFx from the links that I listed in my first reply.

     

    Thanks,

    Dong


    This posting is provided AS IS with no warranties, and confers no rights.
    Wednesday, April 27, 2011 6:21 PM
    Moderator
  • Hi Dong,

    Good news. Thanks for shedding some light!

    Let me ask you few questions while I'm doing my research on the custom provider that you mentioned:

    1. Do I have to install Sync Fx or can I only selectively deploy only dlls that are needed?

    2. What are those dlls and what is their approximate total size?

    3. What is the future development of Sync Fx (4.0+) in this respect? Will it support or improve file-based sync and how?

    The reason I'm asking is that we would like to minimize any impact on our customers' environment to avoid maintenance issues.

    I've spent a reasonable amount of time doing my research on the technologies for our new project. Now it's a difficult times for developers using Silverlight, SQL Server CE, Sync Fx because of the new standards (HTML5, OData) emerging on the market. Silverlight 5.0 release is postponed, Sync Fx 4.0 release is postponed, SQL Server CE 4.0 implementation is not complete (merge replication is missing in particular).  Some teams even consider moving to Java/Oracle.

    Waiting for yours.


    Everything is possible
    • Edited by Alexander Yarushin Thursday, April 28, 2011 11:15 AM new technologies (HTML5, OData) -> new standards (HTML5, OData)
    Wednesday, April 27, 2011 6:38 PM
  • Hi Alexander,

    Here are the answers:

    1. Do I have to install Sync Fx or can I only selectively deploy only dlls that are needed?

    You can install SyncFx through release MSI from MSDN download page. Selective dll deployment is not supported by SyncFx team, but it is doable. SyncFx includes both native COM dlls and managed dlls. You need to register and gac the ones that you need.

    2. What are those dlls and what is their approximate total size?

    For KnowledgeSyncProvider, if you plan to code with native code, you only need Synchronization21.dll (511,344 bytes for X86, and 870,768 bytes for X64). If you will implement with managed code, you need Microsoft.Synchronization.dll (288,624 bytes) as well.

    3. What is the future development of Sync Fx (4.0+) in this respect? Will it support or improve file-based sync and how?

    We don't have a release plan for next version of SyncFx yet. If we see enough customer demonds for this intermediate file-based sync scenario, we will consider it.

     

    Thanks,
    Dong


    This posting is provided AS IS with no warranties, and confers no rights.
    Thursday, April 28, 2011 6:23 AM
    Moderator
  • Hi Dong,

    Thanks. Further to my last questions:

    1. In case if dlls have to be registered it is not a "private deployment". The spirit of private deployment is to make installation independent from any resources but the OS. Copying files is enough for a private deployment. Registering dlls can cause conflicts and "Dll Hell".

    2. We will be implementing in managed code. Please confirm/clarify the following for enabling Sync Fx on the target site:

    Synchronization21.dll - register as a COM object or as an isolated COM object?

    Microsoft.Synchronization.dll - register with gacutil? What are thet suggested parameters?

    How should I register dlls to avoid conflicts with future installations of other programs which may require using Sync Fx?

    3. Demand in a file-based sync persists on the market. There is even a competitive solution "Mobilink" being delivered by Sybase. The problem with existing "offline" synchronization solution by Microsoft is that it only supports HTTP/HTTPS application protocols, but communication is not limited by only Internet connections. There is a huge niche on the market which consumes satelite communication tools, for example. Those allow POP/SMTP exchange in some cases, but not limited to that. Serialization of synchronization packages and establishing mechanism for true deferred synchronization would allow to make synchronization solutions connection and protocol independent. Hence, completely platform independent as well.

    Waiting for yours.


    Everything is possible

    Thursday, April 28, 2011 11:49 AM
  • Hi Alexander,

    private deployment is not officially supported by syncfx. But if you know how to do native COM dll private deployment, you can do so for SyncFx COM dlls. For detailed of syncfx assemblies, please look at syncfx msdn site and try it out. Here is link of the SyncFx installation in MSDN: http://msdn.microsoft.com/en-us/library/dd937137(v=SQL.110).aspx.

    Thanks,

    Dong


    This posting is provided AS IS with no warranties, and confers no rights.
    Thursday, April 28, 2011 10:41 PM
    Moderator
  • Hi Dong,

    Your noted. Thank you!

    You mentioned that it's possible to synchronize databases using n-tier scenario. As I understand it can be both file-based (serialized changes) and service-based (WCF-service). Please confirm.

    Here is what I found for that scenario: http://code.msdn.microsoft.com/Database-SyncSQL-Server-e97d1208 The difference with what we need is the SQL Server edition - for us that would be SQL Server Compact. Do you think the example is what we need or is there something we need to be aware of?

     


    Everything is possible
    Friday, April 29, 2011 1:15 PM
  • Hi Alexander,

    There is another n-tier between SQL Server and SQL CE: http://code.msdn.microsoft.com/Database-Sync-SQL-Server-7e88adab. You can consider starting with this sample and keep all serialized contents in file format following my previous logical flow. Since you don't have connection to Service, you need to build your change application side knowledge from old change enumeration knowledge (keep a copy of old change enumeration knowledge after previous on-line sync, and convert it to the change application knowledge). 

    I never tried your scenario with syncfx database prorider, and I cannot confirm that your can convert this sample to work for your scenario. If you plan to build your scenarios with SyncFx KnowledgeSyncProvider, I feel it should be possible since you can write your sync providers for any storages and any protocols.

    Thanks,
    Dong

     


    This posting is provided AS IS with no warranties, and confers no rights.
    Friday, April 29, 2011 5:30 PM
    Moderator
  • Hi Dong,

    The connection to a Service is an option which is sometimes available. So, again, both file-based and service-based have equal chance to be used. That means that we'll need to implement WCF client part anyway.

    I'd appreciate if you could elaborate your comment below:

    ... Since you don't have connection to Service, you need to build your change application side knowledge from old change enumeration knowledge (keep a copy of old change enumeration knowledge after previous on-line sync, and convert it to the change application knowledge)...

    Thank you!
    Everything is possible
    Friday, April 29, 2011 5:55 PM