Sync Framework 4... What is it good for?
-
2012년 4월 9일 월요일 오전 2:40
I'm asking this question having developed a system for the last two years based partly around Sync Framework 4. I'm currently in the position that the only solution I can see to my current problem is to rip the Sync Framework out completely and re-engineer the synchronisation, when I'm on the verge of commercialisation of the system. I apologise for the length of the question but I'd like an opinion from the moderators/experts of this forum as to whether this is a salvageable situation or not, so I'm going to spell it all out.
Part 1: The app for the users
We're targeting iOS for now (with plans for Android, WinPhone7 and Win8 down the track). We have an app out on the market now which is designed to gather large amounts of data and to make the users' lives easier we provide a large body of reference data. This data doesn't change very often but we do need to be able to change it (due to legislative requirements, to provide more efficient data, etc). We want thousands of users to be able to receive a notification that new data is available, to go to the 'sync' screen, press the sync button and have a small packet of changed data downloaded by their app to be stored locally (better yet keep the users oblivious and do it all in the background automatically). For this purpose we have a 'reference server' available at a standard URL.
Part 2: The server (ie The upsell)
We can't make enough from app sales to survive (can anyone?). In order to generate enough revenue to function as a business we need to 'upsell' our users to make use of additional services. These services are in the form of a server that they can access to store and share their 'live' data between multiple devices (using OData), as well as being able to customise and configure the 'reference' data to their way of working (as opposed to using our cookie cutter data designed for everyone) using a web app. Initially the server will be installed on-site at any office that signs up for it, although it is our full intention to go to a cloud based solution as soon as possible.
Part 3: The assumption (horrible mistake?)
We didn't want every new user who downloaded the app to have to also pull down a large blob of data and wait for the app to process it all. As such we wanted every client to be pre-loaded with the most current reference data combined with a SyncBlob/SyncKnowledge indicating 'this is what I know' about the reference data. The assumption was that by sending this sync knowledge to the server the server would then be able to calculate what changes it had locally which needed to be downloaded to the client and would then send those changes along with a new sync knowledge indicating its updated state. We realised through some trial and error with this that the sync scope id (being scope+randomly generated guid for each 'new' client that requested a sync with an empty sync blob) was intimately tied into the sync blob itself. The seemingly functional workaround for this was to pre-load the same sync blob onto each client and then to make certain that the associated sync scope info entry for that Blob was created on ALL servers (including the reference server). On receiving the 'initialised' sync blob the server (be it the reference server or the locally installed on-site server) would recognise the blob as its sync scope info entry, compare the sync knowledge with the current (updated) state of the reference data on the server, be able to calculate what new/modified/deleted items there were, before sending those changes plus the new sync knowledge back to the client.
Part 4: Where it all falls apart
We've started installing on-site servers at some beta test sites. One of the sites is very keen to make a lot of changes to the reference data. We've created a web (Silverlight) app that uses Sync Framework to download the intial data (as a new client/sync scope info entry), store the data in the local cache and then use UploadChanges to upload the modifications back to the server. This all worked fine and the changes are now sitting in the database on the server. The next step was to download the changes to the iOS app and this is where it all falls apart. The iOS client sends a DownloadChanges request to the server with its syncblob (created from the original reference data set). The server receives the syncblob, recognises the sync scope and proceeds through the processing. It all gets very messy and difficult to understand as it passes through SqlSyncProviderService.cs GetChanges(byte[] serverBlob). Fundamentally NO entities are detected as having been changed. What is sent back to the client is a message with no modified entities, BUT A MODIFIED SYNC BLOB (about 80 chars longer and with around 80 chars altered when viewing Base64 encoded, so 160 chars difference out of 1200 chars total).
From my perspective the fact that a sync against an unmodified reference data set returns an identical sync blob while a sync against a modified reference data set returns a modified sync blob shows that there should be SOME changes being synced. I can't work out what the hell is actually going on because all the actual entity delta generation appears to happen inside either SqlSyncProvider or SyncKnowledge objects which are both (as far as I can tell) black boxes from the Microsoft Sync 2.1 Framework.
Part 5: Help?
I'm basically looking for someone to do one of the following:
1) is to tell me that I was crazy to think that this scenario would ever work so I can safely ditch that part of the architecture and handle the whole sync and comms process with OData (I've got a design for this already, it's just going to set me back time I don't really have...).
2) is to tell me that I've made some fundamental misunderstanding about the sync framework but that with a couple of relatively easy code changes I can get it working (anything major and I'll go with option 1 above).
3) is to tell me that there is just something a little bit funky with my data and that it can be fixed using some tools that are available.
4) no-one responds and I run with option 1.
Really looking forward to some prompt replies on this. Any and all help that can be offered will be massively appreciated!
모든 응답
-
2012년 4월 9일 월요일 오후 6:27소유자
Peter, can you paste an email address where I could send you some code? That code will deserialize the knowledge and help you see what the replica ids are, so you can better understand what is happening during the sync.
thanks
sid
-
2012년 4월 9일 월요일 오후 10:02peter at rednightingale dot com dot au
-
2012년 4월 10일 화요일 오전 1:34중재자
hi peter,
just to further clarify on how you initialized the clients:
when you say you pre-loaded a client with data, do you you synched an empty client and had sync framework populate the data and the sync knowledge? or did you pre-load the data outside of sync? if it's the latter, how did you generate the sync blob?
did you create separate scopes for each client in the server? e.g., one for SL, one for IOS, etc...
did you copy a sync blob from one client unto another and simply created the corresponding scope entry in the server?
to answer your questions:
1. - no, i dont think you're crazy as this is exactlyt the scenario that the Sync Framework Toolkit is supposed to address. re-writing may seem to be a viable option but you may end-up duplicating exactly the same functionalities that the framework already provides out of the box (change tracking, change enumeration, change application, conflict handing, serialization,etc...)
2 & 3 - i think you're issue is more on how you initialize clients and there may be workarounds for it.
feel free to shoot me an email at junetidlethoughts at hotmail dot com if you prefer a lenghty email over a lenghty public forum post.
-
2012년 4월 10일 화요일 오후 11:22
Hi June,
Initial setup was as follows. Empty sqlite database on iOS client, sql database on server populated with reference data and provisioned for sync. DownloadChanges on iOS client fully populated the client sqlite database with reference data and syncblob, and also created 1x scope_info and 1x scope_parameter row on sql database.
sqlite database file copied and used as resource in iOS apps. On startup if no database is present in the data folders the sqlite file is copied there from the resources and then the copy is used to store live data, etc.
sql database was backed up (full backup) with the addition of the scope_info and scope_parameter. The reference server's database (global reference data for all clients who don't have their 'own' sync server) was populated directly from the backup (I did have the problems mentioned below on the reference server from recollection). Syncing a new client against the reference server results in no new rows and identical syncblob returned compared with sent (client perspective).
The on-site server installations have their db generated from the sql scripts. They should (once everything is running smoothly... see below) be a carbon copy of the starting point for the reference data server.
Things went a bit awry with the first installs at beta sites (and the reference server). I very quickly ran into this problem (http://social.msdn.microsoft.com/Forums/en-US/synclab/thread/df763577-b4f7-45ae-9b02-27bc2fcf3fba) although didn't find that post until just recently (have uninstalled all the Sync1.0 components from my devbox I could find and still getting the problem... Sync framework 1.0 files are still present in my GAC). The solution I found during the installations was to deprovision and reprovision the database using the config file. In order to maintain the 'single client' concept I ran sql scripts after the provisioning process to clear the change tracking tables and scope_* tables then repopulated them with the data from my backup of the initial sync. I've also recently discovered that InstallShield sql script generation uses string.empty in place of DBNULL (WTF???!) so the beta site reference data was a little bit off kilter relative to its change tracking data.
Does this provide any insights as to why syncs of subsequently modified reference data wouldn't send the changes? I'm going to shortly be trying the code Sid sent me to see what the actual internals of the syncknowledge for the various blobs I'm seeing are.
-
2012년 4월 10일 화요일 오후 11:45
Ok, here's the output from Sid's code. I have no idea what it means but hopefully one of you will better understand ;).
Reference Server Database (or newly initialised on-site database):
SCOPE: 249 coldfusionsync_b31c70fb-791b-414d-84d0-d1721b541209 70677111-078f-42be-ade5-ddda75970d0c
No Knowledge!
No Forgotten Knowledge.
From the heavily modified database:
SCOPE: 249 coldfusionsync_b31c70fb-791b-414d-84d0-d1721b541209 70677111-078f-42be-ade5-ddda75970d0c
knowledge: ReplicaKeyMap: [
(0:e20f581e466b40368e1476e71909819c)
(1:b6cbf3c5ce0c4cffa843e146a25d337a)]
ScopeRangeSet: [00:[
(0:58000)
(1:7)]
]
forgotten: ReplicaKeyMap: [
(0:e20f581e466b40368e1476e71909819c)(1:b6cbf3c5ce0c4cffa843e146a25d337a)]
ScopeRangeSet: [00:[]
]
Thoughts???
Edit: Just realised that the second result is on account of my playing around with the scope_info table (tried copying the syncknowledge from another scope to try and trigger some action on sync). From what I've seen the scope_sync_knowledge column stays null no matter how much the reference data is changed (and therefore I'm assuming will give the 'No Knowledge!' response)- 편집됨 Peter Mauger 2012년 4월 10일 화요일 오후 11:51 Invalid Data
-
2012년 4월 11일 수요일 오전 1:45중재자
hi peter,
i'm pretty sure the problem is around copying of provisioned databases.
in sync framework, each replica gets its own replica id. so when you provision a scope on a particular database, that gets an unique id. you provision a scope on the client and that gets its own unique id as well. (it's the guid you see in the scope_info table and the guid that you see in the ReplicaKeyMap of the sync knowledge)
when you sync, the replicas are effectively telling each other: "here's what i know about you, do you have any changes for me since the last sync?").
so if you have duplicate Ids around, that may result to problems synching. one client say I am ClientA and server says here's what i know about you. then you sync another client that says i'm ClientA as well, so server sends what it knows about ClientA from the previous client and so on... in your case, all your IOS sqllite databases will have this same id.
for sql ce databases, the workaround for distributing a pre-populated database is to use the GenerateSnapshot method. for SQL databases, you can do a back up of the db, restore and then run PerformPostRestoreFixup. Both GenerateSnapshot and PerformPostRestoreFixup causes Sync Framework to re-adjust these Ids and the sync metadata. for your onsite-servers, you can use the backup->restore->performpostrestorefixup route.
unfortunately, you dont have these functionalities for other stores like sqllite or IsolatedStorage. so it's going to be a challenge distributing a "generic" pre-populated database and get its metadata adjusted on first sync (just like the snapshot generated using the GenerateSnapshot in SQL Ce).
i may have some crappy workarounds to test but i'll defer to Sid first if he has any workaround for this issue.
-
2012년 8월 27일 월요일 오후 7:48Are there any workarounds out there for this? I have a working iOS sync implementation and need to be able to utilize snapshot initialization for a sqlite database. Please tell me someone's solved this problem! :)
-
2012년 8월 27일 월요일 오후 11:39Hi Cody, Unfortunately I abandoned sync framework completely not long after posting this. I've now re-implemented the entire sync process for my devices to a manually processed 'proper' OData link with hand-cut tracking triggers and tables. Set me back about 3 months but I can now move ahead with certainty...
-
2012년 9월 26일 수요일 오전 7:08
Hi Sid,
I am very interested in your code to deserialize a knowledge.
Can you please send it to me at the following address:
olivier[dot] gauchard [at] raynet-it [dot] com
Thanks in advance.
-
2012년 10월 2일 화요일 오후 1:20
I'm also in need of this knowledge deserialization code.
ccarse [at] gmail [.] com
Thanks!
