locked
Distribute a single snapshot SDF that can be used to generate multiple production SDFs … each with a different filtered scope? RRS feed

  • Question

  • We have one unfiltered scope and one filtered scope.  The unfiltered scope inserts 60mb of data into an SDF; the filtered scope inserts less than 1mb.  Currently, every time we add a new user a new 61mb SDF is created and pushed to their device … we want to avoid this file transfer … we want a snapshot SDF already available at the new device to be used to configure the new user’s SDF.

     

    The goal is to create a snapshot SDF (that contains the unfiltered scope) while directly connected to our LAN and SQL server (doing the heavy work over a fast/stable connection) … then to copy that single snapshot SDF onetime to a remote location where it can be copied onto each of the new remote devices.  Each remote client can then sync (over WIFI) to receive its own filtered data.

     

    The question boils down to: How do we add each user’s scope to their individual copy of the snapshot SDF?

     

    Is this a workable solution?

    1)      SERVER: Create one snapshot SDF with the unfiltered scope as well as a filtered scope for a “dummy” user.

    2)      Transfer the snapshot SDF to the remote location where the new clients are located

    3)      SERVER: Provision the server for each new user by adding user-specific scopes (ex: scope name = “User100”).

    4)      CLIENT: Copy the snapshot SDF onto each new device.

    5)      CLIENT: Sync the filtered scope on each new device against their own scope – not the “dummy” scope the SDF was created with.

     

    Will this work?  Will the client’s sync knowledge for the dummy user be replace with new sync knowledge for “User100”?

    Friday, January 6, 2012 9:11 PM

Answers

  • actually, the scope defined in the SDF does not have any information about the filters. the filters can only be defined on the server (SqlSyncProvider). so yes, you can pre-provision in the SDF the corresponding scope that will sync with the filtered scope on the server.

    each replica in a sync topology gets their unique id to identify them in the sync community. Sync Fx processes a snapshot differently in that processes the sync knowledge  almost similar to the way PerformPostRestoreFixup does when processing a restored database.

    So, you should only distribute a snapshot. once you have synched an SDF, it already has its unique id and the knowledge adjusted. so if you copy it, you might end up with SDFs having the same identity.

    make sure that when you distribute the SDF from your district office to the client, it is a snapshot SDF, not an SDF that has been synched already.

    better, if you want to further reduce bandwith consumption, you can get your clients to sync to an SDF in your district office for the unfiltered scope.

    so your district office syncs with corporate and your client SDFs syncs with the district office SDF.

     

     

    • Marked as answer by P H T Thursday, January 12, 2012 1:09 PM
    Wednesday, January 11, 2012 1:46 PM
  • 1. any scope will do for as long it has the same scope definition (tables, columns). in reality the scope definition in the client and the server are totally independent (the client doesnt know about the scope definition of the server and vice versa). but of course, since we want a sync, we make sure both scope definitions matches otherwise the sync will fail (you dont want to sync a 5 table scope with a 10 table scope).

    Server: provision unfiltered scope, provision filtered scope template, provision dummy scope based on template

    Template SDF: provision unfiltered scope, provision scope with same structure as the dummy scope

    Sync Template SDF, Generate Snapshot SDF from Template SDF, Distribute Snapshot SDF to District Office

    When you add a new client:

    Server: Provision filtered scope for client based on scope template

    Client : Sync unfiltered scope, sync dummy scope with newly created filtered scope for that client

    2. yes, make a copy of the snapshot and use that to initialize a client.

    • Marked as answer by P H T Thursday, January 12, 2012 1:09 PM
    Thursday, January 12, 2012 1:04 AM

All replies

  • i dont see any benefit of provisioning a scope with a dummy user. the snapshot approach is used to distribute an SDF with data already. the dummy user data is irrelevant to your user anyway if you're going to provision a new scope with the actual filter.

     

    i suggest you provision the unfiltered scope on the server. sync with one SDF client, generate a snapshot with that client and distribute.

    Once distributed, just create an additional scope for the filtered scope. 

    Saturday, January 7, 2012 1:33 AM
  • Once distributed, just create an additional scope for the filtered scope. 


    Thanks JuneT.  I guess this is where I am lost ... are you saying to provision the server for the new user's filtered scope and then just sync the client SDF by specifying that scope?  In this scenario how does the SDF get provisioned for the new scope?

    Here is what we are doing now:

    1) Create an SDF with the unfiltered scope

    2) Create a snapshot from that SDF

    3) Provision a new SDF using the shapshot and filtered scope (using SqlCeSyncScopeProvisioning)

    4) Sync the filtered scope

     

    From your suggestion i would either have to do #3 over WCF ... or #3 is unnecessary as an explicit step.

    Thanks!

     

     

    Monday, January 9, 2012 1:56 PM
  • here's a more detailed step:

    1. Provision unfiltered scope in server

    2. Provision unfiltered scope in SDF

    3. Sync unfiltered scope between SDF and server

    4. Generate snapshot SDF from the SDF in step 3

    5. Distribute a copy of snapshot SDF to client.

    6. every time you add a new client, provision filtered scope for that client in server

    7. every time you distribute snapshot SDF to new client, provision the corresponding filtered scope in the SDF

    8. Sync both unfiltered and filtered scopes.

    Tuesday, January 10, 2012 2:36 AM
  • Thanks again JuneT, that sounds like what i am currently doing (with #6, #7 actually being done before #5).

    I guess i need to make this part more clear: I currently create every user's 60mb SDF at a corporate office and transfer each one to a district office where it is loaded onto a client PC.  What i would like to do is create one 60mb snapshot SDF, transfer it one time to a district office where the SDF can be provisioned (at the district office) for any new user's filtered scope as needed.

    In other words, I am trying to avoid transferring a 60mb SDF for each new client by doing #7 at the client.

    How can #7 be done at the client?

    (FYI: this is why i was toying with the 'dummy' scope idea ... i would essentially provision the SDF with a template scope (before it leaves corporate) ... then district would hopefully hijack the scope for each real user.)

     

    Thanks!

    Tuesday, January 10, 2012 2:18 PM
  • so how does the SDF in the district office get to the actual clients?
    Wednesday, January 11, 2012 12:38 AM
  • The district office copies the SDF onto a new user's Netbook PC and assigns the PC to the user.

    Currently every time they do this they have to ask Corporate for a new SDF (occasionally they have asked for up to 100 SDFs at a time) ... I want them to create their own SDF based off of a snapshot they have locally (after Corporate provisions the server of course - which I intend to eventually empower district to do via a web interface).

    Also of note ... typically the filtered scope for a brand new user has no data at all ... so the whole process of provisioning new users really ends up with the exact same data in the SDF ... just different user provisioning metadata!

    Thanks!

    Wednesday, January 11, 2012 1:25 PM
  • actually, the scope defined in the SDF does not have any information about the filters. the filters can only be defined on the server (SqlSyncProvider). so yes, you can pre-provision in the SDF the corresponding scope that will sync with the filtered scope on the server.

    each replica in a sync topology gets their unique id to identify them in the sync community. Sync Fx processes a snapshot differently in that processes the sync knowledge  almost similar to the way PerformPostRestoreFixup does when processing a restored database.

    So, you should only distribute a snapshot. once you have synched an SDF, it already has its unique id and the knowledge adjusted. so if you copy it, you might end up with SDFs having the same identity.

    make sure that when you distribute the SDF from your district office to the client, it is a snapshot SDF, not an SDF that has been synched already.

    better, if you want to further reduce bandwith consumption, you can get your clients to sync to an SDF in your district office for the unfiltered scope.

    so your district office syncs with corporate and your client SDFs syncs with the district office SDF.

     

     

    • Marked as answer by P H T Thursday, January 12, 2012 1:09 PM
    Wednesday, January 11, 2012 1:46 PM
  • That sounds like exactly what I was hoping for and believed to be the case … thanks for clarifying!

     

    For finer clarification on the key issue …

     

    When making the snapshot, regarding the filtered scope … I use a scope for any existing user … right? 

    (I will choose a user whose scope does not return data as the objective at this point is just to get the table structures in the SDF.)

     

    Also, going from snapshot SDF to final production SDF is just a matter of making a copy of the snapshot file and then syncing the copy using the scope of the new user …right (regardless of the scope that was used to create it)?

     

    Thanks again JuneT … your help in this forum and the resources on your own site have been a huge help to us as we have developed our product!

    Wednesday, January 11, 2012 3:17 PM
  • 1. any scope will do for as long it has the same scope definition (tables, columns). in reality the scope definition in the client and the server are totally independent (the client doesnt know about the scope definition of the server and vice versa). but of course, since we want a sync, we make sure both scope definitions matches otherwise the sync will fail (you dont want to sync a 5 table scope with a 10 table scope).

    Server: provision unfiltered scope, provision filtered scope template, provision dummy scope based on template

    Template SDF: provision unfiltered scope, provision scope with same structure as the dummy scope

    Sync Template SDF, Generate Snapshot SDF from Template SDF, Distribute Snapshot SDF to District Office

    When you add a new client:

    Server: Provision filtered scope for client based on scope template

    Client : Sync unfiltered scope, sync dummy scope with newly created filtered scope for that client

    2. yes, make a copy of the snapshot and use that to initialize a client.

    • Marked as answer by P H T Thursday, January 12, 2012 1:09 PM
    Thursday, January 12, 2012 1:04 AM
  • When creating snapshots for redistribution that will allow clients to finalize the process of creating SDFs, the following observations were made that lead me to believe our business model prevents this from being a usable solution.

    1. Generating a snapshot using the following code works as expected for UNFILTERED scopes: SqlCeSyncStoreSnapshotInitialization syncStoreSnapshot = new SqlCeSyncStoreSnapshotInitialization(); syncStoreSnapshot.GenerateSnapshot(SqlCeConnection, NewSdfLocation);

    2. When creating FILTERED scopes, this is the current method used to provision the SDFs for a certain scope: DbSyncScopeDescription scopeDesc = SqlSyncDescriptionBuilder.GetDescriptionForScope(scopeName, SqlServerConnection); SqlCeSyncScopeProvisioning clientProvision = new SqlCeSyncScopeProvisioning(connection, scopeDesc);

    In our business model, we have two(2) UNFILTERED scopes that every sdf will have. Then each user ( each individual SDF ) will have his/her own scope. The preceding steps works very well for us in that it adds the UNFILTERED scopes as a snapshot and we overlay an individual users scope on top of that ( step 2 ) resulting in an SDF that is unique in data for each user ( with the exception of the UNFILTERED scopes which is common/shared data ).

    The goal that we’re attempting to accomplish is we want to allow those individual users ( existing as well as new ) to create the SDF on their own machines ( which just happen to reside outside of the intranet and therefore have no access to the Sql Server for provisioning ). This is to allow expediting the distribution of the SDFs. Please note that NONE of our users will EVER be inside our firewall therefore NEVER have access to SqlServer and cannot provision on their own. Here is what our testing has come to prove:

    1. Using the analogy of a scope being a ‘region’ such as a Northwest region or a Southeast region, allowing users to finalize the SDF creation process will work ONLY if they are sharing a filtered scope such as those ‘region’s. Creating a snapshot as laid out above provisions the SDF to include the necessary table structure and gives the SDF the necessary sync knowledge, scope names, etc. pertaining to the ‘region’. This makes sense and works as expected. This coincides with point number 1 above and works well for our business model to create a snapshot based on UNFILTERED scopes that all of our users will be using. In other words, a ‘region’ is ‘shared’ data just like our UNFILTERED scopes above are shared.
    2. Since our business model dictates each user is his/her own scope, basically his/her own ‘region’, the use of snapshots seems it will not be a logical solution for our goal. Is this a correct conclusion?

    If my conclusion is correct in that using snapshots for (basically) unshared data will not work, is there a work around for this? Is my conclusion correct? Is there a possibility for a more efficient business model that WILL allow us to go the rout in which we desire?


    One more question I have is the use of this method. In what scenario would this be called? Is this maybe a missing piece of the snapshot puzzle?: SqlCeSyncStoreSnapshotInitialization.InitializeSnapshot(SqlCeConnection);

     

    • Edited by P H T Tuesday, February 14, 2012 4:43 PM
    Tuesday, February 14, 2012 4:42 PM
  • how many filtered scopes or "regions" do you have? does the user have the ability to specify dynamically which "region" or filter they want to work on? how are you distributing the SDFs?

    quite frankly, i can't see why your model above cant work.

    as for the InitializeSnapshot method, you dont have to call it explicitly. Sync Framework can detect if the SDF is a snapshot and will run the metadata functions automatically on first sync even if you havent called InitializeSnapshot explicitly.

    Wednesday, February 15, 2012 1:06 AM
  • Each user has 2 filtered scopes that they sync to for the life of their sdf (the only time they will dynamically choose a "region" is during their first sync).

    Yes, when provisioning the sdf during the first sync we will have a user interface (over WCF) that allows the user to select from a list of available scopes to sync to ... after the first sync the user's scope will never change.

    When we get this to work, we intend to distribute the sdfs as follows: 1) create a snapshot while on the LAN 2) FTP the snapshot sdf to the district IT office (in another state) 3) the district office will place a COPY of the snapshot sdf on each new user's device 4) during the first sync, each new user will sync against their own previously provisioned 2 filtered scopes (implemented using WCF UI mentioned above).

    With regard to InitializeSnapshot, the issue we have is that we: 1) create an sdf for any currently provisioned scope 2) create a snapshot from that sdf 3) provision the server with a new/similar scope 4) attempt to sync a copy of the snapshot file to the new scope ... the result is an error that says the new scope cannot be found in the copy of the snapshot sdf.

    Wednesday, February 15, 2012 1:48 PM
  • the WCF part and the distribution looks fine.

    am a bit lost on the last part where you get the error.

    1) create an sdf for any currently provisioned scope - does this include the filtered scope?

    2) create a snapshot from that sdf

    3) provision the server with a new/similar scope  - what's this scope for?

    4) attempt to sync a copy of the snapshot file to the new scope - you're synching an existing scope to a new scope? the error you stated says it cant find the scope in the sdf, when you provisioned a new scope in #3, did you provision the same on the SDF?

    Wednesday, February 15, 2012 2:02 PM
  • #1) yes it includes the filtered scope in this way ... let's say the existing scope is named "100" and i have a new scope "200" (each scope has the same table structure with only the filters being different) ... i create a snapshot from the sdf created with scope "100" then make a copy of that snapshot which will become the sdf for scope "200" when it is sync'd with the "200" scope.

    #3) this scope is the "200" scope mentioned in answer #1 above.

    #4a) "you're synching an existing scope to a new scope?" ... yes to the extent that the snapshot was created using scope "100" whereas a copy of the snapshot is being sync'd using scope "200".

    #4b) "did you provision the same on the SDF?" ... the initial sdf was provisioned for only scope "100", the snapshot sdf implicitly has only scope "100" ... after the snapshot is sent to the district office scope "200" becomes a new scope in the sync community ... this is the root question ... how do we remotely get scope "200" into the sync community?  We can provision the server with scope "200" ... but how do we create an sdf with scope "200" without creating a new snapshot in the main office and going through the whole process of sending it to district etc.?

    Wednesday, February 15, 2012 2:22 PM
  • When u sync, set the scope name to "100" for the local provider, set "200" as scope name for remote. Am assuming the table in scope "100" is actually empty when u initially provisioned it and is just a placeholder for the "real" filtered scope that we discussed earlier in this thread. Scope definitions in SQL CE has no idea about filters even if you created a scope From the server scope using PopulateFromScopeDescription.
    • Edited by JuneT Wednesday, February 15, 2012 2:42 PM
    Wednesday, February 15, 2012 2:36 PM
  • Now the light came on!!! 

    I have incorrectly been trying to get each local sdf to have it's own unique scope name that matches the remote's scope name ... what i instead need is a generic local scope name that can be sync'd against any remote scope name!

    We will play with this ... Thanks!

    Wednesday, February 15, 2012 2:58 PM