Another Batch Processing Issue

All replies

• 

  

  0

  Sign in to vote

  hi steve,

  am assuming in your test above, you ran the tests using the same set of data, meaning same number of rows ( and same size).

  in the tests where it didnt do batching (batch > 2102, w/c is in ur test is 2103) have you tried adding up the rowsizes for all the rows in the change set to see if they are less than or equal to the batch size * 110% (ie., 2103*1.1)?

   

   

  Thursday, April 29, 2010 10:48 AM
• 

  

  0

  Sign in to vote

  June,

  Same data. In fact, I have been simply breakpointing and stopping the test at the point in the ProcessChangeBatch() in the client where you get an indication of the batching status for thge changes to be uploaded to the server:

  Code in client:

   
  public override void ProcessChangeBatch(ConflictResolutionPolicy resolutionPolicy, ChangeBatch sourceChanges, object changeDataRetriever, SyncCallbacks syncCallbacks, SyncSessionStatistics sessionStatistics)
  {
   
    this.Log("ProcessChangeBatch called...");
   
    DbSyncContext context = changeDataRetriever as DbSyncContext;
   
    if (context != null && context.IsDataBatched)
  

  This is different from the situation you are thinkng about wrt to the 110% of batchsize. In that situation the server side would throw an exception because a record would not fit in its batchsize. In this situation, when batching does not occur, a changeset message containing the 18382 inserts is actually being sent over the WCF wire as a single non-batched transaction (and the remote end ie the client end is throwing an exception because it exceeds my 6mb WCF message limit).

  In other words, if I made the receiving side have a maximum WCF message size of 10MB, this sync works. But that's besides the point because I want to have batching. And, if I set the value to 2102, I get it!

  I guess that the para above indirectly answers your question about the sum of the rowsizes for all rows: more than 6mb and less than 10mb.

  So, there are two ways for me to get this particular change set to work:

  a) Have a maximum WCF message size of 10mb. This change I fully understand why it would work but it is not a solution because another changeset coule be bigger than 10mb and besides I want batching.

  b) Have the batchsize selected be less than 2103. This gets me my batches. But I don't understand why, so this is not a solution either.

  Steve

   p.s. There is reference code for this at http://msdn.microsoft.com/en-us/library/dd918908(SQL.105).aspx

  • Edited by Speedware Thursday, April 29, 2010 2:16 PM Put correct code snippet in post

  Thursday, April 29, 2010 1:49 PM
• 

  

  0

  Sign in to vote

  Hi,

  When you send the changes to the other provider, the batches are generated on the client side. It's not related to what the other provider's architecture is. i.e, it should not matter whether the other provider is on WCF service or just a local 2-tier provider. That means, if data changes are exactly same and the min batch size is same, if in n-tier the batching is not happening then in 2-tier it will not happen either.

  How do you set the server batch size in above scenarios, have you checked the proxyProvider's GetSyncBatchParameter is really returning 2102 or 2103 for batch size?

  After you set the batching (Batch directory, batch size and whether cleanup batch dir when sync is done ) on the client sending provider, you can check the batch directory and see whether batch files are generated, when the breakpoint is hit at the Proxy provider's ProcessChangeBatch or use batching events to verify.

  You could check the same for a 2 tier scenario as well, though no breakpoint can be set, but you can set cleanupBatchingDir to false and check whether the batch directory has any files or use batching events to verify.

  The data changes for all experiments should be enough for a change batch to be generated. You could generate 2 - 3 * batchSize changes in all cases.

  Thanks. 

  Thursday, April 29, 2010 5:31 PM

  Answerer
• 

  

  0

  Sign in to vote

  Hello Jin,

  First let me just say that what I reported is what actually happened, not what "should be enough for a change batch to be generated". So, when I said no batch was generated, I meant that context.IsDataBatched was false (see code snippet above). My results reported were not theoretical ("should"), they were actual.

  How do I set the batch server batch size in the above scenarios? Provider.MemoryDataCacheSize = someValue; As you correctly identify in another of my postings on batch processing (http://social.microsoft.com/Forums/en/syncdevdiscussions/thread/9aa6eae9-5ac5-404f-9e16-f7aca840b05a), the GetSyncBatchParameters() method in its BatchSize parameter returns the value set for Provider.MemoryDataCacheSize.

  So when I say that the client had value X, I meant that there is a statement Provider.MemoryDataCacheSize = X; ... and that debug reinforced this.

  Have I checked the the proxyProvider's GetSyncBatchBatchParameter is really returning 2102 or 2103? Simple answer: Yes.

  When I set Provider.MemoryDataCacheSize to X, Provider.GetSyncBatchParameters() returns X. So when X is 2102, 2102 is returned. When X is 2103, then 2103 is returned. And so forth.

  Regarding your last paragraph, I agree with what should happen. The point of the post is that it does not.

  The original post closed with these questions:

  What is special about 2102 and why is batching being disabled when the smallest value of client and server is larger than 2102?

  Thanks

  Steve 

  Friday, April 30, 2010 4:50 AM
• 

  

  0

  Sign in to vote

  I have forwarded the reported issue to relevant people for investigation.

  From my experience, I had created different batches with size from a few kbs to dozens of MBs. With enough data changes to sync, I never see batching disabled.

  We will contact you if we can not reproduce this issue on our side.

  Thanks.

   

  Monday, May 3, 2010 4:41 PM

  Answerer
• 

  

  0

  Sign in to vote

  Hello Jin,

  Thank you for the update.

  I just want to stress that this is an uploading batching issue only (that is, the source provider is the one that decides that there should be no batching).

  I have 200,000+ record downloads containing 50MB of data that are successfully broken up into 2MB+ batch files when the batch size/MemoryDataCahcheSize is set to 3000 at both ends.

  Here is an overview of the architecture.

  Sql2005 - SqlSyncProvider - SyncOrchestration - CustomRelationalSyncProvider(Proxy) - WCF - CustomServiceClass - SqlSyncProvider - Sql 2008

  Here is the setup that reproduces the problem for me:

  • Local Sql2005 has a 3 tables with a sum of 18,000 updates.
  • Local SqlSyncProvider has MemoryDataCacheSize set to 3000.
  • Local SyncOrchestrator initiates an upload sync.
  • Remote Sql2008 has no records in the 3 tables.
  • Remote SqlSyncProvider has MemoryDataCacheSize set to 3000.
  • Scope has never been sync'ed before.
  • Local database was provisioned with SqlSyncScopeProvisioning.
  • Remote database was provisioned with SqlSyncScopeProvisioning.
  • During provisioning SetPopulateTrackingTableDefault was set to create (the default, but I thought I'd identify it for you).
  • SyncOrchestrator calls GetKnowledge() on proxy provider.
  • ProxyProvider, via WCF, calls GetKnowledge on remote service.
  • Remote service calls GetKowledge() on remote provider which returns the MemoryDataCacheSize in the BatchSize parameter, namely 3000.
  • Remote service returns BatchSize, again 3000.
  • ProxyProvider receives BatchSize (3000) and returns it to the SyncOrchestrator.
  • SyncOrchestrator does whatever it does with the sourve provider to create the change set and ends up calling ProcessChangeBatch() in proxy provider.
  • Proxy provider examines context.IsDatabatched (see code snippet previously posted above). It is not batched and, by inspection, the change batch is seen to contain the 18,000 inserts.

  Now, in the above, if the MemoryDataCacheSize at the remote provider is 2103 or above, then no batching occurs as per above. If it is 2102 or less, but not 0, batching does occur and the first batch consist of about 6,000 records (IIRC). If it is 0, batching does not occur and there are the 18,000 records in the change batch.

  The same results (of batching or no batching) can be achieved by modifying the value from 3000 to X (either in debug or simply by hard coded assignment) in the 3rd from last step, namely

• ProxyProvider receives BatchSize (3000) and the value is modified to X and then the proxy returns it to the SyncOrchestrator.

  That is, it does not appear to be anything to do with the remote side of things.

  HTH

  Steve