locked
Batching in Sync Framework 2.0 RTM

    Kérdés

  • Hello,

    I did not dive into the new 2.0 RTM until now because I application runs fine.
    Now I am running into some memory problems and thinking to implement batching.

    I´ve tried it before but could not solve it, because I am using DateTime for change tracking instead of timestamps in my solution.

    Therefore my question now...

    AFAIK the new 2.0 RTM provides some new features for batching.
    Do I still need timestamps or is there another stable way to implement batching?

    And are there any other memory improvements in the new release?

    For example the posted surrogates:
    http://blogs.msdn.com/mahjayar/archive/2008/10/01/dbsyncprovider-improving-memory-performance-in-wcf-based-synchronization.aspx


    Regards,

    Martin
    • Áthelyezte: Max Wang_Chinasoft 2011. április 19. 23:04 Forum consolidation (From:SyncFx - Microsoft Sync Framework Database Providers [ReadOnly])
    2009. december 11. 15:11

Válaszok

  • I have seen a couple interesting posts around batching wtih datetime these days. as Jin said, this approach is possible techinically. however I think it is worth to emphasize that timestamp or min_active_rowversion is the highly recommended anchor type to accuracy and data convergence's puspose. there are a lot of sample code using Datetime as the anchor type as this is the easy and most straitfoward way to show the idea of sync services and for demo purposes.

    Also, the new db sync providers shipped with MSF V2 (http://msdn.microsoft.com/en-us/sync/default.aspx ) has much robust batching support so I also suggest to evaluate that to see if this can be a better fit to your sceanrio and business needs.

    BTW, I have played with the batching with anchor type a bit and proved ( as the concept level ) that it can be used with batching, there are  a few issues ( and some a bit difficult to resolve ) around it and I believe there will be other issues too if we play with it further -- that is also a reason I wont' recommend this.

    below is the stord proc I used.
    /*
    *	Important Note: this script is merely used to verify that datetime can be used as anchor type for Sync Service V1 batching.
    *		       Datetime as anchor is hightly NOT recommended for sync services, regardless batching is used or not
    *	This proc assumes 1 second for batch size as default and '1901-01-01 00:00:000' is the earliest datetime used in your tracking layer
    *
    */
    create  procedure dbo.sp_new_batch_anchor (
            @sync_last_received_anchor datetime, 
            @sync_batch_size Int,
                  @sync_max_received_anchor datetime output,
                  @sync_new_received_anchor datetime output,            
                  @sync_batch_count Int output)            
    as            
    	print @sync_last_received_anchor
    	print @sync_batch_size
    	print @sync_max_received_anchor
    	print @sync_new_received_anchor
    	print @sync_batch_count
    	
    	-- convert the datetime value to timestamp for easy comuputing
    	declare @sync_last_received_anchor_dbts bigint
    	declare @sync_max_received_anchor_dbts bigint
    	declare @sync_new_received_anchor_dbts bigint
    	
    	declare @minDateTimeVal bigint
    	declare @minDateTime datetime
    	declare @maxDateTimeVal bigint
    	declare @maxDateTime datetime
    	
    	declare @tempBatchCount bigint
    	
    	print 'start proc'
    	
    	select @minDateTime = '1901-01-01 00:00:000'
    	select @minDateTimeVal  = convert ( bigint, convert( timestamp, convert ( binary(8), @minDateTime ) ) )
    	
    	select @maxDateTime = getdate()
    	select @maxDateTimeVal  = convert ( bigint, convert( timestamp, convert ( binary(8), @maxDateTime ) ) )
    	
    	select @tempBatchCount = @sync_batch_count
    	
    	print 'min and max datetime values'
    	print @minDateTime
    	print @minDateTimeVal
    	print @maxDateTime
    	print @maxDateTimeVal
    	
    	if ( @sync_last_received_anchor is null OR @sync_last_received_anchor <= @minDateTime )
    		select @sync_last_received_anchor_dbts = @minDateTimeVal
    	else
    		select @sync_last_received_anchor_dbts =  convert ( bigint, convert ( timestamp, convert ( binary(8), @sync_last_received_anchor) ) )
    	
    	print 'last rec anchor value'
    	print @sync_last_received_anchor_dbts
    	
    	if ( @sync_max_received_anchor is null OR @sync_max_received_anchor <= @minDateTime )
    		select @sync_max_received_anchor_dbts = @maxDateTimeVal
    	else
    		select @sync_max_received_anchor_dbts = convert ( bigint, convert (timestamp, convert ( binary(8), @sync_max_received_anchor ) ) )
    	
        if  @sync_batch_size is null or @sync_batch_size <= 0
                  set @sync_batch_size = 300 -- 1  SECOND
    
    	print 'batch size : '
    	print @sync_batch_size
    	
           -- simplest form of batching
           if @sync_last_received_anchor_dbts is null or @sync_last_received_anchor_dbts <= @minDateTimeVal 
           begin                
    			 print 'simple batching for inital sync'
               set @sync_new_received_anchor_dbts = @maxDateTimeVal
               print @sync_new_received_anchor_dbts
               
               print 'get batch count'
               if @sync_batch_count is null or @sync_batch_count <= 0
                         set @tempBatchCount = (@sync_max_received_anchor_dbts /  @sync_batch_size) + 1
           end
           else
           begin
    			 print 'simple batching for subsequent sync'
               set @sync_new_received_anchor_dbts = @sync_last_received_anchor_dbts + @sync_batch_size
               if @sync_batch_count is null or @sync_batch_count <= 0
    			begin
    					print 'get batch count'
                         set @tempBatchCount = (@sync_max_received_anchor_dbts / @sync_batch_size) - (@sync_new_received_anchor_dbts / @sync_batch_size) + 1
                end
           end
        
        
        --- ensure batch count is Int32 value
        --- better logic will be needed
        print 'temp batch count:' 
        print @tempBatchCount
        
        if ( @tempBatchCount > 1000 )
    		set @sync_batch_count = 100;
    	else
    		set @sync_batch_count = @tempBatchCount
    		
        print @sync_batch_count
        
        -- check if this is the last batch       
        if @sync_new_received_anchor_dbts >= @sync_max_received_anchor_dbts
        begin
            set @sync_new_received_anchor_dbts = @sync_max_received_anchor_dbts        
            if @sync_batch_count <= 0
                         set @sync_batch_count = 1
    
    	end
    	
    	print @sync_batch_count
        
    	print 'max rec and new rec anchor dbts values'
    	print @sync_max_received_anchor_dbts
    	print @sync_new_received_anchor_dbts
    	
    	print 'convert back to datetime for return values'
    	-- convert the values back to datetime before proc returns
    	select @sync_max_received_anchor = convert ( datetime, convert (timestamp, convert ( binary(8), @sync_max_received_anchor_dbts ) )  )
    	select @sync_new_received_anchor = convert ( datetime, convert (timestamp,convert ( binary(8), @sync_new_received_anchor_dbts ) )  ) 
    	
    	print 'max rec and new rec anchor date values'
    	print @sync_max_received_anchor
    	print @sync_new_received_anchor
    
    -- end of proc	
    	

    This posting is provided "AS IS" with no warranties, and confers no rights.
    2009. december 18. 17:59
    Moderátor

Az összes válasz

  • Are you using offline scenario with DbServerSyncProvider &  SqlCeClientSyncProvider instead of collaboration scenario with SqlSyncProvider or DbSyncProvider?
    The Sync framework v2.0 batching support is for collaboration scenario.
    Please check whether that applies to your situation.

    If you are using Sync V2 collaboration providers, then enabling batching should be simple, just need to specify pooling directory and max memory sizes on client and server sides etc.
    There is a code sample with batching support at place like C:\Program Files\Microsoft SDKs\Microsoft Sync Framework\2.0\Samples\SharingAppDemo-CEProviderEndToEnd if you have installed Sync Framework V2. You can take a look.

    2009. december 11. 18:38
    Válaszadó
  • Hi Jin, yes I am using an offline Szenario. Windows 2003 Server with SQL 2005 Server on the one side, WM 2005 and WM 6.1 with SqlCE on the other side. So batching still does not really work in this situation without using timestamps? Regards, Martin
    2009. december 14. 8:23
  • It's possible to use Datetime with batching in offline scenario.
    The batching document is at http://msdn.microsoft.com/en-us/library/bb902828.aspx
    The anchor calculation needs to changed to work on Datetime as the scenario needed.

    2009. december 14. 18:46
    Válaszadó
  • Sorry! I only believe that if you show me a working example :-)

    I tried to change the anchor calculation of that example to DateTime and it did not work
    as well as many others here in the board tried to make Batching working with DateTime.

    Until now I did not heard about anybody who made the current offline-scenario-batching working with DateTime columns :-(


    Regards,
    Martin
    2009. december 15. 8:41
  • Bump.

    Just curious if a batching example using 'DateTime' does exist.

    Damian
    2009. december 17. 22:35
  • I have seen a couple interesting posts around batching wtih datetime these days. as Jin said, this approach is possible techinically. however I think it is worth to emphasize that timestamp or min_active_rowversion is the highly recommended anchor type to accuracy and data convergence's puspose. there are a lot of sample code using Datetime as the anchor type as this is the easy and most straitfoward way to show the idea of sync services and for demo purposes.

    Also, the new db sync providers shipped with MSF V2 (http://msdn.microsoft.com/en-us/sync/default.aspx ) has much robust batching support so I also suggest to evaluate that to see if this can be a better fit to your sceanrio and business needs.

    BTW, I have played with the batching with anchor type a bit and proved ( as the concept level ) that it can be used with batching, there are  a few issues ( and some a bit difficult to resolve ) around it and I believe there will be other issues too if we play with it further -- that is also a reason I wont' recommend this.

    below is the stord proc I used.
    /*
    *	Important Note: this script is merely used to verify that datetime can be used as anchor type for Sync Service V1 batching.
    *		       Datetime as anchor is hightly NOT recommended for sync services, regardless batching is used or not
    *	This proc assumes 1 second for batch size as default and '1901-01-01 00:00:000' is the earliest datetime used in your tracking layer
    *
    */
    create  procedure dbo.sp_new_batch_anchor (
            @sync_last_received_anchor datetime, 
            @sync_batch_size Int,
                  @sync_max_received_anchor datetime output,
                  @sync_new_received_anchor datetime output,            
                  @sync_batch_count Int output)            
    as            
    	print @sync_last_received_anchor
    	print @sync_batch_size
    	print @sync_max_received_anchor
    	print @sync_new_received_anchor
    	print @sync_batch_count
    	
    	-- convert the datetime value to timestamp for easy comuputing
    	declare @sync_last_received_anchor_dbts bigint
    	declare @sync_max_received_anchor_dbts bigint
    	declare @sync_new_received_anchor_dbts bigint
    	
    	declare @minDateTimeVal bigint
    	declare @minDateTime datetime
    	declare @maxDateTimeVal bigint
    	declare @maxDateTime datetime
    	
    	declare @tempBatchCount bigint
    	
    	print 'start proc'
    	
    	select @minDateTime = '1901-01-01 00:00:000'
    	select @minDateTimeVal  = convert ( bigint, convert( timestamp, convert ( binary(8), @minDateTime ) ) )
    	
    	select @maxDateTime = getdate()
    	select @maxDateTimeVal  = convert ( bigint, convert( timestamp, convert ( binary(8), @maxDateTime ) ) )
    	
    	select @tempBatchCount = @sync_batch_count
    	
    	print 'min and max datetime values'
    	print @minDateTime
    	print @minDateTimeVal
    	print @maxDateTime
    	print @maxDateTimeVal
    	
    	if ( @sync_last_received_anchor is null OR @sync_last_received_anchor <= @minDateTime )
    		select @sync_last_received_anchor_dbts = @minDateTimeVal
    	else
    		select @sync_last_received_anchor_dbts =  convert ( bigint, convert ( timestamp, convert ( binary(8), @sync_last_received_anchor) ) )
    	
    	print 'last rec anchor value'
    	print @sync_last_received_anchor_dbts
    	
    	if ( @sync_max_received_anchor is null OR @sync_max_received_anchor <= @minDateTime )
    		select @sync_max_received_anchor_dbts = @maxDateTimeVal
    	else
    		select @sync_max_received_anchor_dbts = convert ( bigint, convert (timestamp, convert ( binary(8), @sync_max_received_anchor ) ) )
    	
        if  @sync_batch_size is null or @sync_batch_size <= 0
                  set @sync_batch_size = 300 -- 1  SECOND
    
    	print 'batch size : '
    	print @sync_batch_size
    	
           -- simplest form of batching
           if @sync_last_received_anchor_dbts is null or @sync_last_received_anchor_dbts <= @minDateTimeVal 
           begin                
    			 print 'simple batching for inital sync'
               set @sync_new_received_anchor_dbts = @maxDateTimeVal
               print @sync_new_received_anchor_dbts
               
               print 'get batch count'
               if @sync_batch_count is null or @sync_batch_count <= 0
                         set @tempBatchCount = (@sync_max_received_anchor_dbts /  @sync_batch_size) + 1
           end
           else
           begin
    			 print 'simple batching for subsequent sync'
               set @sync_new_received_anchor_dbts = @sync_last_received_anchor_dbts + @sync_batch_size
               if @sync_batch_count is null or @sync_batch_count <= 0
    			begin
    					print 'get batch count'
                         set @tempBatchCount = (@sync_max_received_anchor_dbts / @sync_batch_size) - (@sync_new_received_anchor_dbts / @sync_batch_size) + 1
                end
           end
        
        
        --- ensure batch count is Int32 value
        --- better logic will be needed
        print 'temp batch count:' 
        print @tempBatchCount
        
        if ( @tempBatchCount > 1000 )
    		set @sync_batch_count = 100;
    	else
    		set @sync_batch_count = @tempBatchCount
    		
        print @sync_batch_count
        
        -- check if this is the last batch       
        if @sync_new_received_anchor_dbts >= @sync_max_received_anchor_dbts
        begin
            set @sync_new_received_anchor_dbts = @sync_max_received_anchor_dbts        
            if @sync_batch_count <= 0
                         set @sync_batch_count = 1
    
    	end
    	
    	print @sync_batch_count
        
    	print 'max rec and new rec anchor dbts values'
    	print @sync_max_received_anchor_dbts
    	print @sync_new_received_anchor_dbts
    	
    	print 'convert back to datetime for return values'
    	-- convert the values back to datetime before proc returns
    	select @sync_max_received_anchor = convert ( datetime, convert (timestamp, convert ( binary(8), @sync_max_received_anchor_dbts ) )  )
    	select @sync_new_received_anchor = convert ( datetime, convert (timestamp,convert ( binary(8), @sync_new_received_anchor_dbts ) )  ) 
    	
    	print 'max rec and new rec anchor date values'
    	print @sync_max_received_anchor
    	print @sync_new_received_anchor
    
    -- end of proc	
    	

    This posting is provided "AS IS" with no warranties, and confers no rights.
    2009. december 18. 17:59
    Moderátor
  • Hi,

    thanks for the answer. A last question...
    is there a blog post or anything else about the pros/cons of using timestamps instead of DateTime objects?

    I am thinking about adding 2 new timestamp columns to my tables and leave the DateTime columns as it is for the rest of my app.

    Do I understand right that timestamps fix the problem of e.g. mobile devices with wrong time settings (because of flashing the memory or something else etc.).

    Regards,

    Martin
    2009. december 22. 11:06
  • Yunwen,

    I think I speak for alot of people that 'simple examples' of how sync technology works is appreciated, but if I had of known that 'timestamp / min_active_rowversion' was the more preferred method of Sync tracking (and for batching) I would have spent the extra time to understand it and implement it.  Now I have a Sync application with 18 months of development into the 'DateTime' method that follows a 'Highly Not Recommended' technique.

    I dont want to distract from this original threads intention, but if MSDN examples could be more up front with which different techniques are available, and at least provide examples of all techniques instead of 'what is easy to explain' it would go a long way.  Developers dont mind steep learning curb as long as its the correct method.

    On that note, thank you very much for your source code!!!  It was very helpful.

    Damian
    2009. december 23. 4:33