locked
Sync Framework uses too much memory RRS feed

  • Question

  • Hello,

     I work with sync by using this example: http://msdn.microsoft.com/en-us/library/dd918908(v=sql.105).aspx .

    But when I run upload on the small database (about 100Mb of bussiness data)

    SyncOrchestrator syncOrchestrator = new SyncOrchestrator();

    syncOrchestrator.LocalProvider = new SqlSyncProvider(clientScope, this.clientConnection,

    string.Empty, objectSchema);

    syncOrchestrator.RemoteProvider = new SqlSyncProviderProxy(this.syncServerUri.AbsoluteUri,

    scopeName);

    syncOrchestrator.Direction = SyncDirectionOrder.Upload;

    var statistics = syncOrchestrator.Synchronize();

    where class SqlSyncProviderProxy inherit RelationalProviderProxy.

     SyncFramework uses a lot of memory (about 3 Gb) and send 4Gb of data to the server.
    How can I reduce memory using and data transmitting?

    P.S.
    How can I enable DataBatching mode?
      I set syncOrchestrator.LocalProvider.MemoryDataCacheSize and syncOrchestrator.RemoteProvider.MemoryDataCacheSize to 50000 but it doesn't help.

    In RelationalProviderProxy.cs :
    DbSyncContext context = changeDataRetriever as DbSyncContext;
    if (context != null && context.IsDataBatched)
    {
    context.IsDataBatched always false.


    Regards, Denis.



    • Edited by kp_denis_r Tuesday, December 9, 2014 7:33 PM
    Tuesday, December 9, 2014 7:21 PM

All replies

  • do you have blob columns in your tables?
    Wednesday, December 10, 2014 2:11 AM
  • do you have blob columns in your tables?
    No I don't.
    I have nvarchar(MAX) columns, but they have no data.

    Extra memory used when SF do this (from trace log):

    ----- Checking for Outdated Peer -----
    Source Forgotten ReplicaKeyMap: [....
    Destination is not outdated
    --- End Checking for Outdated Peer ---


    Also in public override void ProcessChangeBatch(...)
    ChangeBatch sourceChanges, object changeDataRetriever have a lot of data
    when I send it by wcf it produces about 4Gb of tcp traffic.
    Its intersting because my database has about 100 Mb of data.

    Maybe using batching mode with calls UploadBatchFile, DownloadBatchFile can help me?
    But I don't understand how to enable it, context.IsDataBatched always false.
    Wednesday, December 10, 2014 4:59 AM
  • setting the Memory Cache Size should enable batching already.

    how did you measure the memory consumption of the sync app? is this a newly provisioned scope? can you check the actual size of the knowledge columns in the scope_info tables?

    Thursday, December 11, 2014 5:08 AM
  • setting the Memory Cache Size should enable batching already.

    how did you measure the memory consumption of the sync app? is this a newly provisioned scope? can you check the actual size of the knowledge columns in the scope_info tables?

    1. By windows process manager, this process makes only synchronization.
    All memory of this process is a sum of "Sync Framework" + "Wcf buffer" + "small other part".

    2. Yes it is new provisioned scope and empty database. 

    3.  knowledge size is 13188

    >>  setting the Memory Cache Size should enable batching already.

    I noticed that batch files in batching dir are created, but this code never called:

    DbSyncContext context = changeDataRetriever as DbSyncContext;
    if (context != null && context.IsDataBatched)
    {
        string fileName = new FileInfo(context.BatchFileName).Name;
        string peerId = context.MadeWithKnowledge.ReplicaId.ToString();
        if (!this.proxy.HasUploadedBatchFile(fileName, peerId))
        {
            FileStream stream = new FileStream(context.BatchFileName, FileMode.Open, FileAccess.Read);
            byte[] contents = new byte[stream.Length];
            using (stream)
            {
                stream.Read(contents, 0, contents.Length);
            }
            this.proxy.UploadBatchFile(fileName, contents, peerId);
        }
    
        context.BatchFileName = fileName;
    }

    because context.IsDataBatched is always false and context.BatchFileName is always null

    How can I enable network batching i.e. send data by batches?

    Thursday, December 11, 2014 6:41 AM
  • am a bit confused about your setup, are you running all of this on one machine?

    if its a new scope, knowledge should be empty.

    and you shouldn't be seeing that much traffic either because the dataset is being sent as a byte array.

    run a WCF trace and see what's being sent.

    nothing special about batching. if you set the memory data cache size, that should kick in batching automatically.

    the only reason that will not kick in, is the data to be synched is lower than batch size or the data is greater than the batch size and it will not split row if it contains blob/varchar(max)/etc...

    Thursday, December 11, 2014 7:30 AM