DbSyncException - The internal array cannot expand to greater than Int32.MaxValue RRS feed

  • Question

  • Im syncing a larg DB using the 2.1 sync framework. more that 10GB.

    My MemoryDataCacheSize = 200000, ApplicationTransactionSize= 100000. So i use batch sync. I use 64 bit bould and the program is compiled in 64 bits.

    The error that i get after the generation of the 20 batch file is:

    Microsoft.Synchronization.Data.DbSyncException: An unexpected error occured when spooling changes to a batch file. See the inner exception for more details. ---> System.Runtime.Serialization.SerializationException: The internal array cannot expand to greater than Int32.MaxValue elements.

    Full stack trace:

    at System.Runtime.Serialization.ObjectIDGenerator.Rehash()
       at System.Runtime.Serialization.ObjectIDGenerator.GetId(Object obj, Boolean& firstTime)
       at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.InternalGetId(Object obj, Boolean assignUniqueIdToValueType, Type type, Boolean& isNew)
       at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteString(NameInfo memberNameInfo, NameInfo typeNameInfo, Object stringObject)
       at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteKnownValueClass(NameInfo memberNameInfo, NameInfo typeNameInfo, Object data)
       at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteArrayMember(WriteObjectInfo objectInfo, NameInfo arrayElemTypeNameInfo, Object data)
       at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteArray(WriteObjectInfo objectInfo, NameInfo memberNameInfo, WriteObjectInfo memberObjectInfo)
       at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Write(WriteObjectInfo objectInfo, NameInfo memberNameInfo, NameInfo typeNameInfo)
       at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Serialize(Object graph, Header[] inHeaders, __BinaryWriter serWriter, Boolean fCheck)
       at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize(Stream serializationStream, Object graph, Header[] headers, Boolean fCheck)
       at Microsoft.Synchronization.Data.DbSyncBatchInfoFactory.Serialize(DbSyncBatchInfo batchInfo, String batchFileName)
       at Microsoft.Synchronization.Data.DbSyncBatchProducer.EnqueueBatchAndRaiseEvent(DbSyncBatchInfo batchInfo)
       at Microsoft.Synchronization.Data.DbSyncBatchProducer.PrepareAndQueueBatch(Guid batchId, DataSet batchDataSet, Int64 dataCacheSize, Exception exception, Boolean isLastBatch)
       --- End of inner exception stack trace ---
       at Microsoft.Synchronization.Data.DbSyncBatchProducer.DequeueBatch()
       at Microsoft.Synchronization.Data.RelationalSyncProvider.ConsumeBatchFromProducer(DbSyncScopeMetadata scopeMetadata)
       at Microsoft.Synchronization.Data.RelationalSyncProvider.GetChanges(DbSyncScopeMetadata scopeMetadata, DbSyncSession DbSyncSession, UInt32 memoryBatchSize)
       at Microsoft.Synchronization.Data.RelationalSyncProvider.GetChangeBatch(UInt32 batchSize, SyncKnowledge destinationKnowledge, Object& changeDataRetriever)
       at Microsoft.Synchronization.KnowledgeProviderProxy.GetChangeBatch(UInt32 dwBatchSize, ISyncKnowledge pSyncKnowledge, ISyncChangeBatch& ppChangeBatch, Object& ppUnkDataRetriever)
       at Microsoft.Synchronization.CoreInterop.ISyncSession.Start(CONFLICT_RESOLUTION_POLICY resolutionPolicy, _SYNC_SESSION_STATISTICS& pSyncSessionStatistics)
       at Microsoft.Synchronization.KnowledgeSyncOrchestrator.DoOneWaySyncHelper(SyncIdFormatGroup sourceIdFormats, SyncIdFormatGroup destinationIdFormats, KnowledgeSyncProviderConfiguration destinationConfiguration, SyncCallbacks DestinationCallbacks, ISyncProvider sourceProxy, ISyncProvider destinationProxy, ChangeDataAdapter callbackChangeDataAdapter, SyncDataConverter conflictDataConverter, Int32& changesApplied, Int32& changesFailed)
       at Microsoft.Synchronization.KnowledgeSyncOrchestrator.DoOneWayKnowledgeSync(SyncDataConverter sourceConverter, SyncDataConverter destinationConverter, SyncProvider sourceProvider, SyncProvider destinationProvider, Int32& changesApplied, Int32& changesFailed)
       at Microsoft.Synchronization.KnowledgeSyncOrchestrator.Synchronize()
       at Microsoft.Synchronization.SyncOrchestrator.Synchronize()

    What can i do to make it work? I need a urgent fix or a workaround! Please help anybody knows what could  the sollution.


    Friday, August 31, 2012 10:22 AM

All replies

  • do you have blobs in your columns? can you check if you have rows bigger than your batch size?
    Monday, September 3, 2012 1:21 AM
  • No there are just many columns in the table. After i lowered MemoryDataCacheSize to 50000 and ApplicationTransactionSize= 10000 there was not anymore any problem.

    I think it was the problem in serializing the batch xml file. to many objects but that is strange anyway the value for Int32.MaxValue is 2,147,483,647. there is no chance that so many objects ware really there. On gogole it seams that is a "bug" from microsoft in xml binary serialization .net. I could not found any solution for this or any other workaround.Lowered the MemoryDataCacheSize  is not good for performance.

    Is there any way to use the sinhronization in nativ from c#? It would be better that the batching would do the nativ layer! Is there any example of a custom implemented SqlSyncProvider on the internet? Has anyone in the community made it it would be great to have just some pointers what can be done and how!


    Tuesday, September 4, 2012 6:41 AM
  • what do you mean native with c#? native layer?
    Tuesday, September 4, 2012 7:17 AM
  • I mean calling unamanaged api from c#.  I sure the core if sync framework is coded in nativ, unmanaged code not in .net or not? Basically it would be nice to have a whole provider writen in unmanaged and  just calling the API of the native provider. Maybe an unmanaged sqlprovider already exists? That means the batching xml serialization would be all be in unmanaged code.

    So all the extra logic of the provider that is now in .NET (batching, xml serialization, db connection,... don't know what else)  would be better performance.

    Wednesday, September 5, 2012 9:27 AM
  • the unmanaged API is in the docs.

    there is no unmanaged SQLSyncProvider though, you'll have to write one if you want. am not entirely sure how much performance gain you will get when most of the activities by a database sync provider is spent on the database.

    Wednesday, September 5, 2012 10:26 AM