locked
Uploading large data amount of image blobs using ServerSyncProviderProxy, WCF issue RRS feed

  • Question

  •  

    Hi all

     

    In the evaluation of Sync Framework for ADO.NET I am trying to upload large amounts of image data in blobs.

     

    This is what I found:

    • Uploading large amounts of data (tested 3 images with 1.2 mb each) in 2 Tier config works like a charm
    • Uploading very small abounts of data in 3 tier config (ServerSyncProviderProxy, tested 3 images with 5 kb each) works fine too.
    • Downloading medium amounts of data in 3 tier config works.

    Uploading large amounts of data with ServerSyncProviderProxy fails with 400, BadRequest error. I tried to give enough reader quotas etc in the WCF configuration, but could not resolve the issue.

     

    Any ideas? Thanks!

    • Moved by Hengzhe Li Friday, April 22, 2011 2:36 AM (From:SyncFx - Microsoft Sync Framework Database Providers [ReadOnly])
    Wednesday, June 11, 2008 8:03 AM

Answers

  • Hi Sam, Everyone

    I finished building a custom proxy which allows streaming the changes over http from client to server. It works very well, I am pleased with the results ;-).

    Streaming can help in the following scenarios:
    - Clients are offline for a long time and thus will post a very big collection of changes to the server the next time the clients go online. I was unhappy with the possibility that those changes could no more be posted due to upload limits of WCF
    - Clients produce a lot of data in relatively short time, such as documenting real live objects in the form of lots of pictures (which is the case in the project I am working on). In this scenario, even with frequent synchronizations, there is a lot of data transfered between client and server.

    For testing purpose, I uploaded 6 pictures with 1.2 mb size each. This took around 1 minute and 15 seconds to upload to the server as base64 encoded xml message using default streaming configuration.

    Steps I took to build the custom proxy:
    1. Build wrapper classes for the requests and responses issued by the sync framework (GetChanges, ApplyChanges, ...)
    2. Configure the wrapper classes for using them in a WCF Service (DataContract and DataMember Attributes)
    3. Write the server code to serialize and deserialize the wrapper classes (using DataContractSerializer)
    4. Configure the server app.config file for http streaming
    5. Configure the client app.config file for http streaming
    6. Write the Custom proxy which serializes and deserializes the wrapper classes, again by using DataContractSerializer
    7. Replace the Standard Proxy (ServerSyncProviderProxy) with the custom proxy
    Cheers
    Oliver

    Sunday, June 15, 2008 4:18 PM

All replies

  • Hi

    can you take a look at this post and tell me if it answer to you question :
    http://forums.microsoft.com/Forums/ShowPost.aspx?PostID=3365118&SiteID=1

    Thanks

    Sam
    Wednesday, June 11, 2008 7:41 PM
  • Hi Sam

     

    Thanks Sam for your input.

    However, I am not (yet? Wink ) talking about a performance issue. I have the problem of syncing lob data and was not yet able to make it run.

     

    I think the solution will be using streaming to avoid the upload limits using buffered WCF.

    Thursday, June 12, 2008 5:19 AM
  • Ok, am getting smarter concerning WCF....

    With WCF, one solution for large uploads is to use HTTP Streaming instead of buffered mode. As Streaming does change the signature of the involved service methods, I will try to implement a custum proxy which uses HTTP Streaming to pass on the Sync Framework data.

    Will post when I managed to accomplish this
    Oliver
    Friday, June 13, 2008 5:31 PM
  • Hi,

    I am interested in your solution of streaming to see how it is implemented but i am not sure that you will get interesting performances.

    Thanks for keeping me uploaded

    Sam
    Friday, June 13, 2008 9:59 PM
  • Hi Sam, Everyone

    I finished building a custom proxy which allows streaming the changes over http from client to server. It works very well, I am pleased with the results ;-).

    Streaming can help in the following scenarios:
    - Clients are offline for a long time and thus will post a very big collection of changes to the server the next time the clients go online. I was unhappy with the possibility that those changes could no more be posted due to upload limits of WCF
    - Clients produce a lot of data in relatively short time, such as documenting real live objects in the form of lots of pictures (which is the case in the project I am working on). In this scenario, even with frequent synchronizations, there is a lot of data transfered between client and server.

    For testing purpose, I uploaded 6 pictures with 1.2 mb size each. This took around 1 minute and 15 seconds to upload to the server as base64 encoded xml message using default streaming configuration.

    Steps I took to build the custom proxy:
    1. Build wrapper classes for the requests and responses issued by the sync framework (GetChanges, ApplyChanges, ...)
    2. Configure the wrapper classes for using them in a WCF Service (DataContract and DataMember Attributes)
    3. Write the server code to serialize and deserialize the wrapper classes (using DataContractSerializer)
    4. Configure the server app.config file for http streaming
    5. Configure the client app.config file for http streaming
    6. Write the Custom proxy which serializes and deserializes the wrapper classes, again by using DataContractSerializer
    7. Replace the Standard Proxy (ServerSyncProviderProxy) with the custom proxy
    Cheers
    Oliver

    Sunday, June 15, 2008 4:18 PM
  • Hi Olivier,

    your solution seems nice =) But with my actual level in C# I dont really see how did you implement that. Could you send your code in order to check how it s working, at braxivamov@hotmail.fr ?

    In both cases thank you.

    Sam
    Monday, June 16, 2008 6:09 PM
  • Hi Sam, all

     

    I can provide you with the prototype code. The code is a Modification of the original 2 Tier Sample I found in this Forum (See my Post Getting started.....)

     

    I changed the demo in the following significant ways:

    - Use guids instead of ints for primary keys

    - Use n tier sync over WCF Webservice instead of 2 Tier Sync with direct Db connection to the server db

    - Use Http streaming to up- and download large amounts of data

     

    Steps you need to make to run the proto (Hope I remember all of them):

    - Run the demo.sql code in sqlserver 2005 server database to create server side tables, triggers and data

    - In ServiceForSync.css line 33 and following: change connection settings to your server db

    - In SyncForm.css line 28 change the path to the local database

    - In SyncForm.css line 750 change the code to Point to a jpeg Picture you would like to load to the server

    - In client and server project check the WCF parameters in app.config to make sure the communication fits your environment

     

    I uploaded the VS2008 solution to Megaupload.com so all forum users can access the prototype solution. Sorry for the blinking and annoying contact advertisements .

    http://www.megaupload.com/de/?d=980IIC2A

     

    Regards

    Oliver

     

    Oh.. If this post is helpful, please click the helpful button.....

     

    • Proposed as answer by loopmaster Wednesday, July 22, 2009 5:09 PM
    Wednesday, June 18, 2008 9:32 AM
  • Hi Oliver,
    thanks for your response. This is really nice from you to upload your code. Unfortunately for me I am using VS2005, so it wont fit but it will help me to see how to implement a streaming solution. Thanks again.

    Oh.. Ok I'll click the helpfull button even if you already click it for your own post Wink

    See you
    Wednesday, June 18, 2008 6:33 PM
  • Cheers Sam

     

    I simply marked the post concerning http streaming as answer to my question, which is not the same as marking any post as helpful..

     

    Oliver

     

    Thursday, June 19, 2008 5:51 AM
  • Hi Oliver,

    Thanks for posting the source example, I found it really useful to get me on my way, thank you.

    One thing for others reading this post.

    Having implemented a streaming solution, two of my test suites still failed with a

    The server did not provide a meaningful reply; this might be caused by a contract mismatch, a premature session shutdown or an internal server error.

    error message.

    The problem is that I was hosting my service on IIS, which means that I also need to tell IIS to expect large files being uploaded. You can do this by adding the following to the <configuration> section in your host's web.config file

    Code Snippet

    <httpRuntime maxRequestLength="64000" executionTimeout="100" />




    maxRequestLength is in Kbytes.

    Hope this helps.

    Regards

    Aidan


    Monday, September 1, 2008 2:28 PM
  • Hi Oliver,
    Your Solution works perfectly fine, thanks for the code.

    Aidan - Thanks for the tip in the Web.Config.
    Saturday, May 23, 2009 5:52 AM
  • J Oliver -

    U DA MAN!!!!

    I have been building an application similar to yours that synchronizes data between SQL Server compact and SQL Server 2008 using the sync framework.  My application stores files uploaded as BLOBs in the database.  In building it I have stumbled across several gotchas in the technologies I have been using:

    WPF
    Composite Application Guidance
    Entity Framework
    SQL Server CE
    SQL Server 2008
    Microsoft Sync Framework
    WCF
    ClickOnce Deployment
    and others...

    I knew when I came up against the problem with pushing the file BLOBs through the sync framework across WCF I had hit a difficult problem to solve.  I figured I had a couple of days ahead of me to find a solution.  Then I came across this post.

    You have saved me countless hours!!  Your sample was excellent!  Perfectly streamlined (no bloated code) and easy to understand.  Thanks a million!!

    On a related note, you don't have a similar code example for layering batching on top of this do you?  ;)  Not only does my application need to send files but at times it will also need to send a large number of changes.  While the streaming helps with the files, I figure i'll have to implement batching when the changeset has 50 files at roughly 1MB a  piece.  Do you think I am right in this assumption?

    Thanks Again!
    Stewart
    Thursday, July 16, 2009 12:23 AM
  • Wow!

    That sample is great!

    You saved my day (an some further days of googling around the www)!

    Thank you very much, Oliver!

    Regards & Grüße
    Sebastian
    Wednesday, July 22, 2009 5:09 PM