Android - How to implement batch processing and transaction in MS Sync Framework 4.0 with httpclient+JSON in android .... Plz help ... Urgent

• maandag 9 april 2012 9:36

   

   
  

  0

  Hi

  we are developing an android app for tablet pc where we are implementing synchronization based on "ListSyncSample" by "Selvin" and in the server side we are using "MS SQL server " with "Microsoft Sync Framework 4.0". In our application we need to sync almost 170 tables with huge data . As per that ListSyncSample Initially it has to download all the data from the server in ONE GO and our app was crashing because of lack of memory to store and process that huge JSON string.... we modified that code and implement batch processing in the server side like this ..... 

  public static void InitializeService(Microsoft.Synchronization.Services.ISyncServiceConfiguration config)
          {
              // TODO: MUST set these values
              // config.ServerConnectionString = "connection string here";
              // config.SetEnableScope("scope name goes here");
              logger.Debug("started : Initialize Service");
              config.ServerConnectionString = ConfigurationManager.ConnectionStrings["DbConnectionString"].ToString();
              config.SetEnableScope("ListSync_Scope");
              config.SetDownloadBatchSize(500); // batch size is 500 //
              bool isBatchEnabled = config.IsBatchingEnabled;            
              config.UseVerboseErrors = true;
              config.EnableDiagnosticPage = true;
          }

  And the modification of the Java code in the android side ......

  public String downloadOnlySync(Context context) throws Exception
  {
  ///Log.d(TAG, "<<<<<<<<<<<<<<<<<<<< downloadOnlySync() >>>>>>>>>>>>>>>> ");
  try
  {
  //Log.d(TAG + "downloadOnlySync", "Inside of downloadOnlySync method");
  DefaultHttpClient httpClient = GetClient();
  HttpRequestBase request = null;
  String defaultserverBlob = null;
  
  defaultserverBlob = getServerBlobFromDatabase(context, ZenUtility.DOWNLOAD_SERVICE_SCOPE);
  //start of download changes 
  boolean moreChanges = false;
  displayProgress("Download Changes started......" , context, ZenUtility.DOWNLOAD_SERVICE_SCOPE);
  int loopCountTest = 0;
  do
  {
  recordsDeleted = 0;
  recordsInserted = 0;
  if (defaultserverBlob != null)
  {
  HttpPost p = (HttpPost) (request = new HttpPost(syncServerUrl + ZenUtility.DOWNLOAD_SERVICE_URL));
  // p.setEntity(new StringEntity(String.format(send, defaultserverBlob, "")));
  Log.d("MySyncService", "Down String Entity :::: " +String.format(send, defaultserverBlob, ""));
  p.setEntity(new GzipCompressingEntity(new StringEntity(String.format(send, defaultserverBlob, ""))));
  
  p = null;
  }
  else
  {
  request = new HttpGet(syncServerUrl + ZenUtility.DOWNLOAD_SERVICE_URL);
  }
  StringBuffer responseText = sendRequest(httpClient, request, ZenUtility.DOWNLOAD_SERVICE_SCOPE);
  //process response header data
  JSONObject obj = new JSONObject(responseText.toString());
  responseText.setLength(0);
  responseText = null;
  JSONObject d = obj.getJSONObject("d");
  obj.remove("d");
  obj = null;
  JSONObject sync = d.getJSONObject("__sync");
  defaultserverBlob = sync.getString("serverBlob");
  moreChanges = sync.getBoolean("moreChangesAvailable");
  JSONArray results = d.getJSONArray("results");
  //process results data
  processResults(context, results, ZenUtility.DOWNLOAD_SERVICE_SCOPE);
  //memory management for JSON Object
  sync.remove("moreChangesAvailable");
  sync.remove("serverBlob");
  d.remove("results");
  d.remove("__sync");
  d = null;
  sync = null;
  displayProgress("Processing Loop:" + loopCountTest +" Info:Deleted(" + recordsDeleted + ") Inserted/Updated:(" + recordsInserted +"\n" , context, ZenUtility.DOWNLOAD_SERVICE_SCOPE);
  //results = null;
  loopCountTest++;
  Runtime.getRuntime().runFinalization();
  }while (moreChanges);
  //update or insert the serverblob in the settings table
  updateServerBlobInDatabase(context, defaultserverBlob, ZenUtility.DOWNLOAD_SERVICE_SCOPE);
  return "DownloadOnlySync :  Completed Successfully \n ";
  }
  catch(Exception ex) {
  ex.printStackTrace();
  //return ("DownloadOnlySync(with Error): \n" + ex.toString() + " Message: " + ex.getMessage());
  throw new Exception("DownloadOnlySync(with Error): \n" + ex.toString() + " Message: " + ex.getMessage());
  }
  }

  The JSON from server looks like this ....

  {"d":{"__sync":{"moreChangesAvailable":true,"serverBlob":"AAEAAAD/////AQAA.......","results":[...........]}}

  Clearly as long as "moreChangesAvailable" is true the app is making request to server and we are getting another batch of data as response and in the final batch "moreChangesAvailable" is false like this ....

  {"d":{"__sync":{"moreChangesAvailable":false,"serverBlob":"AAEAAAD/////AQAA.......","results":[...........]}}

  This part is working fine.

  Same thing we are doing for sending updates from device and getting response for these requests .... the logic is like this

  1) Parse the modified(inserted/updated/deleted) rows of the tables and build the JSON for the modified row (A modified GetChanges() method from listsync  sample)

  of the tables and set isDeleted = -1 and isDirty = -1 of the currently parsed row.

  2) If the length of the JSON is greater than 62000 then create a file(request_file) and store the JSON and go to step 1.

  3) When all the data in the device is parsed(we get a set of files for requests to make requests) then 

  try{

  do {

  3.1      Read one file and send the JSON string to server.

  3.2      create a response_file and store the response for that request.

  } while(total No Of Request Files)

  3.3  Changed all the (-1) marked field(did it in step-1)  into 0.

  do {

  3.4    process one by one response files and modify the local database.

  }while(total No Of Response Files)

  }catch(Network Exception) {

  3.5 Changed all the (-1) marked field(did it in step-1)  into 1.

  }

  Therefore the upload strings for successive three requests look like this(I am using same instance of httpclient for posing this JSON data to the server) .....

  i) {"d":{"__sync":{"moreChangesAvailable":true,"serverBlob":"AAEAAAD/////AQAA.......","results":[...........]}}

  ii) {"d":{"__sync":{"moreChangesAvailable":true,"serverBlob":"AAEAAAD/////AQAA.......","results":[...........]}}

  iii) {"d":{"__sync":{"moreChangesAvailable":false,"serverBlob":"AAEAAAD/////AQAA.......","results":[...........]}}

  The problem is when server get the 1st JSON data it immediately start updating the MS SQL server database without waiting for 2nd and 3rd requests.It seems that "moreChangesAvailable":true or "moreChangesAvailable":false does not matter for the server when data is going from device to server.What I actually want is that server will not update it's database as long as  "moreChangesAvailable" is true ... when server get "moreChangesAvailable":false then it start update it's database with all the data or maintaining a transaction for this batch processing.

  Another problem is SERVER side does not accept non UTF-8 characters.

  NOTE :::: I am not able to send able all the modified data from the device to server in ONE GO because server is not accepting the huge JSON string (although we increased the request data limit in the IIS server and I am sending JSON string in gzip and successfully unzipping that gzip request in the server).Any way we need to implement this batch processing in our project.

  
  

  • Bewerkt door Soubhab maandag 9 april 2012 9:42
  •  
  • Reageren
  • Citeren