locked
Order of Synchronisation RRS feed

  • Question

  • We are looking at using Sync Framework to store transactional data on a remote web device (Amazon S3) as an intermediate store between two systems which may not necessarily be connected at the same time - hence the intermediary store. We use the fact that S3 uses UTC to guarantee the order of packets as part of their meta data and it works very well.

     

    We are looking at implementing our solution using the partial participant paradym in that S3 will hold the file and the Sync Framework metadata as an autonomous unit. Our issue is not necessarily getting the storage on the S3 server right with the Sync Framework - the issue is attempting to retrieve the informaton in the order they were produced.

     

    Consider this:

    A Quote is raised at time X - this is formed as a transaction file and sent to S3 - it is assigned its own Item in the Knowledge sync on the producers side.

    The Quote is revised at time Y and only the deltas are sent. Again this is formed as a file and sent to S3 - it is assigned its own item in the knowledge sync on the producers side.

     

    If X is consumed before Y, we do not have an issue. However, it we get Y first, we get an exception as the Quote is not known.

     

    My question is simple - is there anyway via the Knowledge that we can guarantee that the Item associated with X and therefore the file is guaranteed to be consumed prior to the Item associated with the revised quote at time Y.

     

    Looking at it simplistically, the consumer will retrieve the metadata from the S3 file which contains the Knowledge information as part of its metadata. It will look at its local store of Knowledge and determine that the file on the S3 is new, so will attempt to consume that transaction from the S3 server. My thoughts are that if the knowledge information returned for comparison between the S3 and the consumers local copy guarantees that the S3 knowledge is stored in oldest-to-youngest order, will the Sync occur in that order  ?????

     

    I have run the FileSync sample and it appears to Sync based on the timestamp ordering - which appears to be the order a standard FindFirst/FindNext or DirInfo would be returned.

     

    Thanks for any enlightenment.

    Friday, September 5, 2008 7:36 AM

Answers

  • Assuming that you will write you own provider, you will be able to control the order of change enumeration, and hence the destination provider will see changes in the order the source provider enumerated them.

     

    To make it easier for the source provider to enumerate changes in the right order, you could have the ItemId such that it is pre-fixed by the S3 in-order metadata. This will also allow you to report changes in "ranges", which will make the whole sync more performant.

     

    Even if you don't report changes in the right order, the destination provider could reject changes it cannot apply. Then you would sync again till you have no more changes to apply. This is of course not so performant, so the best option is to report changes in the right order.

    Friday, September 5, 2008 5:10 PM
    Moderator

All replies

  • Assuming that you will write you own provider, you will be able to control the order of change enumeration, and hence the destination provider will see changes in the order the source provider enumerated them.

     

    To make it easier for the source provider to enumerate changes in the right order, you could have the ItemId such that it is pre-fixed by the S3 in-order metadata. This will also allow you to report changes in "ranges", which will make the whole sync more performant.

     

    Even if you don't report changes in the right order, the destination provider could reject changes it cannot apply. Then you would sync again till you have no more changes to apply. This is of course not so performant, so the best option is to report changes in the right order.

    Friday, September 5, 2008 5:10 PM
    Moderator
  • You might consider using a Prerequisite Knowledge check to accomplish what you are asking.  Prerequisite Knowledge is a piece of Knowledge that must be known by the destination before a set of changes can be applied.  Each change batch for instance has a prerequisite knowledge associated with it which is the base level of knowledge that must be known in order to safely apply the change batch. 

     

    To check to see if a prerequisite knowledge is contained you can simply take the destination’s knowledge and call ContainsKnowledge with the prerequisite as the parameter.  If the knowledge is contained then it is safe to apply your change.  If not you can defer the change to a later point and try again later when some other changes have been applied.

     

    Hope that helps.

     

    Michael Clark

     

    Friday, September 5, 2008 5:36 PM
    Moderator