Large amount of data/files - my experience. RRS feed

  • General discussion

  • Hi!

    First of all, Mesh is a great service. I use(d) foldershare before and Mesh offered larger limits in terms of amount of files and datasize. So I gave it a try, especially because I have a large directory with about 1000000 files and 100gb size. If found a blogbost that said you tried it with 40gb until now, but those limits aren't hardcoded.

    Ok so first, I tried with two small directories on two pcs. I already had that dir sycned between 2 pcs, using robocopy manually. So they actually were in sync before I applied live to it. So I just added those dirs to live. After a while messages kept popping up telling me that I added a few files and afterwards afew more files. That dir consists of approx. 6000 files and 120mb in size. It somehow posted msgs about chunks of between 100 - 1000 files. I didn't care about that, but what surprises me is, from time to time it said, I updated a file. Looking at those files in both dirs, they were still the same, except for the timestamp. Their timestamps were identical before, after such a message the timestamp of one file was like before, but the other changed to the current time. Apart from that the files were still identical. I don't know the reason for this. Maybe it's related to the fact that both dirs weren't empty at the beginning? Maybe one had to be empty? I used foldershare a while back and it didn't have such issues.

    Ok, then I applied it to my big directory - after 12 hours, new message still popped up about me adding 100 - 1000 files. The directories were already in sync like before. robocopy takes about 40 min for one run on that data. After 12 hours, mesh wasn't able to see that they are in sync? What's the bottleneck here? I also got the problem discribed above about updated files where timestamps were set to now for some reason. I guess that's too much data for now for mesh. Oh, and I didn't sync it to live desktop of course it wouldn't fit. The pcs are connected with gige lan. I guess it would/could have workes, if I waited longer - but how long? There is no progress bar or anything. Foldershare had an activity window at least, those messages don't tell me what's the current status. :(

    So for now, I'm back to manually syncing unless you have some tips to speed it up/fix this. The whole point is to save time and reduce mistakes by forgetting to sync manually etc. It's probably too much data for now.

    One PC is vista sp1 64 and the other vista sp1 32. Both 4gb ram. This remeinds me, after those 12 hours of letting mesh work, the memory was filled. One PC was ahrdly usable. I used one to disable syncing in live desktop for both pcs and deleted the folder. The pc with the large memory consumption still thinks that folder is a live folder, the mesh bar apars next to it. And it's been 30 min since then. This was also the pc where I added the dir initially to the mesh.

    Best regards

    Friday, May 1, 2009 11:12 AM

All replies

  • I just robocopied my large folder again from one pc to the other to get rid of these strange timestamp changes. Now I realize, it actually copies a lot of files telling me they are new. Looking at my 2nd pc, a lot of files were actually deleted by live mesh. Why did that happen? Those files were in sync - there was no need to delete them?!?

    Hopefully, all files on my initial live mesh folder creation pc are still there and not deleted.
    Friday, May 1, 2009 12:19 PM
  • The Live Mesh team is working to improve the sync capabilities of Live Mesh to improve the experience for conflict resolution. As you have encountered, the process for adding a folder where the files are already in sync often causes the files to be transferred over anyway and updated. Any files deleted by Live Mesh should be in the local PC Recycle Bin.
    Microsoft MVP Windows Live / Windows Live OneCare & Live Mesh Forum Moderator
    Friday, May 1, 2009 2:58 PM
  • I realized the files get moved to the recycle bin. But may recycle bin is set to 100mb only and as I said I treid syncing 100gb. Luckily, only on the pc I added 2nd, files were deleted. So I disabled Live Mesh for that folder and ran robocopy again to fix it. Besides that, I have a backup as well.

    If this scales to my needs some time, I'll try it again.
    Friday, May 1, 2009 3:27 PM
  • Glad to read that you had the foresight to have a backup of your data - just in case. Do check back as I'm sure that Live Mesh will improve moving forward. As i said, sync works well, but can behave oddly, especially in conflict resolution.
    Microsoft MVP Windows Live / Windows Live OneCare & Live Mesh Forum Moderator
    Friday, May 1, 2009 4:22 PM
  • Marc,

    my personal experience is, that you should start wie only ONE PC that has the data you want to syncronize. Keep the sync-folders on the second PC empty, and let Live Mesh do the job.

    Keeping many, many files and folders in-sync will slow down the start-up process. I finally gave up and now use Live Mesh only for folders that are updated frequently. And it works like a charm.

    Sunday, May 3, 2009 12:50 PM
  • Hi 3Thinkpad,

    what do you mean by "start-up process"? The initial synchronization run or every time a pc boots and starts Live Mesh again?

    Tuesday, June 23, 2009 1:23 PM