Hi,
According to your description, my understanding is that VSS writers went to timed out status on WHS 2011 once Arcserve initiate the backup.
It is recommended to fully patches the WHS 2011 with Windows Update/Hotfix, it is helpful for resolving some known issues and improving the performance.
Besides, you may check the Event Viewer and confirm that if any related event has been logged. It might be helpful for further identify this problem.
As your description, WHS built-in backup works well, and this problem only happens when using the tool – Arcserve. Compatibility should be considered.
As it is a third-party product which is manufactured by company that are independent of Microsoft, I would recommend you to contact this tool’s official website or supporter to identify this problem.
Best Regards,
Eve Wang
Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Support, contact tnmff@microsoft.com.