(For those who are wondering, handbrake is an open source video transcoder...)
I don't know anything specific about the tool (I don't use it) but probably you would want to rewrite it as a WHS add-in. I can think of several ways to approach the problem, but none of them are going to be simple. One that might work is to "segment" the tool, putting UI elements in an add-in and the code that does the grunt work in a service.
I'm not on the WHS team, I just post a lot. :)