I would plug a regular telephone into the phone line and try to call it. See if you can answer a regular call to that phone.
If not- then the problem is with your phone line.
If the call does go through - I would try a different modem.
Faxing from SBS server is rather simple in that it just works - there isn't a lot of troubleshooting or specific things you can do to fix problems like this. Most of the problems that I have had with faxing from SBS have been hardware/modem related - solved by trying a different modem.
I've had poor results with external USB modems. I've had the best results with external serial modems like the old school US Robotics modems.
I hope this helps.
-Eric