Hi Ryan,
According to the documentation no you do not need to have users enabled for enterprise voice but you will need medaition servers deployed to have PSTn users able to dial in to the conference. I think you will find that use of conferencing is dictated by meeting policy in OCS.
"Conferencing Attendant is designed to use components and features of Enterprise Voice by default, but you can configure support for dial-in conferencing in your organization whether or not you have deployed Enterprise Voice for Office Communications Server 2007 R2."
Hope this helps.
Cheers
Chris
http://voipnorm.blogspot.com/