Hi StanJo,
The Compute Cluster Scheduler is "process-aware" but not "thread-aware" in the sense that you describe. The scheduler will not distribute 6 threads (belonging to 1 process) such that 4 execute on 1 node and 2 on another. It will schedule the 1 submitted task (corresponding to 1 process) on 1 node and then all 6 threads will execute on that node.
For CCS 2003, the primary schedule-able resources are nodes and processors. For this application (which I assume isn't using MPI, for instance), you would want to schedule exclusive use of 1 node. The application itself (being thread and multi-cpu aware) will utilize all the processors on that node. Of course, if you're application is written to use thread-level cpu affinity, then you'll need at least 6 processors on the node. This, I assume, is why you want to use the scheduler to distribute threads for you automatically, right?
You'll actually need to rethink the application architecture a bit to accomplish your goal. Unfortunately, this forum lacks the capabilities to adequately address the breadth of architectural considerations. You might start with a look at MPI. There is a nice online tutorial at http://www.mhpcc.edu/training/workshop/mpi/MAIN.html and the MPI.net work at http://osl.iu.edu/research/mpi.net/ is also very interesting. The Concurrency and Coordination Runtime (CCR) is an option along with other resources at http://msdn2.microsoft.com/en-us/concurrency/default.aspx.
I hope this helps some...
PhilPenn