The server is not counted, but sometimes you have record for a formerly joined machine stuck in the registry of the Home Server.
Login to the server and run regedit.
Check the subkeys below registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Home Server\Transport\Clients (I assume you will find 10 keys here) for a obsolete machine. If you think you found one delete the key after exporting it to a .reg file, just in case.
Reboot and try again to add a client.
Best greetings from Germany
Olaf