I've looked through the forums and all the threads concerning PS3 streaming are a bit out of date.
Currently streaming working fine for me, the PS3 sees the WHS and I can play all PS3-supported codec/media types.
Where it gets complicated is that I have a lot of x264 mkv files that I would like to transcode on-the-fly with software that will run directly from the WHS server. Is anyone transcoding on-the-fly? If so, what are you using?
You can try using Tversity it supports on-the-fly transcoding of mkv files. I have PlayON running on my WHS which transcodes and streams hulu to my Xbox 360 and it works perfectly.
I found that PS3 Media Server is much better than TVercity. TVercity often cannot convert formats and has performance issues with high bitrate movies where PS3 Media Server is fine.
Have you specifically tried to transcode x264 mkvs on the fly with PS3 Media Server? I was using it prior to setting up my WHS server and I couldn't get the transcoding to work, the process seemed kind of hacked together.