locked
Recommended performance counters RRS feed

  • Question

  • Is there any documentation on what performance counters to monitor for Speech Server? I need to help a client with this and I was wondering if there are any suggested or recommended counter to watch.

     

    I have my own ideas as to what to watch. I'm also familiar with UPL and pass rate etc but I would like to know the "official" stance on this..

    Sunday, February 24, 2008 12:36 AM

Answers

  • Answering from the point of view of expecting things to go wrong during testing, here's what I capture in a log file (using "Add Objects" rather than "Add Counters" in perfmon):

     

    .NET CLR LocksAndThreads(*)\*

    .NET CLR Memory(*)\*

    Process(*)\*

    Processor(*)\*

    SpeechService\*

    TIMC\* (if installed)

    System\*

    Memory\*

     

    For day-to-day use, I store this lot to a .blg file (which can be re-opened in perfmon), with a sample data interval of 15 secs.  When performing a long-running stress test it is beneficial to schedule a new log file periodically to keep the file size down.  The sample interval can also be extended, but this risks missing vital data.  Also, the .csv file format is useful, because it can be re-opened in perfmon or Excel (if not too big) and can also be run through standard text-parsing tools.

     

    When monitoring a deployed system, your needs are different, since the main goal is to keep a check that things are running smoothly, with a secondary goal of capturing useful information in the event of a problem occurring.  For the former, you only really need the SpeechService latency counters, Processor(_Total)\% Processor Time and Memory\Available MBytes.  However for the latter it is useful to have some of the others (favouring more data rather than less, within whatever file-size constraints you have), e.g.

    .NET CLR Memory([w3wp*|SESWorker|SESWorker#1])\*

    Process(*)\% Processor Time

    Process([w3wp*|SESWorker|SESWorker#1])\*

    SpeechService\*

    TIMC\*

    System\Context Switches/sec

    .NET CLR LocksAndThreads([w3wp*|SESWorker|SESWorker#1])\Contention Rate / sec

     

    It's a good idea to track this secondary list all the time, so that if something bad does happen, you have the data available 1st time.  You don't need to keep the data forever; a couple of days worth of data is probably ample to give sufficient coverage of the system prior to the problem occurring to show stable state & events leading up to the problem.  Add to this however much time is needed to cover the period between a problem occurring & someone saving off the logs. 

     

    Word of warning about .NET CLR Memory Process ID and Process\ID Process; if there are multiple instances of w3wp, Process(w3wp) is not necessarily (and commonly isn't) the same process as .NET CLR Memory(w3wp) - use these PID counters to identify which corresponds to which.  Also, when a process quits, the identifies shift up: e.g. if w3wp stops, w3wp#1 becomes w3wp.  Be aware of this when looking at performance counters which cross process recycles - use PID counters to track this change.

     

    For live monitoring of an application (again, during development), I watch a subset, tuned to what can fit on my screen.  An example subset might be the following counters for w3wp (all instances) and SESWorker/SESWorker#1 (where all the SR/TTS happens).  This gives me a good idea of how the system is performing, and I can dig into the log files for more info if needed. 

    .NET CLR Memory\#Bytes in all Heaps

    .NET CLR Memory\#Gen [0|1|2] Collections

    .NET CLR Memory\Induced GC (should remain flat except for steps at app domain recycles)

    .NET CLR Memory\% Time in GC

    .NET CLR Memory\Allocated Bytes / sec

    .NET CLR Memory\Finalization Survivors

    .NET CLR Memory\Gen [0|1|2] heap size

    .NET CLR Memory\Large Object Heap size

    .NET CLR Memory\Process ID

     

    SpeechService\*

     

    Process\% Processor Time

    Process\Elapsed Time (indicates how recently the last process recycle was)

    Process\ID Process

    Process\Priority Base (identifies active / idle instances of SESWorker).

    Process\Private Bytes

    Process\Thread Count

    Process\Virtual Bytes

    Process\Working Set

     

    Monday, February 25, 2008 11:49 AM

All replies

  • Answering from the point of view of expecting things to go wrong during testing, here's what I capture in a log file (using "Add Objects" rather than "Add Counters" in perfmon):

     

    .NET CLR LocksAndThreads(*)\*

    .NET CLR Memory(*)\*

    Process(*)\*

    Processor(*)\*

    SpeechService\*

    TIMC\* (if installed)

    System\*

    Memory\*

     

    For day-to-day use, I store this lot to a .blg file (which can be re-opened in perfmon), with a sample data interval of 15 secs.  When performing a long-running stress test it is beneficial to schedule a new log file periodically to keep the file size down.  The sample interval can also be extended, but this risks missing vital data.  Also, the .csv file format is useful, because it can be re-opened in perfmon or Excel (if not too big) and can also be run through standard text-parsing tools.

     

    When monitoring a deployed system, your needs are different, since the main goal is to keep a check that things are running smoothly, with a secondary goal of capturing useful information in the event of a problem occurring.  For the former, you only really need the SpeechService latency counters, Processor(_Total)\% Processor Time and Memory\Available MBytes.  However for the latter it is useful to have some of the others (favouring more data rather than less, within whatever file-size constraints you have), e.g.

    .NET CLR Memory([w3wp*|SESWorker|SESWorker#1])\*

    Process(*)\% Processor Time

    Process([w3wp*|SESWorker|SESWorker#1])\*

    SpeechService\*

    TIMC\*

    System\Context Switches/sec

    .NET CLR LocksAndThreads([w3wp*|SESWorker|SESWorker#1])\Contention Rate / sec

     

    It's a good idea to track this secondary list all the time, so that if something bad does happen, you have the data available 1st time.  You don't need to keep the data forever; a couple of days worth of data is probably ample to give sufficient coverage of the system prior to the problem occurring to show stable state & events leading up to the problem.  Add to this however much time is needed to cover the period between a problem occurring & someone saving off the logs. 

     

    Word of warning about .NET CLR Memory Process ID and Process\ID Process; if there are multiple instances of w3wp, Process(w3wp) is not necessarily (and commonly isn't) the same process as .NET CLR Memory(w3wp) - use these PID counters to identify which corresponds to which.  Also, when a process quits, the identifies shift up: e.g. if w3wp stops, w3wp#1 becomes w3wp.  Be aware of this when looking at performance counters which cross process recycles - use PID counters to track this change.

     

    For live monitoring of an application (again, during development), I watch a subset, tuned to what can fit on my screen.  An example subset might be the following counters for w3wp (all instances) and SESWorker/SESWorker#1 (where all the SR/TTS happens).  This gives me a good idea of how the system is performing, and I can dig into the log files for more info if needed. 

    .NET CLR Memory\#Bytes in all Heaps

    .NET CLR Memory\#Gen [0|1|2] Collections

    .NET CLR Memory\Induced GC (should remain flat except for steps at app domain recycles)

    .NET CLR Memory\% Time in GC

    .NET CLR Memory\Allocated Bytes / sec

    .NET CLR Memory\Finalization Survivors

    .NET CLR Memory\Gen [0|1|2] heap size

    .NET CLR Memory\Large Object Heap size

    .NET CLR Memory\Process ID

     

    SpeechService\*

     

    Process\% Processor Time

    Process\Elapsed Time (indicates how recently the last process recycle was)

    Process\ID Process

    Process\Priority Base (identifies active / idle instances of SESWorker).

    Process\Private Bytes

    Process\Thread Count

    Process\Virtual Bytes

    Process\Working Set

     

    Monday, February 25, 2008 11:49 AM
  • Great stuff there Anthony. Thanks.

     

    The client is really interested in what to monitor at run time so your post helps a lot.

     

    I tend to look at things like active calls, total calls and calls rejected along with the latency counters and error counters. I also monitor CPU but I don't normally monitor the .NET related stuff.

     

    Monday, February 25, 2008 12:55 PM