none
Feed HPC logs into splunk RRS feed

  • Question

  • Hi,

    We use splunk to collect and analyse logs across our machines.
    Is there a way to feed hpc node logs into splunk ?

    Thanks,
    Candide
    Monday, July 18, 2016 11:30 AM

Answers

  • All HPC logs are ETW trace logs and by default we provided our own listener to dump the logs into disk. You can easily replace them with your own listener and feed the logs into splunk.

    For example, in %CCP_HOME%bin\hpcscheduler.exe.config file, there is session about listeners, where you can add your own and replace with yours.

        <sharedListeners>
          <!-- Define listener -->
          <add name="SchedulerLog"
               type="Microsoft.Hpc.Trace.HpcTraceListener, Microsoft.Hpc.Trace"
               initializeData="%CCP_LOGROOT_SYS%\Scheduler\HpcScheduler" FileSizeMB="4" MaxAllowedDiskUsageInMB="10000" />
        </sharedListeners>

        <trace autoflush="false" useGlobalLock="false">
          <listeners>
            <!-- Listen to trace -->
            <remove name="Default" />
            <add name="SchedulerLog" />
          </listeners>
        </trace>
         
        <sources>
          <!--enable tracing for specific source-->
          <source name="HpcScheduler" switchValue="All">
            <listeners>
              <remove name="Default" />
              <add name="SchedulerLog" />
            </listeners>
          </source>
          <source name="HpcFileStaging" switchValue="All">
            <listeners>
              <remove name="Default" />
              <add name="SchedulerLog" />
            </listeners>
          </source>
        </sources>    

    Other services are similar.


    Qiufang Shi

    Tuesday, July 19, 2016 2:33 AM

All replies

  • All HPC logs are ETW trace logs and by default we provided our own listener to dump the logs into disk. You can easily replace them with your own listener and feed the logs into splunk.

    For example, in %CCP_HOME%bin\hpcscheduler.exe.config file, there is session about listeners, where you can add your own and replace with yours.

        <sharedListeners>
          <!-- Define listener -->
          <add name="SchedulerLog"
               type="Microsoft.Hpc.Trace.HpcTraceListener, Microsoft.Hpc.Trace"
               initializeData="%CCP_LOGROOT_SYS%\Scheduler\HpcScheduler" FileSizeMB="4" MaxAllowedDiskUsageInMB="10000" />
        </sharedListeners>

        <trace autoflush="false" useGlobalLock="false">
          <listeners>
            <!-- Listen to trace -->
            <remove name="Default" />
            <add name="SchedulerLog" />
          </listeners>
        </trace>
         
        <sources>
          <!--enable tracing for specific source-->
          <source name="HpcScheduler" switchValue="All">
            <listeners>
              <remove name="Default" />
              <add name="SchedulerLog" />
            </listeners>
          </source>
          <source name="HpcFileStaging" switchValue="All">
            <listeners>
              <remove name="Default" />
              <add name="SchedulerLog" />
            </listeners>
          </source>
        </sources>    

    Other services are similar.


    Qiufang Shi

    Tuesday, July 19, 2016 2:33 AM
  • I see, so basically I could hook up some of the standard splunk listeners for .net like this one https://github.com/splunk/splunk-library-dotnetlogging
    Tuesday, July 19, 2016 10:13 AM