The Performance Log

The performance log contains details about how long it takes for a job or service to run. It includes overall performance information for the job or service as well as performance information for each stage in the job or service dataflow. You can use this information to identify bottlenecks in your dataflow by looking at the execution time and processing time for each stage. A large difference between execution time and processing time means that the stage is spending time waiting for data from upstream stages. This may indicate that an upstream stage is a bottleneck in the dataflow. Note that for sinks, a large difference between execution time and processing time does not necessarily indicate a performance problem because sinks typically have to wait for the first records from the rest of the dataflow.

To enable performance monitoring for a job or service, use the performancemonitor enabled set command in the Administration Utility.

The performance log is located on your Spectrum Technology Platform server in the following location:

SpectrumDirectory\server\logs\performance.log

The performance log contains one row for each run of a monitored job or service. It is a rolling log that consists of a maximum of five files. Each file is limited to 10 MB in size. Once this limit is reached, the oldest performance data is deleted when new performance data is logged.

Note: The log file path name, maximum file size, and maximum number of files are specified by the PERFORMANCE appender settings in the logback.xml configuration file. For more information, see Logback configuration file.

Each entry in the performance log contains the following information. For ease of reading, line breaks and indentation are shown below. In the actual log, the entry is on one line.

Date Time [performance]
{
     "username":"UserName",
     "dataflowId":"DataflowName",
     "runMode":"BatchOrRealTime",
     "elapsedTime":Nanoseconds,
     "stageInfo":[
          {
               "stageName":"Name",
               "stageLabel":"Label",
               "options":{
                    OptionsList
               },
               "recordsRead":Count,
               "recordsWritten":Count,
               "executionTime":Nanoseconds,
               "processingTime":Nanoseconds
               "readBlockingTime":Nanoseconds
               "writeBlockingTime":Nanoseconds
               "readBlockingPercent":Percentage
               "writeBlockingPercent":Percentage
          }
     ]
}

Where:

username

The user who executed the job or service.

dataflowID

The name of the service or job as defined in Spectrum Enterprise Designer.

runMode

Indicates whether the log entry is for a job or a service. One of the following:

Batch: The log entry is for a job.
RealTime: The log entry is for a service.

elapsedTime

The time in nanoseconds that it took to run the job or service request.

stageInfo

Lists runtime information for each stage in the dataflow. The following information is listed for each stage:

stageName: The permanent name of the stage.
stageLabel: The user-defined name of the stage. The stage label is shown on the canvas in Spectrum Enterprise Designer.
options: If any options were specified at runtime, those options and their settings are listed here.
recordsRead: The total number of records that passed into the stage through all the stage's input ports.
recordsWritten: The total number of records that the stage wrote to all its output ports.
executiontime: The amount of time from when the stage processed its first record to when it processed its last record. This includes the time the stage was idle while waiting for data from other stages in the dataflow.
processingtime: The amount of time the stage spent actively processing records, not including the time it was idle while waiting for other stages in the dataflow.
readBlockingTime: The amount of time blocking while waiting to read the next record. A high read blocking time means that a prior process is taking longer than this stage and may indicate additional tuning is needed.
writeBlockingTime: The amount of time blocking while waiting to write the next record. A high write blocking time means that a prior process is taking longer than this stage and may indicate additional tuning is needed.
readBlockingPercent: The percentage of the total execution time that a stage was blocking on reading a record.
writeBlockingPercent: the percentage of the total execution time that a stage was blocking on writing a record.