This chapter provides information about recording files, annotating files, and other support programs.
This chapter explains the contents of recording files and how the files can be created and processed. It then describes each of the following programs collectively known as the recording support programs:
The main program for analyzing recordings is the azizo program described in "Analyzing Performance Recordings with azizo" .
Note: You can access these programs from the xmperf Command Menu Interface.
Recording files are binary files whose first record is a configuration record. This record identifies the file as a recording file, names the source of the recording, and states the version of the file. To be valid, recording files must also contain the definition of one or more statistics and must contain at least one value record.
Recording files are created by one of the Agent or Manager programs. They can be created by the xmperf and 3dmon programs during monitoring, by the xmservd daemon at any time it is running, by the program a2ptx from ASCII files that adhere to a certain format, or by the ptxrlog program.
From the definition of statistics, a program reading the recording file can determine how the statistics are grouped into sets of statistics (statsets). By the nature of the record layout, at least one such set exists but the definition records may define multiple. Sets of statistics are defined by the program that creates the recording:
Recordings created by xmperf also contain console definition records. This record type describes the layout and other properties of the console that was used to create the recording and is used by xmperf to reconstruct the console when the recording is played back by xmperf. Recordings created by other programs do not contain console definition records. When such recordings are played back with xmperf, default consoles are constructed as explained in the section entitled "Creation of Playback Consoles" .
Records carrying the observations are called value records. They correspond to the sets defined in the recording file and contain one reading for each of the statistics in the set plus the time stamp of the reading and the elapsed time since the previous reading. Each reading consists of two fields:
A special type of value record is the stop record which signals that recording was stopped for a statset and gives the time it happened. This allows programs using the recording file to distinguish between gaps in the recording and variances in recording interval.
Several programs can modify recording files. As they do so, they may preserve or discard information about sets of statistics and consoles in the files. The following sections describe how recording files can be modified. More detail is provided in the detailed description of each of the programs later in this chapter and in "Monitoring Remote Systems" .
The azizo program allows you to write a filtered recording file from an input recording file. Filtering can write a subset of statistics for a subset of the time span covered by the input recording file. If the input recording file contains more than one definition of sets, or if it contains a console definition, you can elect to discard these definitions and create the filtered file with only one set and no console definition. Creation of filtered recording files with the azizo program is described in the article entitled "Filtered Recordings" .
The program ptxmerge allows you to merge multiple recording files into one. When merging only two recordings, the program can be asked to adjust the time stamps of one of the recordings. If the multiple recordings you merge into one all contain identical console definitions, the console and set definitions are retained. Otherwise, all console definitions are discarded while set definitions are retained.
The ptxmerge program also allows you to reorganize recording files where multiple recordings are concatenated into one file. It does so by first splitting the files into separate files, then merging them together.
Very large recording files can be time consuming to analyze. The program ptxsplit can be used to divide such files into smaller files. Splits can be done as simply as dividing the file into sections where the only change to the sequence and contents of records is that each output file has a copy of the configuration and definition records.
More advanced features allow you to split the file into groups of selected statistics. When this is done, you have the option of preserving or discarding definition records, depending on how you have specified the split to take place. For some splits, it may not be possible to preserve the definition records.
Recording files created by Version 1.1 of the Performance Toolbox for AIX can be converted to the format used by Versions 1.2 or 2. If so desired, the program also does conversion the other way. Recording files created by Version 1.1 do not have the special stop records and neither do files converted from Version 1.1 format. When you analyze such files with azizo, a stop in the recording is interpreted as an extraordinarily long sampling interval.
Note: Annotation files are available with Version 2.2 or later only.
Annotation files are plain ASCII text files. They are associated with recording files only through a naming convention. For example, the annotation file for a recording file named R.time_off would beN.time_off. If the name of a recording file is changed to something that does not begin with the R. prefix, the association with the annotation file is lost, even if the annotation file is renamed.
If the recording file is moved to a different directory and the annotation file is not moved to the same directory, then the association with the annotation file is lost.
If the recording file is processed with the recording support programs, then the modified (and differently named) new recording file will no longer have any association to the annotation file. However, none of the recording support programs change the original recording file so the assocation to those files still exists.
Annotation files can be created and modified from xmperf, 3dmon, 3dplay, and azizo. From xmperf and 3dmon, annotation files can only be created and modified if recording is, or has been, activated. From 3dplay and azizo, the user can create or modifiy annotation files at any time.
The a2ptx program takes a file with a tabulated list of data as input and produces a recording file in a format that allows the file to be processed by any recording support program and by xmperf and azizo. The purpose is to extend the usability of Performance Toolbox for AIX to cover other types of data or performance data produced by programs that are not part of Performance Toolbox for AIX.
For a2ptx to successfully create a valid recording file from an input file, the latter must be in a certain format. When used with their -s command line flags, the programs ptxtab and ptxrlog produce output in a format suitable for a2ptx. For ptxrlog, the -t flag can also be used. There are rules for the following parts of the input file:
For the following explanation, please refer to "Figure 3. example of ptxtab spreadsheet Output Format " , which shows an example of a valid a2ptx input file.
If a string in the format hostname: xxxxx is at the end of the very first line of the input file, this causes a2ptx to prefix all statistic names it later reads with the string hosts/xxxxx. The keyword hostname: may start with a capital H and must be followed by a colon. The xxxxx part can be any value you want. The two parts must be separated by white space. If the host name string is not found on the very first line of the file, no prefix is added to the statistic names.
Before any time stamps or data values appear in the input file, a line beginning with the string"Timestamp" must exist. The double quotes are optional. Following the identifier string on the line must be one column heading for each column of data values on data value lines. The column headings may optionally be enclosed in double quotes and are used as the fully qualified name of the statistic records written to the output recording file. The line's identifier string must be separated from the column headings with white space, which must also separate the column headings.
Each line with data values must begin with a time stamp optionally enclosed in double quotes and followed by white space. The format of the string is:
Time stamps should appear in ascending order to make the resulting output file usable in other Performance Toolbox for AIX programs.
Following the time stamp on data lines must be a number of data values separated by white space. Data values may have a decimal point or be integers. The number of data values on each line must be the same as the number of column headings. If one or more data lines do not have data available for a value, a dash must be inserted in place of the missing data value.
The command line to invoke a2ptx is:
The ptxmerge program has two modes of operation:
When you supply only one input file name on the ptxmerge command line, it is assumed that you want to rearrange the records in that file. In all other cases, merging is assumed.
The actual implementation of the rearrange function divides the input file into separate temporary files, one for each set of control information in the file. Those temporary files are then merged into one to create the final output file and deleted.
The created output file always has only one group of control records at the beginning of the file. This set is created from the groups of control records in the input files. If all input files were created by xmperf and have identical definitions (were created from identically configured consoles that contain identical instruments), all control records, including the console definition, are preserved and written to the output file.
If there's the slightest difference between the control records of the input files, then ptxmerge creates the output file so that any console information is discarded. Definitions of sets of statistics (statsets) is then only retained if the -z command line argument is used. In all other cases, the resulting output file contains only a single set of statistics (statset).
The value records that carry observations of statistics are arranged so they appear in time order in the resulting output recording file.
The ptxmerge program is safe to use because it always leaves the original files unchanged and it is reasonably fast in doing what it does. It can be used to organize recordings for optimal analysis or playback, and it works in concert with the other recording support programs as well as the filtering function of azizo. The following two sections give some examples of when to use ptxmerge.
The xmservd daemon can produce recording files with more than one set of control information. It happens whenever xmservd resumes recording to an existing recording file and the xmservd recording configuration file (xmservd.cf) has been changed since the recording file was created. It is done to prevent the recording from being corrupted when changes are made to the sets of statistics being recorded.
The xmservd daemon makes no attempt at keeping track of what the changes to its configuration file were, so it is very possible that the two or more sets of control information in the recording file are identical. On the other hand, there's no guarantee that they are. The ptxmerge program allows you to rearrange the file and detects if the sets are indeed identical. If they are, the resulting output file are identical to the input file, except that only the first set of control records is preserved.
If a recording file produced by xmservd has different sets of statistics, or if a recording file was produced by concatenating two or more recording files with different sets of statistics, then the sets are merged into one single set of statistics unless the -z command line argument is used.
The only other program that can create recording files with more than one set of control information is ptxrlog .
There can be many reasons to merge one or more recording files into one. Generally, it is done because you want to analyze multiple recordings as one with azizo or because you want to play multiple recordings back as one with xmperf. The separate recording files may represent recordings from the same invocation of xmperf, but from different instruments. They may be produced simultaneously on different hosts, possibly to record the effects of a distributed application on the application's server and client sides. Or they may represent identical recordings for different time periods that you want to analyze together.
If the intention is to play the recordings back with xmperf, then it is often a good idea to use the -z flag to preserve instrument definitions. This allows you to keep track of the original sets of statistics, which is especially important if you use ptxmerge to adjust the time stamps of one of the input files.
The ptxmerge program allows the user to specify up to 10 input files that are to be merged into one file. All files must be valid Performance Toolbox for AIX recording files in Version 2 format. When more than one input file is specified and one or more of the input files contain multiple sets of control information, only the records belonging to the first such set participate in the merge operation.
If only one input file is given, the program assumes you want it to rearrange the records in that file. If this file contains only one set of control information, then the output file is identical to the input file.
The command line to invoke ptxmerge is:
ptxmerge [ -m| -p incr|-t inc r] [ -z] outfile input1 [ input2 [ input3...]]
The command line flags have the following meaning:
At times, it is advantageous to divide one recording file into multiple. This may be because the file is too large to allow timely analysis or playback, because it contains statistics that are irrelevant for the current use of the file, or because it contains more than one set of control records. In either case, ptxsplit splits a recording file to your specifications.
The ptxsplit program is invoked with the following command line:
ptxsplit { -p parts| -s size| -h| -b| -f cfile| -d hhmm [ -t dhhmm]} infile
The command line arguments are all mutually exclusive, except that the -t argument is only valid if the -d argument is given. One of the arguments must be specified. The arguments are:
The program ptxls can produce a list of the statistics contained in a recording file. The output from the program has the format required for the control file. Use it by redirecting ptxls output to a file; then edit the file to include only the statistics you want in the file infile.sel.
If the -t argument is omitted, the time period begins with the earliest value record in the input file; otherwise with the time specified on the -t argument. The output files all begin with a set of control records. The output file names are infile.d1, infile.d2, ... infile.dn. Statsets are preserved in the output as are any console records.
d = Day of week, Sunday = day 0.
The time given may lie outside the time period covered by the input recording file. If the time given differs from the time stamp of the first value record in the input file, the first output file contains data for an interval smaller than that requested with the -d argument.
For example, assume a recording file's first value record has a time stamp corresponding to 30830 (day 3, at 8:30 a.m.) and you invoke ptxsplit with the command line:
ptxsplit -d0600 -t00000 recording_file
This causes the first file to cover the interval from 8:30 a.m. until 11:59 a.m., the next one from 12:00 noon until 5:59 p.m., and so on until there's no more value records in the input file.
Consider splitting the same file with the command line:
ptxsplit -d0600 -t40800 recording_fileThe -t argument, in this case, gives a point in time later than the first value record's time stamp. The program determines the time to place the first split point by stepping backwards in time from day 4 at 8:00 a.m. in steps of six hours (as per the -d argument) until it has passed the time stamp of the first value record. This would be on day 3 at 8:00 a.m. This is the reference point. The first output file covers day 3 from 8:30 a.m. to 1:59 p.m., the next from 2 p.m. to 7:59 p.m., and so forth.
In Version 1.1 of the Performance Toolbox for AIX, recording files could only be created with the xmperf program. The only other program in Version 1.1 that used recording files was the xmtab program, which is renamed ptxtab in later versions of Performance Toolbox for AIX. Neither program was concerned with the possibility of a recording file being created from a monitoring session where the recording of instruments was stopped and started one or more times. Similarly, neither program ever attempted to read a recording file backwards.
With the introduction of the azizo program and the recording support programs, this changed. The azizo program wants to know if a recording session was stopped and restarted, and to speed the analysis, the program reads some records off the end of the recording file. Consequently, the recording file format was changed so that stop records are added when a recording stops, and so a recording file can be read backwards. The changes mean that Version 1.1 files can not be processed by azizo or other Version 2 or Version 1.2 Performance Toolbox for AIX programs.
To allow a user of Version 1.1 of Performance Toolbox for AIX to upgrade to later versions without losing the ability to view existing recording files, the ptxconv program was added to Performance Toolbox for AIX. This program allows conversion between Version 1.1 and Version 2 and Version 1.2 recording files in either direction.
Note: Once a recording file has the Version 1.1 format, whether it was created that way or was converted from a later format, then a conversion to the later format will not add stop records to the file. This causes azizo to treat gaps in the recording of a set of statistics as an extraordinarily long sampling interval.
The ptxconv program is invoked with the following command line:
ptxconv -v{1|2} input_file output_file
All command line arguments are required and have the following meaning:
A simple utility program, ptxtab lets you format a recording for tabulated output. In earlier versions of Performance Toolbox for AIX, this program was called xmtab. The program takes a recording file as input and produces one output file for each set of statistics (statset) in the recording file.
Each of the output files are named by suffixing the statset sequence number to the name of the recording file. If the recording file has an original file name as created by xmperf or 3dmon, the initial "R". is changed to "A". to distinguish between Recordings and ASCII output files. This also ensures that ASCII output files do not show up in the dialog window used to select recordings in the programs xmperf and azizo. As an example, assume the recording file $HOME/XmRec/R.NiceMonitor was created by xmperf and has instruments (statsets) with sequence numbers 3 and 8. Running the ptxtab program would then produce the ASCII output files $HOME/XmRec/A.NiceMonitor_3 and $HOME/XmRec/A.NiceMonitor_8.
By default, each of the output files produced by ptxtab is formatted for printing with a multiline heading that begins with a page eject. The first line lists the name of the console (if console information is available in the recording file) or program (when the recording file was not created by xmperf) that created the recording and the host name of the host providing the data. The second line is blank and the remaining lines provide headings for each column of tabulated data. The following is an example of output from ptxtab as produced with no command line flags.
#Monitor: Nice Monitor --- hostname: nchris
Mem Mem
PagSp PagSp Virt Virt
Timestamp %totalused %totalfree pagein pageout
1994/01/07 15:36:03 27.8 72.2 8 20
1994/01/07 15:36:07 27.8 72.2 7 17
1994/01/07 15:36:11 27.8 72.2 3 283
1994/01/07 15:36:15 27.8 72.2 28 48
1994/01/07 15:36:19 28.2 71.8 56 41
1994/01/07 15:36:23 29.5 70.5 29 38
1994/01/07 15:36:27 31.5 68.5 0 62
1994/01/07 15:36:31 32.4 67.6 70 1
1994/01/07 15:36:35 32.6 67.4 73 32
1994/01/07 15:36:39 32.8 67.2 156 0
1994/01/07 15:36:43 34.5 65.5 167 4
1994/01/07 15:36:47 34.4 65.6 163 0
1994/01/07 15:36:51 31.1 68.9 12 57
1994/01/07 15:36:55 30.2 69.8 35 34
1994/01/07 15:36:59 28.0 72.0 15 0
1994/01/07 15:37:04 28.0 72.0 15 0
The ptxtab command line looks as follows:
ptxtab [-l lines|-c|-s] [-r|-t] recording_file
The following shows an example of ptxtab comma-separated output format:
#Monitor: Nice Monitor --- hostname: nchris Time="1994/01/07 15:36:03", PagSp/%totalused=27.82 Time="1994/01/07 15:36:03", PagSp/%totalfree=72.18 Time="1994/01/07 15:36:03", Mem/Virt/pagein=8 Time="1994/01/07 15:36:03", Mem/Virt/pageout=20 Time="1994/01/07 15:36:07", PagSp/%totalused=27.82 Time="1994/01/07 15:36:07", PagSp/%totalfree=72.18 Time="1994/01/07 15:36:07", Mem/Virt/pagein=7 Time="1994/01/07 15:36:07", Mem/Virt/pageout=17
The following shows an example of ptxtab spreadsheet output format:
#Monitor: Nice Monitor --- hostname: nchris
"Timestamp" "PagSp/%totalused" "PagSp/%totalf
ree" "Mem/Virt/pagein "Mem/Virt/pageout"
"1994/01/07 15:36:03" 27.8 72.2 8
20
"1994/01/07 15:36:07" 27.8 72.2 7
17
"1994/01/07 15:36:11" 27.8 72.2 3
283
"1994/01/07 15:36:15" 27.8 72.2 28
48
"1994/01/07 15:36:19" 28.2 71.8 56
41
"1994/01/07 15:36:23" 29.5 70.5 29
38
"1994/01/07 15:36:27" 31.5 68.5 0
62
"1994/01/07 15:36:31" 32.4 67.6 70
1
"1994/01/07 15:36:35" 32.6 67.4 73
32
"1994/01/07 15:36:39" 32.8 67.2 156
0
"1994/01/07 15:36:43" 34.5 65.5 167
4
"1994/01/07 15:36:47" 34.4 65.6 163
0
"1994/01/07 15:36:51" 31.1 68.9 12
57
"1994/01/07 15:36:55" 30.2 69.8 35
34
"1994/01/07 15:36:59" 28.0 72.0 15
0
"1994/01/07 15:37:04" 28.0 72.0 15
0
The ptxls program allows you to list the control records of a recording file. You can use it to find out if more than one set of control records exist, and it tells you which statistics are contained in a recording file. In addition, it lists the time period covered by the recording and the number of value records (observation count) for each set of control records in the file.
The output created by ptxls is suitable for use with the -f option of the ptxsplit program. That option requires a file name of a file that contains a list of statistics to extract from a recording. This file can be created by redirecting the output of ptxls to a file and then deleting the path names that should not be extracted.
The ptxls command line takes no flags but you must specify the name of a recording file to process.
The first output line names the set of control records in the line saying "Configuration." Then, each instrument in the console is identified by one line saying "Console" giving the number assigned to the instrument plus one line for each of the statistics belonging to that instrument (statset). Each statistic line ends with two numbers. The first identifies the statset and the second is the statistic identifier within the statset.
The following is an example of ptxls output. The example is created from a recording of the xmperf console "Combo Style Sample" as supplied in the sample xmperf configuration file:
# Configuration: ID=Combo Style Sample # Console #00001: ID=Combo Style Sample hosts/nchris/SysIO/writech 00001/00002 hosts/nchris/SysIO/readch 00001/00003 # Console #00002: ID=Combo Style Sample hosts/nchris/Mem/Real/%local 00002/00001 hosts/nchris/Mem/Real/%clnt 00002/00003 hosts/nchris/Mem/Real/%free 00002/00005 # Console #00006: ID=Combo Style Sample hosts/nchris/Disk/hdisk0/xfer 00006/00001 hosts/nchris/Disk/hdisk1/xfer 00006/00002 hosts/nchris/Disk/hdisk2/xfer 00006/00003 hosts/nchris/Proc/pswitch 00006/00005 hosts/nchris/Proc/runque 00006/00007 # Console #00003: ID=Combo Style Sample hosts/nchris/CPU/cpu0/wait 00003/00001 hosts/nchris/CPU/cpu0/kern 00003/00002 hosts/nchris/CPU/cpu0/user 00003/00004 hosts/nchris/Syscall/total 00003/00005 # Console #00008: ID=Combo Style Sample hosts/nchris/CPU/cpu0/kern 00008/00001 hosts/nchris/CPU/cpu0/wait 00008/00002 hosts/nchris/Mem/Real/%free 00008/00003 hosts/nchris/PagSp/%totalfree 00008/00004 hosts/nchris/Proc/swpque 00008/00005 # Console #00004: ID=Combo Style Sample hosts/nchris/Disk/hdisk0/busy 00004/00001 hosts/nchris/Disk/hdisk1/busy 00004/00002 hosts/nchris/Disk/hdisk2/busy 00004/00003 # Console #00009: ID=Combo Style Sample hosts/nchris/Mem/Virt/pagein 00009/00001 hosts/nchris/Mem/Virt/pageout 00009/00002 hosts/nchris/Mem/Virt/pgrclm 00009/00003 hosts/nchris/PagSp/hd6/%free 00009/00004 hosts/nchris/Mem/Real/%free 00009/00006 # Statistics for above: Start time Wed Jan 12 18:11:19 1994 # End time Wed Jan 12 18:20:09 1994 # Observation count 1881 #
The ptxrlog program can produce recordings in either ASCII format, which allows you to print the output or postprocess it with database or spreadsheet programs or with the a2ptx program to produce a standard Performance Toolbox for AIX recording file, or it can produce a standard Performance Toolbox for AIX recording file in binary format. Statistics to record are specified from a control file, the command line, or both. If ptxrlog is executed in the background, the list of statistics to record must be specified in a control file.
The ptxrlog program uses the RSI to access statistics and can collect and record statistics from one host at a time across the network. All statistics are defined in one statset.
The ptxrlog command line looks as follows:
ptxrlog {-f infile|-m|-mfinfile} [-h hostname] [-i seconds] [-o outfile [-c|-s|-t]|-r binoutfile] [-l pagelen] [-b hhmm] [-e hh.mm]
hh = Number of hours to record.
mm = Number of minutes to record.
If this argument is omitted, the recording continues for 12 hours. A maximum of 24 hours can be specified. When the time specified by this argument has elapsed, ptxrlog terminates.
When the -r flag is used, output is written to the file name specified after the flag. If the file exists when recording starts, it is opened for append. After opening the binary output file, whether for creation or append, ptxrlog writes the control records to the file. For existing files, this causes the file to contain more than one set of control records and may require you to process the file with ptxmerge or ptxsplit before you can process the file with xmperf or azizo.
The ptxrlog program initiates a resynchronizing with the data-supplier host if the data-supplier host sends an i_am_back packet. This usually happens if the data-supplier host's xmservd daemon has died and is restarted.
The ptxrlog initiates a resynchronizing with the data-supplier host if no data_feed packets have been received for ten times the specified sampling interval.