[ Previous | Next | Contents | Glossary | Search ]
Performance Toolbox Version 1.2 and 2 for AIX: Guide and Reference

Chapter 6. 3D Monitor

This chapter provides information about the 3dmon program.

A system administrator is often responsible for monitoring the performance of several hosts in a network. To cope with this job, it is desirable to have a monitor that allows the monitoring of selected key statistics on multiple hosts. One such monitor is available in the program 3dmon.

The 3dmon program is an X Window System based program that displays statistics in a 3-dimensional graph where each of the two sides may have up to 24 statistics for a maximum of 576 statistics plotted in a single graph.

Overview of the 3dmon Program

The graph layout of 3dmon is that of a chessboard except that each side may have from 1 to 24 fields. The right side always represents a context in the statistics hierarchy. This context may be any context that can have more than one instance of its subcontexts. The currently valid contexts are:

CPU
Disk
FS
FS/*
IP/NetIF
LAN
Mem/Kmem
PagSp
Proc
RTime/ARM
RTime/ARM/*
RTime/LAN
hosts
Note: Not all wildcards are available on systems other than IBM RS/6000 systems.

User Interface

For each context you want to use, you must supply a list of statistics in the 3dmon configuration file. The list of statistics you specify is used to define a set of statistics (statset). In the configuration file, each set is defined by a wildcard stanza which names the configuration set. When you start 3dmon, you specify which of the defined configuration sets should be used and as 3dmon starts, it displays a list of all instances, which match the wildcard specified in the configuration set. An example of a selection list is shown in the following figure.

3dmon Selection List

You must then pick the instances you want to monitor. The number of instances you select determines the number of fields on the right side of the "chessboard". For certain configuration sets, you have to first choose from a list of hosts to monitor, then from a list of context instances on the selected hosts. If the selection window in 3dmon Selection List represents the first selection list, then the second could look like the one shown in the Second Level 3dmon Selection List figure below. In this case, the first selection list lets you pick the hosts to monitor and the second shows a complete list of all disks on the selected hosts and lets you select the ones you want to monitor.

Second Level 3dmon Selection List

To select from the lists displayed by 3dmon, move the mouse pointer to the first instance you want, then press the left mouse button and move the mouse while holding the button down. When all instances you want are selected, release the mouse button. If you want to select instances that are not adjacent in the list, press and hold the Ctrl key on the keyboard while you select. When all instances are selected, release the key. After all selections have been made, click with the left mouse button on the button at the top of the window. This will create the 3dmon monitor window. A window created with the selection shown in Second Level 3dmon Selection Listis shown in the following figure.

3dmon Graph

The left side of the grid lists the statistics you chose to include in the configuration set. Each must be specified in the configuration file with its full value path name without the part up to and including the primary wildcard. This is explained in detail in Customizing the 3dmon Program .

The third dimension is represented by the actual statistics values as received from the data supplier daemon(s). The values are plotted as rectangular pillars placed on the fields of the chessboard, each filling its field except for a user modifiable spacing between the pillars. The actual value is displayed at the top of each pillar.

The 3dmon graph shows the incoming observations of the values as they are received depending on the type of statistic selected. Statistics can be of type SiCounter or of type SiQuantity:

SiCounter
Value is incremented continuously. The graph shows the change (delta) in the value between observations, divided by the elapsed time, representing a rate per second.
 
SiQuantity
Value represents a level, such as memory used or available disk space. The actual observation value is shown by the graph.

A user-modifiable color selection is in effect so that each of the statistics represented by the left side of the grid is drawn in a different color.

Pulldown Menus

3dmon File Menu

The File menu of the 3dmon graph window is shown in the preceding figure. Valid selections on the File pulldown menu include:

Playback
This option invokes 3dplay. See 3D Play for details.
 
Exit
Stops recording and exits program.

The Recording pulldown menu is shown in 3dmon Recording Menu . Valid selections on the Recording pulldown menu include:

Begin Recording
This option starts writing data values to the recording files as they are received. Recording continues until you stop it or the console is closed.
 
End Recording
Stops recording and closes the recording file if no other instrument in the console is still recording.
 
Annotation
Allows you to add an annotation to the currently active eller dormant recording file. When you select this item, you'll see the window shown in Adding an Annotation and you can proceed as described in Annotating while Recording .

3dmon Recording Menu

Autoscaling

In order to determine the height of a "pillar," 3dmon must select a scale. This scale states how many units of the statistic to be drawn correspond to one pixel on the display. The scale is derived from the expected maximum value for the statistic. However, for many statistics it's hard to set a realistic scaling. Because of that, 3dmon attempts to adjust the scale when the drawn pillars would otherwise disappear off the top of the window. This is done as follows:

When, prior to a pillar being drawn, 3dmon sees that the pillar would be more than 10 percent taller than the expected maximum height, an autoscaling function is invoked. The autoscaling function slowly adjusts the scale of a statistic until the scale corresponds to actual measurements. The scale is adjusted by approximately 50 percent each time autoscaling is invoked. Also, autoscaling is always done for all instances of the statistic as represented by a row on the chessboard.

Resynchronizing with Multiple Hosts

When 3dmon uses the wildcard value hosts, the user selects the hosts to monitor from the initial selection list. If the xmservd daemon on one or more of those hosts dies while 3dmon runs, the program periodically attempts to resynchronize with the host(s). Note that with the Performance Toolbox Local feature, only the local host is available for selection.

Resynchronizing is attempted every 30 seconds if more than 30 seconds have expired since the last data feed was received from a host. The interval between checking if resynchronizing is required can be changed with the command line argument -t.

Viewing Obscured Statistics

As 3dmon displays the statistics, high values for statistics in the front part of the grid can obscure the drawing of statistics behind them. You can, at any time, move a row or column to the front by clicking on the name associated with the row or column. When you click on a name at the left side of the grid, the colors of statistics are preserved as the rows of statistics are rearranged.

Another way to see obscured statistics is to reduce the height of the 3dmon window. Because the height of the pillars is calculated from the ratio of window height to window width, decreasing the height reduces the height until, eventually, only the base of the pillar is drawn.

How to Record with 3dmon

In the lower left corner of the 3dmon window, you see a small icon resembling a tape reel. Initially, this icon has the word "Start" shown in green. When you click on the icon with mouse button 1, recording of the statistics received by 3dmon begins and the text changes to the word "Stop" shown in red. The recording facility is described in Recording from 3dmon .

You can also start and stop recording from the pulldown shown in 3dmon Recording Menu .

Path Name Display

The right side of the grid shows the path names of the selections you made from the selection list up to the part that is displayed at the left side of the grid. If all the names at the right side begin with the same string, that string is removed from the names and shown in the upper left corner of the window. This reduces path name lengths and leaves more space to show the graph.

The 3dmon Command Line

To avoid clashes with X Window System command line options, never leave a blank between a command line option and its argument. For example, do not specify

3dmon -i 1 -p 75 -n

Instead, use:

3dmon -i1 -p75 -n

The 3dmon program takes the following command line arguments, all of which are optional:

3dmon [-vng] [-f config_file] [-i seconds_interval] [-h hostname] [-w weight_percent] [-s spacing] [-p filter_percent] [-c config] [-a "wildcard_match_list"] [-t resync_timeout] [-d invitation_delay] [-l left_side_tile] [-r right_side_tile] [-m top_tile]

v
Verbose. Causes the program to display warning messages about potential errors in the configuration file to stderr. Also causes 3dmon to print a line for each statset created and for each statistic added to the statset, including the results of resynchronizing.
 
n
Only has an effect if a filter percentage is specified with the -p argument. When specified, draws only a simple outline of the grid rectangles for statistics with values that are filtered out. If not specified, a full rectangle is outlined and the numerical value is displayed in the rectangle.
 
g
Normally, 3dmon will attempt to resynchronize for each statset it doesn't receive data-feeds for for resync-timeout seconds. If more than half of the statsets for any host are found to not supply data-feeds, resynchronizing is attempted for all the statsets of that host. By specifying the -g option, you can force resynchronization of all the statsets of a host if any one of them becomes inactive.
 
f
Allows you to specify a configuration file name other than the default. If not specified, 3dmon looks for the file $HOME/3dmon.cf. If that file does not exist the file is searched for as described in Overview of File Placement .
 
i
Sampling interval. If specified, this argument is taken as the number of seconds between sampling of the statistics. If omitted, the sampling interval is 5 seconds. You can specify from 1 to 60 seconds sampling interval.
 
h
Used to specify which host to monitor. This argument is ignored if the specified wildcard is "hosts." If omitted, the local host is assumed. With the Performance Toolbox Local feature, this flag always uses the local host name.
 
w
Modifies the default weight percentage used to calculate a weighted average of statistics values before plotting them. The default value for the weight is 50%, meaning that the value plotted for statistics is composed of 50 percent of the previously plotted value for the same statistic and 50 percent of the latest observation. The percentage specified is taken as the percentage of the previous value to use. For example, if you specify 40 with this argument the value plotted is:
 
.4 * previous + (1 - .4) * latest

Weight can be specified as any percentage from 0 to 100.

s
Spacing (in pixels) between the pillars representing statistics. The default space is 4 pixels. You can specify from 0 to 20 pixels.
 
p
Filtering percentage, -p. If specified, only statistics with current values of at least -p percent of the expected maximum value for the statistic are drawn. The idea is to allow you to specify monitoring "by exception" so statistics that are approaching a limit stand out while others are not drawn. Filtering can be specified as any percentage from 0 to 100. Default is 0%.
 
c
Configuration set. When specified, overrides the default configuration set and causes 3dmon to configure its graph using the named configuration set. The argument specified after the -c must match one of the wildcard stanzas in the configuration file. If this argument is omitted, the configuration set used is the first one defined in the configuration file.
 
a
Wildcard match list. When specified, is assumed to be a list of host names. If the primary wildcard in the selected configuration set is hosts, then the list to display host names is suppressed as 3dmon automatically selects the supplied hosts from the list of active remote hosts. Depending on the configuration set definition, 3dmon then either goes directly on with displaying the monitoring screen or, when additional wildcards are present, displays the secondary selection list.
Note: With the Performance Toolbox Local feature, this flag always uses the local host name.

The list of host names must be enclosed in double quotes if it contains more than one host name. Individual host names must be separated by white space or commas.

The primary purpose of this option is to allow the invocation of 3dmon from other programs. For example, you could customize NetView to invoke 3dmon with a list of host names, corresponding to hosts selected in a NetView window.

t
Resynchronizing timeout. When specified, overrides the default time between checks for whether resynchronizing is required. The default is 30 seconds; any specified timeout value must be at least 30 seconds.
 
d
Invitation delay. Allows you to control the time 3dmon waits for remote hosts to respond to an invitation. The value must be given in seconds and defaults to 10 seconds. Use this flag if the default value results in the list of hosts being incomplete when you want to monitor remote hosts.
l
(Lowercase L). Specifies the number of the tile to use when painting the left side of the pillars. Specify a value in the range 0 to 8. The values correspond to the tile names:

The default tile number for the left side is 1 (75_foreground).

r
Specifies the number of the tile to use when painting the right side of the pillars. Specify a value in the range 0 to 8. The values correspond to the tile names specified above for option -l. The default tile number for the right side is 8 (slant_left).
 
m
Specifies the number of the tile to use when painting the top of the pillars. Specify a value in the range 0 to 8. The values correspond to the tile names specified above for option -l. The default tile number for the top is 0 (foreground).

Hardware Dependencies

On some graphics adapters in certain configurations, the 3dmon program might not give you proper tiling. If you notice this, use the following command line arguments to suppress tiling:

3dmon -l0 -r0 -m0

Use the flags shown in addition to any other flags you may require. You can substitute the digit 4 for any of the zeroes shown above. The digit 0 means to paint the pillar in the foreground color; the digit 4 means to paint it in the backgound color.

Exiting 3dmon

To exit the 3dmon program, select Exit from the pulldown shown in 3dmon File Menu or close the program's window with the window manager. In case of the mwm window manager, click in the upper left corner of the window frame, then select the Close option from the menu.

Customizing the 3dmon Program

3dmon Configuration File

The 3dmon program requires a configuration file to describe the set of statistics (statsets) to display. A configuration file can be specified with the -f command line argument. If it is not, 3dmon first looks for the file 3dmon.cf in your home directory. If no such file is found, the file is searched for as described in Overview of File Placement .

The configuration file may have comment lines beginning with the character # (number sign). It can contain multiple configuration sets, each of which must begin with a wildcard stanza. The wildcard stanza must have one or two arguments. The last argument must be a valid context name. Configuration sets can be constructed as single wildcard sets or dual wildcard sets. Single wildcard sets present only one selection list to the user, while dual wildcard sets require the user to select from two lists before 3dmon begins to display statistics.

If the command line argument -c is not used to override the default, the first configuration set in the configuration file is chosen by 3dmon. Otherwise the set named by the -c argument is chosen. If no set in the configuration file matches the -c argument, 3dmon terminates with an error message.

Single Wildcard Configuration Sets

The following is an example of a definition of a configuration set where only one argument is provided on the wildcard stanza. The argument is hosts, and is used both to identify the configuration set and to specify the wildcard context:

wildcard: hosts

If two arguments are supplied with the wildcard stanza, the first identifies the configuration set. This name is used to match any name passed with the command line argument -c. The second argument must be the name of the wildcard context.

The remaining lines of a set must each specify a statistic, which is valid for the given wildcard. A set is terminated when another wildcard stanza is met or at end of file. The following example shows statistics defined for the wildcard hosts:

wildcard:        hosts     # remote hosts   
Mem/Virt/steal   
PagSp/pgspgout   
PagSp/pgspgin   
Proc/swpque   
Proc/runque   
PagSp/%totalused   
Syscall/total   
SysIO/writech   
SysIO/readch

Notice how the full path name is specified except for the wildcard part. The resulting path name for the instance (host name) birte for the first statistic would be hosts/birte/Mem/Virt/steal.

If you want more than one configuration set that uses the hosts wildcard, enter two arguments on the wildcard line:

wildcard:      nodecpu    hosts   
CPU/cpu0/user   
CPU/cpu0/kern   
CPU/cpu0/wait   
CPU/cpu0/idle

The above configuration set would be activated by the command line:

3dmon -cnodecpu

For process statistics, the statistic path name includes the name of the process context, which is constructed from the process ID, a ~ (tilde), and the name of the executing program. To reach a specific process, you can add a line that specifies either the process ID followed by the ~ (tilde), or the name of the executing program. The example below shows how to specify a statistic for the wait pseudo process, which (on AIX UPs) always has a process ID of 516. Both lines point to the same statistic.

Proc/516~/usercpu   
Proc/wait/usercpu

If you specify a name of a program currently executing in more than one process, only the first one encountered is found.

The distributed sample configuration file for 3dmon defines two single wildcard configuration sets for the hosts wildcard. One (the default) is called hosts. The other is called 24 because it shows 24 statistics for each host selected.

To monitor selected statistics for processes, the configuration file might contain:

wildcard: Proc   
usercpu   
kerncpu   
workmem   
pagsp

The above configuration set would cause 3dmon to present you with a list of all processes in the host.

Two contexts are special because they themselves may exist in multiple instances and have subcontexts that may also exist in multiple instances. One of these contexts describes file systems and is named FS. To allow 3dmon to find all instances in both context levels, you can begin a statistics line with the wildcard character * (asterisk). While, strictly speaking, this is a use of dual wildcards, it is described here because it, unlike other dual wildcard configuration sets, presents you with only one selection list. The use of the asterisk in a path name is used in the supplied configuration set shown below:

wildcard: FS   
*/%totfree   
*/size   
ppsize   
free

The statistics %totfree and size exist for every logical volume, while the remaining two exist for every volume group. In a system with two volume groups and a total of five logical volumes, this would yield five columns (right side of graph) and four rows (left side of graph). For example, the right side might show:

rootvg/hd4   
rootvg/hd9var   
rootvg/hd3   
rootvg/hd1   
newvg/hd10

The left side would show the four statistic names. Note that because the ppsize and free statistics are at the volume group level, they are identical across all four columns that are derived from rootvg.

The supplied configuration file contains the following single wildcard configuration sets:

wildcard: hosts           # remote hosts
wildcard: 24 hosts        # remote hosts, large set
wildcard: Disk            # local disks
wildcard: Kmem  Mem/Kmem  # local kernel memory
wildcard: Proc            # local processes
wildcard: LAN             # local lan adapters
wildcard: FS              # local file systems
wildcard: IP/NetIF        # local IP interfaces
wildcard: CPU             # local processors
wildcard: RTime/ARM       # local application response time
wildcard: lanresp hosts   # response time between multiple hosts

The configuration set "lanresp" for response time between multiple hosts is special because it will have identical labels on the left and right side of the grid. It uses the top context RTime/LAN, which always creates this type of graph and thus allows only a single metric to be specified. This is describes in more detail in Monitoring IP Response Time from 3dmon .

Dual Wildcard Configuration Sets

If a configuration set begins with a wildcard stanza that defines the wildcard as hosts, then the path names of the statistics belonging to that set may contain one or more asterisks in place of contexts that may exist in more than one instance. This is shown in the supplied configuration set called proc:

wildcard: proc hosts # processes on remote hosts   
Proc/*/usercpu   
Proc/*/kerncpu   
Proc/*/workmem   
Proc/*/codemem   
Proc/*/pagsp   
Proc/*/majflt   
Proc/*/minflt

When you invoke 3dmon to use this configuration set, you are presented first with a list of all matches of the primary wildcard, which is the list of hosts that responded to invitation. After you select the hosts you want to monitor, you are shown a list of all processes on those hosts. To proceed, select the processes you want to monitor. If you select more than 3dmon can show, excess instances are ignored.

If you use the command line argument -a to specify a list of host names, the first selection list is not shown. Rather, all responding hosts that match a name in the -a list are automatically selected.

Another example of a dual wildcard configuration set is shown below:

wildcard: fs hosts # remote file systems   
FS/*/*/%totfree LV   
FS/*/*/size     LV   
FS/*/ppsize     VG   
FS/*/free       VG   
FS/iget         FS

This time the statistics are at different levels so that the first two are logical volume statistics, the next two are volume group statistics, and the last one is a statistic directly under the FS context. To show this in the final graph, a suffix is specified after the name of each statistic. The first word of such a suffix is suffixed to the statistic name on the graph. Its only purpose is to show you the levels of the statistics in the graph.

The third example of a dual wildcard configuration set is the following set to create a 3dmon graph to monitor application response time and activity:

wildcard:  armresp hosts # application response time
RTime/ARM/*/*/resptime
RTime/ARM/*/*/respavg
RTime/ARM/*/*/respmax
RTime/ARM/*/*/respmin
RTime/ARM/*/*/good
RTime/ARM/*/*/aborted
RTime/ARM/*/*/failed

When one or more lines have the wildcard character * (asterisk), all lines in that configuration set must either be without wildcards or match the same pattern of wildcards. It would be invalid to specify the following two metrics under the same wildcard:

FS/*/*/%totfree   
Proc/*/usercpu

3dmon warns about such errors in the configuration file and ignores the offending lines.

The supplied configuration file contains the following dual wildcard configuration sets:

wildcard: disk hosts     # disks on remote hosts
wildcard: kmem hosts     # kernel memory on remote hosts
wildcard: proc hosts     # processes on remote hosts
wildcard: lan hosts      # LAN adapters on remote hosts
wildcard: fs hosts       # file systems on remote hosts
wildcard: ip hosts       # IP interfaces on remote hosts
wildcard: cpu hosts      # Processors on remote hosts
wildcard: armresp hosts  # application response time

Rsi.hosts File

The 3dmon program uses the RSi application programming interface throughout. This means that to locate potential data supplier hosts it uses the RSiInvite function call. This function relies on the file $HOME/Rsi.hosts to specify the rules for broadcasting are_you_there messages. (See Overview of File Placement for alternative locations of the Rsi.hosts file).

If the file does not exist, broadcasting is done only to hosts on the same subnet as defined for the network adapters in the host where 3dmon runs. For details about the $HOME/Rsi.hosts file, refer to How Data-Suppliers are Identified .

The 3dmon X Resources

The X Window System resource file for 3dmon defines resources you can use to enhance the appearance of 3dmon and is installed as /usr/lib/X11/app-defaults/3Dmon.

X Window System Resources for 3dmon lists the defined resources for 3dmon.

The sample file first defines the font to use. The resource name to define the font specifically for 3dmon is GraphFont. If you don't define this resource, 3dmon tries to get a font name from the following resources:

If none are defined, a suitable fixed-pitch font is used.

The next two lines define the foreground and background colors of the graph area. Finally, 24 lines define the default colors for the up to 24 statistics that can be plotted. The colors shown in "Figure 3. X Window System Resources for 3dmon" are the default colors. Only if you do not like the defaults do you need to modify the ValueColorxx resources.

  #
  #    3dmon  options
  #
  *GraphFont:  
-ibm-block-medium-r-normal--15-100-100-100-c-70-iso8859-1
  *DrawArea.background:  black
  *DrawArea.foreground:  white
  *ValueColor1:   ForestGreen
  *ValueColor2:   goldenrod
  *ValueColor3:   red
  *ValueColor4:   MediumVioletRed
  *ValueColor5:   LightSteelBlue
  *ValueColor6:   SlateBlue
  *ValueColor7:   green
  *ValueColor8:   yellow
  *ValueColor9:   BlueViolet
  *ValueColor10:  SkyBlue
  *ValueColor11:  pink
  *ValueColor12:  GreenYellow
  *ValueColor13:  SandyBrown
  *ValueColor14:  OrangeRed
  *ValueColor15:  plum
  *ValueColor16:  MediumTurquoise
  *ValueColor17:  LimeGreen
  *ValueColor18:  khaki
  *ValueColor19:  coral
  *ValueColor20:  magenta
  *ValueColor21:  turquoise
  *ValueColor22:  salmon
  *ValueColor23:  white
  *ValueColor24:  blue

X Window System Resources for 3dmon

Recording from 3dmon

The icon shown in the lower left corner of the 3dmon window controls recording of the observations received by 3dmon. Initially, this icon has the word "Start" shown in green. When you click on the icon with mouse button 1, recording to a disk file of the statistics received by 3dmon begins and the text changes to the word "Stop" shown in red.

3dmon Recording File Name

The first time you click on the icon, the window shown in the preceding figure pops open and prompts you to select a file name for the recording file. The window has a default file name constructed from the path name of the xmperf recording directory $HOME/XmRec followed by R.3dmon.set, where the part following the last period is the name of the configuration set you are using. Change the name, if required, and then click on the OK button to start the recording. If the name you select exists, the window shown in the following figure allows you to either overwrite the file or select a different name.

3dmon Existing File Dialog

After recording has been started, it can be stopped and restarted by clicking on the icon. Whenever recording is active, the icon shows the word "Stop" in red. After you stop recording, the recording file is kept open by 3dmon to allow quick resumption of the recording without going through the prompting for a recording file name. The recording file is closed when 3dmon exits. You can also start and stop recording from the 3dmon Recording pull-down menu.

Recordings produced by 3dmon defines sets of statistics (statsets) for each selection you did from the selection list (one for each name at the right side of the grid). The value records in the recording file correspond to sets of statistics (statsets).All 3dmon recordings can be played back by xmperf but they lack a console definition and therefore use the default playback console format described in Creation of Playback Consoles .

3dmon recordings can also be played back by 3dplay.

Recordings produced by 3dmon can be analyzed by the azizo program and processed by the recording support programs.


[ Previous | Next | Contents | Glossary | Search ]