[ Previous | Next | Contents | Glossary | Home | Search ]
AIX Versions 3.2 and 4 Performance Tuning Guide

How Much Memory Is Really Being Used?

Several performance tools provide reports of memory usage. The reports of most interest are from vmstat, ps, and svmon.

vmstat

vmstat summarizes the total "active" virtual memory used by all of the processes in the system, as well as the number of real-memory page frames on the free list. Active virtual memory is defined as the number of virtual-memory working-segment pages that have actually been touched. It is usually equal to the number of paging-space slots that have been assigned. This number can be larger than the number of real page frames in the machine, since some of the active virtual-memory pages may have been written out to paging space.

ps

ps provides several different reports of memory use, depending on the flag used. The most comprehensive comes with ps v, which displays the following memory-related columns:

SIZE Virtual size in kilobytes of the data section of the process. (Displayed as SZ by other flags.) This number is equal to the number of working-segment pages of the process that have been touched (that is, the number of paging-space slots that have been allocated) times 4. If some working-segment pages are currently paged out, this number is larger than the amount of real memory being used.
RSS Real-memory (resident set) size in kilobytes of the process. This number is equal to the sum of the number of working-segment and code-segment pages in memory times 4. Remember that code-segment pages are shared among all of the currently running instances of the program. If 26 ksh processes are running, only one copy of any given page of the ksh executable would be in memory, but ps would report that code-segment size as part of the RSS of each instance of ksh.
TSIZ Size of text (shared-program) image. This is the size of the text section of the executable file. Pages of the text section of the executable are only brought into memory when they are touched, i.e., branched to or loaded from. This number represents only an upper bound on the amount of text that could be loaded.
TRS Size of the resident set (real memory) of text. This is the number of code-segment pages times 4. As was noted earlier, this number exaggerates memory use for programs of which multiple instances are running.
%MEM Calculated as the sum of the number of working-segment and code-segment pages in memory times 4 (that is, the RSS value), divided by the size of the real memory of the machine in KB, times 100, rounded to the nearest full percentage point. This value attempts to convey the percentage of real memory being used by the process. Unfortunately, like RSS, it tends the exaggerate the cost of a process that is sharing program text with other processes. Further, the rounding to the nearest percentage point causes all of the processes in the system that have RSS values under .005 times real memory size to have a %MEM of 0.0.

As you can see, reporting memory statistics in a format that was designed for earlier, simpler systems sometimes results in distorted data.

svmon

svmon provides both global, process-level, and segment-level reporting of memory use. For tuning purposes, the -G and -P options are most interesting.

-G Summarizes the memory use for the entire system.
-P Shows the memory use for one or more processes.

In AIX Version 4, the svmon command is packaged as part of the Performance Toolbox for AIX. To determine whether svmon is available, use:

lslpp -lI perfagent.tools

If this package has been installed, svmon is available.

Example of vmstat, ps, and svmon Output

The following example shows the output of these commands on a large system. vmstat was run in a separate window while ps and svmon were running consecutively. The vmstat summary (first) line has been removed:

$ vmstat 5
procs    memory             page              faults        cpu     
----- ----------- ------------------------ ------------ -----------
 r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa 
 0  0 25270  2691   0   0   0   0    0   0 142 2012  41  4 11 86  0
 1  0 25244  2722   0   0   0   0    0   0 138 6752  39 20 70 10  0
 0  0 25244  2722   0   0   0   0    0   0 128   61  34  0  1 99  0
 0  0 25244  2722   0   0   0   0    0   0 137  163  41  1  4 95  0

The global svmon report below shows related numbers. The number that vmstat reports as Active Virtual Memory (avm) is reported by svmon as page-space slots in use (25270). The number of page frames on the free list (2691) is identical in both reports. The number of pages pinned (2157) is a separate report, since the pinned pages are included in the pages in use.

$ svmon -G      
       m e m o r y            i n  u s e            p i n        p g  s p a c e
  size inuse  free   pin   work  pers  clnt   work  pers  clnt     size   inuse
 24576 21885  2691  2157  13172  7899   814   2157     0     0    40960   25270

Singling out a particular, long-running process on this machine, we can compare the ps v and svmon -P reports. The actual program has been renamed anon.

$ ps v 35851    
   PID    TTY STAT  TIME PGIN  SIZE   RSS   LIM  TSIZ   TRS %CPU %MEM COMMAND
 35851      - S     0:03  494  1192  2696    xx  1147  1380  0.2  3.0 anon

The SIZE value (1192) is the svmon Pgspace number (298) times four. The RSS value (2696) is equal to the number of pages in the process private segment (329) plus the number of pages in the code segment (345) times four. The TSIZE number is not related to real-memory use. The TRS value (1380) is equal to the number of pages in use in the code segment (345) times four. The %MEM is the RSS value, divided by the size of real memory in KB, times 100, rounded to the nearest full percentage point.

$ svmon -P 35851
  Pid                         Command        Inuse        Pin      Pgspace
35851                         anon            2410          2         4624
Pid:  35851
Command:  anon
Segid  Type  Description         Inuse   Pin  Pgspace   Address Range
 18a3  pers  /dev/hd2:5150           1     0       0    0..0
 9873  pers  /dev/hd2:66256          1     0       0    0..0
 4809  work  shared library       1734     0    4326    0..4668 : 60123..65535
 748e  work  private               329     2     298    0..423 : 65402..65535
 2105  pers  code,/dev/hd2:4492    345     0       0    0..402

As we analyze various processes in the environment, we observe that the shared library is indeed shared among almost all of the processes in the system, so its memory requirement is part of overall system overhead. Segment 9873 is also widely used, so we can include its memory in overhead. If one were estimating the memory requirement for program anon, the formula would be:

The total memory requirement for anon is equal to 345*4KB for program text (shared among all users) plus the estimated number of simultaneous users of anon times the sum of the working-segment size (329*4KB) and 4KB for the mapped segment (segment ID 18a3 in this example).

Memory-Leaking Programs

A memory leak is a program bug that consists of repeatedly allocating memory, using it, and then neglecting to free it. A memory leak in a long-running program, such as an interactive application, is a serious problem, because it can result in memory fragmentation and the accumulation of large numbers of mostly garbage-filled pages in real memory and page space. Systems have been known to run out of page space because of a memory leak in a single program.

A memory leak can be detected with svmon, by looking for processes whose working segment continually grows. Identifying the offending subroutine or line of code is more difficult, especially in AIXwindows applications, which generate large numbers of malloc and free calls. Some third-party programs exist for analyzing memory leaks, but they require access to the program source code.

Some uses of realloc, while not actually programming errors, can have the same effect as a memory leak. If a program frequently uses realloc to increase the size of a data area, the process's working segment can become increasingly fragmented if the storage released by realloc cannot be re-used for anything else. ( Appendix F, "Application Memory Management" contains background information on malloc and realloc.)

In general, memory that is no longer required should be released with free, if the memory will probably be re-used by the program. On the other hand, it is a waste of CPU time to free memory after the last malloc. When the program terminates, its working segment is destroyed and the real-memory page frames that contained working-segment data are added to the free list.

Related Information

The vmstat, ps, and svmon commands.


[ Previous | Next | Contents | Glossary | Home | Search ]