AIX Versions 3.2 and 4 Performance Tuning Guide

About This Book

This book, the AIX Versions 3.2 and 4 Performance Tuning Guide, provides information on concepts, tools, and techniques for assessing and tuning the performance of AIX on RS/6000 systems. Topics covered include efficient system and application design and implementation, as well as post-implementation tuning of CPU use, memory use, disk I/O, and communications I/O. Most of the tuning recommendations were developed or validated on AIX Version 3.2.5. Some information on AIX Version 4 is also provided. Information that applies only to AIX Version 4 is so identified in the text.

The material in this guide is designed for programmers, system managers, and end users concerned with performance tuning of AIX systems. You should be familiar with the AIX operating environment. Introductory sections are included to assist the less experienced and to acquaint experienced users with AIX performance-tuning terminology.

This is a major revision of the AIX Version 3.2 Performance Monitoring and Tuning Guide, including both changes in organization and new tuning recommendations. Of particular interest to the performance analyst are the changes in the packaging and content of the AIX performance tools in AIX Version 4.

AIX Performance Management Structure

There are appropriate tools for each phase of AIX system performance management. Some of the tools are available from IBM; others are the products of third parties. The figure illustrates the phases of performance management in a simple LAN environment and some of the tools packages that apply in each phase. This tuning guide also provides information about each phase.

Performance Tools Packages and Their Documentation

The packaging of the performance tools lets the performance analyst install on any given system only those tools that are required to monitor and tune that system. The packages and their main functions are described in the following sections.

BEST/1

BEST/1 is a capacity-planning tool that uses queuing models to predict the performance of a given configuration when processing a specific workload. The prediction can be based on:

workload descriptions derived from an application design, or
workload data acquired by monitoring existing systems.

BEST/1 has three main components:

Collect	Collects detailed information about the processing of a workload by an existing system.
Analyze	Transforms the detailed information into reports and a queuing model of the workload-processing activity.
Predict	Uses the queuing model to estimate the performance effects of changes in the workload or the configuration.

BEST/1 for UNIX is a product of BGS Systems, Inc. BGS Systems can be reached at 1-800-891-0000 (in the US).

AIX Performance Diagnostic Tool (PDT)

The Performance Diagnostic Tool, which is an optionally installable component of AIX Version 4, assesses the configuration of the system and tracks trends in resource use. If PDT detects an actual or potential performance problem, it reports the situation to the system administrator. This book contains the primary, detailed documentation of the functions of PDT.

AIX Base Operating System (BOS)

The AIX Base Operating System contains a number of monitoring and tuning tools that have historically been part of UNIX systems or are required to manage the implementation-specific features of AIX. The BOS functions and commands that are most important to performance analysts are:

iostat		Reports CPU and I/O statistics.
lsattr		Displays the attributes of devices.
lslv		Displays information about a logical volume or the logical volume allocations of a physical volume.
netstat		Displays the contents of network-related data structures.
nfsstat		Displays statistics about Network File System (NFS) and Remote Procedure Call (RPC) activity.
nice		Runs a command at higher- or lower-than-normal priority.
no		Displays or sets network options.
ps		Displays the status of processes.
renice		Changes the priority of one or more processes.
reorgvg		Reorganizes the physical-partition allocation within a volume group.
sar		Collects and reports or records system-activity information.
time		Prints the elapsed execution time and the user and system processing time attributed to a command.
trace		Records and reports selected system events.
vmstat		Reports virtual-memory activity and other system statistics.

The documentation of the AIX BOS commands is the AIX Commands Reference Manual.

Performance Toolbox (PTX)

The Performance Toolbox for AIX (PTX) contains tools for local and remote system-activity monitoring and tuning. This licensed product consists of two main components: the PTX Manager and the PTX Agent. The PTX Agent is available as a separate licensed product called the Performance Aide for AIX. The figure shows a simplified LAN configuration in which the PTX Manager is being used to monitor the activity of several systems.

The main purpose of the PTX Manager is to collect and display data from the various systems in the configuration. The primary program for this purpose is xmperf. The primary program used by the Agent to collect and transmit data to the Manager is xmservd.

In addition to the main PTX components, in AIX Version 4 both the Performance Toolbox for AIX and the Performance Aide for AIX licensed products include a set of separate monitoring and tuning tools, most of which are part of the AIX Version 3.2.5 Base Operating System:

fdpr	Optimizes an executable program for a particular workload.
filemon	Uses the trace facility to monitor and report the activity of the AIX file system.
fileplace	Displays the placement of a file's blocks within logical or physical volumes.
lockstat	Displays statistics about contention for kernel locks.
netpmon	Uses the trace facility to report on network I/O and network-related CPU usage.
rmss	Simulates systems with various sizes of memory for performance testing.
svmon	Captures and analyzes information about virtual-memory usage.
syscalls	Records and counts system calls.
tprof	Uses the trace facility to report CPU usage at module and source-code-statement levels.
bf	Reports memory access patterns of processes (AIX Version 4 only).
stem	Permits subroutine-level entry/exit instrumentation of existing executables (AIX Version 4 only).

The primary documentation of the commands and functions of PTX is the Performance Toolbox Version 1.2 and 2 for AIX: Guide and Reference, although the syntax descriptions of the tools listed above are documented in the AIX Version 4.3 Commands Reference. Use of the listed commands is incorporated in various diagnosis and tuning scenarios in this book.

AIX Performance PMR Data Collection Tool (PerfPMR)

The AIX Performance PMR Data Collection Tool (PerfPMR) package is used to collect configuration and performance information to accompany a report of a suspected AIX performance defect. This book contains the primary, detailed documentation of the functions and use of PerfPMR.

Tools That Are Not Supported in AIX Version 4

The rmap trace data reduction and reporting program is not supported in AIX Version 4.

How to Use This Book

Overview of Contents

This book contains the following chapters and appendixes:

Chapter 1, "Performance Concepts," gives an introduction to the basic considerations of performance analysis. For those who are already experienced in performance tuning, this chapter will be of interest mainly as a guide to AIX terminology.
Chapter 2, "AIX Resource Management Overview," describes the structures and principal algorithms of the main resource-management components of AIX.
Chapter 3, "An Introduction to Multiprocessing," provides an overview of the performance aspects of multiprocessor systems.
Chapter 4, "Performance-Conscious Planning, Design, and Implementation," describes the performance considerations that should be taken into account in preparation for an application.
Chapter 5, "System Monitoring and Initial Performance Diagnosis," explains how to prepare for the detection of performance problems and the preliminary steps to take when such a problem is encountered.
Chapter 6, "Monitoring and Tuning CPU Use," describes techniques for ensuring that the CPU resource is being used efficiently.
Chapter 7, "Monitoring and Tuning Memory Use," shows how to determine how much real and virtual storage is being used and how to avoid or detect some common inefficiencies.
Chapter 8, "Monitoring and Tuning Disk I/O," explains the dynamics of disk I/O in AIX and how those dynamics can be affected by user choices.
Chapter 9, "Monitoring and Tuning Communications I/O," gives tuning techniques for various forms of communications I/O.
Chapter 10, "DFS Performance Tuning," describes various parameters of DFS operation that can affect performance.
Chapter 11, "Performance Analysis with the Trace Facility," gives an extended explanation of the use of the trace facility, which is a powerful tool for detailed performance tuning and also is the base of a number of other tools discussed in this book.
Chapter 12, "Performance Diagnostic Tool (PDT)," describes an AIX Version 4 tool that assesses configurations for balance and maintains historical performance data to identify performance trends.
Chapter 13, "Handling a Possible AIX Performance Bug," explains the process of reporting, and providing data about, a possible performance bug in AIX.
Appendix A, "AIX Performance Monitoring and Tuning Commands," lists the AIX commands that are most helpful in carrying out performance monitoring and tuning tasks and provides detailed documentation of the syntax and functions of the schedtune, vmtune, pdt_config, and pdt_report commands.
Appendix B, "Performance-Related Subroutines," describes several subroutines with performance-related uses.
Appendix C, "Cache and Addressing Considerations," provides a conceptual discussion of the ways in which caches can affect the performance of programs.
Appendix D, "Efficient Use of the ld Command," describes techniques for using the AIX binder.
Appendix E, "Performance of the Performance Tools," documents the resource consumption and response time of the performance tools.
Appendix F, "Application Memory Management," describes the distinction between the original and the current versions of the malloc and realloc subroutines.
Appendix G, "Performance Effects of Shared Libraries," describes the performance advantages and disadvantages of shared libraries versus nonshared libraries.
Appendix H, "Accessing the Processor Timer," describes methods of using the processor timer to compute elapsed-time values.
Appendix I, "National Language Support--Locale versus Speed," explains the effect that use of the AIX National Language Support facility can have on performance.
Appendix J, "Summary of Tunable Performance Parameters in AIX," documents the AIX operational parameters that can be changed by the user and that have a direct or indirect effect on performance.

Highlighting

The following highlighting conventions are used in this book:

Bold	Identifies commands, subroutines, keywords, files, structures, directories, and other items whose names are predefined by the system. Also identifies graphical objects such as buttons, labels, and icons that the user selects.
Italics	Identifies parameters whose actual names or values are to be supplied by the user.
`Monospace`	Identifies examples of specific data values, examples of text similar to what you might see displayed, examples of portions of program code similar to what you might write as a programmer, messages from the system, or information you should actually type.

ISO 9000

ISO 9000 registered quality systems were used in the development and manufacturing of this product.

Related Publications

The following books contain information about or related to performance monitoring:

AIX and Related Products Documentation Overview, order number SC23-2456.
AIX Version 4.3 System User's Guide: Operating System and Devices, order number SC23-4121.
AIX Versions 4.1 and 4.2 System User's Guide: Operating System and Devices, order number SC23-2544.
AIX Version 3.2 System User's Guide: Operating System and Devices, order number GC23-2522.
AIX Version 4.3 System Management Guide: Operating System and Devices, order number SC23-4126.
AIX Versions 4.1 and 4.2 System Management Guide: Operating System and Devices, order number SC23-2525.
AIX Version 3.2 System Management Guide: Operating System and Devices, order number GC23-2486.
AIX Version 4.3 System Management Guide: Communications and Networks, order number SC23-4127.
AIX Versions 4.1 and 4.2 System Management Guide: Communications and Networks, order number SC23-2526.
AIX Version 3.2 System Management Guide: Communications and Networks, order number GC23-2487.
AIX Version 4.3 Commands Reference, order number SBOF-1877.
AIX Versions 4.1 and 4.2 Commands Reference, order number SBOF-1851.
AIX Version 3.2 Commands Reference, order number GBOF-1802.
AIX Version 4.3 General Programming Concepts: Writing and Debugging Programs, order number SC23-4128.
AIX Versions 4.1 and 4.2 General Programming Concepts, order number SC23-2533.
AIX Version 3.2 General Programming Concepts, order number SC23-2205.
AIX Version 4.3 Technical Reference: Base Operating System and Extensions Volume 1, order number SC23-4159.
AIX Versions 4.1 and 4.2 Technical Reference, Volume 1: Base Operating System and Extensions, order number SC23-2614.
AIX Version 3.2 Technical Reference, Volume 1: Base Operating System and Extensions, order number SC23-2382.
AIX Version 4.3 Technical Reference: Base Operating System and Extensions Volume 2, order number SC23-4160.
AIX Versions 4.1 and 4.2 Technical Reference, Volume 2: Base Operating System and Extensions, order number SC23-2615.
AIX Version 3.2 Technical Reference, Volume 2: Base Operating System and Extensions, order number SC23-2383.
Performance Toolbox Version 1.2 and 2 for AIX: Guide and Reference, order number SC23-2625.
Optimization and Tuning Guide for XL Fortran, XL C and XL C++, order number SC09-1705
IBM Engineering and Scientific Subroutine Library Guide and Reference, order number SC23-0184.
RISC System/6000 Technology, order number SA23-2619.
Comer, D., Internetworking with TCP/IP Vol I, 2nd ed., Englewood Cliffs: Prentice-Hall, 1991.
Ferrari, D., Serazzi, G., and Zeigner, A., Measurement and Tuning of Computer Systems, New York: Prentice-Hall,1983.
Lazowska, D., Zahorjan, J., Graham, G., and Sevchik, K., Quantitative System Performance, New York: Prentice-Hall,1984.
Leffler, McKusick, Karels, and Quarterman, The Design and Implementation of the 4.3 BSD Unix Operating System, Reading: Addison-Wesley, 1989.
Smith, C. U., Performance Engineering of Software Systems, Reading: Addison-Wesley, 1990.
Stern, H., Managing NFS and NIS, Sebastopol, CA: O'Reilly & Associates, Inc., 1992.

Ordering Publications

You can order publications from your sales representative.

To order additional copies of this book, use order number SC23-2365.

Use AIX and Related Products Documentation Overview for information on related publications and how to obtain them.