Use of Disk-I/O Pacing

Disk-I/O pacing is intended to prevent programs that generate very large amounts of output from saturating the system's I/O facilities and causing the response times of less-demanding programs to deteriorate. Disk-I/O pacing enforces per-segment (which effectively means per-file) high- and low-water marks on the sum of all pending I/Os. When a process tries to write to a file that already has "high-water mark" pending writes, the process is put to sleep until enough I/Os have completed to make the number of pending writes less than or equal to "low-water mark." The logic of I/O-request handling does not change. The output from high-volume processes is just slowed down somewhat. The high- and low-water marks are set with smit by selecting System Environments -> Change / Show Characteristics of Operating System and then entering the number of pages for the high- and low-water marks. The default value for the high- and low-water marks is 0, which disables pacing. New I/O pacing parameters normally take effect immediately and last until they are explicitly changed.

Example

The effect of pacing on performance can be demonstrated with an experiment that consists of starting a vi session on a new file while another process is cping a 64MB file. The file is copied from disk1 to disk0 and the vi executable is located on disk0. For the vi session to start, it must page itself in as well as perform a few other I/Os, which it does randomly one page at a time. This takes about 50 physical I/Os, which can be completed in .71 seconds when there is no contention for the disk. With the high-water mark set to the default of 0, the logical writes from cp run ahead of the physical writes, and a large queue builds up. Each I/O started by vi must wait its turn in the queue before the next I/O can be issued, and thus vi is not able to complete its needed I/O until after cp finishes. The figure "I/O Pacing Test Results" shows the elapsed times for cp execution and vi initialization with different pacing parameters. This experiment was run on a Model 530 with two 857MB disks and 32MB of RAM.

It is important to notice that the cp duration is always longer when pacing is set. Pacing sacrifices some throughput on I/O-intensive programs to improve the response time of other programs. The challenge for a system administrator is to choose settings that result in a throughput/response-time trade-off that is consistent with the organization's priorities.

The high- and low-water marks were chosen by trial and error, based on our knowledge of the I/O path. Choosing them is not straightforward because of the combination of write-behind and asynchronous writes. High-water marks of 4x + 1 work particularly well, because of the following interaction:

• The write-behind feature sends the previous four pages to disk when a logical write occurs to the first byte of the fifth page.
• If the pacing high-water mark were a multiple of 4 (say, 8), a process would hit the high-water mark when it requested a write that extended into the 9th page. It would then be put to sleep--before the write-behind algorithm had a chance to detect that the fourth dirty page is complete and the four pages were ready to be written.
• The process would then sleep with four full pages of output until its outstanding writes fell below the pacing low-water mark.
• If, on the other hand, the high-water mark had been set to 9, write-behind would get to schedule the four pages for output before the process was suspended.

One limitation of pacing is that it does not offer as much control when a process writes buffers larger than 4KB. If, when a write is sent to the VMM, the high-water mark has not been met, the VMM performs Start I/Os on all pages in the buffer, even if that results in exceeding the high-water mark. Pacing works well on cp because cp writes 4KB at a time; but if cp wrote larger buffers, the times in the figure "I/O Pacing Test Results" for starting vi would increase.

Disk-I/O pacing is a tuning parameter that can improve interactive response time in some situations where foreground or background programs that write large volumes of data are interfering with foreground requests. If not used properly, however, it can reduce throughput excessively. The settings in the figure I/O Pacing Test Results are a good place to start, but some experimenting will be needed to find the best settings for your workload.

Programs whose presence in a workload may make imposition of disk-I/O pacing necessary include:

• Programs that generate large amounts of output algorithmically, and thus are not constrained by the time required to read input. Some such programs may need pacing on comparatively fast processors and not need it on comparatively slow processors.
• Programs that write large, possibly somewhat modified, files that have been read in their entirety shortly before writing begins--by a previous command, for example.
• Filters, such as the tar command, that read a file and write it out again with little processing. The need for pacing can be exacerbated if the input is being read from a faster disk drive than the output is being written to.