awk Command

Purpose

Finds lines in files that match patterns and then performs specified actions on them.

Syntax

awk [ -F Ere ] [ -v Assignment ] ... { -f ProgramFile | 'Program' } [ [ File ... | Assignment ... ] ] ...

Description

The awk command utilizes a set of user-supplied instructions to compare a set of files, one line at a time, to extended regular expressions supplied by the user. Then actions are performed upon any line that matches the extended regular expressions.

The pattern searching of the awk command is more general than that of the grep command, and it allows the user to perform multiple actions on input text lines. The awk command programming language requires no compiling, and allows the user to use variables, numeric functions, string functions, and logical operators.

The following topics are covered in this article:

• Input for the awk Command
• Output for the awk Command
• File Processing with Records and Fields
• The awk Command Programming Language
  • Patterns
  • Actions
  • Variables
  • Special Variables
• Flags
• Examples

Input for the awk Command

The awk command takes two types of input: input text files and program instructions.

Input Text Files

Searching and actions are performed on input text files. The files are specified by:

• Specifying the File variable on the command line.
• Modifying the special variables ARGV and ARGC.
• Providing standard input in the absence of the File variable.

If multiple files are specified with the File variable, the files are processed in the order specified.

Program Instructions

Instructions provided by the user control the actions of the awk command. These instructions come from either the `Program' variable on the command line or from a file specified by the -f flag together with the ProgramFile variable. If multiple program files are specified, the files are concatenated in the order specified and the resultant order of instructions is used.

Output for the awk Command

The awk command produces three types of output from the data within the input text file:

• Selected data can be printed to standard output, without alteration to the input file.
• Selected portions of the input file can be altered.
• Selected data can be altered and printed to standard output, with or without altering the contents of the input file.

All of these types of output can be performed on the same file. The programming language recognized by the awk command allows the user to redirect output.

File Processing with Records and Fields

Files are processed in the following way:

1. The awk command scans its instructions and executes any actions specified to occur before the input file is read.

   The BEGIN statement in the awk programming language allows the user to specify a set of instructions to be done before the first record is read. This is particularly useful for initializing special variables.

2. One record is read from the input file.

   A record is a set of data separated by a record separator. The default value for the record separator is the new-line character, which makes each line in the file a separate record. The record separator can be changed by setting the RS special variable.

3. The record is compared against each pattern specified by the awk command's instructions.

   The command instructions can specify that a specific field within the record be compared. By default, fields are separated by white space (blanks or tabs). Each field is referred to by a field variable. The first field in a record is assigned the $1 variable, the second field is assigned the $2 variable, and so forth. The entire record is assigned to the $0 variable. The field separator can be changed by using the -F flag on the command line or by setting the FS special variable. The FS special variable can be set to the values of: blank, single character, or extended regular expression.

4. If the record matches a pattern, any actions associated with that pattern are performed on the record.
5. After the record is compared to each pattern, and all specified actions are performed, the next record is read from input; the process is repeated until all records are read from the input file.
6. If multiple input files have been specified, the next file is then opened and the process repeated until all input files have been read.
7. After the last record in the last file is read, the awk command executes any instructions specified to occur after the input processing.

   The END statement in the awk programming language allows the user to specify actions to be performed after the last record is read. This is particularly useful for sending messages about what work was accomplished by the awk command.

The awk Command Programming Language

The awk command programming language consists of statements in the form:

Pattern { Action }

If a record matches the specified pattern, or contains a field which matches the pattern, the associated action is then performed. A pattern can be specified without an action, in which case the entire line containing the pattern is written to standard output. An action specified without a pattern is performed for every input record.

Patterns

There are four types of patterns used in the awk command language syntax:

• Regular Expressions
• Relational Expressions
• Combinations of Patterns
• BEGIN and END Patterns.

Regular Expressions

The extended regular expressions used by the awk command are similar to those used by the grep or egrep command. The simplest form of an extended regular expression is a string of characters enclosed in slashes. For an example, suppose a file named testfile had the following contents:

smawley, andy
smiley, allen
smith, alan
smithern, harry
smithhern, anne
smitters, alexis

Entering the following command line:

awk '/smi/' testfile

would print to standard output of all records that contained an occurrence of the string smi. In this example, the program '/smi/' for the awk command is a pattern with no action. The output is:

smiley, allen
smith, alan
smithern, harry
smithhern, anne
smitters, alexis

The following special characters are used to form extended regular expressions:

awk '/smith+ern/' testfile

smithern, harry
smithhern, anne

awk '/smith?/' testfile

smith, alan
smithern, harry
smithhern, anne
smitters, alexis

awk '/allen 
| 
alan /' testfile

smiley, allen
smith, alan

awk '/a(ll)?(nn)?e/' testfile

smiley, allen
smithhern, anne

awk '/l{2}/' testfile

smiley, allen

awk '/t{2,}/' testfile

smitters, alexis

awk '/er{1, 2}/' testfile

smithern, harry
smithern, anne
smitters, alexis

awk '/sm[a-h]/' testfile

smawley, andy

awk '/sm[^a-h]/' testfile

smiley, allen
smith, alan
smithern, harry
smithhern, anne
smitters, alexis

awk '$1 ~ /n/' testfile

smithern, harry
smithhern, anne

awk '$2 ~ /^h/' testfile

smithern, harry

awk '$2 ~ /y$/' testfile

smawley, andy
smithern, harry

awk '/a..e/' testfile

smawley, andy
smiley, allen
smithhern, anne

awk '/a.*e/' testfile

smawley, andy
smiley, allen
smithhern, anne
smitters, alexis

/a\/\//

Recognized Escape Sequences

The awk command recognizes most of the escape sequences used in C language conventions, as well as several that are used as special characters by the awk command itself. The escape sequences are:

Note: Except in the gsub, match, split, and sub built-in functions, the matching of extended regular expressions is based on input records. Record-separator characters (the new-line character by default) cannot be embedded in the expression, and no expression matches the record-separator character. If the record separator is not the new-line character, then the new-line character can be matched. In the four built-in functions specified, matching is based on text strings, and any character (including the record separator) can be embedded in the pattern so that the pattern matches the appropriate character. However, in all regular-expression matching with the awk command, the use of one or more NULL characters in the pattern produces undefined results.

Relational Expressions

The relational operators < (less than), > (greater than), <= (less than or equal to), >= (greater than or equal to), = = (equal to), and ! = (not equal to) can be used to form patterns. For example, the pattern:

$1 < $4

matches records where the first field is less than the fourth field. The relational operators also work with string values. For example:

$1 =! "q"

matches all records where the first field is not a q. String values can also be matched on collation values. For example:

$1 >= "d"

matches all records where the first field starts with a character that is a, b, c, or d. If no other information is given, field variables are compared as string values.

Combinations of Patterns

Patterns can be combined using three options:

• Ranges are specified by two patterns separated with a , (comma). Actions are performed on every record starting with the record that matches the first pattern, and continuing through and including the record that matches the second pattern. For example:

  /begin/,/end/

  matches the record containing the string begin, and every record between it and the record containing the string end, including the record containing the string end.
• Parentheses ( ) group patterns together.
• The boolean operators || (or), && (and), and ! (not) combine patterns into expressions that match if they evaluate true, otherwise they do not match. For example, the pattern:

  $1 == "al" && $2 == "123"

  matches records where the first field is al and the second field is 123.

BEGIN and END Patterns

Actions specified with the BEGIN pattern are performed before any input is read. Actions specified with the END pattern are performed after all input has been read. Multiple BEGIN and END patterns are allowed and processed in the order specified. An END pattern can precede a BEGIN pattern within the program statements. If a program consists only of BEGIN statements, the actions are performed and no input is read. If a program consists only of END statements, all the input is read prior to any actions being taken.

Actions

There are several types of action statements:

• Action Statements
• Built-in Functions
• User-Defined Functions
• Conditional Statements
• Output Actions

Action Statements

Action statements are enclosed in { } (braces). If the statements are specified without a pattern, they are performed on every record. Multiple actions can be specified within the braces, but must be separated by new-line characters or ; (semicolons), and the statements are processed in the order they appear. Action statements include:

Expression Operator Expression

$2 = $1 ^ 3

+Expression or -Expression

++Variable or --Variable

Variable++ or Variable--

Variable += Expression

Variable -= Expression

Variable /= Expression

Variable *= Expression

$1 *= $2

Variable1^=Expression1

Variable2%=Expression2

Variable1=pow(Variable1, Expression1)

Variable2=fmod(Variable2, Expression2)

$3 = $1 $2

Built-In Functions

The awk command language uses arithmetic functions, string functions, and general functions. The close Subroutine statement is necessary if you intend to write a file, then read it later in the same program.

Arithmetic Functions

The following arithmetic functions perform the same actions as the C language subroutines by the same name:

String Functions

The string functions are:

General Functions

The general functions are:

Note: All forms of the getline function return 1 for successful input, zero for end of file, and -1 for an error.

User-Defined Functions

User-defined functions are declared in the following form:

function Name (Parameter, Parameter,...)  { Statements }

A function can be referred to anywhere in an awk command program, and its use can precede its definition. The scope of the function is global.

Function parameters can be either scalars or arrays. Parameter names are local to the function; all other variable names are global. The same name should not be used for different entities; for example, a parameter name should not be duplicated as a function name, or special variable. Variables with global scope should not share the name of a function. Scalars and arrays should not have the same name in the same scope.

The number of parameters in the function definition does not have to match the number of parameters used when the function is called. Excess formal parameters can be used as local variables. Extra scalar parameters are initialized with a string value equivalent to the empty string and a numeric value of 0 (zero); extra array parameters are initialized as empty arrays.

When invoking a function, no white space is placed between the function name and the opening parenthesis. Function calls can be nested and recursive. Upon return from any nested or recursive function call, the values of all the calling function's parameters shall be unchanged, except for array parameters passed by reference. The return statement can be used to return a value.

Within a function definition, the new-line characters are optional before the opening { (brace) and after the closing } (brace).

An example of a function definition is:

function average ( g,n) 
  {
        for (i in g)
           sum=sum+g[i]
        avg=sum/n
        return avg
  } 

The function average is passed an array, g, and a variable, n, with the number of elements in the array. The function then obtains an average and returns it.

Conditional Statements

Most conditional statements in the awk command programming language have the same syntax and function as conditional statements in the C programming language. All of the conditional statements allow the use of { } (braces) to group together statements. An optional new-line can be used between the expression portion and the statement portion of the conditional statement, and new-lines or ; (semicolon) are used to separate multiple statements in { } (braces). Six conditional statements in C language are:

Five conditional statements in the awk command programming language that do not follow C-language rules are:

for (i in g)
   delete g[i];

Output Statements

Two output statements in the awk command programming language are:

Note: If the Expression parameter specifies a path name for the Redirection parameter, the Expression parameter should be enclosed in double quotes to insure that it is treated as a string.

Variables

Variables can be scalars, field variables, arrays, or special variables. Variable names cannot begin with a digit.

Variables can be used just by referencing them. With the exception of function parameters, they are not explicitly declared. Uninitialized scalar variables and array elements have both a numeric value of 0 (zero) and a string value of the null string (" ").

Variables take on numeric or string values according to context. Each variable can have a numeric value, a string value, or both. For example:

x = "4" + "8"

assigns the value of 12 to the variable x. For string constants, expressions should be enclosed in " " (double quotation) marks.

There are no explicit conversions between numbers and strings. To force an expression to be treated as a number, add 0 (zero) to it. To force an expression to be treated as a string, append a null string (" ").

Field Variables

Field variables are designated by a $ (dollar sign) followed by a number or numerical expression. The first field in a record is assigned the $1 variable , the second field is assigned to the $2 variable, and so forth. The $0 field variable is assigned to the entire record. New field variables can be created by assigning a value to them. Assigning a value to a non-existent field, that is, any field larger than the current value of $NF field variable, forces the creation of any intervening fields (set to the null string), increases the value of the NF special variable, and forces the value of $0 record variable to be recalculated. The new fields are separated by the current field separator ( which is the value of the FS special variable). Blanks and tabs are the default field separators. To change the field separator, use the -F flag, or assign the FS special variable a different value in the awk command program.

Arrays

Arrays are initially empty and their sizes change dynamically. Arrays are represented by a variable with subscripts in [ ] (square brackets). The subscripts, or element identifiers, can be numbers of strings, which provide a type of associative array capability. For example, the program:

/red/  { x["red"]++ }
/green/ { y["green"]++ }

increments counts for both the red counter and the green counter.

Arrays can be indexed with more than one subscript, similar to multidimensional arrays in some programming languages. Because programming arrays for the awk command are really one dimensional, the comma-separated subscripts are converted to a single string by concatenating the string values of the separate expressions, with each expression separated by the value of the SUBSEP environmental variable. Therefore, the following two index operations are equivalent:

x[expr1, expr2,...exprn]

AND

x[expr1SUBSEPexpr2SUBSEP...SUBSEPexprn]

When using the in operator, a multidimensional Index value should be contained within parentheses. Except for the in operator, any reference to a nonexistent array element automatically creates that element.

Special Variables

The following variables have special meaning for the awk command:

Flags

Exit Status

This command returns the following exit values:

You can alter the exit status within the program by using the exit [ Expression ] conditional statement.

Examples

1. To display the lines of a file that are longer than 72 characters, enter:

   awk  'length  >72'  chapter1

   This selects each line of the chapter1 file that is longer than 72 characters and writes these lines to standard output, because no Action is specified. A tab character is counted as 1 byte.
2. To display all lines between the words start and stop, including "start" and "stop", enter:

   awk  '/start/,/stop/'  chapter1

3. To run an awk command program, sum2.awk, that processes the file, chapter1, enter:

   awk  -f  sum2.awk  chapter1

   The following program, sum2.awk, computes the sum and average of the numbers in the second column of the input file, chapter1:

       {
          sum += $2
       }
   END {
          print "Sum: ", sum;
          print "Average:", sum/NR;
       }

   The first action adds the value of the second field of each line to the variable sum. All variables are initialized to the numeric value of 0 (zero) when first referenced. The pattern END before the second action causes those actions to be performed after all of the input file has been read. The NR special variable, which is used to calculate the average, is a special variable specifying the number of records that have been read.
4. To print the first two fields in opposite order, enter:

   awk '{ print $2, $1 }' chapter1

5. The following awk program sum3.awk prints the first two fields of the file chapter2 with input fields separated by comma and/or blanks and tabs, and then adds up the first column, and prints the sum and average:

   BEGIN  {FS = ",|[ \t]+"}
          {print $1, $2}
          {s += $1}
   END    {print "sum is",s,"average is", s/NR }               

Related Information

Commands: egrep, fgrep, grep, lex, printf, sed.

Subroutines: popen, printf, system.

Books: Aho, A.V., Kernighan, B.W., and Weinberger, P.J. The Awk Programming Language. Bell Telephone Laboratories, Incorporated, 1988.