Converters Overview for Programming

National Language Support (NLS) provides a base for internationalization in which data often can be changed from one code set to another. Support of several standard converters for this purpose is provided. This section discusses the following aspects of conversion:

• Converters Introduction
• Standard Converters
• Understanding libiconv
• Using Converters
• List of Converters

Converters Introduction

Data sent by one program to another program residing on a remote host may require conversion from the code set of the source machine to that of the receiver. For example, when communicating with a VM system, the workstation converts its ISO8859-1 data to an EBCDIC form.

Code sets define graphic characters and control character assignments to code points. These coded characters must also be converted when a program obtains data in one code set but displays it in another code set.

Two interfaces for conversions are provided:

The system provides ready-to-use libraries of converters. You supply the name of the converter you want to use. The converter libraries are found in the /usr/lib/nls/loc/iconv/* and /usr/lib/nls/loc/iconvTable/* directories.

In addition to code set converters, the converter library also provides a set of network interchange converters. In a network environment, the code sets of the communications systems and the protocols of communication determine how the data should be converted.

Interchange converters are used to convert data sent from one system to another. Conversions from one internal code set to another require code set converters. When data must be converted from a sender's code set to a receiver's code set or from 8-bit data to 7-bit data, a uniform interface is required. The iconv subroutines provide this interface.

Standard Converters

The system supports standard converters for use with the iconv command and subroutines. The following list describes the different types of converters:

Understanding libiconv

The iconv application programming interface (API) consists of three subroutines that accomplish conversion:

In a network environment, two factors determine how data should be converted:

• Code sets of the sender and the receiver
• Communication protocol (8-bit or 7-bit data)

The following table outlines the conversion methods and recommends how you should convert data in different situations. See the "List of Interchange Converters--7-bit" and the "List of Interchange Converters--8-bit" for more information.

If the sender uses the same code set as the receiver, there are two possibilities:

• When protocol allows 8-bit data, the data can be sent without conversions.
• When protocol allows only 7-bit data, the 8-bit code points must be mapped to 7-bit values. Use the iconv interface and one of the following methods:

  uucode method	Provides the same mapping as the uuencode and uudecode commands. This is the recommended method.
  fold7 method	Converts internal code sets using 7-bit data. This method passes ASCII without any change.

If the sender uses a code set different from the receiver, there are two possibilities:

• When protocol allows only 7-bit data, use the fold7 method.
• When protocol allows 8-bit data and you know the receiver's code set, use the iconv interface to convert the data. If you do not know the receiver's code set, use the following method:

  fold8 method

  Converts internal code sets to standard interchange formats. The 8-bit data is transmitted and the information is preserved so that the receiver can reconstruct the data in its code set.

Using the iconv_open Subroutine

The following examples illustrate how to use the iconv_open subroutine in different situations:

• Sender and receiver use the same code sets:

  If the protocol allows 8-bit data, you can send data without converting it.

  If the protocol allows only 7-bit data, do the following:

  Sender:
   cd = iconv_open("uucode", nl_langinfo(CODESET));
   
  
  Receiver:
   cd = iconv_open(nl_langinfo(CODESET), "uucode"); 

• Sender and receiver use different code sets:

  If the protocol allows 8-bit data and the receiver's code set is unknown, do the following:

  Sender:
   cd = iconv_open("fold8", nl_langinfo(CODESET));
   
  
  Receiver:
   cd = iconv_open(nl_langinfo(CODESET),"fold8" );

  If the protocol allows only 7-bit data, do the following:

  Sender:
   cd = iconv_open("fold7", nl_langinfo(CODESET));
   
  
  Receiver:
   cd = iconv_open(nl_langinfo(CODESET), "fold7" );

How the iconv_open Subroutine Finds Converters

The iconv_open subroutine uses the LOCPATH environment variable to search for a converter whose name is in the form:

iconv/FromCodeSet_ToCodeSet

The FromCodeSet string represents the sender's code set, and the ToCodeSet string represents the receiver's code set. The underscore character separates the two strings.

Note: All setuid and setgid programs will ignore the LOCPATH environment variable.

Since the iconv converter is aloadable object module, a different object is required when running in the 64-bit environment. In the 64-bit environment, the iconv_open routine will use the LOCPATH environment variable to search for a converter whose name is in the form:

iconv/FromCodeSet_ToCodeSet__64.

The iconv library will automatically choose whether to load the standard converter object or the 64-bit converter object.

If the iconv_open subroutine does not find the converter, it uses the from,to pair to search for a file that defines a table-driven conversion. The file contains a conversion table created by the genxlt command.

The iconvTable converter uses the LOCPATH environment variable to search for a file whose name is in the form:

iconvTable/FromCodeSet_ToCodeSet

If the converter is found, it performs a load operation and is initialized. The converter descriptor, iconv_t, is returned in its initial state.

Converter Programs versus Tables

Converter programs are executable functions that convert data according to a set of rules. Converter tables are single-byte conversion tables that perform stateless conversions. Programs and tables are in separate directories:

After a converter program is compiled and linked with the libiconv.a library, the program is placed in the /usr/lib/nls/loc/iconv directory.

To build a table converter, build a source converter table file. Use the genxlt command to compile translation tables into a format understood by the table converter. The output file is then placed in the /usr/lib/nls/loc/iconvTable directory.

Unicode and Universal Converters

Unicode (or UCS-2) conversion tables are found in:

$LOCPATH/uconvTable/*CodeSet*

The $LOCPATH/uconv/UCSTBL converter program is used to perform the conversion to and from UCS-2 using the iconv utilities. For the iconv utilities to use these uconvTable conversion tables, links must be set up within the $LOCPATH/iconv directory, for example, for code set "X."

ln -s /usr/lib/nls/loc/uconv/UCSTBL /usr/lib/nls/loc/iconv/X_UCS-2
ln -s /usr/lib/nls/loc/uconv/UCSTBL /usr/lib/nls/loc/iconv/UCS-2_X

A "Universal converter" program is provided that can be used to convert between any two code sets whose conversions to and from UCS-2 is defined. Given the following uconvTables:

X     -> UCS-2
UCS-2 -> Y

a universal conversion can be defined that maps

X -> UCS-2 -> Y

by use of the $LOCPATH/iconv/Universal_UCS_Conv. The conversion X->Y is set by defining links to the universal converter, for example:

ln-s /usr/lib/nls/loc/iconv/Universal_UCS_Conv /usr/lib/nls/loc/iconv/X_Y

Using Converters

The iconv interface is a set of subroutines used to open, perform, and close conversions:

• iconv_open
• iconv
• iconv_close

Code Set Conversion Filter Example

The following example shows how you can use these subroutines to create a code set conversion filter that accepts the ToCode and FromCode parameters as input arguments:

#include <stdio.h>
#include <nl_types.h>
#include <iconv.h>
#include <string.h>
#include <errno.h>
#include <locale.h>
 
#define ICONV_DONE() (r>=0)
#define ICONV_INVAL() (r<0) && (errno==EILSEQ))
#define ICONV_OVER() (r<0) && (errno==E2BIG))
#define ICONV_TRUNC() (r<0) && (errno==EINVAL))
 
#define USAGE 1
#define ERROR 2
#define INCOMP 3
 
char ibuf[BUFSIZ], obuf[BUFSIZ];
 
extern int errno;
 
main (argc,argv)
int argc;
char **argv;
{

 size_t  ileft,oleft;
 nl_catd catd;
 iconv_t cd;
 int r;
 char *ip,*op;
 
 setlocale(LC_ALL,"");
 catd = catopen (argv[0],0);
 
 if(argc!=3){
  fprintf(stderr,
   catgets (catd,NL_SETD,USAGE,"usage;conv fromcode tocode\n"));
  exit(1);
 }
 
 cd=iconv_open(argv[2],argv[1]);
 
ileft=0;
 
while(!feof(stdin)) {

 /*
 * After the next operation,ibuf will
 * contain new data plus any truncated
 * data left from the previous read.
 */
 ileft+=fread(ibuf+ileft,1,BUFSIZ-ileft,stdin);
 do {
  ip=ibuf;
  op=obuf;
  oleft=BUFSIZ;
  
  r=iconv(cd,&ip,&ileft,&op,&oleft);
  
  if(ICONV_INVAL()){
   fprintf(stderr,
      catgets(catd,NL_SETD,ERROR,"invalid input\n"));
   exit(2);
 }
  
 fwrite(obuf,1,BUFSIZ-oleft,stdout);
  
 if(ICONV_TRUNC() || ICONV_OVER())
  /*
  *Data remaining in buffer-copy
  *it to the beginning
  */
  
  memcpy(ibuf,ip,ileft);
 
  /*
  *loop until all characters in the input
  *buffer have been converted.
  */
 } while(ICONV_OVER());
}
 
 if(ileft!=0){
  /*
  *This can only happen if the last call
  *to iconv() returned ICONV_TRUNC, meaning
  *the last data in the input stream was
  *incomplete.
  */
 fprintf(stderr,catgets(catd,NL_SETD,INCOMP,"input incomplete\n"));
 exit(3);
 }
 
 iconv_close(cd);
 exit(0);
}

Naming Converters

Code set names are in the form CodesetRegistry-CodesetEncoding where:

The from,to variable used by the iconv command and iconv_open subroutine identifies a file whose name should be in the form /usr/lib/nls/loc/iconv/%f_%t or /usr/lib/nls/loc/iconvTable/%f_%t, where:

List of Converters

Converters change data from one code set to another. The sets of converters supported with the ICONV library are in the following sections. All converters shipped with the BOS Runtime Environment are located in the /usr/lib/nls/loc/iconv/* or /usr/lib/nls/loc/iconvTable/* directory.

These directories also contain private converters; that is, they are used by other converters. However, users and programs should only depend on the converters in the following lists.

Any converter shipped with the BOS Runtime Environment and not listed here should be considered private and subject to change or deletion. Converters supplied by other products can be placed in the /usr/lib/nls/loc/iconv/* or /usr/lib/nls/loc/iconvTable/* directory.

Programmers are encouraged to use registered code set names or code set names associated with an application. The X Consortium maintains a registry of code set names for reference. See the "Code Set Overview" for more information about code sets.

• List of PC, ISO, and EBCDIC Code Set Converters
• List of Multibyte Code Set Converters
• List of Interchange Converters--7-bit
• List of Interchange Converters--;8-bit
• List of Interchange Converters--;Compound Text
• List of Interchange Converters--uucode
• List of UCS-2 Interchange Converters
• List of UTF-8 Interchange Converters
• List of Miscellaneous Converters

List of PC, ISO, and EBCDIC Code Set Converters

These converters provide conversion between PC, ISO, and EBCDIC single-byte stateless code sets. The following types of conversions are supported: PC to/from ISO, PC to/from EBCDIC, and ISO to/from EBCDIC.

Conversion is provided between compatible code sets such as Latin-1 to Latin-1 and Greek to Greek. However, conversion between different EBCDIC national code sets is not supported. For information about converting between incompatible character sets refer to the "List of Interchange Converters--7-bit" and the "List of Interchange Converters--8-bit".

Conversion tables in the iconvTable directory are created by the genxlt command.

Compatible Code Set Names

The following table lists code set names that are compatible. Each line defines to/from strings that may be used when requesting a converter.

Note: The PC and ISO code sets are ASCII-based.

Note: A character that exists in the source code set but does not exist in the target code set is converted to a converter-defined substitute character.

Files

The following table describes the inconvTable converters found in the /usr/lib/nls/loc/iconvTable directory:

List of Multibyte Code Set Converters

Multibyte code-set converters convert characters among the following code-sets:

• PC multibyte code sets
• EUC multibyte code sets (ISO-based)
• EBCDIC multibyte code sets

The following table lists code set names that are compatible. Each line defines to/from strings that may be used when requesting a converter.

1. Conversions between Simplified and Traditional Chinese are provided (IBM-eucTW <--> IBM-eucCN and big5 <--> IBM-eucCN).

2. UTF-8 is an additional code set. See "List of UTF-8 Interchange Converters" for more information.

Files

The following list describes the Multibyte Code Set converters that are found in the /usr/lib/nls/loc/iconv directory.

List of Interchange Converters--7-bit

This converter provides conversion between internal code and 7-bit standard interchange formats (fold7). The fold7 name identifies encodings that can be used to pass text data through 7-bit mail protocols. The encodings are based on ISO2022. For more information about fold7 see "Understanding libiconv".

The fold7 converters convert characters from a code set to a canonical 7-bit encoding that identifies each character. This type of conversion is useful in networks where clients communicate with different code sets but use the same character sets. For example:

The following escape sequences designate standard code sets:

When converting from a code set to fold7, the escape sequence used to designate the code set is chosen according to the order listed. For example, the JISX0208.1983-0 characters use 01/11 01/04 04/02 as the designation.

Files

The following list describes the fold7 converters that are found in the /usr/lib/nls/loc/iconv directory:

List of Interchange Converters--8-bit

This converter provides conversions between internal code and 8-bit standard interchange formats (fold8). The fold8 name identifies encodings that can be used to pass text data through 8-bit mail protocols. The encodings are based on ISO2022. For more information about fold8 see "Understanding libiconv".

The fold8 converters convert characters from a specific code set encoding to a canonical 8-bit encoding that identifies each character. This type of conversion is useful in networks where clients communicate with different code sets but use the same character sets. For example:

The following escape sequences designate standard code sets.

When converting from a code set to fold8, the escape sequence used to designate the code set is chosen according to the order listed. For example, the JISX0208.1983-0 characters use 01/11 02/04 02/08 04/02 as the designation.

Files

The following list describes the fold8 converters found in the /usr/lib/nls/loc/iconv directory:

List of Interchange Converters--Compound Text

Compound text interchange converters convert between compound text and internal code sets.

Compound text is an interchange encoding defined by the X Consortium. It is used to communicate text between X clients. Compound text is based on ISO2022 and can encode most character sets using standard escape sequences. It also provides extensions for encoding private character sets. The supported code sets provide a converter to and from compound text. The name used to identify the compound text encoding is ct.

The following escape sequences are used to designate standard code sets in the order listed below.

Files

The following list describes the compound text converters that are found in the /usr/lib/nls/loc/iconv directory:

List of Interchange Converters--uucode

This converter provides the same mapping as the uuencode and uudecode Command.

During conversion from uucode, 62 bytes at a time (including a new-line character trailing the record) are converted, and generating 45 bytes in outbuf.

Files

The following list describes the uucode converters found in the /usr/lib/nls/loc/iconv directory:

List of UCS-2 Interchange Converters

UCS-2 is a universal, 16-bit encoding described in the "Code Set Overview" . Conversions for each code set are provided in both directions, between the code set and UCS-2.

UCS-2 converters are found in /usr/lib/nls/loc/uconvTable and /usr/lib/nls/loc/uconv directories. The uconvdef command is used to generate new converters or to customize existing UCS-2 converters.

The /usr/lib/nls/loc/iconv/Universal_UCS_Conv converter is used to generate conversions from any code set X to code set Y by setting the proper links:

cd /usr/lib/nls/loc/iconv
ln -s /usr/lib/nls/loc/uconv/Universal_UCS_Conv X_Y
ln -s /usr/lib/nls/loc/uconv/UCSTBL X_UCS-2
ln-s /usr/lib/nls/loc/uconv/UCSTBL UCS-2_Y
ln -s /usr/lib/nls/loc/uconv/UCSTBL X
ln -s /usr/lib/nls/loc/uconv/UCSTBL Y

List of UTF-8 Interchange Converters

UTF-8 is a universal, multibyte encoding described in the "UCS-2 and UTF-8" . Conversions for each code set are provided in both directions, between the code set and UTF-8.

UTF-8 converters are usually done by using the Universal_UCS_Conv (see "List of UCS-2 Interchange Converters" and /usr/lib/nls/loc/uconv/UTF-8 conversion.

List of Miscellaneous Converters

A set of low level converters used by the code set and interchange converters is provided. These converters are called miscellaneous converters. These low-level converters may be used by some of the interchange converters. However, the use of these converters is discouraged because they are intended for support of other converters.

Files

The following list describes the miscellaneous converters found in the /usr/lib/nls/loc/iconv and /usr/lib/nls/loc/iconvTable directories:

Related Information

National Language Support Overview for Programming, List of National Language Support Subroutines.

Code Sets Overview in AIX Kernel Extensions and Device Support Programming Concepts.

The iconv command, uuencode and uudecode commands.

The iconv_open subroutine, iconv subroutine, iconv_close subroutine.