IP Security Problem Determination

AIX Version 4.3 System Management Guide: Communications and Networks

IP Security Problem Determination

This section includes some hints and tips that may assist you when you encounter a problem. We recommend that you set up logging from the start. Logs are very useful in determining what is going on with the filters and tunnels. (See Logging Facilities for detailed log information.)

Error:	Issuing mktun command results in the following error: insert_tun_man4(): write failed : The requested resource is busy. Problem: The tunnel you requested to activate is already active or you have colliding SPI values. To fix: Issue the rmtun command to deactivate, then issue the mktun command to activate. Check to see if the SPI values for the failing tunnel match any other active tunnel. Each tunnel should have its own unique SPI values.
Error:	Issuing mktun command results in the following error: Device ipsec_v4 is in Defined status. Tunnel activation for IP Version 4 not performed. Problem: You have not made the IP Security device available. To fix: Issue the following command: mkdev -l ipsec -t 4 You may have to change -t option to 6 if you are getting the same error for Version 6 tunnel activation. The devices must be in available state. To check the IP Security device state, issue the following command: lsdev -Cc ipsec
Error:	Issuing a chfilt command results in the following error: Cannot modify the first rule. or Cannot modify a pre_defined filter rule. Problem: You are not allowed to modify these filter rules. You may however change whether they log or not. To fix: If you want these rules to log, just issue the command: chfilt -v (4 or 6) -n (filter number) -l y If you want to set up the default rules to pass Authentication Header (AH) or Encapsulating Security Payload (ESP) header packets to specific hosts only, then you may prevent the autogeneration of rules by using the -g parameter with the gentun command. Then you may add in the same rules for the AH and ESP packets with the specific host's IP address for source and the partner host's IP address for destination. Make sure these rules are placed before the actual tunnel traffic rules.
Error:	Issuing a gentun command results in the following error: Invalid Source IP address Problem: You have not entered a valid IP address for the source address. To fix: For IP Version 4 tunnels, please check to see that you have entered an available IP Version 4 address for the local machine. You cannot use host names for the source when generating tunnels, you may only use host names for the destination. For IP Version 6 tunnels, please check to see that you entered an available IP Version 6 address. If you type `netstat -in` and no IP Version 6 addresses exist, run /usr/sbin/autoconf6 (interface) for a link local auto-generated address (using MAC address) or use ifconfig to manually assign an address.
Error:	Issuing mktun command results in the following error: insert_tun_man4(): write failed : A system call received a parameter that is not valid. Problem: Tunnel generation occurred with invalid ESP and AH combination or without the use of the new header format when necessary. To fix: Check to see what authentication algorithms are in use by the particular tunnel in question. Remember that the HMAC_MD5 and HMAC_SHA algorithms require the new header format. The new header format can be changed using the SMIT fast path ips4_basic or the -z parameter with the chtun command. Also remember that DES_CBC_4 cannot be used with the new header format.

Troubleshooting IKE Tunnel Errors

IKE Tunnel Process Flow

The IKE tunnels are set up by the communication of the ike command or the Web-based System Manager VPN panels with two daemons:

tmd: The Tunnel Manager daemon
isakmpd: The ISAKMP daemon

For IKE tunnels to be properly set up, both daemons must be running. If IP Security is set to start at reboot, these daemons start automatically. Otherwise, they must be started manually.

The Tunnel Manager gives requests to isakmpd to start a tunnel. If the tunnel already exists or is not valid (for instance, has an invalid remote address), it reports an error. If negotiation has started, it may take some time, depending on network latency, for the negotiation to complete. The ike cmd=list command can list the state of the tunnel to determine if the negotiation was successful. Also, the Tunnel Manager logs events to syslog to the levels of debug, event, and information, which can be used to monitor the progress of the negotiation.

The sequence is:

Use Web-based System Manager or the ike command to initiate a tunnel.
The tmd daemon gives the isakmpd daemon a connection request for key management (phase 1).
The isakmpd daemon responds with SA created or an error.
The tmd daemon gives the isakmpd daemon a connection request for a data management tunnel (phase 2).
The isakmpd daemon responds with SA created or an error.
Tunnel parameters are inserted into the kernel tunnel cache.
Filter rules are added to the kernel dynamic filter table.

When the machine is acting as a responder, the isakmpd daemon notifies the Tunnel Manager tmd daemon that a tunnel has been negotiated successfully and a new tunnel is inserted into the kernel. In such cases, the process starts with step 3 and continues until step 7, without the tmd daemon issuing connection requests.

isakmpd Logging

The isakmpd daemon logs to a separate log because of the number and size of the logging messages. Logging is enabled using the ike cmd=log command. The /etc/isakmpd.conf configuration file can be set up to specify the output files for each logging level. Levels may be set to none, errors, events, and information.

The isakmpd daemon code will either initiate or respond by sending or evaluating a proposal. If the proposal is accepted, a security association is created and the tunnel will be set up. If the proposal is not accepted or the connection times out before the negotation completes, the daemon indicates an error. The entries in syslog from the tmd indicate whether the negotiation succeeded. To find out the exact cause of a failed negotiation, the isakmpd log needs to be checked.

Tracing facilities

Tracing is a debug facility for tracing kernel events. Traces can be used to get more specific information about events or errors occuring in the kernel filter and tunnel code.

SMIT has an IP Security trace facility available through the Advanced IP Security Configuration menu. The information captured by this trace facility includes information on Error, Filter, Filter Information, Tunnel, Tunnel Information, Capsulation/Decapsulation, Capsulation Information, Crypto, and Crypto Information. By design, the error trace hook provides the most critical information. The info trace hook can generate a lot of information and may have an impact on system performance. This tracing will provide clues to you as to what a problem may be. Tracing information will also be required when speaking with an IBM IP Security Technician. To access the tracing facility, use the SMIT fast path smit ips4_tracing (for IP Version 4) or smit ips6_tracing (for IP Version 6).

ipsecstat

You can issue the ipsecstat command to generate the following sample report. This sample report shows that the IP Security devices are in the available state, that there are three authentication algorithms installed, three encryption algorithms installed, and that there is a current report of packet activity. This information may be useful to you in determining where a problem exists if you are troubleshooting your IP Security traffic.

IP Security Devices:
ipsec_v4 Available
ipsec_v6 Available

Authentication Algorithm:
HMAC_MD5 -- Hashed MAC MD5 Authentication Module
HMAC_SHA -- Hashed MAC SHA Hash Authentication Module
KEYED_MD5 -- Keyed MD5 Hash Authentication Module
 
Encryption Algorithm:
CDMF -- CDMF Encryption Module
DES_CBC_4 -- DES CBC 4 Encryption Module
DES_CBC_8 -- DES CBC 8 Encryption Module
3DES_CBC -- Triple DES CBC Encryption Module
 
IP Security Statistics -
Total incoming packets:  1106
Incoming AH packets:326
Incoming ESP packets:  326
Srcrte packets allowed:  0
Total outgoing packets:844
Outgoing AH packets:527
Outgoing ESP packets:  527
Total incoming packets dropped:  12
  Filter denies on input:  12
  AH did not compute: 0
  ESP did not compute:0
  AH replay violation:0
  ESP replay violation:  0
Total outgoing packets dropped:0
  Filter denies on input:0
Tunnel cache entries added: 7
Tunnel cache entries expired:  0
Tunnel cache entries deleted:  6

Interoperability Notes

The following sections describe interoperability solutions. For related information, see Coexistence of IP Security and IBM Secured Network Gateway 2.2/IBM Firewall 3.1 or 3.2 .

IBM Firewall 3.1/3.2, IBM Secured Network Gateway (SNG) 2.2

The IBM Firewall 3.1/3.2, and IBM SNG 2.2 products operate as a tunnel partner with the IP Security feature of AIX 4.3. The tunnel may be created on the firewall and exported, then imported into an AIX 4.3 host running IP Security by using the -n option with the imptun command. There is, however, a script called ipsec_convert that is shipped as a sample shell script that transforms an IP Security tunnel export file into the necessary files needed by the IBM Firewall 3.1/3.2, or IBM SNG 2.2 to import.

There are several items to note when exporting a tunnel that will have the IBM Firewall 3.1/3.2 or the IBM SNG 2.2 as a tunnel partner. They are as follows:

The IBM Firewall 3.1/3.2, and IBM SNG 2.2 only use the KEYED_MD5 algorithm for AH, and will not be able to import a tunnel specifying HMAC_MD5 nor HMAC_SHA1 for AH.
The IBM Firewall 3.1/3.2 and IBM SNG 2.2 do not support replay prevention.
The IBM Firewall 3.1/3.2 and IBM SNG 2.2 do not accept a tunnel lifetime of zero (unlimited lifetime).
Also, IP Security will create tunnels with its own ordered numbers for tunnel IDs; when importing into the firewall, make sure these numbers are not already in use.
The IBM Firewall 3.1/3.2 and IBM SNG 2.2 do not support IP Version 6 tunnels.
Make sure that SNG 2.2 has been been updated with the correct service level.

FTP Software's IP Security

FTP Software's TCP/IP stack and IP Security function will operate as a tunnel partner with the IP Security feature of AIX 4.3. Follow the instructions from FTP Software to add IP Security. From the FTP Software's IP Security configuration table, you can choose to add an address for setting up secure communication. After that, a page comes up with the IP Security configuration entry fields. The source AH SPI and shared secret key (for AH) have been generated for you, but you may enter the destination AH SPI and shared secret key in the fields provided. The page also contains autogenerated source ESP SPI and source ESP key. When the box for encryption is selected, the source ESP SPI and source ESP key are shown.

For interoperability, follow these steps:

On the AIX 4.3 host, add a tunnel using FTP Software's IP Security parameters for the destination AH SPI and key and the destination ESP SPI and key.
Note that FTP Software's IP Security configuration page allows only hexadecimal numbers into their entry fields. You will have to convert the AH and ESP SPI values generated by FTP Software into decimal numbers before giving them to AIX 4.3 IP Security.
When entering SPI values and keys into FTP's IP Security configuration page, leave off the 0x. FTP Software will also drop any leading zeros.
Note that the policy can only be authentication after encryption (encr/auth), authentication only (auth), or encryption only (encr).
Only DES_CBC_4 and DES_CBC_8 can be used for encryption.
Only Keyed_MD5 should be used for authentication.
Be careful entering the key values, if they are not entered correctly, the tunnel will not function.
You will have to reboot the Windows 95 box that has FTP Software's IP Security function when a new tunnel is added or when an existing tunnel is changed.