Introduction : This guide will present the basic information required to troubleshoot problems in establishing an IKE IPSec VPN Tunnel. The guide will first present the basic premise of IKE negotiation, protocol support, and noteworthy configuration details. This guide will then provide a methodology to test and troubleshoot using the IKE log messages.

IKE Negotiation : An IKE VPN tunnel is established by negotiations between two IPSec security devices. The VPN Security Association (SA) configured on each device defines the Security Policy settings proposed in the IKE negotiation. For negotiations to succeed, these Security Policies must be in agreement.

When a VPN/Firewall receives a packet (e.g. a PING) destined for a subnet located behind the remote peer VPN/Firewall and the tunnel is not established, it will initiate the IKE negotiations to establish the VPN tunnel.

The IPSec security devices negotiating an IKE VPN are referred to as an IKE Initiator and an IKE Responder. This distinction is evident in the logs.

They are defined as follows:

IKE Initiator: Device initiating the IKE VPN tunnel negotiation request.

IKE Responder: Device receiving the request to establish an IKE VPN tunnel

IKE VPN Protocols : The IKE protocol is used during the entire negotiation phase. The negotiation defines policy settings and keys used by the IPSec tunnel protocol. The protocols used for the IKE negotiation and VPN tunnel are as follows:

Standard

• TCP port 50 for IPSec Encapsulating Security Protocol (ESP) traffic
• TCP port 51 for IPSec Authentication Header (AH) traffic
• UDP port 500 for Internet Key Exchange (IKE) negotiation traffic

With NAT Traversal (NAT-T) active

• UDP port 500 for Internet Key Exchange (IKE) negotiation traffic
• UDP port 4500 for IPSec Encapsulating Security Protocol (ESP) traffic

These protocols must not be blocked by any firewalls or the ISP networks between the two IPSec security devices attempting to establish the tunnel.

NAT Traversal (NAT-T) NAT Traversal (NAT-T) is a VPN option used on many IPSec security devices. It is typically enabled by default. With this option, a NAT discovery process runs after the IKE initiation request to determine if there are any NAT devices in the tunnel path. If a NAT device is detected in the tunnel path, the IPSec security devices will use UDP encapsulated IPSec packets for the VPN tunnel. NAT discovery messages are displayed in the logs, but typically only in the IKE Responder log with Aggressive Mode.

Main Mode vs. Aggressive Mode

There are two phases of the IKE negotiations, called Phase 1 and Phase 2. Phase 1 can be configured to use either Main Mode or Aggressive Mode. Main Mode is more secure in providing identity protection for the negotiating nodes. However, Main Mode requires a static IP address on both IPSec security devices negotiating the VPN tunnel.

Aggressive Mode is used when one IPSec security device has a dynamic WAN IP address (i.e., uses DHCP, PPPoE, PPPoA, PPTP, etc.). Aggressive Mode has more configuration requirements than Main Mode and may be difficult or impossible to achieve with some IPSec security device pairings.

Routing Multiple Subnets Through a VPN (optional)

The SA on each IPSec security device also defines the subnet behind the remote peer IPSec security device to which the VPN tunnel will route. In some cases, the IPSec security devices may support defining multiple subnets via a single SA. But this is device dependant and specific.

If there are multiple remote destination subnets defined in an SA, there will typically be a separate tunnel established for each subnet. Thus, a separate negotiation would occur for each subnet.

Testing The VPN Tunnel

Initiating traffic through the VPN will start the IKE negotiation. Check the VPN Summary page of your IPSec security device to see the status of the VPN and any active tunnels. If a tunnel does not show as active after traffic for it has been generated, review the log messages on both IPSec security devices to determine the problem. The final sections of this document provide troubleshooting tips based on the error messages displayed.

Running a simple PING test will typically generate sufficient traffic to initiate the VPN. Note: If a server is not responding to PING across a tunnel that appears to be active, verify that there is not a routing or connectivity problem between the server and the IPSec security device local to it. Also, verify the TCP/IP configurations of the server.

Troubleshooting IKE VPN Tunnel Establishment

There are 3 basic scenarios to investigate if the IKE VPN tunnel fails to establish:

1. No Response to IKE Initiation Request
2. IKE Negotiation Fails
3. IKE Negotiation Completes, but No Traffic is Passing

The log messages generated during the IKE negotiations will show quickly which of the scenarios is occurring and where to focus the troubleshooting. The log entries in the Initiator log are different than those in the Responder log. In most cases, the responder will provide more precise information when there is an SA proposal mismatch during the negotiations. The logs of both the IKE Initiator and IKE Responder should be checked when troubleshooting establishment of a VPN.

How to troubleshoot a tunnel :

1. Check the WAN to WAN connectivity

Before your VPN can work, you obviously need connectivity between the WANs on each peer router. This step describes how to test this.

To check that the IPsec peers can connect through the intermediary network or Internet, perform a ping or traceroute from one peer router to the other peer’s public interface. Note that this is not a payload (LAN to LAN) check.

Note that both ping and traceroute may not work if you have a firewall enabled on either peer. Firewalls are more likely to allow pings, so start with ping. If you suspect that the firewall is blocking pings, move to “2. Confirm VPN establishment” on page 7—those tests are not affected by the firewall.

Scenario 1: No Response to IKE Initiation Request
Use the following table of log entries to determine troubleshooting steps to resolve a problem initiating the IKE negotiations.

IKE Initiator Log Messages Troubleshooting Notes

No IKE Initiator Start Negotiation message in log Check the following: ‘Disable this SA’ box is not checked in SA of IKE Initiator. Destination subnet defined in SA of IKE Initiator.
IKE Initiator Log Messages – No IKE Initiator Start Negotiation message in log

Check the following:

• ‘Disable this SA’ box is not checked in SA of IKE Initiator.
• Destination subnet defined in SA of IKE Initiator.

IKE Initiator Log Messages – 

• IKE Initiator: No response – Remote Peer Timeout
• IKE Initiator: No response – Remote Peer Timeout
• IKE Initiator: No response – Remote Peer Timeout
• IKE negotiation aborted due to timeout

Note: If IKE Initiator Log only shows several timeout messages and negotiation aborted after a short delay, then there is a communication problem between the Initiator and Responder.

Check the following:

• Network connectivity between the units. (Hint: Try to access remote unit HTTPS management console from a host behind the local unit)
• ‘Disable this SA’ box is not checked in SA of IKE Responder.
• IPSec Gateway address in Initiator SA specifies WAN address of IKE Responder.
• IPSec Gateway Name (if used) resolves to WAN address of IKE Responder.
• IKE Access Rules are enabled on both IPSec security devices.
• No other firewalls in path blocking IKE (UDP 500) or IPSec (IP 50) protocols.

Contact ISP to see if they are blocking IKE (UDP 500) or IPSec (IP 50) protocols.

Scenario 2: IKE Negotiation Failures

Use the following tables to identify pertinent log entries in the IKE Initiator and IKE Responder and appropriate troubleshooting steps to resolve the problem. Troubleshooting with IKE Initiator Log Messages