Troubleshooting

This section contains tips to help you with some common challenges of IPsec VPNs.

A VPN connection has multiple stages that can be confirmed to ensure the connection is working properly. It is easiest to see if the final stage is successful first since if it is successful the other stages will be working properly. Otherwise, you will need to work back through the stages to see where the problem is located.

When a VPN connection is properly established, traffic will flow from one end to the other as if both ends were physically in the same place. If you can determine the connection is working properly then any problems are likely problems with your applications.

On some FortiGate units, such as the FortiGate 94D, you cannot ping over the IPsec tunnel without first setting a source-IP. In this scenario, you must assign an IP address to the virtual IPSEC VPN interface. Anything sourced from the FortiGate going over the VPN will use this IP address.

If the egress/outgoing interface (determined by kernel route) has an IP address, then use the IP address of the egress/outgoing interface. Otherwise, use the IP address of the first interface from the interface list (that has an IP address).

The first diagnostic command worth running, in any IPsec VPN troubleshooting situation, is the following:

diagnose vpn tunnel list

 

This command is very useful for gathering statistical data such as the number of packets encrypted versus decrypted, the number of bytes sent versus received, the SPI identifier, etc. This kind of information in the resulting output can make all the difference in determining the issue with the VPN.

Another appropriate diagnostic command worth trying is:

diagnose debug flow

 

This command will inform you of any lack of firewall policy, lack of forwarding route, and of policy ordering issues.

The following is a list of such potential issues. Bear in mind that the troubleshooting suggestions below are not exhaustive, and may not reflect your network topology.

The options to configure policy-based IPsec VPN are unavailable.

Go to System > Feature Select. Select Show More and turn on Policy-based IPsec VPN.

The VPN connection attempt fails.

If your VPN fails to connect, check the following:

  • Ensure that the pre-shared keys match exactly (see The pre-shared key does not match (PSK mismatch error). below).
  • Ensure that both ends use the same P1 and P2 proposal settings (seeThe SA proposals do not match (SA proposal mismatch). below).
  • Ensure that you have allowed inbound and outbound traffic for all necessary network services, especially if services such as DNS or DHCP are having problems.
  • Check that a static route has been configured properly to allow routing of VPN traffic.
  • Ensure that your FortiGate unit is in NAT/Route mode, rather than Transparent.
  • Check your NAT settings, enabling NAT traversal in the Phase 1 configuration while disabling NAT in the security policy. You might need to pin the PAT/NAT session table, or use some of kind of NAT-T keepalive to avoid the expiration of your PAT/NAT translation.
  • Ensure that both ends of the VPN tunnel are using Main mode, unless multiple dial-up tunnels are being used.
  • If you have multiple dial-up IPsec VPNs, ensure that the peer ID is configured properly on the FortiGate and that clients have specified the correct local ID. Furthermore, in circumstances where multiple remote dialup VPN tunnels exist, each tunnel must have a peer ID set.
  • If you are using FortiClient, ensure that your version is compatible with the FortiGate firmware by reading the FortiOS Release Notes.
  • If you are using Perfect Forward Secrecy (PFS), ensure that it is used on both peers. You can use the diagnose vpn tunnel list command to troubleshoot this.
  • Ensure that the Quick Mode selectors are correctly configured. If part of the setup currently uses firewall addresses or address groups, try changing it to either specify the IP addresses or use an expanded address range. This is especially useful if the remote endpoint is not a FortiGate device.
  • If XAUTH is enabled, ensure that the settings are the same for both ends, and that the FortiGate unit is set to Enable as Server.
  • Check IPsec VPN Maximum Transmission Unit (MTU) size. A 1500 byte MTU is going to exceed the overhead of the ESP-header, including the additional ip_header,etc. You can use the diagnose vpn tunnel list command to troubleshoot this.
  • If your FortiGate unit is behind a NAT device, such as a router, configure port forwarding for UDP ports 500 and 4500.
  • Remove any Phase 1 or Phase 2 configurations that are not in use. If a duplicate instance of the VPN tunnel appears on the IPsec Monitor, reboot your FortiGate unit to try and clear the entry.

If you are still unable to connect to the VPN tunnel, run the following diagnostic command in the CLI:

diagnose debug application ike -1

diagnose debug enable

 

The resulting output may indicate where the problem is occurring. When you are finished, disable the diagnostics by using the following command:

diagnose debug reset

diagnose debug disable

 

The VPN tunnel goes down frequently.

If your VPN tunnel goes down often, check the Phase 2 settings and either increase the Keylife value or enable Autokey Keep Alive.

The pre-shared key does not match (PSK mismatch error).

It is possible to identify a PSK mismatch using the following combination of CLI commands:

diag vpn ike log filter name <phase1-name>

diag debug app ike -1

diag debug enable

 

This will provide you with clues as to any PSK or other proposal issues. If it is a PSK mismatch, you should see something similar to the following output:

ike 0:TRX:322: PSK auth failed: probable pre-shared key mismatch

ike Negotiate SA Error:

The SA proposals do not match (SA proposal mismatch).

The most common problem with IPsec VPN tunnels is a mismatch between the proposals offered between each party. Without a match and proposal agreement, Phase 1 can never establish. Use the following command to show the proposals presented by both parties.

diag debug app ike -1

diag debug enable

 

The resulting output should include something similar to the following, where blue represents the remote VPN device, and green represents the local FortiGate.

responder received SA_INIT msg

incoming proposal:

proposal id = 1:

protocol = IKEv2:

encapsulation = IKEv2/none

type=ENCR, val=AES_CBC (key_len = 256)

type=INTEGR, val=AUTH_HMAC_SHA_96

type=PRF, val=PRF_HMAC_SHA

type=DH_GROUP, val=1536.

proposal id = 2:

protocol = IKEv2:

encapsulation = IKEv2/none

type=ENCR, val=3DES_CBC

type=INTEGR, val=AUTH_HMAC_SHA_2_256_128

type=PRF, val=PRF_HMAC_SHA2_256

type=DH_GROUP, val=1536.

proposal id = 1:

protocol = IKEv2:

encapsulation = IKEv2/none

type=ENCR, val=AES_CBC (key_len = 128)

type=INTEGR, val=AUTH_HMAC_SHA_96

type=PRF, val=PRF_HMAC_SHA

type=DH_GROUP, val=1536.

 

Pre-existing IPsec VPN tunnels need to be cleared.

Should you need to clear an IKE gateway, use the following commands:

diagnose vpn ike restart

diagnose vpn ike gateway clear

LAN interface connection

To confirm whether a VPN connection over LAN interfaces has been configured correctly, issue a ping or traceroute command on the network behind the FortiGate unit to test the connection to a computer on the remote network. If the connection is properly configured, a VPN tunnel will be established automatically when the first data packet destined for the remote network is intercepted by the FortiGate unit.

If the ping or traceroute fail, it indicates a connection problem between the two ends of the tunnel. This may or may not indicate problems with the VPN tunnel. You can confirm this by going to Monitor > IPsec Monitor where you will be able to see your connection. A green arrow means the tunnel is up and currently processing traffic. A red arrow means the tunnel is not processing traffic, and this VPN connection has a problem.

If the connection has problems, see Troubleshooting VPN connections.

Dialup connection

A dialup VPN connection has additional steps. To confirm that a VPN between a local network and a dialup client has been configured correctly, at the dialup client, issue a ping command to test the connection to the local network. The VPN tunnel initializes when the dialup client attempts to connect.

If the ping or traceroute fail, it indicates a connection problem between the two ends of the tunnel. This may or may not indicate problems with the VPN tunnel, or dialup client. As with the LAN connection, confirm the VPN tunnel is established by checking Monitor > IPsec Monitor.

Troubleshooting VPN connections

If you have determined that your VPN connection is not working properly through Troubleshooting, the next step is to verify that you have a phase2 connection.

If traffic is not passing through the FortiGate unit as you expect, ensure the traffic does not contain IPcomp packets (IP protocol 108, RFC 3173). FortiGate units do not allow IPcomp packets, they compress packet payload, preventing it from being scanned.

Testing Phase 1 and 2 connections is a bit more difficult than testing the working VPN. This is because they require diagnose CLI commands. These commands are typically used by Fortinet customer support to discover more information about your FortiGate unit and its current configuration.

Before you begin troubleshooting, you must:

  • Configure FortiGate units on both ends for interface VPN
  • Record the information in your VPN Phase 1 and Phase 2 configurations - for our example here the remote IP address is 10.11.101.10 and the names of the phases are Phase 1 and Phase 2
  • Install a telnet or SSH client such as putty that allows logging of output
  • Ensure that the admin interface supports your chosen connection protocol so you can connect to your FortiGate unit admin interface.

For this example, default values were used unless stated otherwise.

Obtaining diagnose information for the VPN connection - CLI
  1. Log into the CLI as admin with the output being logged to a file.
  2. Stop any diagnose debug sessions that are currently running with the CLI command

diagnose debug disable

 

  1. Clear any existing log-filters by running

diagnose vpn ike log-filter clear

 

  1. Set the log-filter to the IP address of the remote computer (10.11.101.10). This filters out all VPN connections except ones to the IP address we are concerned with. The command is

diagnose vpn ike log-filter dst-addr4 10.11.101.10.

 

  1. Set up the commands to output the VPN handshaking. The commands are:

diagnose debug app ike 255

diagnose debug enable

 

  1. Have the remote FortiGate initiate the VPN connection in the web-based manager by going to

    VPN > IPsec Tunnels     and selecting Bring up.



    This makes the remote FortiGate the initiator and the local FortiGate becomes the responder. Establishing the connection in this manner means the local FortiGate will have its configuration information as well as the information the remote computer sends. Having both sets of information locally makes it easier to troubleshoot your VPN connection.
  1. Watch the screen for output, and after roughly 15 seconds enter the following CLI command to stop the output.

diagnose debug disable

 

  1. If needed, save the log file of this output to a file on your local computer. Saving the output to a file can make it easier to search for a particular phrase, and is useful for comparisons.
Troubleshooting a Phase 1 VPN connection

Using the output from Obtaining diagnose information for the VPN connection - CLI, search for the word proposal in the output. It may occur once indicating a successful connection, or it will occur two or more times for an unsuccessful connection — there will be one proposal listed for each end of the tunnel and each possible combination in their settings. For example if 10.11.101.10 selected both Diffie-Hellman Groups 1 and 5, that would be at least 2 proposals set.

A successful negotiation proposal will look similar to

IPsec SA connect 26 10.12.101.10->10.11.101.10:500

config found

created connection: 0x2f55860 26 10.12.101.10->10.11.101.10:500

IPsec SA connect 26 10.12.101.10->10.11.101.10:500 negotiating

no suitable ISAKMP SA, queuing quick-mode request and initiating ISAKMP SA negotiation

initiator: main mode is sending 1st message...

cookie 3db6afe559e3df0f/0000000000000000

out [encryption]

sent IKE msg (ident-i1send): 10.12.101.10:500->10.11.101.10:500, len=264, id=3db6afe559e3df0f/0000000000000000

diaike 0: comes 10.12.101.1:500->10.11.101.1:500,ifindex=26....

 

Note the phrase “initiator: main mode is sending 1st message...” which shows you the handshake between the ends of the tunnel is in progress. Initiator shows the remote unit is sending the first message.

Troubleshooting invalid ESP packets using Wireshark

The following section provides information to help debug an encryption key mismatch. The ESP packet invalid error is due to an encryption key mismatch after a VPN tunnel has been established. When an IPsec VPN tunnel is up, but traffic is not able to pass through the tunnel, Wireshark (or an equivalent program) can be used to determine whether there is an encryption mismatch. A mismatch could occur for many reasons, one of the most common is the instability of an ISP link (ADSL, Cable), or it could effectively be any device in the physical connection.

The following information is required to troubleshoot the problem.

  • Take a packet sniffer trace on both FortiGates.
  • Run the diag vpn tunnel list command a few times on both FortiGates when generating traffic that will pass through the tunnel.

In the following example, the error message was seen on the recipient FortiGate:


date=2010-12-28 time=18:19:35 devname=Kosad_VPN device_id=FG300B3910600118 log_id=0101037132 type=event subtype=ipsec pri=critical vd="root" msg="IPsec ESP" action="error" rem_ip=180.87.33.2 loc_ip=121.133.8.18 rem_port=32528 loc_port=4500 out_intf="port2" cookies="88d40f65d555ccaf/05464e20e4afc835" user="N/A" group="N/A" xauth_user="N/A" xauth_group="N/A" vpn_tunnel="fortinet_0" status=esp_error error_num=Invalid ESP packet detected (HMAC validation failed). spi=c32b09f7 seq=00000012

This is the output of the command diag vpn tunnel list on the FortiGate:

inet ver=1 serial=2 192.168.1.205:4500->121.133.8.18:4500 lgwy=dyn tun=intf mode=auto bound_if=4
proxyid_num=1 child_num=0 refcnt=7 ilast=0 olast=0
stat: rxp=41 txp=56 rxb=4920 txb=3360
dpd: mode=active on=1 idle=5000ms retry=3 count=0 seqno=696
natt: mode=keepalive draft=32 interval=10 remote_port=4500
proxyid=P2_60C_Fortinet proto=0 sa=1 ref=2 auto_negotiate=0 serial=1 src: 0:182.40.101.0/255.255.255.0:0
dst: 0:100.100.100.0/255.255.255.0:0
SA: ref=3 options=0000000d type=00 soft=0 mtu=1428 expire=1106 replaywin=0 seqno=15
life: type=01 bytes=0/0 timeout=1777/1800
dec: spi=29a26eb6 esp=3des key=24 bf25e69df90257f64c55dda4069f01834cd0382fe4866ff2
ah=sha1 key=20 38b2600170585d2dfa646caed5bc86d920aed7ff
enc: spi=c32b09f7 esp=3des key=24 0abd3c70032123c3369a6f225a385d30f0b2fb1cd9687ec8
ah=sha1 key=20 214d8e717306dffceec3760464b6e8edb436c6

This is the packet capture from the FortiGate:

How to verify if the original packet has been encrypted correctly

To verify, it is necessary to decrypt the ESP packet using Wireshark. Open the packet capture that is taken from initiator FortiGate using Wireshark. Go to Edit > Preferences, expand Protocol and look for ESP. Select "Attempt to detect/decode encrypted ESP payloads", and fill in the information for the encryption algorithm and the keys. This information can be obtained from the output of the command diag vpn tunnel list.

If the packet was encrypted correctly using the correct key, then the decryption will be successful and it will be possible to see the original package as shown below:

Repeat the decryption process for the packet capture from the recipient firewall. If the decryption failed using the same key, the packet may be corrupted and the interface should then be checked for CRC or packet errors