Defining IKE negotiation parameters

In Phase 1, the two peers exchange keys to establish a secure communication channel between them. As part of the Phase 1 process, the two peers authenticate each other and negotiate a way to encrypt further communications for the duration of the session. The Phase 1 Proposal parameters select the encryption and authentication algorithms that are used to generate keys for protecting negotiations.

The IKE negotiation parameters determine:

  • Which encryption algorithms may be applied for converting messages into a form that only the intended recipient can read
  • Which authentication hash may be used for creating a keyed hash from a preshared or private key
  • Which Diffie-Hellman group (DH Group) will be used to generate a secret session key

Phase 1 negotiations (in main mode or aggressive mode) begin as soon as a remote VPN peer or client attempts to establish a connection with the FortiGate unit. Initially, the remote peer or dialup client sends the FortiGate unit a list of potential cryptographic parameters along with a session ID. The FortiGate unit compares those parameters to its own list of advanced Phase 1 parameters and responds with its choice of matching parameters to use for authenticating and encrypting packets. The two peers handle the exchange of encryption keys between them, and authenticate the exchange through a preshared key or a digital signature.

Generating keys to authenticate an exchange

The FortiGate unit supports the generation of secret session keys automatically using a Diffie-Hellman algorithm. These algorithms are defined in RFC 2409. The Keylife setting in the Phase 1 Proposal area determines the amount of time before the Phase 1 key expires. Phase 1 negotiations are re-keyed automatically when there is an active security association. See Dead Peer Detection.

You can enable or disable automatic re-keying between IKE peers through the phase1-rekey attribute of the config system global CLI command. For more information, see the “System” chapter of the FortiGate CLI Reference.

note icon

When in FIPS-CC mode, the FortiGate unit requires DH key exchange to use values at least 3072 bits long. However most browsers need the key size set to 1024. You can set the minimum size of the DH keys in the CLI.

config system global
   set dh-params 3072

When you use a preshared key (shared secret) to set up two-party authentication, the remote VPN peer or client and the FortiGate unit must both be configured with the same preshared key. Each party uses a session key derived from the Diffie-Hellman exchange to create an authentication key, which is used to sign a known combination of inputs using an authentication algorithm (such as HMAC-MD5, HMAC-SHA-1, or HMAC-SHA-256). Hash-based Message Authentication Code (HMAC) is a method for calculating an authentication code using a hash function plus a secret key, and is defined in RFC 2104. Each party signs a different combination of inputs and the other party verifies that the same result can be computed.

note icon For information regarding NP accelerated offloading of IPsec VPN authentication algorithms, see Hardware acceleration.

When you use preshared keys to authenticate VPN peers or clients, you must distribute matching information to all VPN peers and/or clients whenever the preshared key changes.

As an alternative, the remote peer or dialup client and FortiGate unit can exchange digital signatures to validate each other’s identity with respect to their public keys. In this case, the required digital certificates must be installed on the remote peer and on the FortiGate unit. By exchanging certificate DNs, the signed server certificate on one peer is validated by the presence of the root certificate installed on the other peer.

The following procedure assumes that you already have a Phase 1 definition that describes how remote VPN peers and clients will be authenticated when they attempt to connect to a local FortiGate unit. For information about the Local ID and XAuth options, see Defining IKE negotiation parameters and Defining IKE negotiation parameters. Follow this procedure to add IKE negotiation parameters to the existing definition.

Defining IKE negotiation parameters

  1. Go to VPN > IPsec Tunnels and create the new custom tunnel or edit an existing tunnel.
  2. Edit the Phase 1 Proposal (if it is not available, you may need to click the Convert to Custom Tunnel button).
  3. Select Phase 1 Proposal and include the appropriate entries as follows:
Phase 1 Proposal Select the encryption and authentication algorithms that will be used to generate keys for protecting negotiations.

Add or delete encryption and authentication algorithms as required. Select a minimum of one and a maximum of three combinations. The remote peer must be configured to use at least one of the proposals that you define.

It is invalid to set both Encryption and Authentication to null.
Encryption Select a symmetric-key algorithms:

NULL — Do not use an encryption algorithm.
DES — Digital Encryption Standard, a 64-bit block algorithm that uses a 56-bit key.
3DES — Triple-DES; plain text is encrypted three times by three keys.
AES128 — A 128-bit block algorithm that uses a 128-bit key.
AES192 — A 128-bit block algorithm that uses a 192-bit key.
AES256 — A 128-bit block algorithm that uses a 256-bit key.
ChaCha20/Poly1305— A 128-bit block algorithm that uses a 128-bit key and a symmetric cipher. Only available for IKEv2.
Authentication You can select either of the following message digests to check the authenticity of messages during an encrypted session:

NULL — Do not use a message digest.
MD5 — Message Digest 5.
SHA1 — Secure Hash Algorithm 1 - a 160-bit message digest.

To specify one combination only, set the Encryption and Authentication options of the second combination to NULL. To specify a third combination, use the Add button beside the fields for the second combination.

For information regarding NP accelerated offloading of IPsec VPN authentication algorithms, see Hardware acceleration.
Diffie-Hellman Group Select one or more Diffie-Hellman groups from DH groups 1, 2, 5, 14 through 21, and 27 through 30. When using aggressive mode, DH groups cannot be negotiated. By default, DH group 14 is selected, to provide sufficient protection for stronger cipher suites that include AES and SHA2. If you select multiple DH groups, the order they appear in the configuration is the order in which they are negotiates.

If both VPN peers (or a VPN server and its client) have static IP addresses and use aggressive mode, select a single DH group. The setting on the FortiGate unit must be identical to the setting on the remote peer or dialup client.

When the remote VPN peer or client has a dynamic IP address and uses aggressive mode, select up to three DH groups on the FortiGate unit and one DH group on the remote peer or dialup client. The setting on the remote peer or dialup client must be identical to one of the selections on the FortiGate unit.

If the VPN peer or client employs main mode, you can select multiple DH groups. At least one of the settings on the remote peer or dialup client must be identical to the selections on the FortiGate unit.
Keylife Type the amount of time (in seconds) that will be allowed to pass before the IKE encryption key expires. When the key expires, a new key is generated without interrupting service. The keylife can be from 120 to 172800 seconds.
Nat-traversal Enable this option if a NAT device exists between the local FortiGate unit and the VPN peer or client. The local FortiGate unit and the VPN peer or client must have the same NAT traversal setting (both selected or both cleared). When in doubt, enable NAT-traversal. See NAT traversal .
Keepalive Frequency If you enabled NAT traversal, enter a keepalive frequency setting. The value represents an interval from 0 to 900 seconds where the connection will be maintained with no activity. For additional security this value must be as low as possible. See NAT keepalive frequency .
Dead Peer Detection Enable this option to reestablish VPN tunnels on idle connections and clean up dead IKE peers if required. This feature minimizes the traffic required to check if a VPN peer is available or unavailable (dead). See Dead Peer Detection.

NAT traversal

Network Address Translation (NAT) is a way to convert private IP addresses to publicly routable Internet addresses and vise versa. When an IP packet passes through a NAT device, the source or destination address in the IP header is modified. FortiGate units support NAT version 1 (encapsulate on port 500 with non-IKE marker), version 3 (encapsulate on port 4500 with non-ESP marker), and compatible versions.

NAT cannot be performed on IPsec packets in ESP tunnel mode because the packets do not contain a port number. As a result, the packets cannot be demultiplexed. To work around this, the FortiGate unit provides a way to protect IPsec packet headers from NAT modifications. When the Nat-traversal option is enabled, outbound encrypted packets are wrapped inside a UDP IP header that contains a port number. This extra encapsulation allows NAT devices to change the port number without modifying the IPsec packet directly.

To provide the extra layer of encapsulation on IPsec packets, the Nat-traversal option must be enabled whenever a NAT device exists between two FortiGate VPN peers or a FortiGate unit and a dialup client such as FortiClient. On the receiving end, the FortiGate unit or FortiClient removes the extra layer of encapsulation before decrypting the packet.

Additionally, you can force IPsec to use NAT traversal. If NAT is set to Forced, the FortiGate will use a port value of zero when constructing the NAT discovery hash for the peer. This causes the peer to think it is behind a NAT device, and it will use UDP encapsulation for IPsec, even if no NAT is present. This approach maintains interoperability with any IPsec implementation that supports the NAT-T RFC.

NAT keepalive frequency

When a NAT device performs network address translation on a flow of packets, the NAT device determines how long the new address will remain valid if the flow of traffic stops (for example, the connected VPN peer may be idle). The device may reclaim and reuse a NAT address when a connection remains idle for too long.

To work around this, when you enable NAT traversal specify how often the FortiGate unit sends periodic keepalive packets through the NAT device in order to ensure that the NAT address mapping does not change during the lifetime of a session. To be effective, the keepalive interval must be smaller than the session lifetime value used by the NAT device.

The keepalive packet is a 138-byte ISAKMP exchange.

Dead Peer Detection

Sometimes, due to routing issues or other difficulties, the communication link between a FortiGate unit and a VPN peer or client may go down. Packets could be lost if the connection is left to time out on its own. The FortiGate unit provides a mechanism called Dead Peer Detection (DPD), sometimes referred to as gateway detection or ping server, to prevent this situation and reestablish IKE negotiations automatically before a connection times out: the active Phase 1 security associations are caught and renegotiated (rekeyed) before the Phase 1 encryption key expires.

By default, Dead Peer Detection sends probe messages every five seconds by default (see dpd-retryinterval in the FortiGate CLI Reference). If you are experiencing high network traffic, you can experiment with increasing the ping interval. However longer intervals will require more traffic to detect dead peers which will result in more traffic.

In the web-based manager, the Dead Peer Detection option can be enabled when you define advanced Phase 1 options. The config vpn ipsec phase1 CLI command supports additional options for specifying a retry count and a retry interval.

For more information about these commands and the related config router gwdetect CLI command, see the FortiGate CLI Reference.

For example, enter the following CLI commands to configure dead peer detection on the existing IPsec Phase 1 configuration called test to use 15 second intervals and to wait for 3 missed attempts before declaring the peer dead and taking action.

config vpn ipsec phase1-interface

edit <value>

set dpd [disable | on-idle | on-demand]

set dpd-retryinveral 15

set dpd-retrycount 3



note icon

The default for vpn ipsec phase1 dpd is on-idle when the type is dynamic to encourage dialup server configurations to more pro-actively delete tunnels if the peer goes away.

DPD scalability

On a dial-up server, if a multitude of VPN connections are idle, the increased DPD exchange could negatively impact the performance/load of the daemon. For this reason, an option is available in the CLI to send DPD passively in a mode called "on-demand".

note icon
  • When there is no traffic and the last DPD-ACK had been received, IKE will not send DPDs periodically.
  • IKE will only send out DPDs if there are outgoing packets to send but no inbound packets had since been received.

Set DPD to on-demand to trigger DPD when IPsec traffic is sent but no reply is received from the peer.

config vpn ipsec phase1-interface

edit <value>

set dpd [disable | on-idle | on-demand]



Certificate key size control

Proxy will choose the same SSL key size as the HTTPS server. If the key size from the server is 512, the proxy will choose 1024. If the key size is bigger than 1024, the proxy will choose 2048.

As a result, the firewall ssl-ssh-profile commands certname-rsa, certname-dsa, and certname-ecdsa have been replaced with more specific key size control commands under vpn certificate setting.

CLI syntax

config vpn certificate setting

set certname-rsa1024 <name>

set certname-rsa2048 <name>

set certname-dsa1024 <name>

set certname-dsa2048 <name>

set certname-ecdsa256 <name>

set certname-ecdsa384 <name>


Quantum resistant IKEv2 SA negotiation

An IKEv2 extension is available that changes the key generation mechanism to include a Post-quantum Pre-shared Key.

The addition of PPK in the calculation means that even if a quantum computer can break the Diffie-Hellman calculation to derive the DH-generated secret key, the inclusion of the PPK in the key generation algorithm means that the attacker is still unable to derive the keys used to authenticate the IKE SA negotiation (and so cannot impersonate either party in the negotiation) nor the keys used in negotiating an IPsec SA (or IKE SA).


config vpn ipsec phase1-interface

edit <name>

set ike-version 2

set type dynamic

set ppk {disable | allow | require}

set ppk-secret <ASCII string or hexadecimal encoded with a leading 0x>

set ppk-identity <string>



config user local

edit <name>

set type password

set ppk-secret <ASCII string or hexadecimal encoded with a leading 0x>




For troubleshooting, diagnose vpn ike ga list can indicate whether PPK was negotiated.

  • The 'PPK' at the gateway level indicates whether PPK was negotiated during the initial IKE SA negotiation.
  • The 'PPK' at the IKE SA level indicates whether PPK was negotiated on this IKE SA.
  • The 'child' at the IKE SA level indicates whether the IKE SA is an initial IKE SA or whether it is a child IKE SA. The above has 'child: no' and so it is initial IKE SA.