FortiOS 5.6 Online Help Link FortiOS 5.4 Online Help Link FortiOS 5.2 Online Help Link FortiOS 5.0 Online Help Link

Home > Online Help

> Chapter 6 - FortiGate-7000 > High Availability

High Availability

FortiGate-7000 supports active-passive FortiGate Clustering Protocol (FGCP) high availability between two identical FortiGate-7000 chassis. With active-passive FortiGate-7000 HA, you create redundant network connections to two identical FortiGate-7000s and add redundant HA heartbeat connections. Then you configure the FIM interface modules for HA. A cluster forms and a primary chassis is selected.

Example FortiGate-7040 inter-chassis HA

All traffic is processed by the primary (or master) chassis. The backup chassis operates in hot standby mode. The configuration, active sessions, routing information and so on is synchronized to the backup chassis. If the primary chassis fails, traffic automatically fails over to the backup chassis.

The primary chassis is selected based on a number of criteria including the configured priority, the bandwidth, the number of FIM interface failures, and the number of FPM or FIM modules that have failed. As part of the HA configuration you assign each chassis a chassis ID and you can set the priority of each FIM interface module and configure module failure tolerances and the link failure thresholds.

Before you begin configuring HA

Before you begin, the chassis should be running the same FortiOS firmware version and interfaces should not be configured to get their addresses from DHCP or PPPoE. Register and apply licenses to the each FortiGate-7000 before setting up the HA cluster. This includes licensing for FortiCare, IPS, AntiVirus, Web Filtering, Mobile Malware, FortiClient, FortiCloud, and additional virtual domains (VDOMs). Both FortiGate-7000s in the cluster must have the same level of licensing for FortiGuard, FortiCloud, FortiClient, and VDOMs. FortiToken licenses can be added at any time because they are synchronized to all cluster members.

If required, you should configure split ports on the FIMs on both chassis before configuring HA. For example, to split the C1, C2, and C4 interfaces of an FIM-7910E in slot 1, enter the following command:

config system global

set split-port 1-C1 2-C1 2-C4

end

After configuring split ports the chassis reboots and the configuration is synchronized.

On each chassis, make sure configurations of the modules are synchronized before starting to configure HA. You can use the following command to verify that the configurations of all of the modules are synchronized:

diagnose sys confsync chsum | grep all

all: c0 68 d2 67 e1 23 d9 3a 10 50 45 c5 50 f1 e6 8e

all: c0 68 d2 67 e1 23 d9 3a 10 50 45 c5 50 f1 e6 8e

all: c0 68 d2 67 e1 23 d9 3a 10 50 45 c5 50 f1 e6 8e

all: c0 68 d2 67 e1 23 d9 3a 10 50 45 c5 50 f1 e6 8e

all: c0 68 d2 67 e1 23 d9 3a 10 50 45 c5 50 f1 e6 8e

all: c0 68 d2 67 e1 23 d9 3a 10 50 45 c5 50 f1 e6 8e

all: c0 68 d2 67 e1 23 d9 3a 10 50 45 c5 50 f1 e6 8e

all: c0 68 d2 67 e1 23 d9 3a 10 50 45 c5 50 f1 e6 8e

If the modules are synchronized, the checksums displayed should all be the same.

You can also use the following command to list the modules that are synchronized. The example output shows all four FIM modules have been configured for HA and added to the cluster.

diagnose sys configsync status | grep in_sync

Master, uptime=692224.19, priority=1, slot_1d=1:1, idx=0, flag=0x0, in_sync=1

Slave, uptime=676789.70, priority=2, slot_1d=1:2, idx=1, flag=0x0, in_sync=1

Slave, uptime=692222.01, priority=17, slot_1d=1:4, idx=2, flag=0x64, in_sync=1

Slave, uptime=692271.30, priority=16, slot_1d=1:3, idx=3, flag=0x64, in_sync=1

Connect the M1 and M2 interfaces for HA heartbeat communication

HA heartbeat communication between chassis happens over the 10Gbit M1 and M2 interfaces of the FIM modules in each chassis. To set up HA heartbeat connections:

  • Connect the M1 interfaces of all FIM modules together using a switch.
  • Connect the M2 interfaces of all FIM modules together using another switch.

Heartbeat packets are VLAN packets with VLAN ID 999 and ethertype 9890. You can use the following command to change the HA heartbeat packet VLAN ID and ethertype values if required for your switches. You must change these settings on each of the FIM interface modules. The M1 and M2 interface heartbeat packets use the same VLAN IDs and ethertypes.

config system ha

set hbdev-vlan-id <vlan>

set ha-eth-type <eth-type>

end

The switches must also support Jumbo frames. Using separate switches for M1 and M2 is recommended for redundancy. If you use the same switch for both, separate the M1 and M2 traffic on the switch and q-in-q must be enabled. It is also recommended that these switches be dedicated to HA heartbeat communication and not used for other traffic.

Sample switch configuration for a Cisco Catalyst switch. This configuration enables jumbo frames, configures the interface speeds, configures the switch to allow vlan 999, and enables trunk mode:

system mtu jumbo 9000

##interface config

interface TenGigabitEthernet1/0/5

description Chassis1 FIM1 M1

switchport trunk allowed vlan 999

switchport mode trunk

If you are using one switch for both M1 and M2 connections, the configuration would be the same except you would add q-in-q support and two different VLANs, one for M1 traffic and one for M2 traffic.

For the M1 connections:

interface Ethernet1/5

description QinQ Test

switchportmode dot1q-tunnel

switchport access vlan 888

spanning-tree port type edge

 

For the M2 connections:

interface Ethernet1/5

description QinQ Test

switchport mode dot1q-tunnel

switchport access vlan 880

spanning-tree port type edge

 

HA packets must have the configured VLAN tag (default 999). If the switch removes or changes this tag, HA heartbeat communication will not work and the cluster will form a split brain configuration. In effect two clusters will form, one in each chassis, and network traffic will be disrupted.

HA configuration

Use the following steps to setup the configuration for HA between two chassis (chassis 1 and chassis 2). These steps are written for a set of two FortiGate-7040E or 7060Es. The steps are similar for the FortiGate-7030E except that each FortiGate-7030E only has one FIM interface module.

Each FIM interface module has to be configured for HA separately. The HA configuration is not synchronized among FIMs. You can begin by setting up chassis 1 and setting up HA on both of the FIM interfaces modules in it. Then do the same for chassis 2.

Each of the FortiGate-7000s is assigned a chassis ID (1 and 2). These numbers just allow you to identify the chassis and do not influence primary unit selection.

Setting up HA on the FIM interface modules in the first FortiGate-7000 (chassis 1)

  1. Log into the CLI of the FIM interface module in slot 1 (FM01) and enter the following command:

config system ha

set mode a-p

set password <password>

set group-id <id>

set chassis-id 1

end

This adds basic HA settings to this FIM interface module.

  1. Repeat this configuration on the FIM interface module in slot 2 (FIM02).

config system ha

set mode a-p

set password <password>

set group-id <id>

set chassis-id 1

end

  1. From either FIM interface module, enter the following command to confirm that the FortiGate-7000 is in HA mode:

diagnose sys ha status

The password and group-id are unique for each HA cluster and must be the same on all FIM modules. If a cluster does not form, one of the first things to check are groupd-id and re-enter the password on both FIM interface modules.

Configure HA on the FIM interface modules in the second FortiGate-7000 (chassis 2)

  1. Repeat the same HA configuration settings on the FIM interfaces modules in the second chassis except set the chassis ID to 2.

config system ha

set mode a-p

set password <password>

set group-id <id>

set chassis-id 2

end

  1. From any FIM interface module, enter the following command to confirm that the cluster has formed and all of the FIM modules have been added to it:

diagnose sys ha status

The cluster has now formed and you can add the configuration and connect network equipment and start operating the cluster. You can also modify the HA configuration depending on your requirements.

Primary unit selection and failover criteria

When the cluster is operating normally, the FIM interface module with the highest serial number becomes the primary unit and as a result the chassis with this FIM interface module becomes the primary chassis. The other chassis operates in a hot standby mode as the backup chassis.

You can select an FIM module to become the primary unit by setting its HA priority higher than all of the other FIM interface modules. Enter the following command to set the priority:

config system ha

set priority <number>

end

You can also enable override to make sure the cluster always renegotiates and selects the FIM interface module with the highest priority as the primary unit.

config system ha

set override enable

end

A failover occurs if the primary chassis encounters the following problems:

  • Bandwidth reduction
  • Interface failure
  • Module failure

If one of these events occurs the cluster renegotiates and the FIM with the highest bandwidth, the most number of operating interfaces, and the most operating modules becomes the primary unit.

HA management configuration

In HA mode, you should connect the MGMT1 interfaces of each of the FIM interface modules to the same switch. Then when you browse to the system management IP address you connect to the primary FIM interface module. To verify which module you have logged into, the GUI header banner or CLI prompt shows the chassis ID and hostname of the module you are logged into plus the slot address in the format <hostname> (<id>-<slot address>).

Only the primary FIM interface module responds to management connections using the system management IP address. If a failover occurs you can connect to the new primary FIM interface module using the same system management IP address.

If the MGMT1 interface of the primary FIM interface module fails, the other FIM interface module in that chassis becomes the primary unit. If the MGMT1 interfaces of both of these FIM interface modules fails the backup chassis becomes the new primary chassis and the FIM interface module with the highest serial number becomes the primary FIM interface module.

You can also set up a more redundant management interface configuration by also connecting the MGMT2 interfaces of each FIM interface module to a different switch. In this configuration, if there are no failures when you connect to the system management IP address you connect to the MGMT1 interface of the primary FIM interface module. If the MGMT1 interface of the primary interface module fails, a failover will not occur as long as the MGMT2 interface of the primary FIM interface module is still connected. In this case, when you connect to the system management IP address you connect to the MGMT2 interface of the primary FIM interface module.

If both MGMT1 and MGMT2 of the primary FIM interface module fail, the other FIM interface module in that chassis becomes the primary FIM interface module. And so on.

Managing individual modules in HA mode

In some cases you may want to connect to an individual FIM or FPM module in a specific chassis. For example, you may want to view the traffic being processed by the FPM module in slot 3 of chassis 2. You can connect to the GUI or CLI of individual modules in the chassis using the system management IP address with a special port number.

For example, if the system management IP address is 1.1.1.1 you can browse to https://1.1.1.1:44323 to connect to the FPM module in chassis 2 slot 3. The special port number (in this case 44323) is a combination of the service port, chassis ID, and slot number. The following table lists the special ports for common admin protocols:

FortiGate-7000 HA special administration port numbers
Chassis and Slot Number Slot Address HTTP (80) HTTPS (443) Telnet (23) SSH (22) SNMP (161)
Ch1 slot 5 FPM05 8005 44305 2305 2205 16105
Ch1 slot 3 FPM03 8005 44303 2303 2203 16103
Ch1 slot 1 FIM01 8003 44301 2301 2201 16101
Ch1 slot 2 FIM02 8002 44302 2302 2202 16102
Ch1 slot 4 FPM04 8004 44304 2304 2204 16104
Ch1 slot 6 FPM06 8006 44306 2306 2206 16106
Ch2 slot 5 FPM05 8005 44325 2325 2225 16125
Ch2 slot 3 FPM03 8005 44323 2323 2223 16123
Ch2 slot 1 FIM01 8003 44321 2321 2221 16121
Ch2 slot 2 FIM02 8002 44322 2322 2222 16122
Ch2 slot 4 FPM04 8004 44324 2324 2224 16124
Ch2 slot 6 FPM06 8006 44326 2326 2226 16126

For example:

  • To connect to the GUI of the FPM module in chassis 1 slot 3 using HTTPS you would browse to https://1.1.1.1:44313.
  • To send an SNMP query to the FPM module in chassis 2 slot 6 use the port number 16126.

Firmware upgrade

All of the modules in a FortiGate-7000 HA cluster run the same firmware image. You upgrade the firmware from the GUI or CLI by logging into the primary FIM interface module using the system management IP address and uploading the firmware image.

During the upgrade process, the firmware of all of the modules in the cluster is upgraded in one step. The primary FIM interface module uploads the new firmware image to all of the modules in the cluster. All modules, including the primary FIM interface module all upgrade their firmware, reboot and rejoin the cluster. This process temporarily interrupts network traffic.

You can use the following command to enable uninterruptable firmware upgrades:

config system ha

set uninterruptible-upgrade enable

end

When enabled, the primary FIM interface module uploads firmware to all modules, but in this case, the modules in the backup chassis install their new firmware and reboot and rejoin the cluster and resynchronize.

Then all traffic fails over to the backup chassis which becomes the new primary chassis. Then the modules in the new backup chassis upgrade their firmware and rejoin the cluster. Unless override is enabled, the new primary chassis continues to operate as the primary chassis.