FortiOS 5.6 Online Help Link FortiOS 5.4 Online Help Link FortiOS 5.2 Online Help Link FortiOS 5.0 Online Help Link
> Chapter 15 - High Availability > HA and failover protection > Cluster virtual MAC addresses

Home > Online Help

Cluster virtual MAC addresses

When a cluster is operating, the FGCP assigns virtual MAC addresses to each primary unit interface. HA uses virtual MAC addresses so that if a failover occurs, the new primary unit interfaces will have the same virtual MAC addresses and IP addresses as the failed primary unit. As a result, most network equipment would identify the new primary unit as the exact same device as the failed primary unit.

If the MAC addresses changed after a failover, the network would take longer to recover because all attached network devices would have to learn the new MAC addresses before they could communicate with the cluster.

If a cluster is operating in NAT/Route mode, the FGCP assigns a different virtual MAC address to each primary unit interface. VLAN subinterfaces are assigned the same virtual MAC address as the physical interface that the VLAN subinterface is added to. Redundant interfaces or 802.3ad aggregate interfaces are assigned the virtual MAC address of the first interface in the redundant or aggregate list.

If a cluster is operating in Transparent mode, the FGCP assigns a virtual MAC address for the primary unit management IP address. Since you can connect to the management IP address from any interface, all of the FortiGate interfaces appear to have the same virtual MAC address.

note icon A MAC address conflict can occur if two clusters are operating on the same network. See Diagnosing packet loss with two FortiGate HA clusters in the same broadcast domain for more information.
note icon Subordinate unit MAC addresses do not change. You can verify this by connecting to the subordinate unit CLI and using the get hardware interface nic command to display the MAC addresses of each FortiGate interface.
note icon The MAC address of a reserved management interface is not changed to a virtual MAC address. Instead the reserved management interface keeps its original MAC address.

When the new primary unit is selected after a failover, the primary unit sends gratuitous ARP packets to update the devices connected to the cluster interfaces (usually layer-2 switches) with the virtual MAC address. Gratuitous ARP packets configure connected network devices to associate the cluster virtual MAC addresses and cluster IP address with primary unit physical interfaces and with the layer-2 switch physical interfaces. This is sometimes called using gratuitous ARP packets (sometimes called GARP packets) to train the network. The gratuitous ARP packets sent from the primary unit are intended to make sure that the layer-2 switch forwarding databases (FDBs) are updated as quickly as possible.

Sending gratuitous ARP packets is not required for routers and hosts on the network because the new primary unit will have the same MAC and IP addresses as the failed primary unit. However, since the new primary unit interfaces are connected to different switch interfaces than the failed primary unit, many network switches will update their FDBs more quickly after a failover if the new primary unit sends gratuitous ARP packets.

Changing how the primary unit sends gratuitous ARP packets after a failover

When a failover occurs it is important that the devices connected to the primary unit update their FDBs as quickly as possible to reestablish traffic forwarding.

Depending on your network configuration, you may be able to change the number of gratuitous ARP packets and the time interval between ARP packets to reduce the cluster failover time.

You cannot disable sending gratuitous ARP packets, but you can use the following command to change the number of packets that are sent. For example, enter the following command to send 20 gratuitous ARP packets:

config system ha

set arps 20

end

You can use this command to configure the primary unit to send from 1 to 60 ARP packets. Usually you would not change the default setting of 5. In some cases, however, you might want to reduce the number of gratuitous ARP packets. For example, if your cluster has a large number of VLAN interfaces and virtual domains and because gratuitous ARP packets are broadcast, sending a higher number gratuitous ARP packets may generate a lot of network traffic. As long as the cluster still fails over successfully, you could reduce the number of gratuitous ARP packets that are sent to reduce the amount of traffic produced after a failover.

If failover is taking longer that expected, you may be able to reduce the failover time by increasing the number gratuitous ARP packets sent.

You can also use the following command to change the time interval in seconds between gratuitous ARP packets. For example, enter the following command to change the time between ARP packets to 3 seconds:

config system ha

set arps-interval 3

end

The time interval can be in the range of 1 to 20 seconds. The default is 8 seconds between gratuitous ARP packets. Normally you would not need to change the time interval. However, you could decrease the time to be able send more packets in less time if your cluster takes a long time to failover.

There may also be a number of reasons to set the interval higher. For example, if your cluster has a large number of VLAN interfaces and virtual domains and because gratuitous ARP packets are broadcast, sending gratuitous ARP packets may generate a lot of network traffic. As long as the cluster still fails over successfully you could increase the interval to reduce the amount of traffic produced after a failover.

For more information about gratuitous ARP packets see RFC 826 and RFC 3927.

Disabling gratuitous ARP packets after a failover

You can use the following command to turn off sending gratuitous ARP packets after a failover:

config system ha

set gratuitous-arps disable

end

Sending gratuitous ARP packets is turned on by default.

In most cases you would want to send gratuitous ARP packets because its a reliable way for the cluster to notify the network to send traffic to the new primary unit. However, in some cases, sending gratuitous ARP packets may be less optimal. For example, if you have a cluster of FortiGates in Transparent mode, after a failover the new primary unit will send gratuitous ARP packets to all of the addresses in its Forwarding Database (FDB). If the FDB has a large number of addresses it may take extra time to send all the packets and the sudden burst of traffic could disrupt the network.

If you choose to disable sending gratuitous ARP packets you must first enable the link-failed-signal setting. The cluster must have some way of informing attached network devices that a failover has occurred.

For more information about the link-failed-signal setting, see Updating MAC forwarding tables when a link failover occurs.

How the virtual MAC address is determined

The virtual MAC address is determined based on following formula:

00-09-0f-09-<group-id_hex>-(<vcluster_integer> + <idx>)

where

<group-id_hex> is the HA Group ID for the cluster converted to hexadecimal.The following table lists the virtual MAC address set for each group ID.

HA group ID in integer and hexadecimal format
Integer Group ID Hexadecimal Group ID
0 00
1 01
2 02
3 03
4 04
... ...
10 0a
11 0b
... ...
63 3f
... ...
255 ff

<vcluster_integer> is 0 for virtual cluster 1 and 20 for virtual cluster 2. If virtual domains are not enabled, HA sets the virtual cluster to 1 and by default all interfaces are in the root virtual domain. Including virtual cluster and virtual domain factors in the virtual MAC address formula means that the same formula can be used whether or not virtual domains and virtual clustering is enabled.

<idx> is the index number of the interface. Interfaces are numbered from 0 to x (where x is the number of interfaces). Interfaces are numbered according to their has map order. See Interface index and display order. The first interface has an index of 0. The second interface in the list has an index of 1 and so on.

note icon Only the <idx> part of the virtual MAC address is different for each interface. The <vcluster_integer> would be different for different interfaces if multiple VDOMs have been added.
note icon Between FortiOS releases interface indexing may change so the virtual MAC addresses assigned to individual FortiGate interfaces may also change.

Example virtual MAC addresses

An HA cluster with HA group ID unchanged (default=0) and virtual domains not enabled would have the following virtual MAC addresses for interfaces port1 to port12:

  • port1 virtual MAC: 00-09-0f-09-00-00
  • port10 virtual MAC: 00-09-0f-09-00-01
  • port2 virtual MAC: 00-09-0f-09-00-02
  • port3 virtual MAC: 00-09-0f-09-00-03
  • port4 virtual MAC: 00-09-0f-09-00-04
  • port5 virtual MAC: 00-09-0f-09-00-05
  • port6 virtual MAC: 00-09-0f-09-00-06
  • port7 virtual MAC: 00-09-0f-09-00-07
  • port8 virtual MAC: 00-09-0f-09-00-08
  • port9 virtual MAC: 00-09-0f-09-00-
  • port11 virtual MAC: 00-09-0f-09-00-0a
  • port12 virtual MAC: 00-09-0f-09-00-0b

If the group ID is changed to 34 these virtual MAC addresses change to:

  • port1 virtual MAC: 00-09-0f-09-22-00
  • port3 virtual MAC: 00-09-0f-09-22-03
  • port4 virtual MAC: 00-09-0f-09-22-04
  • port5 virtual MAC: 00-09-0f-09-22-05
  • port6 virtual MAC: 00-09-0f-09-22-06
  • port7 virtual MAC: 00-09-0f-09-22-07
  • port8 virtual MAC: 00-09-0f-09-22-08
  • port9 virtual MAC: 00-09-0f-09-22-
  • port11 virtual MAC: 00-09-0f-09-22-0a
  • port12 virtual MAC: 00-09-0f-09-22-0b
  • port10 virtual MAC: 00-09-0f-09-22-01
  • port2 virtual MAC: 00-09-0f-09-22-02

A cluster with virtual domains enabled where the HA group ID has been changed to 23, port5 and port 6 are in the root virtual domain (which is in virtual cluster1), and port7 and port8 are in the vdom_1 virtual domain (which is in virtual cluster 2) would have the following virtual MAC addresses:

  • port5 interface virtual MAC: 00-09-0f-09-23-05
  • port6 interface virtual MAC: 00-09-0f-09-23-06
  • port7 interface virtual MAC: 00-09-0f-09-23-27
  • port8 interface virtual MAC: 00-09-0f-09-23-28

Displaying the virtual MAC address

Every FortiGate physical interface has two MAC addresses: the current hardware address and the permanent hardware address. The permanent hardware address cannot be changed, it is the actual MAC address of the interface hardware. The current hardware address can be changed. The current hardware address is the address seen by the network. For a FortiGate not operating in HA, you can use the following command to change the current hardware address of the port1 interface:

config system interface

edit port1

set macaddr <mac_address>

end

end

For an operating cluster, the current hardware address of each cluster unit interface is changed to the HA virtual MAC address by the FGCP. The macaddr option is not available for a functioning cluster. You cannot change an interface MAC address and you cannot view MAC addresses from the system interface CLI command.

You can use the get hardware nic <interface_name_str> command to display both MAC addresses for any FortiGate interface. This command displays hardware information for the specified interface. Depending on their hardware configuration, this command may display different information for different interfaces. You can use this command to display the current hardware address as Current_HWaddr and the permanent hardware address as Permanent_HWaddr. For some interfaces the current hardware address is displayed as MAC. The command displays a great deal of information about the interface so you may have to scroll the output to find the hardware addresses.

note icon You can also use the diagnose hardware deviceinfo nic <interface_str> command to display both MAC addresses for any FortiGate interface.

Before HA configuration the current and permanent hardware addresses are the same. For example for one of the units in Cluster_1:

FGT60B3907503171 # get hardware nic internal

.

.

.

MAC: 02:09:0f:78:18:c9

Permanent_HWaddr: 02:09:0f:78:18:c9

.

.

.

During HA operation the current hardware address becomes the HA virtual MAC address, for example for the units in Cluster_1:

FGT60B3907503171 # get hardware nic internal

.

.

.

MAC: 00:09:0f:09:00:02

Permanent_HWaddr: 02:09:0f:78:18:c9

.

.

.

The following command output for Cluster_2 shows the same current hardware address for port1 as for the internal interface of Cluster_2, indicating a MAC address conflict.

FG300A2904500238 # get hardware nic port1

.

.

.

MAC: 00:09:0f:09:00:02

Permanent_HWaddr: 00:09:0F:85:40:FD

.

.

.

Diagnosing packet loss with two FortiGate HA clusters in the same broadcast domain

A network may experience packet loss when two FortiGate HA clusters have been deployed in the same broadcast domain. Deploying two HA clusters in the same broadcast domain can result in packet loss because of MAC address conflicts. The packet loss can be diagnosed by pinging from one cluster to the other or by pinging both of the clusters from a device within the broadcast domain. You can resolve the MAC address conflict by changing the HA Group ID configuration of the two clusters. The HA Group ID is sometimes also called the Cluster ID.

This section describes a topology that can result in packet loss, how to determine if packets are being lost, and how to correct the problem by changing the HA Group ID.

note icon Packet loss on a network can also be caused by IP address conflicts. Finding and fixing IP address conflicts can be difficult. However, if you are experiencing packet loss and your network contains two FortiGate HA clusters you can use the information in this article to eliminate one possible source of packet loss.

Changing the HA group ID to avoid MAC address conflicts

Change the Group ID to change the virtual MAC address of all cluster interfaces. You can change the Group ID from the FortiGate CLI using the following command:

config system ha

set group-id <id_integer>

end

Example topology

The topology below shows two clusters. The Cluster_1 internal interfaces and the Cluster_2 port 1 interfaces are both connected to the same broadcast domain. In this topology the broadcast domain could be an internal network. Both clusters could also be connected to the Internet or to different networks.

Example HA topology with possible MAC address conflicts

Ping testing for packet loss

If the network is experiencing packet loss, it is possible that you will not notice a problem unless you are constantly pinging both HA clusters. During normal operation of the network you also might not notice packet loss because the loss rate may not be severe enough to timeout TCP sessions. Also many common types if TCP traffic, such as web browsing, may not be greatly affected by packet loss. However, packet loss can have a significant effect on real time protocols that deliver audio and video data.

To test for packet loss you can set up two constant ping sessions, one to each cluster. If packet loss is occurring the two ping sessions should show alternating replies and timeouts from each cluster.

Cluster_1 Cluster_2
reply timeout
reply timeout
reply timeout
timeout reply
timeout reply
reply timeout
reply timeout
timeout reply
timeout reply
timeout reply
timeout reply

Viewing MAC address conflicts on attached switches

If two HA clusters with the same virtual MAC address are connected to the same broadcast domain (L2 switch or hub), the MAC address will conflict and bounce between the two clusters. This example Cisco switch MAC address table shows the MAC address flapping between different interfaces (1/0/1 and 1/0/4).

1 0009.0f09.0002 DYNAMIC Gi1/0/1

1 0009.0f09.0002 DYNAMIC Gi1/0/4