HA heartbeat & synchronization

You can group multiple FortiWeb appliances together as a high availability (HA) cluster (see Configuring a high availability (HA) FortiWeb cluster). The heartbeat traffic indicates to other appliances in the HA cluster that the appliance is up and “alive.” Synchronization ensures that all appliances in the cluster remain ready to process traffic, even if you only change one of the appliances.

Heartbeat and synchronization traffic between cluster appliances occurs over the physical network ports selected in Heartbeat Interface. Heartbeat traffic uses multicast on port number 6065 and the IP address 239.0.0.1. Synchronization traffic uses TCP on port number 6010 and a reserved IP address. The HA IP addresses are hard-coded and cannot be configured.

Ensure that switches and routers that connect to heartbeat interfaces are configured to allow level2 frames. See Heartbeat packet Ethertypes.

Failover is triggered by any interruption to either the heartbeat or a port monitored network interface whose length of time exceeds your configured limits (Detection Interval x Heartbeat Lost Threshold). When the active (“main”) appliance becomes unresponsive, the standby appliance:

1. Notifies the network via ARP that the network interface IP addresses (including the IP address of the bridge, if any) are now associated with its virtual MAC addresses

2. Assumes the role of the active appliance and scans network traffic

To keep the standby appliance ready in case of a failover, HA pairs also use the heartbeat link to automatically synchronize most of their configuration. Synchronization includes:

core CLI-style configuration file (fwb_system.conf)
X.509 certificates, certificate request files (CSR), and private keys
HTTP error pages
FortiGuard IRIS Service database
FortiGuard Security Service files (attack signatures, predefined data types & suspicious URLs, known web crawlers & content scrapers, global white list, vulnerability scan signatures)
FortiGuard Antivirus signatures
Geography-to-IP database

and occurs immediately when an appliance joins the cluster, and thereafter every 30 seconds.

Although they are not automatically synchronized for performance reasons due to large size and frequent updates, you can manually force HA to synchronize. For instructions, see execute ha synchronize in the FortiWeb CLI Reference. For a list of settings and data that are not synchronized, see Data that is not synchronized by HA and Configuration settings that are not synchronized by HA.

If you do not want to configure HA (perhaps you have a separate network appliance implementing HA externally), you can still replicate the FortiWeb’s configuration on another FortiWeb appliance. For more information, see Configuring a high availability (HA) FortiWeb cluster

Data that is not synchronized by HA

In addition to HA configuration, some data is also not synchronized.

FortiWeb HTTP sessions — FortiWeb appliances can use cookies to add and track its own sessions, functionality that is not inherently provided by HTTP. For more information, see HTTP sessions & security. This state-tracking data corresponds in a 1:1 ratio to request volume, and therefore can change very rapidly. To minimize the performance impact on an HA cluster, this data is not synchronized.

Failover will not break web applications’ existing sessions, which do not reside on the FortiWeb, and are not the same thing as FortiWeb’s own HTTP sessions. The new active appliance will allow existing web application sessions to continue. For more information, see FortiWeb sessions vs. web application sessions.

FortiWeb sessions are used by some FortiWeb features. After a failover, these features may not work, or may work differently, for existing sessions. (New sessions are not affected.) See the description for each setting that uses session cookies. For more information, see Sessions & FortiWeb HA.

SSL/TLS sessions — HTTPS connections are stateful in that they must be able to remember states such as the security associations from the SSL/TLS handshake: the mutually supported cipher suite, the agreed parameters, and any certificates involved. Encryption and authentication in SSL/TLS cannot function without this. However, a new primary FortiWeb’s lack of existing HTTPS session information is gracefully handled by re-initializing the SSL/TLS session with the client.This does not impact to the encapsulated HTTP application, has only an initial failover impact during re-negotiation, and therefore is not synchronized.
Log messages — These describe events that happened on that specific appliance. After a failover, you may notice that there is a gap in the original active appliance’s log files that corresponds to the period of its down time. Log messages created during the time when the standby was acting as the active appliance (if you have configured local log storage) are stored there, on the original standby appliance. For more information on configuring local log storage, see Configuring logging.
Generated reports — Like the log messages that they are based upon, PDF, HTML, RTF, and plain text reports also describe events that happened on that specific appliance. As such, report settings are synchronized, but report output is not. For information about this feature, see Reports.
Auto-learning data — Auto-learning is a resource-intensive feature. To minimize the performance impact on an HA cluster, this data is not synchronized. For information about this feature, see Auto-learning.

Configuration settings that are not synchronized by HA

All configuration settings on the active appliance are synchronized to the standby appliance, except the following:

Setting	Explanation
Operation mode	You must set the operation mode of each HA group member before configuring HA. See Setting the operation mode.
Host name	The host name distinguishes each member of the FortiWeb HA cluster. See Changing the FortiWeb appliance’s host name.
Network interfaces (reverse proxy or offline protection mode only) or Bridge (true transparent proxy or transparent inspection mode only)	Only the FortiWeb appliance acting as the main appliance, actively scanning web traffic, is configured with IP addresses on its network interfaces (or bridge). The standby appliance only uses the configured IP addresses if a failover occurs, and the standby appliance therefore assumes the role of the main appliance. See Configuring the network interfaces or Configuring a bridge (V-zone). If you have configured a reserved management port for a cluster member, that configuration, including administrative access and other settings, is not synchronized.
RAID level	RAID settings are hardware-dependent and determined at boot time by looking at the drives (for software RAID) or the controller (hardware RAID), and are not stored in the system configuration. Therefore, they are not synchronized. See RAID level & disk statuses.
HA active status and priority	The HA configuration, which includes Device Priority, is not synchronized because this configuration must be different on the primary and secondary appliances.

How HA chooses the active appliance

An HA pair may or may not resume their active and standby roles when the failed appliance resumes responsiveness to the heartbeat.

Since the current active appliance will by definition have a greater uptime than a failed previous active appliance that has just returned online, assuming each has the same number of available ports, the current active appliance usually retains its status as the active appliance, unless Override is enabled. If Override is enabled, and if the Device Priority setting of the returning appliance is higher, it will be elected as the active appliance in the HA cluster.

If Override is disabled, HA considers (in order)

1. The most available ports

For example, if two FortiWeb appliances, FWB1 and FWB2, were configured to monitor two ports each, and FWB2 has just one port currently available according to Port Monitor, FWB1 would become the active appliance, regardless of uptime or priority. But if both had 2 available ports, this factor alone would not be able to determine which appliance should be active, and the HA cluster would proceed to the next consideration.

2. The highest uptime value

Uptime is reset to zero if an appliance fails, or the status of any monitored port (per Port Monitor) changes.

3. The smallest Device Priority number (that is, 0 has the highest priority)

4. The highest-sorting serial number

Serial numbers are sorted by comparing each character from left to right, where 9 and z are the greatest values, and result in highest placement in the sorted list.

If Override is enabled, HA considers (in order)

1. The most available ports

2. The smallest Device Priority number (that is, 0 has the highest priority)

3. The highest uptime value

4. The highest-sorting serial number

If the heartbeat link occurs through switches or routers, and the active appliance is very busy, it might require more time to establish a heartbeat link through which it can negotiate to elect the active appliance. You can configure the amount of time that a FortiWeb appliance will wait after it boots to establish this connection before assuming that the other appliance is unresponsive, and that it should become the active appliance. For details, see the boot-time <seconds_int> setting in the FortiWeb CLI Reference.

Heartbeat packet Ethertypes

Normal IP packets are 802.3 packets that have an Ethernet type (Ethertype) field value of 0x0800. Ethertype values other than 0x0800 are understood as level2 frames rather than IP packets.

By default, HA heartbeat packets use the following Ethertypes, which are hard-coded and cannot be configured:

Ethertype 0x8890 — For HA heartbeat packets that cluster members use to find other cluster member and to verify the status of other cluster members while the cluster is operating.
Ethertype 0x8893 — For HA sessions that synchronize the cluster configurations.

Because heartbeat packets are recognized as level2 frames, the switches and routers that connect to heartbeat interfaces require a configuration that allows them. If these network devices drop level2 frames, they prevent heartbeat traffic between the members of the cluster.

In some cases, if you connect and configure the heartbeat interfaces so that regular traffic flows but heartbeat traffic is not forwarded, you can change the configuration of the switch that connects the HA heartbeat interfaces to allow level2 frames with Ethertypes 0x8890 and 0x8893 to pass.

Open topic with navigation