Failover scenario 5: Primary heartbeat link fails
If the primary heartbeat link fails, such as when the cable becomes accidentally disconnected, and if you have not configured a secondary heartbeat link, the FortiMail units in the HA group cannot verify that other units are operating and assume that the other has failed. As a result, the secondary unit (S2) changes to operating as a primary unit, and both FortiMail units are acting as primary units.
Two primary units connected to the same network may cause address conflicts on your network because matching interfaces will have the same IP addresses. Additionally, because the heartbeat link is interrupted, the FortiMail units in the HA group cannot synchronize configuration changes or mail data changes.
Even after reconnecting the heartbeat link, both units will continue operating as primary units. To return the HA group to normal operation, you must connect to the web‑based manager of S2 to restore its effective HA operating mode to slave (secondary unit).
1. The FortiMail HA group is operating normally.
2. The heartbeat link Ethernet cable is accidently disconnected.
3. S2’s HA heartbeat test detects that the primary unit has failed.
How soon this happens depends on the HA daemon configuration of S2.
4. The effective HA operating mode of S2 changes to master.
5. S2 sends an alert email similar to the following, indicating that S2 has determined that P1 has failed and that S2 is switching its effective HA operating mode to master.
This is the HA machine at 172.16.5.11.
The following event has occurred
‘MASTER heartbeat disappeared’
The state changed from ‘SLAVE’ to ‘MASTER’
6. S2 records event log messages (among others) indicating that S2 has determined that P1 has failed and that S2 is switching its effective HA operating mode to master.
Recovering from a heartbeat link failure
Because the hardware failure is not permanent (that is, the failure of the heartbeat link was caused by a disconnected cable, not a failed port on one of the FortiMail units), you may want to return both FortiMail units to operating in their configured modes when rejoining the failed primary unit to the HA group.
To return to normal operation after the heartbeat link fails
1. Reconnect the primary heartbeat interface by reconnecting the heartbeat link Ethernet cable.
Even though the effective HA operating mode of S2 is master, S2 continues to attempt to find the other primary unit. When the heartbeat link is reconnected, S2 finds P1 and determines that P1 is also operating as a primary unit. So S2 sends a heartbeat signal to notify P1 to stop operating as a primary unit. The effective HA operating mode of P1 changes to off.
2. P1 sends an alert email similar to the following, indicating that P1 has stopped operating as the primary unit.
This is the HA machine at 172.16.5.10
The following event has occurred
'SLAVE asks us to switch roles (user requested takeover)'
The state changed from 'MASTER' to 'OFF'
3. P1 records event log messages (among others) indicating that P1 is switching to off mode.
The configured HA mode of operation of P1 is master and the effective HA operating mode of P1 is off.
The configured HA mode of operation of S2 is slave and the effective HA operating mode of S2 is master.
P1 synchronizes the content of its MTA queue directories to S2. Email in these directories can now be delivered by S2.
4. Connect to the web‑based manager of P1, go to System > High Availability > Status.
5. Check for synchronization messages.
Do not proceed to the next step until P1 has synchronized with S2.
6. Connect to the web‑based manager of S2, go to System > High Availability > Status and select click HERE to restore configured operating mode.
The HA group should return to normal operation. P1 records the event log message (among others) indicating that S2 asked P1 to return to operating as the primary unit.
7. P1 and S2 synchronize their MTA queue directories. All email in these directories can now be delivered by P1.