• All Files

Home > Online Help

Advanced static routing example: ECMP failover and load balancing

Equal Cost Multi-Path (ECMP) load balancing and failover are methods that extend basic static routing. They allow you to use your network bandwidth more effectively and with less down time than if you used basic static routing alone.

The concepts in this section include:

• Equal-Cost Multi-Path (ECMP)
• Configuring interface status detection for gateway load balancing
• Configuring spillover or usage-based ECMP
• Configuring weighted static route load balancing

Equal-Cost Multi-Path (ECMP)

FortiOS uses equal-cost multi-path (ECMP) to distribute traffic to the same destination such as the Internet or another network. Using ECMP you can add multiple routes to the destination and give each of those routes the same distance and priority.

If multiple routes to the same destination have the same priority but different distances, the route with the lowest distance is used. If multiple routes to the same destination have the same distance but different priorities, the route with the lowest priority is used. Distance takes precedence over priority. If multiple routes to the same destination have different distances and different priorities, the route with the lowest distance is always used even if it has the highest priority.

If more than one ECMP route is available, you can configure how the FortiGate unit selects the route to be used for a communication session. If only one ECMP route is available (for example, because an interface cannot process traffic because interface status detection does not receive a reply from the configured server) then all traffic uses this route.

Previous versions of FortiOS provided source IP-based load balancing for ECMP routes, but now FortiOS includes three configuration options for ECMP route failover and load balancing:

Source IP based (also called source IP based)	The FortiGate unit load balances sessions among ECMP routes based on the source IP address of the sessions to be load balanced. This is the default load balancing method. No configuration changes are required to support source IP load balancing.
Weighted Load Balance (also called weight-based)	The FortiGate unit load balances sessions among ECMP routes based on weights added to ECMP routes. More traffic is directed to routes with higher weights. After selecting weight-based you must add weights to static routes.
Spillover (also called usage-based)	The FortiGate unit distributes sessions among ECMP routes based on how busy the FortiGate interfaces added to the routes are. After selecting spill-over you add route Spillover Thresholds to interfaces added to ECMP routes. The FortiGate unit sends all ECMP-routed sessions to the lowest numbered interface until the bandwidth being processed by this interface reaches its spillover threshold. The FortiGate unit then spills additional sessions over to the next lowest numbered interface. The Spillover Thresholds range is 0-2097000 KBps.
Source-Destination IP based	The FortiGate unit load balances sessions among ECMP routes based on both the source and destination IP addresses of the sessions to be load balanced. This is required particularly for L3 link aggregation hashing.

You can configure only one of these ECMP route failover and load balancing methods in a single VDOM. If your FortiGate unit is configured for multiple VDOM operation, each VDOM can have its own ECMP route failover and load balancing configuration.

To configure the ECMP load balancing method from the web‑based manager

1. Go to Router > Static > Settings.
2. Set ECMP Load Balancing Method to Source IP based, Weighted Load Balance, or Spillover.

To configure the ECMP load balancing method from the CLI

For example, to set the load balancing method to usage-based, enter the following:

config system settings

set v4-ecmp-mode usage‑based

end

ECMP routing of simultaneous sessions to the same destination IP address

When the FortiGate unit selects an ECMP route for a session, a route cache is created that matches the route with the destination IP address of the session. All new sessions to the same destination IP address from the same source IP address use the same route until the route is flushed from the cache. Routes are flushed from the cache after a period of time when no new sessions to the destination IP address are received.

The route cache improves FortiGate unit routing performance by reducing how often the FortiGate unit looks up routes in the routing table.

If the FortiGate unit receives a large number of sessions with the same destination IP address, because all of these sessions will be processed by the same route, it may appear that sessions are not distributed according to the ECMP route failover and load balancing configuration.

Configuring interface status detection for gateway load balancing

Interface status detection is used for ECMP route failover and load balancing. Interface status detection consists of the unit confirming that packets sent from an interface result in a response from a server. You can use up to three different protocols to confirm that an interface can connect to the server. Usually the server is the next-hop router that leads to an external network or the Internet. Interface status detection sends a packet using the configured protocols. If a response is received from the server, the unit assumes the interface can connect to the network. If a response is not received, the unit assumes that the interface cannot connect to the network.

Since it is possible that a response may not be received, even if the server and the network are operating normally, the dead gateway detection configuration controls the time interval between testing the connection to the server and the number of times the test can fail before the unit assumes that the interface cannot connect to the server.

As long as the unit receives responses for at least one of the protocols that you select, the unit assumes the server is operating and can forward packets. Responding to more than one protocol does not enhance the status of the server or interface.

To configure gateway failover detection for an interface - web-based manager

1. Go to Router > Static > Settings.
2. Under Link Health Monitor, select Create New.
3. Enter the following information:

Name	Give the monitor a name
Interface	Select the interface to test.
Gateway	Enter the IP address of the gateway.
Probe Type	Select the method of probe type, either Ping or HTTP.
Server	Address of Server to test. The address can be an IP address or FQDN. In some builds of the firmware if an invalid address is entered an error message comes up stating “Please enter an IP, IP range, FQDN, or comma separated list of IPs and/or FQDNs.” This is incorrect. Ranges cannot be used.
Probe Interval(s)	Enter the interval between pings, in seconds.
Failure Threshold	Enter the number of times the test can fail before the unit assumes that the interface cannot connect to the server.
Recovery Threshold	Configure the threshold for ECMP recovery, ranging from 1 to 10.
HA Priority	Set the HA priority, if configuring an HA cluster.

4. Select OK.

To configure gateway failover detection for an interface - CLI

config system link-monitor

edit "test"

set srcintf "internal4"

set server "8.8.8.8"

set update-cascade-interface disable

end

Configuring spillover or usage-based ECMP

Spill-over or usage-based ECMP routes new sessions to interfaces that have not reached a configured bandwidth limit (called the Spillover Threshold or a route-spillover threshold). To configure spill-over or usage-based ECMP routing, you enable spill-over ECMP, add ECMP routes, and add a Spillover Threshold to the interfaces used by the ECMP routes. Set the Spillover Thresholds to limit the amount of bandwidth processed by each interface. The range is 0 to 2 097 000 Kbps. The threshold counts only outgoing traffic.

With spill-over ECMP routing configured, the FortiGate unit routes new sessions to an interface used by an ECMP route until that interface reaches its Spillover Threshold. Then, when the threshold of that interface is reached, new sessions are routed to one of the other interfaces used by the ECMP routes.

To add Spillover Thresholds to interfaces - web‑based manager

Use the following steps to enable usage based ECMP routing, add Spillover Thresholds to FortiGate interfaces port3 and port4, and then to configure EMCP routes with device set to port3 and port4.

1. Go to Router > Static > Settings.
2. Set ECMP Load Balance Method to Spillover.
3. Go to Router > Static > Static Routes.
4. Add ECMP routes for port3 and port4.

Destination IP/Mask		192.168.20.0/24
Device		port3
Gateway		172.20.130.3
Advanced
	Distance	10

 

Destination IP/Mask		192.168.20.0/24
Device		port4
Gateway		172.20.140.4
Advanced
	Distance	10

5. Go to System > Network > Interfaces.
6. Edit port3 and port4 and add the following spillover-thresholds:

Interface	port3
Spillover Threshold	100

 

Interface	port4
Spillover Threshold	200

To add Spillover Thresholds to interfaces - CLI

config system settings

set v4-ecmp-mode usage-based

end

config router static

edit 1

set device port3

set dst 192.168.20.0 255.255.255.0

set gateway 172.20.130.3

next

edit 2

set device port4

set dst 192.168.20.0 255.255.255.0

set gateway 172.20.140.4

end

config system interface

edit port3

set spillover-threshold 100

next

edit port4

set spillover-threshold 200

end

Detailed description of how spill-over ECMP selects routes

When you add ECMP routes they are added to the routing table in the order displayed by the routing monitor or by the get router info routing-table static command. This order is independent of the configured bandwidth limit.

The FortiGate unit selects an ECMP route for a new session by finding the first route in the routing table that sends the session out a FortiGate unit interface that is not processing more traffic that its configured route spill-over limit.

A new session to a destination IP address that already has an entry in the routing cache is routed using the route already added to the cache for that destination address. See ECMP routing of simultaneous sessions to the same destination IP address.

For example, consider a FortiGate unit with interfaces port3 and port4 both connected to the Internet through different ISPs. ECMP routing is set to usage-based and route spillover for to 100 Kbps for port3 and 200 Kbps for port4. Two ECMP default routes are added, one for port3 and one for port4.

If the route to port3 is higher in the routing table than the route to port4, the FortiGate unit sends all default route sessions out port3 until port3 is processing 100Kbps of data. When port3 reaches its configured bandwidth limit, the FortiGate unit sends all default route sessions out port4. When the bandwidth usage of port3 falls below 100Kbps, the FortiGate again sends all default route sessions out port3.

New sessions with destination IP addresses that are already in the routing cache; however, use the cached routes. This means that even if port3 is exceeding its bandwidth limit, new sessions can continue to be sent out port3 if their destination addresses are already in the routing cache. As a result, new sessions are sent out port4 only if port3 exceeds its bandwidth limit and if the routing cache does not contain a route for the destination IP address of the new session.

Also, the switch over to port4 does not occur as soon as port3 exceeds its bandwidth limit. Bandwidth usage has to exceed the limit for a period of time before the switch over takes place. If port3 bandwidth usage drops below the bandwidth limit during this time period, sessions are not switched over to port4. This delay reduces route flapping.

FortiGate usage-based ECMP routing is not actually load balancing, since routes are not distributed evenly among FortiGate interfaces. Depending on traffic volumes, most traffic would usually be processed by the first interface with only spillover traffic being processed by other interfaces.

If you are configuring usage-based ECMP in most cases you should add spillover thresholds to all of the interfaces with ECMP routes. The default spillover threshold is 0 which means no bandwidth limiting. If any interface has a spillover threshold of 0, no sessions will be routed to interfaces lower in the list unless the interface goes down or is disconnected. An interface can go down if Detect interface status for Gateway Load Balancing does not receive a response from the configured server.

Determining if an interface has exceeded its Spillover Threshold

You can use the diagnose netlink dstmac list CLI command to determine if an interface is exceeding its Spillover Threshold. If the command displays over_bps=1 the interface is exceeding its threshold. If over_bps=0 the interface has not exceeded its threshold.

dev=Wifi mac=00:00:00:00:00:00 src-vis-os src-vis-host src-vis-user rx_tcp_mss=0 tx_tcp_mss=0 overspill-threshold=0 bytes=0 over_bps=0 sampler_rate=0

Configuring weighted static route load balancing

Configure weighted load balancing to control how the FortiGate unit distributes sessions among ECMP routes by adding weights for each route. Add higher weights to routes that you want to load balance more sessions to.

With the ECMP load balancing method set to weighted, the FortiGate unit distributes sessions with different destination IPs by generating a random value to determine the route to select. The probability of selecting one route over another is based on the weight value of each route. Routes with higher weights are more likely to be selected.

Large numbers of sessions are evenly distributed among ECMP routes according to the route weight values. If all weights are the same, sessions are distributed evenly. The distribution of a small number of sessions; however, may not be even. For example, its possible that if there are two ECMP routes with the same weight; two sessions to different IP addresses could use the same route. On the other hand, 10,000 sessions with different destination IPs should be load balanced evenly between two routes with equal rates. The distribution could be 5000:5000 or 50001:4999. Also, 10 000 sessions with different destination IP addresses should be load balanced as 3333:6667 if the weights for the two routes are 100 and 200.

Weights only affect how routes are selected for sessions to new destination IP addresses. New sessions to IP addresses already in the routing cache are routed using the route for the session already in the cache. So in practice sessions will not always be distributed according to the routing weight distribution.

To add weights to static routes from the web‑based manager

1. Go to Router > Static > Settings.
2. Set ECMP Load Balance Method to Weighted Load Balance.
3. Go to Router > Static > Static Routes.
4. If needed, add new static routes, for example:

Destination IP/Mask	192.168.20.0/24
Device	port1
Gateway	172.20.110.1
Distance	10

 

Destination IP/Mask	192.168.20.0/24
Device	port2
Gateway	172.20.120.2
Distance	10

5. Go to Router > Static > Interfaces.
6. Select a number next to an interface name, and choose Edit to change it, or simply double-click the interface name.
7. Set the weight; for example, set the weight of port1 to 100 and the weight of port2 to 200.