Troubleshooting BGP
There are some features in BGP that are used to deal with problems that may arise. Typically the problems with a BGP network that has been configured, involve routes going offline frequently. This is called route flap and causes problems for the routers using that route.
Clearing routing table entries
To see if a new route is being properly added to the routing table, you can clear all or some BGP neighbor connections (sessions) using the execute router clear bgp command.
For example, if you have 10 routes in the BGP routing table and you want to clear the specific route to IP address 10.10.10.1, enter the command:
execute router clear bgp ip 10.10.10.1
To remove all routes for AS number 650001, enter the command:
execute router clear bgp as 650001
Route flap
When routers or hardware along a route go offline and back online that is called a route flap. Flapping is the term if these outages continue, especially if they occur frequently.
Route flap is a problem in BGP because each time a peer or a route goes down, all the peer routers that are connected to that out-of-service router advertise the change in their routing tables which creates a lot of administration traffic on the network. And the same traffic happens again when that router comes back online. If the problem is something like a faulty network cable that wobbles on and offline every 10 seconds, there could easily be overwhelming amounts of routing updates sent out unnecessarily.
Another possible reason for route flap occurs with multiple FortiGate units in HA mode. When an HA cluster fails over to the secondary unit, other routers on the network may see the HA cluster as being offline resulting in route flap. While this doesn’t occur often, or more than once at a time, it can still result in an interruption in traffic which is unpleasant for network users. The easy solution for this problem is to increase the timers on the HA cluster, such as TTL timers, so they do not expire during the failover process. Also configuring graceful restart on the HA cluster will help with a smooth failover.
The first method of dealing with route flap should be to check your hardware. If a cable is loose or bad, it can easily be replaced and eliminate the problem. If an interface on the router is bad, either avoid using that interface or swap in a functioning router. If the power source is bad on a router, either replace the power supply or use a power conditioning backup power supply. These quick and easy fixes can save you from configuring more complex BGP options. However if the route flap is from another source, configuring BGP to deal with the outages will ensure your network users uninterrupted service.
Some methods of dealing with route flap in BGP include:
Holddown timer
The first line of defence to a flapping route is the hold down timer. This timer reduces how frequently a route going down will cause a routing update to be broadcast.
Once activated, the holddown timer won’t allow the FortiGate unit to accept any changes to that route for the duration of the timer. If the route flaps five times during the timer period, only the first outage will be recognized by the FortiGate unit — for the duration of the other outages there will be no changes because the Fortigate unit is essentially treating this router as down. After the timer expires, if the route is still flapping it will happen all over again.
Even if the route isn’t flapping — if it goes down, comes up, and stays back up — the timer still counts down and the route is ignored for the duration of the timer. In this situation the route will be seen as down longer than it really is, but there will be only the one set of route updates. This is not a problem in normal operation because updates are not frequent.
Also the potential for a route to be treated as down when it is really up can be viewed as a robustness feature. Typically you do not want most of your traffic being routed over an unreliable route. So if there is route flap going on, it is best to avoid that route if you can. This is enforced by the holddown timer.
How to configure the holddown timer
There are three different route flapping situations that can occur: the route goes up and down frequently, the route goes down and back up once over a long period of time, or the route goes down and stays down for a long period of time. These can all be handled using the holddown timer.
For example, your network has two routes that you want to set the holddown timer for. One is your main route ( to 10.12.101.4) that all your Internet traffic goes through, and it can’t be down for long if its down. The second is a low speed connection to a custom network that is used infrequently ( to 10.13.101.4). The holddown timer for the main route should be fairly short, lets say 60 seconds instead of the default 180 seconds. The second route timer can be left at the default or even longer since it is rarely used. In your BGP configuration this looks like:
config router bgp
config neighbor
edit 10.12.101.4
set holddown-timer 60
next
edit 10.13.101.4
set holddown-timer 180
next
end
end
Dampening
Dampening is a method used to limit the amount of network problems due to flapping routes. With dampening the flapping still occurs, but the peer routers pay less and less attention to that route as it flaps more often. One flap doesn’t start dampening, but the second starts a timer where the router will not use that route — it is considered unstable. If the route flaps again before the timer expires, the timer continues to increase. There is a period of time called the reachability half-life after which a route flap will only be suppressed for half the time. This half-life comes into effect when a route has been stable for a while but not long enough to clear all the dampening completely. For the flapping route to be included in the routing table again, the suppression time must expire.
If the route flapping was temporary, you can clear the flapping or dampening from the FortiGate units cache by using one of the execute router clear bgp commands:
execute router clear bgp dampening {<ip_address> | <ip/netmask>}
or
execute router clear bgp flap-statistics {<ip> | <ip/netmask>}
For example, to remove route flap dampening information for the 10.10.0.0/16 subnet, enter the command:
execute router clear bgp dampening 10.10.0.0/16
The BGP commands related to route dampening are:
config router bgp
set dampening {enable | disable}
set dampening-max-suppress-time <minutes_integer>
set dampening-reachability-half-life <minutes_integer>
set dampening-reuse <reuse_integer>
set dampening-route-map <routemap-name_str>
set dampening-suppress <limit_integer>
set dampening-unreachability-half-life <minutes_integer>
end
Graceful restart
BGP4 has the capability to gracefully restart.
In some situations, route flap is caused by routers that appear to be offline but the hardware portion of the router (control plane) can continue to function normally. One example of this is when some software is restarting or being upgraded, but the hardware can still function normally.
Graceful restart is best used for these situations where routing will not be interrupted, but the router is unresponsive to routing update advertisements. Graceful restart does not have to be supported by all routers in a network, but the network will benefit when more routers support it.
|
FortiGate HA clusters can benefit from graceful restart. When a failover takes place, the HA cluster will advertise it is going offline, and will not appear as a route flap. It will also enable the new HA main unit to come online with an updated and usable routing table — if there is a flap the HA cluster routing table will be out of date. |
For example, your FortiGate unit is one of four BGP routers that send updates to each other. Any of those routers may support graceful starting—when a router plans to go offline, it will send out a message to its neighbors how long it expects to be before being back online. That way its neighbor routers don’t remove it from their routing tables. However if that router isn’t back online when expected, the routers will mark it offline. This prevents routing flap and its associated problems.
Scheduled time offline
Graceful restart is a means for a router to advertise it is going to have a scheduled shutdown for a very short period of time. When neighboring routers receive this notice, they will not remove that router from their routing table until after a set time elapses. During that time if the router comes back online, everything continues to function as normal. If that router remains offline longer than expected, then the neighboring routers will update their routing tables as they assume that router will be offline for a long time.
FortiGate units support both graceful restart of their own BGP routing software, and also neighboring BGP routers.
For example, if a neighbor of your FortiGate unit, with an IP address of 172.20.120.120, supports graceful restart, enter the command:
config router bgp
config neighbor
edit 172.20.120.120
set capability-graceful-restart enable
end
end
If you want to configure graceful restart on your FortiGate unit where you expect the Fortigate unit to be offline for no more than 2 minutes, and after 3 minutes the BGP network should consider the FortiGate unit offline, enter the command:
config router bgp
set graceful-restart enable
set graceful-restart-time 120
set graceful-stalepath-time 180
end
The BGP commands related to BGP graceful restart are:
config router bgp
set graceful-restart { disable| enable}
set graceful-restart-time <seconds_integer>
set graceful-stalepath-time <seconds_integer>
set graceful-update-delay <seconds_integer>
config neighbor
set capability-graceful-restart {enable | disable}
end
end
execute router restart
Before the restart, the router sends its peers a message to say it is restarting. The peers mark all the restarting router's routes as stale, but they continue to use the routes. The peers assume the router will restart and check its routes and take care of them if needed after the restart is complete. The peers also know what services the restarting router can maintain during its restart. After the router completes the restart, the router sends its peers a message to say it is done restarting.
Bi-directional forwarding detection (BFD)
Bi-directional Forwarding Detection (BFD) is a protocol used to quickly locate hardware failures in the network. Routers running BFD communicate with each other, and if a timer runs out on a connection then that router is declared down. BFD then communicates this information to the routing protocol and the routing information is updated.
While BGP can detect route failures, BFD can be configured to detect these failures more quickly allowing faster responses and improved convergence. This can be balanced with the bandwidth BFD uses in its frequent route checking.
Configurable granularity
BFD can run on the entire FortiGate unit, selected interfaces, or on BGP for all configured interfaces. The hierarchy allows each lower level to override the upper level’s BFD setting. For example, if BFD was enabled for the FortiGate unit, it could be disabled only for a single interface or for BGP. For information about FortiGate-wide BFD options, see config system settings in the FortiGate CLI Reference.
BFD can only be configured through the CLI.
The BGP commands related to BFD are:
config system {setting | interface}
set bfd {enable | disable | global}
set bfd-desired-mix-tx <milliseconds>
set bfd-detect-mult <multiplier>
set bfd-required-mix-rx <milliseconds>
set bfd-dont-enforce-src-port {enable | disable}
config router bgp
config neighbor
edit <neighbor_address_ipv4>
set bfd {enable | disable}
end
end
get router info bfd neighbor
execute router clear bfd session <src_ipv4> <dst_ipv4> <interface>
The config system commands allow you to configure whether BFD is enabled in a particular unit/vdom or individual interface, and how often the interface requires sending and receiving of BFD information.
The config router bgp commands allow you to set the addresses of the neighbor units that are also running BFD. Both units must be configured with BFD in order to make use of it.
Copyright © 2018 Fortinet, Inc. All Rights Reserved. | Terms of Service | Privacy Policy
