This section is about troubleshooting RIP. For general troubleshooting information, see the FortiOS Handbook Troubleshooting chapter.
This section includes:
- Routing Loops
- Holddowns and Triggers for updates
- Split horizon and Poison reverse updates
- Debugging IPv6 on RIPng
Normally in routing, a path between two addresses is chosen and traffic is routed along that path from one address to the other. When there is a routing loop, that normal path doubles back on itself creating a loop. When there are loops, the network has problems getting information to its destination and also prevents it from returning to the source to report the inaccessible destination.
A routing loop happens when a normally functioning network has an outage, and one or more routers are offline. When packets encounter this, an alternate route is attempted to maneuver around the outage. During this phase it is possible for a route to be attempted that involves going back a hop, and trying a different hop forward. If that hop forward is blocked by the outage as well, a hop back and possibly the original hop forward may be selected. You can see if this continues, how it can consume not only network bandwidth but also many resources on those routers affected. The worst part is this situation will continue until the network administrator changes the router settings, or the downed routers come back online.
Routing loops’ effect on the network
In addition to this “traffic jam” of routed packets, every time the routing table for a router changes that router sends an update out to all of the RIP routers connected to it. In a network loop, its possible for a router to change its routes very quickly as it tries and fails along these new routes. This can quickly result in a flood of updates being sent out, which can effectively grind the network to a halt until the problem is fixed.
How can you spot a routing loop
Any time network traffic slows down, you will be asking yourself if it is a network loop or not. Often slowdowns are normal, they are not a full stoppage, and normal traffic resumes in a short period of time.
If the slow down is a full halt of traffic or a major slowdown does not return to normal quickly, you need to do serious troubleshooting quickly.
If you aren’t running SNMP, dead gateway detection, or you have non-Fortinet routers in your network, you can use networking tools such as ping and traceroute to define the outage on your network and begin to fix it. Ping, traceroute, and other basic troubleshooting tools are largely the same between static and dynamic, and are covered in Advanced Static Routing.
Check your logs
If your routers log events to a central location, it can be easy to check the logs for your network for any outages.
On your FortiGate unit, go to Log & Report. You will want to look at both event logs and traffic logs. Events to look for will generally fall under CPU and memory usage, interfaces going offline (due to dead gateway detection), and other similar system events.
Once you have found and fixed your network problem, you can go back to the logs and create a report to better see how things developed during the problem. This type of forensics analysis can better help you prepare for next time.
Use SNMP network monitoring
If your network had no problems one minute and slows to a halt the next, chances are something changed to cause that problem. Most of the time an offline router is the cause, and once you find that router and bring it back online, things will return to normal.
If you can enable a hardware monitoring system such as SNMP or sFlow on your routers, you can be notified of the outage and where it is exactly as soon as it happens.
Ideally you can configure SNMP on all your FortiGate routers and be alerted to all outages as they occur.
To use SNMP to detect potential routing loops
- Go to System > Config > SNMP.
- Enable SMTP Agent and select Apply.
Optionally enter the Description, Location, and Contact information for this device for easier location of the problem report.
- Under SNMP v1/v2 or SNMP v3 as appropriate, select Create New.
|User Name||Enter the SNMP user ID.|
|Security Level||Select authentication or privacy as desired. Select the authentication or privacy algorithms to use and enter the required passwords.|
|Notification Host||Enter the IP addresses of up to 16 hosts to notify.|
|Enable Query||Select. The Port should be 161. Ensure that your security policies allow ports 161 and 162 (SNMP queries and traps) to pass.|
|Hosts||Enter the IP addresses of up to 8 hosts to notify. You can also specify the network Interface, or leave it as ANY.|
|Queries||Enable v1 and/or v2 as needed. The Port should be 161. Ensure that your security policies allow port 161 to pass.|
|Traps||Enable v1 and/or v2 as needed. The Port should be 162. Ensure that your security policies allow port 162 to pass.|
- Select the events for which you want notification. For routing loops this should include CPU usage is high, Memory is low, and possibly Log disk space is low.If there are problems the log will be filling up quickly, and the FortiGate unit’s resources will be overused.
- Configure SNMP host (manager) software on your administration computer. This will monitor the SNMP information sent out by the FortiGate unit. Typically you can configure this software to alert you to outages or CPU spikes that may indicate a routing loop.
Use Link Health Monitor and e-mail alerts
Another tool available to you on FortiGate units is the Link Health Monitor, useful in dead gateway detection. This feature allows the FortiGate unit to ping a gateway at regular intervals to ensure it is online and working. When the gateway is not accessible, that interface is marked as down.
To detect possible routing loops with Link Health Monitor and e-mail alerts
- Go to Router > Static > Settings and select Create New under Link Health Monitor.
- Enter the Ping Server IP address under Gateway and select the Interface that connects to it.
- Set the Probe Interval (how often to send a ping), and Failure Threshold (how many lost pings is considered a failure). A smaller interval and smaller number of lost pings will result in faster detection, but will create more traffic on your network.
To configure notification of failed gateways
- Go to Log & Report > Report > Local and enable Email Generated Reports.
- Enter your email details.
- Select Apply.
You might also want to log CPU and Memory usage as a network outage will cause your CPU activity to spike.
|If you have VDOMs configured, you will have to enter the basic SMTP server information in the Global section, and the rest of the configuration within the VDOM that includes this interface.|
After this configuration, when this interface on the FortiGate unit cannot connect to the next router, the FortiGate unit will bring down the interface and alert you with an email about the outage.
Look at the packet flow
If you want to see what is happening on your network, look at the packets travelling on the network. This is same idea as police pulling over a car and asking the driver where they have been, and what the conditions were like.
The method used in the troubleshooting sections Debugging IPv6 on RIPng and on debugging the packet flow apply here as well. In this situation, you are looking for routes that have metrics higher than 15 as that indicates they are unreachable.
Ideally if you debug the flow of the packets, and record the routes that are unreachable, you can create an accurate picture of the network outage.
Action to take on discovering a routing loop
Once you have mapped the problem on your network, and determined it is in fact a routing loop there are a number of steps to take in correcting it.
- Get any offline routers back online. This may be a simple reboot, or you may have to replace hardware. Often this first step will restore your network to its normal operation, once the routing tables finish being updated.
- Change your routing configuration on the edges of the outage. Even if step 1 brought your network back online, you should consider making changes to improve your network before the next outage occurs. These changes can include configuring features like holddowns and triggers for updates, split horizon, and poison reverse updates.
One of the potential problems with RIP is the frequent routing table updates that are sent every time there is a change to the routing table. If your network has many RIP routers, these updates can start to slow down your network. Also if you have a particular route that has bad hardware, it might be going up and down frequently, which will generate an overload of routing table updates.
One of the most common solutions to this problem is to use holddown timers and triggers for updates. These slow down the updates that are sent out, and help prevent a potential flood.
The holddown timer activates when a route is marked down. Until the timer expires, the router does not accept any new information about that route. This is very useful if you have a flapping route because it will prevent your router from sending out updates and being part of the problem in flooding the network. The potential down side is if the route comes back up while the timer has not expired, that route will be unavailable for that period of time. This is only a problem if this is a major route used by the majority of your traffic. Otherwise, this is a minor problem as traffic can be re-routed around the outage.
Triggered RIP is an alternate update structure that is based around limiting updates to only specific circumstances. The most basic difference is that the routing table will only be updated when a specific request is sent to update, as opposed to every time the routing table changes. Updates are also triggered when a unit is ‘powered on’, which can include addition of new interfaces or devices to the routing structure, or devices returning to being available after being unreachable.
Split horizon is best explained with an example. You have three routers linked serially, let’s call them A, B, and C. A is only linked to B, C is only linked to B, and B is linked to both A and C. To get to C, A must go through B. If the link to C goes down, it is possible that B will try to use A’s route to get to C. This route is A-B-C, so it will loop endlessly between A and B.
This situation is called a split horizon because from B’s point of view the horizon stretches out in each direction, but in reality it only is on one side.
Poison reverse is the method used to prevent routes from running into split horizon problems. Poison reverse “poisons” routes away from the destination that use the current router in their route to the destination. This “poisoned” route is marked as unreachable for routers that cannot use it. In RIP this means that route is marked with a distance of 16.
The debug commands are very useful to see what is happening on the network at the packet level. There are a few changes to debugging the packet flow when debugging IPv6.
The following CLI commands specify both IPv6 and RIP, so only RIPng packets will be reported. The output from these commands will show you the RIPng traffic on your FortiGate unit including RECV, SEND, and UPDATE actions.
The addresses are in IPv6 format.
diagnose debug enable
diagnose ipv6 router rip level info
diagnose ipv6 router rip all enable
These three commands will:
- turn on debugging in general
- set the debug level to information, a verbose reporting level
- turn on all rip router settings
Part of the information displayed from the debugging is the metric (hop count). If the metric is 16, then that destination is unreachable since the maximum hop count is 15.
In general, you should see an update announcement, followed by the routing table being sent out, and a received reply in response.
For more information, see Troubleshooting RIP.