How can you spot a routing loop
Any time network traffic slows down, you will be asking yourself if it is a network loop or not. Often slowdowns are normal, they are not a full stoppage, and normal traffic resumes in a short period of time.
If the slow down is a full halt of traffic or a major slowdown does not return to normal quickly, you need to do serious troubleshooting quickly.
If you aren’t running SNMP, dead gateway detection, or you have non-Fortinet routers in your network, you can use networking tools such as ping and traceroute to define the outage on your network and begin to fix it. Ping, traceroute, and other basic troubleshooting tools are largely the same between static and dynamic, and are covered in
“Troubleshooting static routing”.
Check your logs
If your routers log events to a central location, it can be easy to check the logs for your network for any outages.
On your FortiGate unit, go to Log & Report. You will want to look at both event logs and traffic logs. Events to look for will generally fall under CPU and memory usage, interfaces going offline (due to dead gateway detection), and other similar system events.
Once you have found and fixed your network problem, you can go back to the logs and create a report to better see how things developed during the problem. This type of forensics analysis can better help you prepare for next time.
Use SNMP network monitoring
If your network had no problems one minute and slows to a halt the next, chances are something changed to cause that problem. Most of the time an offline router is the cause, and once you find that router and bring it back online, things will return to normal.
If you can enable a hardware monitoring system such as SNMP or sFlow on your routers, you can be notified of the outage and where it is exactly as soon as it happens.
Ideally you can configure SNMP on all your FortiGate routers and be alerted to all outages as they occur.
To use SNMP to detect potential routing loops
1. Go to System > Config > SNMP.
2. Enable SMTP Agent and select Apply.
Optionally enter the Description, Location, and Contact information for this device for easier location of the problem report.
3. Under SNMP v1/v2 or SNMP v3 as appropriate, select Create New.
SNMP v3
User Name | Enter the SNMP user ID. |
Security Level | Select authentication or privacy as desired. Select the authentication or privacy algorithms to use and enter the required passwords. |
Notification Host | Enter the IP addresses of up to 16 hosts to notify. |
Enable Query | Select. The Port should be 161. Ensure that your security policies allow ports 161 and 162 (SNMP queries and traps) to pass. |
SNMP v1/v2
Hosts | Enter the IP addresses of up to 8 hosts to notify. You can also specify the network Interface, or leave it as ANY. |
Queries | Enable v1 and/or v2 as needed. The Port should be 161. Ensure that your security policies allow port 161 to pass. |
Traps | Enable v1 and/or v2 as needed. The Port should be 162. Ensure that your security policies allow port 162 to pass. |
4. Select the events for which you want notification. For routing loops this should include CPU usage is high, Memory is low, and possibly Log disk space is low.If there are problems the log will be filling up quickly, and the FortiGate unit’s resources will be overused.
5. Configure SNMP host (manager) software on your administration computer. This will monitor the SNMP information sent out by the FortiGate unit. Typically you can configure this software to alert you to outages or CPU spikes that may indicate a routing loop.
Use dead gateway detection and e-mail alerts
Another tool available to you on FortiGate units is the dead gateway detection. This feature allows the FortiGate unit to ping a gateway at regular intervals to ensure it is online and working. When the gateway is not accessible, that interface is marked as down.
To detect possible routing loops with dead gateway detection and e-mail alerts
1. To configure dead gateway detection, go to Router > Static > Settings and select Create New.
2. Enter the Ping Server IP address and select the Interface that connects to it.
3. Set the Ping Interval (how often to send a ping), and Failover Threshold (how many lost pings is considered a failure). A smaller interval and smaller number of lost pings will result in faster detection, but will create more traffic on your network.
To configure notification of failed gateways
1. Go to Log & Report > Log Config > Alert E-mail.
2. Enter your email details.
3. Select the Configuration changes event.
4. Select Apply.
You might also want to log CPU and Memory usage as a network outage will cause your CPU activity to spike.
| If you have VDOMs configured, you will have to enter the basic SMTP server information in the Global section, and the rest of the configuration within the VDOM that includes this interface. |
After this configuration, when this interface on the FortiGate unit cannot connect to the next router, the FortiGate unit will bring down the interface and alert you with an email about the outage.
Look at the packet flow
If you want to see what is happening on your network, look at the packets travelling on the network. This is same idea as police pulling over a car and asking the driver where they have been, and what the conditions were like.
The method used in the troubleshooting sections
“Debugging IPv6 on RIPng” and on debugging the packet flow apply here as well. In this situation, you are looking for routes that have metrics higher than 15 as that indicates they are unreachable.
Ideally if you debug the flow of the packets, and record the routes that are unreachable, you can create an accurate picture of the network outage.