How can you spot a routing loop
Any time network traffic slows down, you will be asking yourself if it is a network loop or not. Often slowdowns are normal, they are not a full stoppage, and normal traffic resumes in a short period of time.
If the slow down is a full halt of traffic or a major slowdown does not return to normal quickly, you need to do serious troubleshooting quickly.
Some methods to troubleshoot your outage include:
If you aren’t running SNMP, dead gateway detection, or you have non-Fortinet routers in your network, you can use networking tools such as ping and traceroute to define the outage on your network and begin to fix it.
Checking your logs
If your routers log events to a central location, it can be easy to check the logs for your network for any outages.
On your FortiGate unit, go to Log & Report > Log & Archive Access. You will want to look at both event logs and traffic logs. Events to look for will generally fall under CPU and memory usage, interfaces going offline (due to dead gateway detection), and other similar system events.
Once you have found and fixed your network problem, you can go back to the logs and create a report to better see how things developed during the problem. This type of forensics analysis can better help you prepare for next time.
Using SNMP network monitoring
If your network had no problems one minute and slows to a halt the next, chances are something changed to cause that problem. Most of the time an offline router is the cause, and once you find that router and bring it back online, things will return to normal.
If you can enable a hardware monitoring system such as SNMP or sFlow on your routers, you can be notified of the outage and where it is exactly as soon as it happens.
Ideally you can configure SNMP on all your FortiGate routers and be alerted to all outages as they occur.
To use SNMP to detect potential routing loops
1. Go to System > Config > SNMP.
2. Enable SNMP Agent.
3. Optionally enter the Description, Location, and Contact information for this device for easier location of the problem report.
4. In either SNMP v1/v2c section or SNMP v3 section, as appropriate, select Create New.
5. Enter the Community Name that you want to use.
6. In Hosts, select Add to add an IP address where you will be monitoring the FortiGate unit. You can add up to 8 different addresses.
7. Ensure that ports 161 and 162 (SNMP queries and traps) are allowed through your security policies.
8. In SNMP Event, select the events you want to be notified of. For routing loops this should include CPU Overusage, Memory Low, and possibly Log disk space low. If there are problems, the log will be filling up quickly, and the FortiGate unit’s resources will be overused.
9. Select OK.
10. Configure SNMP host (manager) software on your administration computer. This will monitor the SNMP information sent out by the FortiGate unit. Typically you can configure this software to alert you to outages or CPU spikes that may indicate a routing loop.
Using dead gateway detection and e-mail alerts
Another tool available to you on FortiGate units is the dead gateway detection. This feature allows the FortiGate unit to ping a gateway at regular intervals to ensure it is online and working. When the gateway is not accessible, that interface is marked as down.
To detect possible routing loops with dead gateway detection and e-mail alerts
1. To configure dead gateway detection, go to Router > Static > Settings and select Create New.
2. Set the Ping Interval (how often to send a ping), and Failover Threshold (how many lost pings is considered a failure). A smaller interval and smaller number of lost pings will result in faster detection, but will create more traffic on your network.
3. To configure interface status change notification, go to Log & Report > Log Config > Alert E-mail.
4. After you enter your email details, select the events you want to be alerted about — in our case Configuration changes. You may also want to log CPU and Memory usage as a network outage will cause your CPU activity to spike.
| If you have VDOMs configured, you will have to enter the basic SMTP server information in the Global section, and the rest of the configuration within the VDOM that includes this interface. |
After this configuration, when this interface on the FortiGate unit cannot connect to the next router, the FortiGate unit will bring down the interface and alert you with an email to the outage.
Looking at the packet flow
If you want to see what is happening on your network, look at the packets travelling on the network. In this situation, you are looking for routes that have metrics higher than 15 as that indicates they are unreachable. Ideally if you debug the flow of the packets, and record the routes that are unreachable, you can create an accurate picture of the network outage.