> Chapter 26 - Troubleshooting > Common questions

Common questions

The general troubleshooting tips include, and can help answer, the following questions:

How to check hardware connections

Are all the cables and interfaces connected properly?
Is the LED for the interface green?

How to check FortiOS network settings

If you are having problems connecting to the management interface, is your protocol enabled on the interface for administrative access?
Is there an IP address on the interface?

How to check CPU and memory resources

Is your CPU running at almost 100 percent usage?
Are you running low on memory?

How to check modem status

Is the modem connected?
Are there PPP issues?

How to run ping and traceroute

Are you experiencing complete packet loss?

How to check the logs

Do you need to identify a problem?

How to verify the contents of the routing table (in NAT mode)

Are there routes in the routing table for default and static routes?
Do all connected subnets have a route in the routing table?
Does a route wrongly have a higher priority than it should?

How to verify the correct route is being used

Has the traffic been routed correctly?

How to verify the correct firewall policy is being used

Is the correct firewall policy applied to the expected traffic?

How to check the bridging information in Transparent mode

Are you having problems in Transparent mode?

How to check number of sessions used by UTM proxy

Have you reached the maximum number of sessions for a protocol?
Are new sessions failing to start for a certain protocol?

How to examine the firewall session list

Are there active firewall sessions?

How to check wireless information

Is the wireless network functioning properly?

How to verify FortiGuard connectivity

Is the FortiGate unit communicating properly with FortiGuard?

How to perform a sniffer trace (CLI and Packet Capture)

Is traffic entering the FortiGate unit and does it arrive on the expected interface?
Is the ARP resolution correct for the next-hop destination?
Is the traffic exiting the FortiGate unit to the destination as expected?
Is the traffic being sent back to the originator?

How to debug the packet flow

Is the traffic entering or leaving the FortiGate unit as expected?

How to check hardware connections

If there is no traffic flowing from the FortiGate unit, it may be a hardware problem.

To check hardware connections:

Ensure the network cables are properly plugged into the interfaces.
Ensure there are connection lights for the network cables on the unit.
Change the cable if the cable or its connector are damaged or you are unsure about the cable’s type or quality—such as straight through or crossover, or possibly exposed wires at the connector.
Connect the FortiGate unit to different hardware.
Ensure the link status is set to Up for the interface, (see Network > Interface > Status). The link status is based on the physical connection and cannot be set in FortiOS.

If any of these solve the problem, it was a hardware connection problem. You should still perform some basic software connectivity tests to ensure complete connectivity. It might also be that the interface is disabled, or has its Administrative Status set to Down.

To enable an interface - web-based manager

Using the web-based management interface, go to System > Network > Interface.
Select and edit the interface to enable, such as port1.
Find Administrative Status at the bottom of the screen, and select Up.
Select Apply.

To enable an interface - CLI

config system interface

edit port1

set status up

end

How to check FortiOS network settings

FortiOS network settings are present in both the web-based manager interface and the CLI. The following information includes troubleshooting and best practice information. The network settings include:

Interface settings

DNS settings

DHCP Server settings

Interface settings

If you can access the FortiGate unit with the management cable only, the first step is to display the interface settings. To display the settings for the internal interface, use the following CLI command:

FGT# show system interface <Interface_mane>

For a complete listing of all the possible interface settings, use the following CLI command:

config system interface

edit <Interface_name>

get

end

Check the interface settings to ensure they are not preventing traffic. Specific things to check include (only the web-based manager names are shown, CLI names may vary slightly):

Link Status — Down until a valid cable is plugged into this interface, after which it will be Up. The Link Status is shown physically by the connection LED for the interface. If it lights up green, it is a good connection. If Link Status is Down, the interface does not work. Link Status is also displayed on the System > Network > Interface screen by default.
Addressing mode — Do not use DHCP if you don’t have a DHCP server —you will not be able to logon to an interface in DHCP mode as it will not have an IP address.
IP/Netmask — An interface needs an IP address to be able to connect to other devices. Ensure there is a valid IP address in this field. The one exception is if DHCP is enabled for this interface to get its IP address from an external DHCP server.
IPv6 address — The same protocol must be used by both ends to complete the connection. Ensure both this interface and the remote connection are both using IPv4 or both using IPv6 addressing.
Administrative access — If no protocols are selected, you will have to use the local management cable to connect to the unit. If you are using IPv6, configure the IPv6 administrative access protocols.
Administrative status — Set to Up or the interface will not work.

DNS settings

While this section is not complicated, many networking problems can be traced back to DNS problems. Things to check in this area include:

Are there values for both primary and secondary entries?
Is the local domain name correct?
Are you using IPv6 addressing? If so, are the IPv6 DNS settings correct?
Are you using Dynamic DNS (DDNS)? If so, is it using the correct server, credentials, and interface?
Can you contact both DNS servers to verify the servers are operational?
If an interface addressing mode is set to DHCP and is set to override the internal DNS, is that interface receiving a valid DNS entry from the DHCP server? Is it a reasonable address and can it be contacted to verify it’s operational?
Are there any DENY security policies that need to allow DNS?
Can any internal device perform a successful traceroute to a location using the FQDN?

DHCP Server settings

DHCP Servers are common on internal and wireless networks. If the DHCP server is not configured properly it can cause problems. Things to check in this area include:

Is the DHCP server entry set to Relay? If so, verify there is another DHCP server to which requests can be relayed. Otherwise, it should be set to Server.
Is the DHCP server enabled?
Does this DHCP server use a valid range of IP addresses? Are those addresses in use by other devices? If one or more devices are using IP addresses in this range, you can use the IP reservation feature to ensure the DHCP server does not use these addresses.
Is there a gateway entry? Include a gateway entry to ensure clients of this server have a default route.
Is the system DNS setting being used? The best practice is to avoid confusion by using the system DNS whenever possible. However, the option to specify up to three custom DNS servers is available, and all three entries should be used for redundancy.

There are some situations, such as a new wireless interface, or during the initial FortiGate unit configuration, where interfaces override the system DNS entries. When this happens, it often shows up as intermittent Internet connectivity. To fix the problem, go to System > Network > DNS and ensure to enable Use FortiGuard Servers.

How to check CPU and memory resources

System resources are shared and a number of processes run simultaneously on the FortiGate unit. If one of these processes consumes nearly all the resources.

A quick way to monitor CPU and memory usage is on the System Dashboard using the System Resources widgets. They have both a visual gauge displayed to show you the usage.

To check the system resources on your FortiGate unit, run the following CLI command:

FGT# get system performance status

This command provides a quick and easy snapshot of the FortiGate.

The first line of output shows the CPU usage by category. A FortiGate that is doing nothing will look like:

CPU states: 0% user 0% system 0% nice 100% idle

However, if your network is running slow you might see something like:

CPU states: 1% user 98% system 0% nice 1% idle

This line shows that all the CPU is used up by system processes. Normally this should not happen as it shows the FortiGate is overloaded for some reason. If you see this overloading, you should investigate farther as it’s possible a process, such as scanunitid, is using all the resources to scan traffic, in which case you need to reduce the amount of traffic being scanned by blocking unwanted protocols, configuring more security policies to limit scanning to certain protocols, or similar actions. It is also possible that a hacker has gained access to your network and is overloading it with malicious activity such as running a spam server or using zombie PCs to attack other networks on the Internet. You can get additional CPU related information with the CLI command get system performance top. This command shows you all the top processes running on the FortiGate unit (names on the left) and their CPU usage. If a process is using most of the CPU cycles, investigate it to determine if it’s normal activity.

The second line of output from get system performance status shows the memory usage. Memory usage should not exceed 90 percent. If memory is too full, some processes will not be able to function properly. For example, if the system is running low on memory, antivirus scanning will go into failopen mode where it will start dropping connections or bypass the antivirus system.

The other lines of output, such as average network usage, average session setup rate, viruses caught, and IPS attacks blocked can also help you determine why system resource usage it high. For example, if network usage is high it will result in high traffic processing on the FortiGate, or if the session setup rate is very low or zero the proxy may be overloaded and not able to do its job.

How to troubleshoot high memory usage

As with any system, FortiOS has a finite set of hardware resources such as memory and all the running processes share that memory. Depending on their workload, each process will use more or less as needed, usually more in high traffic situations. If some processes use all the available memory, other processes will have no memory available and not be able to function.

When high memory usage happens, you may experience services that appear to freeze up and connections are lost or new connections are refused.

If you are seeing high memory usage in the System Resources widget, it could mean that the unit is dealing with high traffic volume, which may be causing the problem, or it could be when the unit is dealing with connection pool limits affecting a single proxy. If the unit is receiving large volumes of traffic on a specific proxy, it is possible that the unit will exceed the connection pool limit. If the number of free connections within a proxy connection pool reaches zero, problems may occur.

Use the following CLI command, which uses the antivirus failopen feature. Setting it to idledrop will drop connections based on the clients that have the most connections open. This helps to determine the behavior of the FortiGate antivirus system if it becomes overloaded in high traffic.

config system global

set av-failopen idledrop

end

Use the following CLI command, which gives you information about current memory usage:

diagnose hardware sysinfo memory

Sample output:

total: used: free: shared: buffers: cached: shm:

Mem: 2074185728 756936704 1317249024 0 20701184 194555904 161046528

Swap: 0 0 0

MemTotal: 2025572 kB

MemFree: 1286376 kB

MemShared: 0 kB

Buffers: 20216 kB

Cached: 189996 kB

SwapCached: 0 kB

Active: 56644 kB

Inactive: 153648 kB

HighTotal: 0 kB

HighFree: 0 kB

LowTotal: 2025572 kB

LowFree: 1286376 kB

SwapTotal: 0 kB

SwapFree: 0 kB

How to troubleshoot high CPU usage

FortiOS has many features. If many of them are used at the same time, it can quickly use up all the CPU resources. When this happens, you will experience connection related problems stemming from the FortiOS unit trying to manage its workload by refusing new connections, or even more aggressive methods.

Some examples of features that are CPU intensive are VPN high level encryption, having all traffic undergo all possible scanning, logging all traffic, and packets, and dashboard widgets that frequently update their data.

Determine how high the CPU usage is currently.There are two main ways to do this. The easiest is to go to System > Dashboard > Status and look at the system resources widget. This is a dial gauge that displays a percentage use for the CPU. If its at the red-line, you should take action. The other method is to use the Dashboard CLI widget to enter diag sys top.

Sample output:

Run Time: 11 days, 23 hours and 36 minutes

0U, 0S, 98I; 1977T, 758F, 180KF

newcli 286 R 0.1 0.8

ipsengine 78 S < 0.0 3.1

ipsengine 64 S < 0.0 3.0

ipsengine 77 S < 0.0 3.0

ipsengine 68 S < 0.0 2.9

ipsengine 66 S < 0.0 2.9

ipsengine 79 S < 0.0 2.9

scanunitd 133 S < 0.0 1.8

pyfcgid 267 S 0.0 1.8

pyfcgid 269 S 0.0 1.7

pyfcgid 268 S 0.0 1.6

httpsd 139 S 0.0 1.6

pyfcgid 266 S 0.0 1.5

scanunitd 131 S < 0.0 1.4

scanunitd 132 S < 0.0 1.4

proxyworker 90 S 0.0 1.3

cmdbsvr 43 S 0.0 1.1

proxyworker 91 S 0.0 1.1

miglogd 55 S 0.0 1.1

httpsd 135 S 0.0 1.0

Where the codes displayed on the second output line mean the following:

U is % of user space applications using CPU. In the example, 0U means 0% of the user space applications are using CPU.
S is % of system processes (or kernel processes) using CPU. In the example, 0S means 0% of the system processes are using the CPU.
I is % of idle CPU. In the example, 98I means the CPU is 98% idle.
T is the total FortiOS system memory in Mb. In the example, 1977T means there are 1977 Mb of system memory.
F is free memory in Mb. In the example, 758F means there is 758 Mb of free memory.
KF is the total shared memory pages used. In the example, 180KF means the system is using 180 shared memory pages.

Each additional line of the command output displays information for each of the processes running on the FortiGate unit. For example, the third line of the output is:

newcli 286 R 0.1 0.8

Where:

newcli is the process name. Other process names can include ipsengine, sshd, cmdbsrv, httpsd, scanunitd, and miglogd.
286 is the process ID. The process ID can be any number.
R is the current state of the process. The process state can be:
R running
S sleep
Z zombie
D disk sleep.
0.1 is the amount of CPU that the process is using. CPU usage can range from 0.0 for a process that is sleeping to higher values for a process that is taking a lot of CPU time.
0.8 is the amount of memory that the process is using. Memory usage can range from 0.1 to 5.5 and higher.

Enter the following single-key commands when diagnose sys top is running:

Press q to quit and return to the normal CLI prompt.
Press p to sort the processes by the amount of CPU that the processes are using.
Press m to sort the processes by the amount of memory that the processes are using.

Determine what features are using most of the CPU resources.
There is a command in the CLI to let you see the top few processes currently running that use the most CPU resources. The CLI command get system performance top outputs a table of information. You are interested in the second most right column, CPU usage by percentage. If the top few entries are using most of the CPU, note which processes they are and investigate those features to try and reduce their CPU load. Some examples of processes you will see include:

ipsengine — the IPS engine that scans traffic for intrusions
scanunitd — antivirus scanner
httpsd — secure HTTP
iked — internet key exchange (IKE) in use with IPsec VPN tunnels
newcli — active whenever you are accessing the CLI
sshd — there are active secure socket connections
cmdbsrv — the command database server application

Go to the features that are at the top of the list and look for evidence of them overusing the CPU. Generally the monitor for a feature is a good place to start.

Check for unnecessary CPU “wasters”.
These are some best practises that will reduce your CPU usage, even if you are not experiencing high CPU usage. Note that if you require a feature this section tells you to turn off, ignore it.

Use hardware acceleration wherever possible to offload tasks from the CPU. Offloading tasks such as encryption frees up the CPU for other tasks.
Avoid the use of GUI widgets that require computing cycles, such as the Top Sessions widget. These widgets are constantly polling the system for their information, which uses CPU and other resources.
Schedule antivirus, IPS, and firmware updates during off peak hours. Usually these don’t consume CPU resources but they can disrupt normal operation.
Check the log levels and which events are being logged. This is the severity of the messages that are recorded. Consider going up one level to reduce the amount of logging. Also if there are events you do not need to monitor, remove them from the list.
Log to FortiCloud instead of memory or Disk. Logging to memory quickly uses up resources. Logging to local disk will impact overall performance and reduce the lifetime of the unit. Fortinet recommends logging to FortiCloud which doesn’t use much CPU.
If the disk is almost full, transfer the logs or data off the disk to free up space. When a disk is almost full it consumes a lot of resources to find the free space and organize the files.
If you have packet logging enabled, consider disabling it. When it’s enabled it records every packet that comes through that policy.
Halt all sniffers and traces.
Ensure you are not scanning traffic twice. If traffic enters the FortiGate unit on one interface, goes out another, and then comes back in again that traffic does not need to be rescanned. Doing so is a waste of resources. However, ensure that traffic truly is being scanned once.
Reduce the session timers to close unused sessions faster. To do this in the CLI enter the following commands and values. These values reduce the values from defaults. Note that tcp-timewait has 10 seconds added by the system by default.

config system global

set tcp-halfclose-timer 30

set tcp-halfopen-timer 30

set tcp-timewait-timer 0

set udp-idle-timer 60

end

Enable only features that you need under System > Config > Features.

When CPU usage is under control, use SNMP to monitor CPU usage. Alternately, use logging to record CPU and memory usage every 5 minutes.
Once things are back to normal, you should set up a warning system to alert you of future CPU overusage. A common method to do this is with SNMP. SNMP monitors many values on the FortiOS and allows you to set high water marks that will generate events. You run an application on your computer to watch for and record these events. Go to System > Config > SNMP to enable and configure an SNMP community. If this method is too complicated, you can use the System Resources widget to record CPU usage. However, this method will not alert you to problems - it will just record them as they happen.

How to check modem status

Sometimes the modem may not work properly, or the unit may not be detecting the modem. Use the following diagnostic commands to help you troubleshoot issues with the modem.

diagnose sys modem {cmd | com | detect | history | external-modem| query| reset}

You should always run the following diagnose command after inserting the USB modem into the unit:

diagnose sys modem detect

You can view the modem configuration by using the get system modem command. You can also view the modem’s vendor identification as well as the custom product identification number from the information output from the get system modem command.

When there are connectivity issues, use the following to help you resolve them:

diag debug enable – activates the debug on the console
diag debug application modemd – dumps communication between the modem and the unit.
diag debug application ppp – dumps the PPP negotiating messages.
execute modem dial – displays modem debug output.

The modem diagnose output should not contain any error on the way to initializing. You should also verify the number that is used to dial with your ISP.

How to run ping and traceroute

Ping and traceroute are useful tools in network troubleshooting. Alone, either one can determine network connectivity between two points. However, ping can be used to generate simple network traffic to view with diagnose commands on the FortiGate unit. This combination can be very powerful when locating network problems.

In addition to their normal uses, ping and traceroute can tell you if your computer or network device has access to a domain name server (DNS). While both tools can use IP addresses alone, they can also use domain names for devices. This is an added troubleshooting feature that can be useful in determining why particular services, such as email or web browsing, may not be working properly.

If ping does not work, you likely have it disabled on at least one of the interface settings, and security policies for that interface.

Both ping and traceroute require particular ports to be open on firewalls, or else they cannot function. Since you typically use these tools to troubleshoot, you can allow them in the security policies and on interfaces only when you need them, and otherwise keep the ports disabled for added security.

Ping

The ping command sends a very small packet to the destination, and waits for a response. The response has a timer that may expire, indicating the destination is unreachable. The behavior of ping is very much like a sonar ping from a submarine, where the command gets its name.

Ping is part of Layer-3 on the OSI Networking Model. Ping sends Internet Control Message Protocol (ICMP) “echo request” packets to the destination, and listens for “echo response” packets in reply. However, many public networks block ICMP packets because ping can be used in a denial of service (DoS) attack (such as Ping of Death or a smurf attack), or by an attacker to find active locations on the network. By default, FortiGate units have ping enabled while broadcast-forward is disabled on the external interface.

What ping can tell you

Beyond the basic connectivity information, ping can tell you the amount of packet loss (if any), how long it takes the packet to make the round trip, and the variation in that time from packet to packet.

If there is some packet loss detected, you should investigate the following:

Possible ECMP, split horizon, or network loops.
Cabling to ensure no loose connections.
Verify which security policy was used (use the packet count column on the Policy & Objects > Policy page).

If there is total packet loss, you should investigate the following:

Hardware — ensure cabling is correct, and all equipment between the two locations is accounted for.
Addresses and routes — ensure all IP addresses and routing information along the route is configured as expected.
Firewalls — ensure all firewalls, including FortiGate unit security policies allow PING to pass through.

How to use ping

Ping syntax is the same for nearly every type of system on a network.

To ping from a FortiGate unit

Connect to the CLI either through telnet or through the CLI widget on the web-based manager dashboard.
Enter exec ping 10.11.101.101 to send 5 ping packets to the destination IP address. There are no options for this command.

Sample output:

Head_Office_620b # exec ping 10.11.101.101

PING 10.11.101.101 (10.11.101.101): 56 data bytes

64 bytes from 10.11.101.101: icmp_seq=0 ttl=255 time=0.3 ms

64 bytes from 10.11.101.101: icmp_seq=1 ttl=255 time=0.2 ms

64 bytes from 10.11.101.101: icmp_seq=2 ttl=255 time=0.2 ms

64 bytes from 10.11.101.101: icmp_seq=3 ttl=255 time=0.2 ms

64 bytes from 10.11.101.101: icmp_seq=4 ttl=255 time=0.2 ms

--- 10.11.101.101 ping statistics ---

5 packets transmitted, 5 packets received, 0% packet loss

round-trip min/avg/max = 0.2/0.2/0.3 ms

To ping from an MS Windows PC

Open a command window.

In Windows XP, select Start > Run, enter cmd, and select OK.
In Windows 7, select the Start icon, enter cmd in the search box, and select cmd.exe from the list.

Enter ping 10.11.101.100 to ping the default internal interface of the FortiGate unit with four packets.

Other options include:

-t to send packets until you press “Control-C”
-a to resolve addresses to domain names where possible
-n X to send X ping packets and stop

Sample output:

C:\>ping 10.11.101.101

Pinging 10.11.101.101 with 32 bytes of data:

Reply from 10.11.101.101: bytes=32 time=10ms TTL=255

Reply from 10.11.101.101: bytes=32 time<1ms TTL=255

Reply from 10.11.101.101: bytes=32 time=1ms TTL=255

Ping statistics for 10.11.101.101:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 10ms, Average = 3ms

To ping from a Linux PC

Go to a shell prompt.
Enter “ping 10.11.101.101”.

Traceroute

Where ping will only tell you if it reached its destination and came back successfully, traceroute will show each step of its journey to its destination and how long each step takes. If ping finds an outage between two points, traceroute can be used to locate exactly where the problem is.

What is traceroute

Traceroute works by sending ICMP packets to test each hop along the route. It will send out three packets, and then increase the time to live (TTL) setting by one each time. This effectively allows the packets to go one hop farther along the route. This is the reason why most traceroute commands display their maximum hop count before they start tracing the route — that is the maximum number of steps it will take before declaring the destination unreachable. Also, the TTL setting may result in steps along the route timing out due to slow responses. There are many possible reasons for this to occur.

By default, traceroute uses UDP datagrams with destination ports numbered from 33434 to 33534. The traceroute utility usually has an option to specify use of ICMP echo request (type 8) instead, as used by the Windows tracert utility. If you have a firewall and if you want traceroute to work from both machines (Unix-like systems and Windows) you will need to allow both protocols inbound through your FortiGate security policies (UDP with ports from 33434 to 33534 and ICMP type 8).

You can also use the packet count column of the Policy & Objects > Policy page to track traceroute packets. This allows you to verify the connection, but also confirm which security policy the traceroute packets are using.

What traceroute can tell you

Ping and traceroute have similar functions—to verify connectivity between two points. The big difference is that traceroute shows you each step of the way, where ping does not. Also, ping and traceroute use different protocols and ports, so one may succeed where the other fails.

You can verify your DNS connection using traceroute. If you enter an FQDN instead of an IP address for the traceroute, DNS will try to resolve that domain name. If the name does not get resolved, you know you have DNS issues.

How to use traceroute

The traceroute command varies slightly between operating systems. Note that in MS Windows the command name is shortened to “tracert”. Also, your output will list different domain names and IP addresses along your route.

To use traceroute on an MS Windows PC

Open a command window.

In Windows XP, select Start > Run, enter cmd, and select OK.
In Windows 7, select the Start icon, enter cmd in the search box, and select cmd.exe from the list.

Enter “tracert fortinet.com” to trace the route from the PC to the Fortinet web site.

Sample output:

C:\>tracert fortinet.com

Tracing route to fortinet.com [208.70.202.225]

over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 172.20.120.2

2 66 ms 24 ms 31 ms 209-87-254-xxx.storm.ca [209.87.254.221]

3 52 ms 22 ms 18 ms core-2-g0-0-1104.storm.ca [209.87.239.129]

4 43 ms 36 ms 27 ms core-3-g0-0-1185.storm.ca [209.87.239.222]

5 46 ms 21 ms 16 ms te3-x.1156.mpd01.cogentco.com [38.104.158.69]

6 25 ms 45 ms 53 ms te8-7.mpd01.cogentco.com [154.54.27.249]

7 89 ms 70 ms 36 ms te3-x.mpd01.cogentco.com [154.54.6.206]

8 55 ms 77 ms 58 ms sl-st30-chi-.sprintlink.net [144.232.9.69]

9 53 ms 58 ms 46 ms sl-0-3-3-x.sprintlink.net [144.232.19.181]

10 82 ms 90 ms 75 ms sl-x-12-0-1.sprintlink.net [144.232.20.61]

11 122 ms 123 ms 132 ms sl-0-x-0-3.sprintlink.net [144.232.18.150]

12 129 ms 119 ms 139 ms 144.232.20.7

13 172 ms 164 ms 243 ms sl-321313-0.sprintlink.net [144.223.243.58]

14 99 ms 94 ms 93 ms 203.78.181.18

15 108 ms 102 ms 89 ms 203.78.176.2

16 98 ms 95 ms 97 ms 208.70.202.225

Trace complete.

The first, or the left column, is the hop count, which cannot go over 30 hops. When that number is reached, the traceroute ends.

The second, third, and fourth columns display how much time each of the three packets takes to reach this stage of the route. These values are in milliseconds and normally vary quite a bit. Typically a value of <1ms indicates a local connection.

The fifth, or the column farthest to the right, is the domain name of that device and its IP address or possibly just the IP address.

To perform a traceroute on a Linux PC

Go to a command line prompt.
Enter “traceroute fortinet.com”.

The Linux traceroute output is very similar to the MS Windows tracert output.

To perform a traceroute from the FortiGate

Connect to the CLI either through telnet or through the CLI widget on the web-based manager dashboard.
Enter exec traceroute www.fortinet.com to trace the route to the destination IP address. There are no options for this command.

Output appears as follows:

# execute traceroute www.fortinet.com

traceroute to www.fortinet.com (66.171.121.34), 32 hops max, 84 byte packets

1 172.20.120.2 0.637 ms 0.653 ms 0.279 ms

2 209.87.254.221 <static-209-87-254-221.storm.ca> 2.448 ms 2.519 ms 2.458 ms

3 209.87.239.129 <core-2-g0-2.storm.ca> 2.917 ms 2.828 ms 9.324 ms

4 209.87.239.199 <core-3-bdi1739.storm.ca> 13.248 ms 12.401 ms 13.009 ms

5 216.66.41.113 <v502.core1.tor1.he.net> 17.181 ms 12.422 ms 12.268 ms

6 184.105.80.9 <100ge1-2.core1.nyc4.he.net> 21.355 ms 21.518 ms 21.597 ms

7 198.32.118.41 <ny-paix-gni.twgate.net> 83.297 ms 84.416 ms 83.782 ms

8 203.160.228.217 <217-228-160-203.TWGATE-IP.twgate.net> 82.579 ms 82.187 ms 82.066 ms

9 203.160.228.229 <229-228-160-203.TWGATE-IP.twgate.net> 82.055 ms 82.455 ms 81.808 ms

10 203.78.181.2 82.262 ms 81.572 ms 82.015 ms

11 203.78.186.70 83.283 ms 83.243 ms 83.293 ms

12 66.171.127.177 84.030 ms 84.229 ms 83.550 ms

13 66.171.121.34 <www.fortinet.com> 84.023 ms 83.903 ms 84.032 ms

14 66.171.121.34 <www.fortinet.com> 83.874 ms 84.084 ms 83.810 ms

How to check the logs

This step in troubleshooting can be forgotten, but its an important one. Logging records the traffic passing through the FortiGate unit to your network and what action the FortiGate unit took during its scanning process of the traffic. This recorded information is called a log message.

When you configure FortiOS initially, log as much information as you can. If needed, logging of unused features can be turned off or scaled back if the logs generated are too large.

As with most troubleshooting steps, before you can determine if the logs indicate a problem, you need to know what logs result from normal operation. Without a baseline it is difficult to properly troubleshoot.

When troubleshooting with log files:

Compare current logs to a recorded baseline of normal operation.
If needed increase the level of logging (such as from Warning to Information) to obtain more information.

When increasing logging levels, ensure that alert email is configured and both disk usage and log quota are selected. This ensures you will be notified if the increased logging causes problems. You can also use Logging Monitor (located in Log&Report > Monitor > Logging volume Monitor) to determine the activities that generate the most log entries.

check all logs to ensure important information is not overlooked
filter or order log entries based on different fields (such as level, service, or IP address) to look for patterns that may indicate a specific problem (such as frequent blocked connections on a specific port for all IP addresses)

Logs will help identify and locate any problems, but they will not solve the problems. The job of logs is to speed up your problem solving and save you time and effort.

For more information on Logging and Log Reports, see the Logging and Reporting handbook chapter.

How to verify the contents of the routing table (in NAT mode)

When you have some connectivity, or possibly none at all a good place to look for information is the routing table.

The routing table is where all the currently used routes are stored for both static and dynamic protocols. If a route is in the routing table, it saves the time and resources of a lookup. If a route is not used for a while and a new route needs to be added, the oldest least used route is bumped if the routing table is full. This ensures the most recently used routes stay in the table. If your FortiGate unit is in Transparent mode, you are unable to perform this step.

If the FortiGate is running in NAT mode, verify that all desired routes are in the routing table: local subnets, default routes, specific static routes, and dynamic routing protocols.

To check the routing table in the web-based manager, use the Routing Monitor by going to Router > Monitor > Routing Monitor.

In the CLI, use the command get router info routing-table all. Sample output:

FGT# get router info routing-table all

Codes:

K - kernel, C - connected, S - static, R - RIP, B - BGP

O - OSPF, IA - OSPF inter area

N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

E1 - OSPF external type 1, E2 - OSPF external type 2

i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area

* - candidate default

S* 0.0.0.0/0 [10/0] via 172.20.120.2, wan1

C 10.31.101.0/24 is directly connected, internal

C 172.20.120.0/24 is directly connected, wan1

How to verify the correct route is being used

If you have more than one default route and wants to make sure that traffic is flowing as expected via the right route, you can run a trace route from a machine in the local area network, this will indicate you the first hop that the traffic goes through.

Sample output:

C:\>tracert www.fortinet.com

Tracing route to www.fortinet.com [66.171.121.34]

over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 10.10.1.99

2 1 ms <1 ms <1 ms 172.20.120.2

3 3 ms 3 ms 3 ms static-209-87-254-221.storm.ca [209.87.254.221]

4 3 ms 3 ms 3 ms core-2-g0-2.storm.ca [209.87.239.129]

5 13 ms 13 ms 13 ms core-3-bdi1739.storm.ca [209.87.239.199]

6 12 ms 19 ms 11 ms v502.core1.tor1.he.net [216.66.41.113]

7 22 ms 22 ms 21 ms 100ge1-2.core1.nyc4.he.net [184.105.80.9]

8 84 ms 84 ms 84 ms ny-paix-gni.twgate.net [198.32.118.41]

9 82 ms 84 ms 82 ms 217-228-160-203.TWGATE-IP.twgate.net [203.160.22

8.217]

10 82 ms 81 ms 82 ms 229-228-160-203.TWGATE-IP.twgate.net [203.160.22

8.229]

11 82 ms 82 ms 82 ms 203.78.181.2

12 84 ms 83 ms 83 ms 203.78.186.70

13 84 ms * 85 ms 66.171.127.177

14 84 ms 84 ms 84 ms fortinet.com [66.171.121.34]

15 84 ms 84 ms 83 ms fortinet.com [66.171.121.34]

Trace complete.

In this scenario, the first hop contains the IP address 10.10.1.99, which is the internal interface of the FortiGate. The second hop contains the IP address 172.20.120.2, to which the wan1 interface of the FortiGate is connected, so we can conclude that the route via wan1 interface is being used for this traffic.

Also debug the packet flow in the CLI shows the route taken for each session.

Sample output:

id=20085 trace_id=319 func=vf_ip4_route_input line=1597 msg="find a route: gw-172.20.120.2 via wan1"

For more information on debuging the packet flow, see How to debug the packet flow.

How to verify the correct firewall policy is being used

If you have more than one firewall policy, use the count column to check which policy is being used, the count must show traffic increasing. To do so, go to Policy & Objects > Policy page.

Also debuging the packet flow in the CLI shows the policy id allowing the traffic.

Sample output:

id=13 trace_id=1 func=fw_forward_handler line=650 msg="Allowed by Policy-14: SNAT"

For more information on debuging the packet flow, see How to debug the packet flow.

How to check the bridging information in Transparent mode

When FortiOS is in Transparent mode, the unit acts like a bridge sending all incoming traffic out on the other interfaces. The bridge is between interfaces on the FortiGate unit.

Each bridge listed is a link between interfaces. Where traffic is flowing between interfaces, you expect to find bridges listed. If you are having connectivity issues, and there are no bridges listed, that is a likely cause. Check for the MAC address of the interface or device in question.

How to check the bridging information

To list the existing bridge instances on the FortiGate unit, use the following command:

diagnose netlink brctl list

Sample output:

#diagnose netlink brctl list

list bridge information

1. root.b fdb: size=256 used=6 num=7 depth=2 simple=no

Total 1 bridges

How to display forwarding domain information

Forwarding domains, or collision domains, are used in routing to limit where packets are forwarded on the network. Layer-2 broadcasts are limited to the same group. By default, all interfaces are in group 0. For example, if the FortiGate unit has 12 interfaces, only two may be in the same forwarding domain, which will limit packets that are broadcast to only those two interfaces. This reduces traffic on the rest of the network.

Collision domains prevent the forwarding of ARP packets to all VLANs on an interface. Without collision domains, duplicate MAC addresses on VLANs may cause ARP packets to be duplicated. Duplicate ARP packets can cause some switches to reset. It is important to know what interfaces are part of which forwarding domains as this determines which interfaces can communicate with each other.

To manually configure forwarding domains in Transparent mode, use the following FortiOS CLI command:

config system interface

edit <interface_name>

set forward-domain <integer>

end

To display the information for forward domains

Use the following command:

diagnose netlink brctl domain <name> <id>

where <name> is the name of the forwarding domain to display and <id> is the domain id.

Sample output

diagnose netlink brctl domain ione 101

show bridge root.b ione forward domain.

id=101 dev=trunk_1 6

To list the existing bridge MAC table, use the following command:

diagnose netlink brctl name host <name>

Sample output

show bridge control interface root.b host.

fdb: size=256, used=6, num=7, depth=2, simple=no

Bridge root.b host table

`port no`	`device`	`devname`	`mac addr`	`ttl`	`attributes`
`2`	`7`	`wan2`	`02:09:0f:78:69:00`	`0`	`Local Static`
`5`	`6`	`vlan_1`	`02:09:0f:78:69:01`	`0`	`Local Static`
`3`	`8`	`dmz`	`02:09:0f:78:69:01`	`0`	`Local Static`
`4`	`9`	`internal`	`02:09:0f:78:69:02`	`0`	`Local Static`
`3`	`8`	`dmz`	`00:80:c8:39:87:5a`	`194`
`4`	`9`	`internal`	`02:09:0f:78:67:68`	`8`
`1`	`3`	`wan1`	`00:09:0f:78:69:fe`	`0`	`Local Static`

To list the existing bridge port list, use this command:

diagnose netlink brctl name port <name>

Sample Output:

show bridge root.b data port.

trunk_1 peer_dev=0

internal peer_dev=0

dmz peer_dev=0

wan2 peer_dev=0

wan1 peer_dev=0

How to check number of sessions used by UTM proxy

Each FortiGate model has a set limit of the maximum number of sessions the UTM proxy supports. The UTM proxy handles all the traffic for the following protocols: HTTP, SMTP, POP3, IMAP, FTP, and NNTP. If the proxy for a protocol fills up its session table, the FortiGate unit will enter conserve mode, where it behaves differently, until entries and memory free up again.

Conserve or failopen mode

Once you reach the limit, depending on your FortiGate unit’s conserve mode configuration, no new sessions are created until an old ones end. You can configure your FortiGate unit’s behavior when memory is running low or the proxy connection limit has been reached. There are two related commands for this in the CLI:

config system global

set av-failopen-session {enable | disable}

set av-failopen { idledrop | off | one-shot | pass}

end

av-failopen-session must be enabled to set the behavior for these conditions. When it is enabled, and a proxy for a protocol runs out of room in its session table that protocol goes into failopen mode and behaves as defined in the av-failopen command.

av-failopen determines the behavior of the proxy until entries are free in the session table again for that proxy.

idledrop — This option removes idle sessions from the session table, starting with the clients that have the most sessions currently open. This method assumes that idle sessions are not being used and it will not cause problems to close these sessions. This is usually true, but some applications may have problems with this and start complaining about either not having or being able to open a session. If this occurs, try another method to check if this is really the problem. This is a secure option as no unscanned traffic is allowed to pass.
off — This option turns off accepting any new AV sessions, but will continue to process any existing AV sessions that are currently active. All the protocols listed (HTTP, SMTP, POP3, IMAP, FTP, and NNTP) are scanned by FortiGate Antivirus. If AV scanning is enabled, av-failopen off is selected, and the proxy session table fills up, then no new sessions of that type will be accepted. For example, if POP3 session table is filled and email AV scanning is enabled, no more POP3 connections will be allowed until the session table gets some free space. This is a secure option because no unscanned traffic is allowed to pass.
one-shot — When memory is low, bypass the antivirus system. The name one-shot comes from the fact that once you are in one-shot av-failopen mode, you must set av-failopen to either pass or off to restart AV scanning. This is a very unsecure option because it allows all traffic without AV scanning, and it never reverts to normal without manual assistance.
pass — When memory is low, bypass the antivirus system much as one-shot. The difference is that when memory is freed up, the system will start AV scanning automatically again. This is an unsecure option because it allows traffic to pass without AV scanning. However, it is better than one-shot because it automatically restarts AV scanning when possible.

If the proxy session table is full for one or more protocols and your FortiGate unit enters into conserve or failopen mode, it will appear as if you have lost connections, network services are intermittent or non-existent, and yet other services work normally for a while until their sessions end and they join the queue of session-starved applications.

Checking sessions in use

To make troubleshooting this type of problem easier, sessions are broken down by which protocol they use. This provides you with statistics and errors specific to one of the protocols.

Due to the amount of output from this command, you should connect to the CLI with a terminal program, such as puTTY, that logs output. Otherwise, you will likely not be able to access all the output information from the command.

In the following output, only the HTTP entries are displayed. The other protocols have been removed in an attempt to shorten the output. There will be separate entries for each supported protocol (HTTP, SMTP, POP3, IMAP, FTP, and NNTP) in each section of the output.

To check sessions in use and related errors - CLI

FGT# # get test proxyworker 4

Worker[0]

HTTP Common

Current Connections 8/8032

Max Concurrent Connections 76

Worker Stat

Running time (HH:MM:SS:usec) 29:06:27:369365

Time in loop scanning 2:08:000198

Error Count (accept) 0

Error Count (read) 0

Error Count (write) 0

Error Count (poll) 0

Error Count (alloc) 0

Last Error 0

Acceptor Read 6386

Acceptor Write 19621

Acceptor Close 0

HTTP Stat

Bytes sent 667012 (kb)

Bytes received 680347 (kb)

Error Count (alloc) 0

Error Count (accept) 0

Error Count (bind) 0

Error Count (connect) 0

Error Count (socket) 0

Error Count (read) 134

Error Count (write) 0

Error Count (retry) 40

Error Count (poll) 0

Error Count (scan reset) 2

Error Count (urlfilter wait) 3

Last Error 104

Web responses clean 17950

Web responses scan errors 23

Web responses detected 16

Web responses infected with worms 0

Web responses infected with viruses 0

Web responses infected with susp 0

Web responses file blocked 0

Web responses file exempt 0

Web responses bannedword detected 0

Web requests oversize pass 16

Web requests oversize block 0

Last Server Scan errors 102

URL requests exempt 0

URL requests blocked 0

URL requests passed 0

URL requests submit error 0

URL requests rating error 0

URL requests rating block 0

URL requests rating allow 10025

URL requests infected with worms 0

Web requests detected 0

Web requests file blocked 0

Web requests file exempt 0

POST requests clean 512

POST requests scan errors 0

POST requests infected with viruses 0

POST requests infected with susp 0

POST requests file blocked 0

POST requests bannedword detected 0

POST requests oversize pass 0

POST requests oversize block 0

Web request backlog drop 0

Web response backlog drop 0

Worker Accounting

poll=721392/649809/42 pollfail=0 cmdb=85 scan=19266 acceptor=25975

HTTP Accounting

setup_ok=8316 setup_fail=0 conn_ok=0 conn_inp=8316

urlfilter=16553/21491/20 uf_lookupf=0

scan=23786 clt=278876 srv=368557

SMTP Accounting

setup_ok=12 setup_fail=0 conn_ok=0 conn_inp=12

scan=12 suspend=0 resume=0 reject=0 spamadd=0 spamdel=0 clt=275 srv=279

POP3 Accounting

setup_ok=30 setup_fail=0 conn_ok=0 conn_inp=30

scan=3 clt=5690 srv=5836

IMAP Accounting

setup_ok=0 setup_fail=0 conn_ok=0 conn_inp=0

scan=0 clt=0 srv=0

FTP Accounting

setup_ok=0 setup_fail=0 conn_ok=0 conn_inp=0

scan=0 clt=0 srv=0 datalisten=0 dataclt=0 datasrv=0

NNTP Accounting

setup_ok=0 setup_fail=0 conn_ok=0 conn_inp=0

scan=0 clt=0 srv=0

The output from this command falls into the following sections:

HTTP Common current connections — There is an entry for each protocol that displays the connections currently used, and the maximum connections allowed. This maximum is for the UTM proxy, which means all the protocols connections combined cannot be larger than this number. To support this, note that the maximum session count for each protocol is the same. You may also see a line titled Max Concurrent Connections for each protocol. This number is the maximum connections of this type allowed at one time. If VDOMs are enabled, this value is defined either on the global or per-VDOM level at VDOM > Global Resources.
Worker Stat — This is statistics about the UTM proxy including how long it has been running, and how many errors it has found.
HTTP Stat — This section includes statistics about the HTTP protocol proxy. This is a very extensive list covering errors, web responses, and any UTM positive matches. There are similar sections for each protocol, but the specific entries in each vary based on what UTM scanning is looking for in each — spam control for email, file transfer blocking for FTP, and so on.
Worker Accounting — Lists accounting information about the UTM proxy such as polling statistics, how many sessions were scanned, and how many were just accepted. This information can tell you if expect AV scanning is taking place or not. Under normal operation there should be no errors or fails.
HTTP Accounting — The accounting sections for each protocol provide information about successful session creation, failures, how many sessions are being scanned or filtered, and how many are client or server originated. If setup_fail is larger than zero, run the command again to see if it is increasing quickly. If it is, your FortiGate unit may be in conserve mode.

Related commands

To dump memory usage:

# get test proxyworker 1

To display statistics per VDOM:

# get test proxyworker 4444

To restart the proxy:

# get test proxyworker 99

How to examine the firewall session list

One further step is to examine the firewall session. The firewall session list displays all the sessions the FortiGate unit has open. You will be able to see if there are strange patterns such as no sessions apart from the internal network, or all sessions are only to one IP address.

When examining the firewall session list in the CLI, filters may be used to reduce the output. In the web-based manager, the filters are part of the interface.

To examine the firewall session list - web-based manager

Go to System > FortiView> All Sessions.

To examine the firewall session list - CLI

When examining the firewall session list, there may be too many sessions to display. In this case it will be necessary to limit or filter the sessions displayed by source or destination address, or NATed address or port. If you want to filter by more than one of these, you need to enter a separate line for each value.

The following example shows filtering the session list based on a source address of 10.11.101.112.

FGT# diag sys session filter src 10.11.101.112

FGT# diag sys session list

The following example shows filtering the session list based on a destination address of 172.20.120.222.

FGT# diag sys session filter dst 172.20.120.222

FGT# diag sys session list

To clear all sessions corresponding to a filter - CLI

FGT# diag sys session filter dst 172.20.120.222

FGT# diag sys session clear

Check source NAT information

Remember NAT when troubleshooting connections. NAT is especially important if you are troubleshooting from the remote end of the connection outside the FortiGate unit firewall. On the dashboard session list, pay attention to Src address after NAT, and Src port after NAT. These columns display the IP and port values after NAT has been applied.

The NAT values can be helpful to ensure they are the values you expect, and to ensure the remote end of the sessions can see the expected IP address and port number.

When displaying the session list in the CLI, you can match the NATed source address (nsrc) and port (nport). This can be useful if multiple internal IP addresses are NATed to a common external facing source IP address.

FGT# diag sys session filter nsrc 172.20.120.122

FGT# diag sys session filter nport 8888

FGT# diag sys session list

How to check wireless information

Wireless connections, stations, and interfaces have different issues than other physical interfaces.

Troubleshooting station connection issue

To check whether station entry is created on Access Control:

FG600B3909600253 # diagnose wireless-controller wlac -d sta

* vf=0 wtp=70 rId=2 wlan=open ip=0.0.0.0 mac=00:09:0f:db:c4:03 rssi=0 idle=148 bw=0 use=2

vf=0 wtp=70 rId=2 wlan=open ip=172.30.32.122 mac=00:25:9c:e0:47:88 rssi=-40 idle=0 bw=9 use=2

Enable diagnostic for particular station

This example uses the station MAC address to find where it is failing:

FG600B3909600253 # diagnose wireless-controller wlac sta_filter 00:25:9c:e0:47:88 1

Set filter sta 00:25:9c:e0:47:88 level 1

FG600B3909600253 # 71419.245 <ih> IEEE 802.11 mgmt::disassoc <== 00:25:9c:e0:47:88 vap open rId 1 wId 0 00:09:0f:db:c4:03

71419.246 <dc> STA del 00:25:9c:e0:47:88 vap open rId 1 wId 0

71419.246 <cc> STA_CFG_REQ(34) sta 00:25:9c:e0:47:88 del ==> ws (0-192.168.35.1:5246) rId 1 wId 0

71419.246 <cc> STA del 00:25:9c:e0:47:88 vap open ws (0-192.168.35.1:5246) rId 1 wId 0 00:09:0f:db:c4:03 sec open reason I2C_STA_DEL

71419.247 <cc> STA_CFG_RESP(34) 00:25:9c:e0:47:88 <== ws (0-192.168.35.1:5246) rc 0 (Success).

How to verify FortiGuard connectivity

You can verify the FortiGuard connectivity in the License Information widget under System > Dashboard > Status. When FortiGate is connected to FortiGuard, a green check mark appears for available FortiGuard services.

From CLI, execute ping “service.fortiguard.net” and “update.fortiguard.net”.

Sample output:

FG100D# execute ping service.fortiguard.net

PING guard.fortinet.net (208.91.112.196): 56 data bytes

64 bytes from 208.91.112.196: icmp_seq=0 ttl=51 time=61.0 ms

64 bytes from 208.91.112.196: icmp_seq=1 ttl=51 time=60.0 ms

64 bytes from 208.91.112.196: icmp_seq=2 ttl=51 time=59.6 ms

64 bytes from 208.91.112.196: icmp_seq=3 ttl=51 time=58.9 ms

64 bytes from 208.91.112.196: icmp_seq=4 ttl=51 time=59.2 ms

--- guard.fortinet.net ping statistics ---

5 packets transmitted, 5 packets received, 0% packet loss

round-trip min/avg/max = 58.9/59.7/61.0 ms

FG100D# execute ping update.fortiguard.net

PING fds1.fortinet.com (208.91.112.68): 56 data bytes

64 bytes from 208.91.112.68: icmp_seq=0 ttl=53 time=62.0 ms

64 bytes from 208.91.112.68: icmp_seq=1 ttl=53 time=61.8 ms

64 bytes from 208.91.112.68: icmp_seq=2 ttl=53 time=61.3 ms

64 bytes from 208.91.112.68: icmp_seq=3 ttl=53 time=61.9 ms

64 bytes from 208.91.112.68: icmp_seq=4 ttl=53 time=61.8 ms

How to perform a sniffer trace (CLI and Packet Capture)

When troubleshooting networks and routing in particular, it helps to look inside the headers of packets to determine if they are traveling along the expected route. Packet sniffing can also be called a network tap, packet capture, or logic analyzing.

If your FortiGate unit has NP2/NP4 interfaces that are offloading traffic, this will change the sniffer trace. Before performing a trace on any NP2/NP4 interfaces, you should disable offloading on those interfaces.

What can sniffing packets tell you

If you are running a constant traffic application such as ping, packet sniffing can tell you if the traffic is reaching the destination, what the port of entry is on the FortiGate unit, if the ARP resolution is correct, and if the traffic is being sent back to the source as expected.

Sniffing packets can also tell you if the FortiGate unit is silently dropping packets for reasons such as Reverse Path Forwarding (RPF), also called Anti Spoofing, which prevents an IP packet from being forwarded if its Source IP does not either belong to a locally attached subnet (local interface), or be part of the routing between the FortiGate unit and another source (static route, RIP, OSPF, BGP). Note that RPF can be disabled by turning on asymmetric routing in the CLI (config system setting, set asymetric enable), however this will disable stateful inspection on the FortiGate unit and cause many features to be turned off.

If you configure virtual IP addresses on your FortiGate unit, it will use those addresses in preference to the physical IP addresses. You will notice this when you are sniffing packets because all the traffic will be using the virtual IP addresses. This is due to the ARP update that is sent out when the VIP address is configured.

How do you sniff packets

The general form of the internal FortiOS packet sniffer command is:

diag sniffer packet <interface_name> <‘filter’> <verbose> <count>

To stop the sniffer, type CTRL+C.

<interface_name>	The name of the interface to sniff, such as “port1” or “internal”. This can also be “any” to sniff all interfaces.
<‘filter’>	What to look for in the information the sniffer reads. “none” indicates no filtering, and all packets will be displayed as the other arguments indicate. The filter must be inside single quotes (‘).
<verbose>	The level of verbosity as one of: 1 - print header of packets 2 - print header and data from IP of packets 3 - print header and data from Ethernet of packets 4 - print header of packets with interface name
<count>	The number of packets the sniffer reads before stopping. If you do not put a number here, the sniffer will run forever unit you stop it with <CTRL C>.

For a simple sniffing example, enter the CLI command diag sniffer packet port1 none 1 3. This will display the next three packets on the port1 interface using no filtering, and using verbose level 1. At this verbosity level you can see the source IP and port, the destination IP and port, action (such as ack), and sequence numbers.

In the output below, port 443 indicates these are HTTPS packets, and 172.20.120.17 is both sending and receiving traffic.

Head_Office_620b # diag sniffer packet port1 none 1 3

interfaces=[port1]

filters=[none]

0.545306 172.20.120.17.52989 -> 172.20.120.141.443: psh 3177924955 ack 1854307757

0.545963 172.20.120.141.443 -> 172.20.120.17.52989: psh 1854307757 ack 3177925808

0.562409 172.20.120.17.52988 -> 172.20.120.141.443: psh 4225311614 ack 3314279933

For a more advanced example of packet sniffing, the following commands will report packets on any interface travelling between a computer with the host name of “PC1” and the computer with the host name of “PC2”. With verbosity 4 and above, the sniffer trace will display the interface names where traffic enters or leaves the FortiGate unit. Remember to stop the sniffer, type CTRL+C.

FGT# diagnose sniffer packet any "host <PC1> or host <PC2>" 4

FGT# diagnose sniffer packet any "(host <PC1> or host <PC2>) and icmp" 4

The following sniffer CLI command includes the ARP protocol in the filter which may be useful to troubleshoot a failure in the ARP resolution (for instance PC2 may be down and not responding to the FortiGate ARP requests).

FGT# diagnose sniffer packet any "host <PC1> or host <PC2> or arp" 4

Packet Capture

When troubleshooting networks, it helps to look inside the header of the packets. This helps to determine if the packets, route, and destination are all what you expect. Packet capture can also be called a network tap, packet sniffing, or logic analyzing.

To use the packet capture:

Go to System > Network > Packet Capture.
Select the interface to monitor and select the number of packets to keep.
Select Enable Filters.
Enter the information you want to gather from the packet capture.
Select OK.

To run the capture, select the play button in the progress column in the packet capture list. If not active, Not Running will also appear in the column cell. The progress bar will indicate the status of the capture. You can stop and restart it at any time.

When the capture is complete, click the Download icon to save the packet capture file to your hard disk for further analysis.

Packet capture tells you what is happening on the network at a low level. This can be very useful for troubleshooting problems, such as:

Finding missing traffic.
Seeing if sessions are setting up properly.
Locating ARP problems such as broadcast storm sources and causes.
Confirming which address a computer is using on the network if they have multiple addresses or are on multiple networks.
Confirming routing is working as you expect.
Wireless client connection problems.
Intermittent missing PING packets.
A particular type of packet is having problems, such as UDP, which is commonly used for streaming video.

If you are running a constant traffic application such as ping, packet capture can tell you if the traffic is reaching the destination, how the port enters and exits the FortiGate unit, if the ARP resolution is correct, and if the traffic is returning to the source as expected. You can also use packet switching to verify that NAT or other configuration is translating addresses or routing traffic the way that you want it to.

Before you start capturing packets, you need to have a good idea of what you are looking for. Capture is used to confirm or deny your ideas about what is happening on the network. If you try capture without a plan to narrow your search, you could end up with too much data to effectively analyze. On the other hand, you need to capture enough packets to really understand all of the patterns and behavior that you are looking for.

How to debug the packet flow

Traffic should come in and leave the FortiGate unit. If you have determined that network traffic is not entering and leaving the FortiGate unit as expected, debug the packet flow.

Debugging can only be performed using CLI commands. Debugging the packet flow requires a number of debug commands to be entered as each one configures part of the debug action, with the final command starting the debug.

If your FortiGate unit has FortiASIC NP4 interface pairs that are offloading traffic, this will change the packet flow. Before performing the debug on any NP4 interfaces, you should disable offloading on those interfaces.

The following configuration assumes that PC1 is connected to the internal interface of the FortiGate unit and has an IP address of 10.11.101.200. PC1 is the host name of the computer.

To debug the packet flow in the CLI, enter the following commands:

FGT# diag debug disable

FGT# diag debug flow filter add <PC1>

FGT# diag debug flow show console enable

FGT# diag debug flow show function-name enable

FGT# diag debug flow trace start 100

FGT# diag debug enable

The start 100 argument in the above list of commands will limit the output to 100 packets from the flow. This is useful for looking at the flow without flooding your log or displaying too much information.

To stop all other debug activities, enter the command:

FGT# diag debug flow trace stop

The following is an example of debug flow output for traffic that has no matching security policy, and is in turn blocked by the FortiGate unit. The denied message indicates that the traffic was blocked.

id=20085 trace_id=319 func=resolve_ip_tuple_fast line=2825 msg="vd-root received a packet(proto=6, 192.168.129.136:2854->192.168.96.153:1863) from port3."

id=20085 trace_id=319 func=resolve_ip_tuple line=2924 msg="allocate a new session-013004ac"

id=20085 trace_id=319 func=vf_ip4_route_input line=1597 msg="find a route: gw-192.168.150.129 via port1"

id=20085 trace_id=319 func=fw_forward_handler line=248 msg=" Denied by forward policy check"