FortiOS 5.4 Online Help Link FortiOS 5.2 Online Help Link FortiOS 5.0 Online Help Link FortiOS 4.3 Online Help Link

Home

Video

FortiGuard

Fuse

KB

Support

  • All Files

Home > Online Help

> Chapter 26 - Troubleshooting > Troubleshooting tools

Troubleshooting tools

FortiOS provides a number of tools that help with troubleshooting both hardware and software issues. These tools include diagnostics and ports; ports are used when you need to understand the traffic coming in or going out on a specific port, for example, UDP 53, which is used by the FortiGate unit for DNS lookup and RBL lookup.

This section also contains information about troubleshooting FortiGuard issues.

This section contains the following topics:

FortiOS diagnostics

A collection of diagnostic commands are available in FortiOS for troubleshooting and performance monitoring. Within the CLI commands, the two main groups of diagnostic commands are get and diagnose commands. Both commands display information about system resources, connections, and settings that enable you to locate and fix problems, or to monitor system performance.

This topic includes diagnostics commands to help with:

Check date and time

The system date and time are important for FortiGuard services, when logging events, and when sending alerts. The wrong time will make the log entries confusing and difficult to use.

Use Network Time Protocol (NTP) to set the date and time if possible. This is an automatic method that does not require manual intervention. However, you must ensure the port is allowed through the firewalls on your network. FortiToken synchronization requires NTP in many situations.

How to check the date and time - web-based manager
  1. Go to System Information > System Time on the dashboard.

Alternately, you can check the date and time using the CLI commands execute date and execute time.

  1. If required, select Change to adjust the date and time settings.

You can set the time zone, date and time, and select NTP usage. In the CLI, use the following commands to change the date and time:

config system global

set timezone (use ? to get a list of IDs and descriptions of their timezone)

end

config system ntp

set type custom

config ntpserver

edit 1

set server “ntp1.fortinet.net”

next

edit 2

set server “ntp2.fortinet.net”

next

end

set ntpsync enable

set syncinterval 60

end

Resource usage

Each program running on a computer has one or more processes associated with it. For example if you open a Telnet program, it will have an associated telnet process. The same is true in FortiOS. All the processes have to share the system resources in FortiOS including memory and CPU.

Use get system performance status command to show the FortiOS performance status.

Sample output:

FGT#get system performance status

CPU states: 0% user 0% system 0% nice 100% idle

CPU0 states: 0% user 0% system 0% nice 100% idle

CPU1 states: 0% user 0% system 0% nice 100% idle

CPU2 states: 0% user 0% system 0% nice 100% idle

CPU3 states: 0% user 0% system 0% nice 100% idle

Memory states: 25% used

Average network usage: 0 kbps in 1 minute, 0 kbps in 10 minutes, 0 kbps in 30 minutes

Average sessions: 5 sessions in 1 minute, 5 sessions in 10 minutes, 4 sessions in 30 minutes

Average session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes

Virus caught: 0 total in 1 minute

IPS attacks blocked: 0 total in 1 minute

Uptime: 0 days, 12 hours, 7 minutes

 

Monitor the CPU/memory usage of internal processes using the following command:

get system performance top <delay> <max_lines>

 

The data listed by the command includes the name of the daemon, the process ID, whether the process is sleeping or running, the CPU percentage being used, and the memory percentage being used.

Sample output:

  FGT#get system performance top 10 100
Run Time: 0 days, 11 hours and 30 minutes
0U, 0S, 100I; 1977T, 1470F, 121KF
  pyfcgid
pyfcgid
pyfcgid
pyfcgid
ipsengine
ipsengine
ipsengine
ipsengine
ipsengine
ipsengine
cmdbsvr
proxyworker
proxyworker
httpsd
httpsd
httpsd
newcli
newcli
fgfmd
iked
120
121
122
53
75
66
73
74
79
80
43
110
111
125
52
124
141
128
102
86 		S
S
S
S
S <
S <
S <
S <
S <
S <
S
S
S
S
S
S
R
S
S
S 		0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0 		1.3
1.3
1.3
1.3
1.3
1.3
1.3
1.3
1.3
1.3
1.0
1.0
1.0
0.8
0.8
0.8
0.7
0.7
0.7
0.7

Proxy operation

Monitor proxy operations using the following command:

diag test application <application> <option>

The <application> value can include the following:

http HTTP proxy.
smtp SMTP proxy.
ftpd FTP proxy.
pop3 POP3 proxy.
imap IMAP proxy.
nntp NNTP proxy.
proxyacceptor Proxy acceptor.
proxyworker Proxy worker.
scanunit Scanning unit.
sslacceptor SSL proxy.
sslworker SSL proxy.
ssh SSH proxy.
harelay HA relay daemon.
hasync HA sync daemon.
forticldd FortiCloud daemon.
miglogd Miglog logging daemon.
urlfilter URL filter daemon.
ovrd Override daemon.
ipsmonitor ips monitor
ipsengine ips sensor
ipldbd IP load balancing daemon.
ddnscd DDNS client daemon.
snmpd SNMP daemon.
acd Aggregate Controller.
dnsproxy DNS proxy.
sflowd sFlow daemon.
init init process.
l2tpcd L2TP client daemon.
dhcprelay DHCP relay daemon.
pptpcd PPTP client.
wccpd WCCP daemon.
wad WAD related processes.
radiusd RADIUS daemon.
sqldb SQL database daemon.
reportd Report daemon.
dlpfingerprint DLP fingerprint daemon.
dlpfpcache DLP fingerprint cache daemon.
wpad WPA daemon.
fsd FortiExplorer daemon.
ipsufd IPS urlfilter daemon.
stp Spanning Tree Protocol daemon.
lted USB LTE daemon.
swctrl_authd Switch controller authentication daemon.
forticron Forticron daemon.
uploadd Upload daemon.
quarantined Quarantine daemon.
dhcp6c DHCP6 client daemon.
info-sslvpnd SSL-VPN info daemon.
thmd Traffic history monitor daemon.
dsd DLP Statistics daemon.
lnkmtd Link monitor daemon.
dhcp6r DHCP6 relay daemon.
fnbamd Fortigate non-blocking auth daemon.

The <option> value depends from the application value used in the command. Here are some examples:

  • If the application is http, the CLI command will be:

diag test application http <option>

 

The <option> value can be one from the following:

2 Drop all connections.
22 Drop max idle connections.
222 Drop all idle connections.
4 Display connection stat.
44 Display info per connection.
444 Display connections per state.
4444 Display per-VDOM statistics.
44444 Display information about idle connections.
55 Display tcp info per connection.
6 Display ICAP information.
70 Disable ICAP 'Allow: 204' (default).
71 Enable ICAP 'Allow: 204' .
72 Drop all ICAP server connections.
11 Display the SSL session ID cache statistics.
12 Clear the SSL session ID cache statistics.
13 Display the SSL session ID cache.
14 Clear the SSL session ID cache.
80 Show Fortinet bar SSL-VPN bookmark info.
81 Show Fortinet bar SSL-VPN bookmark cache.
82 Show Fortinet bar SSL-VPN bookmark LRU list.
  • If the application is ipsmonitor, the CLI command will be

diag test application ipsmonitor <option>

 

The <option> value can be one from the following:

1 Display IPS engine information
2 Toggle IPS engine enable/disable status
3 Display restart log
4 Clear restart log
5 Toggle bypass status
6 Submit attack characteristics now
10 IPS queue length
11 Clear IPS queue length
12 IPS L7 socket statistics
13 IPS session list
14 IPS NTurbo statistics
15 IPSA statistics
97 Start all IPS engines
98 Stop all IPS engines
99 Restart all IPS engines and monitor

Hardware NIC

Monitor hardware network operations using the following command:

diag hardware deviceinfo nic <interface>

 

The information displayed by this command is important as errors at the interface are indicative of data link or physical layer issues which may impact the performance of the FortiGate unit.

The following is sample output when <interface> = internal:

System_Device_Name port5

Current_HWaddr 00:09:0f:68:35:60

Permanent_HWaddr 00:09:0f:68:35:60

Link up

Speed 100

Duplex full

[……]

Rx_Packets=5685708

Tx_Packets=4107073

Rx_Bytes=617908014

Tx_Bytes=1269751248

Rx_Errors=0

Tx_Errors=0

Rx_Dropped=0

Tx_Dropped=0

[…..]

 

The diag hardware deviceinfo nic command displays a list of hardware related error names and values. The following table explains the items in the list and their meanings.

Possible hardware errors and meanings
Field Definition
Rx_Errors = rx error count Bad frame was marked as error by PHY.
Rx_CRC_Errors +
Rx_Length_Errors -
Rx_Align_Errors		 This error is only valid in 10/100M mode.
Rx_Dropped or
Rx_No_Buffer_Count		 Running out of buffer space.
Rx_Missed_Errors Equals Rx_FIFO_Errors + CEXTERR (Carrier Extension Error Count). Only valid in 1000M mode, whichis marked by PHY.
Tx_Errors = Tx_Aborted_Errors ECOL (Excessive Collisions Count). Only valid in half-duplex mode.
Tx_Window_Errors LATECOL (Late Collisions Count). Late collisions are collisions that occur after 64-byte time into the transmission of the packet while working in 10 to100Mb/s data rate and 512-byte timeinto the transmission of the packet while working in the 1000Mb/s data rate. This register only increments if transmits are enabled and the device is in half-duplex mode.
Rx_Dropped See Rx_Errors.
Tx_Dropped Not defined.
Collisions Total number of collisions experienced by the transmitter. Valid in half-duplex mode.
Rx_Length_Errors Transmission length error.
Rx_Over_Errors Not defined.
Rx_CRC_Errors Frame CRC error.
Rx_Frame_Errors Same as Rx_Align_Errors. This error is only valid in 10/100M mode.
Rx_FIFO_Errors Same as Rx_Missed_Errors - a missed packet count.
Tx_Aborted_Errors See Tx_Errors.
Tx_Carrier_Errors The PHY should assert the internal carrier sense signal during every transmission. Failure to do so may indicate that the link has failed or the PHY has an incorrect link configuration. This register only increments if transmits are enabled. This register is not valid in internal SerDes 1 mode (TBI mode for the 82544GC/EI) and is only valid when the Ethernet controller is operating at full duplex.
Tx_FIFO_Errors Not defined.
Tx_Heartbeat_Errors Not defined.
Tx_Window_Errors See LATECOL.
Tx_Single_Collision_Frames Counts the number of times that a successfully transmitted packed encountered a single collision. The value only increments if transmits are enabled and the Ethernet controller is in half-duplex mode.
Tx_Multiple_Collision_Frames A Multiple Collision Count which counts the number of times that a transmit encountered more than one collision but less than 16. The value only increments if transmits are enabled and the Ethernet controller is in half-duplex mode.
Tx_Deferred Counts defer events. A defer event occurs when the transmitter cannot immediately send a packet due to the medium being busy because another device is transmitting, the IPG timer has not expired, half-duplex deferral events are occurring, XOFF frames are being received, or the link is not up. This register only increments if transmits are enabled. This counter does not increment for streaming transmits that are deferred due to TX IPG.
Rx_Frame_Too_Longs The Rx frame is over size.
Rx_Frame_Too_Shorts The Rx frame is too short.
Rx_Align_Errors This error is only valid in 10/100M mode.
Symbol Error Count Counts the number of symbol errors between reads - SYMERRS. The count increases for every bad symbol received, whether or not a packet is currently being received and whether or not the link is up. This register only increments in internal SerDes mode.

Traffic trace

Traffic tracing allows a specific packet stream to be followed. This is useful to confirm packets are taking the route you expected on your network.

View the characteristics of a traffic session though specific security policies using:

diag sys session

 

Trace per-packet operations for flow tracing using:

diag debug flow

 

Trace per-Ethernet frame using:

diag sniffer packet

Session table

A session is a communication channel between two devices or applications across the network. Sessions enable FortiOS to inspect and act on a sequential group of packets in a session all together instead of inspecting each packet individually. Each of these sessions has an entry in the session table that includes important information about the session.

Use as a tool

Session tables are useful troubleshooting tools because they allow you to verify connections that you expect to see open. For example, if you have a web browser open to browse the Fortinet website, you would expect a session entry from your computer, on port 80, to the IP for the Fortinet website. Another troubleshooting method is if there are too many sessions for FortiOS to process, you can examine the session table for evidence why this is happening.

The FortiGate session table can be viewed from either the CLI or the web-based manager. The most useful troubleshooting data comes from the CLI. The session table in web-based manager also provides some useful summary information, particularly the current policy number that the session is using.

Web-based manager session information

In the web-based manager you can view session information in the FortiView page. Sessions are categorized by Sources, Applications, Destinations, and All Sessions.

How to find which security policy a specific connection is using

Every program and device on your network must have a communication channel, or session, open to pass information. The FortiGate unit manages these sessions with its many features from traffic shaping, to antivirus scanning, and even blocking known bad web sites. Each session has an entry in the session table.

You may want to find information for a specific session, say a secure web browser session, for troubleshooting. For example if that web browser session is not working properly, you can check the session table to ensure the session is still active, and that it is going to the proper address. It can also tell you the security policy number it matches, so you can check what is happening in that policy.

  1. Know your connection information.

You need to be able to identify the session you want. For this you need the source IP address (usually your computer), the destination IP address if you have it, and the port number which is determined by the program being used. Some commons ports are:

  • port 80 (HTTP for web browsing),
  • port 22 (SSH used for secure login and file transfers)
  • port 23 (telnet for a text connection)
  • port 443 (HTTPS for secure web browsing

  1. Find your session and policy ID.

Follow System > FortiView> All Sessions. Find your session by finding your source IP address, destination IP address if you have it, and port number. The policy ID is listed after the destination information. If the list of sessions is very long, you can filter the list to make it easier to find your session.

  1. When there are many sessions, use a filter to help you find your session.

If there are multiple pages of sessions it is difficult to find a single session. To help you in your search you can use a filter to block out sessions that you don’t want. Click the search icon on the column heading to select the filter. Select Source IP and enter your source IP address. Now only sessions that originate from your IP address will be displayed in the session table. If the list is still too long, you can do the same for the Source port. That will make it easy to find your session and the security policy ID. When you are finished remember to clear the filters.

CLI session information

The session table output from the CLI (diag sys session list) is very verbose. Even on a system with a small amount of traffic, displaying the session table will generate a large amount of output. For this reason, filters are used to display only the session data of interest.

You can filter a column in the web-based manager by clicking the search icon on the column heading or from the CLI by creating a filter.

An entry is placed in the session table for each traffic session passing through a security policy. The following command will list the information for a session in the table:

diag sys session list

 

Sample Output:

FGT# diag sys session list

session info: proto=6 proto_state=05 expire=89 timeout=3600 flags=00000000 av_idx=0 use=3

bandwidth=204800/sec guaranteed_bandwidth=102400/sec traffic=332/sec prio=0 logtype=session ha_id=0 hakey=4450

tunnel=/

state=log shape may_dirty

statistic(bytes/packets/err): org=3408/38/0 reply=3888/31/0 tuples=2

orgin->sink: org pre->post, reply pre->post oif=3/5 gwy=192.168.11.254/10.0.5.100

hook=post dir=org act=snat 10.0.5.100:1251->192.168.11.254:22(192.168.11.105:1251)

hook=pre dir=reply act=dnat 192.168.11.254:22->192.168.11.105:1251(10.0.5.100:1251)

pos/(before,after) 0/(0,0), 0/(0,0)

misc=0 domain_info=0 auth_info=0 ftgd_info=0 ids=0x0 vd=0 serial=00007c33 tos=ff/ff

 

Since output can be verbose, the filter option allows specific information to be displayed, for example:

diag sys session filter <option>

 

The <option> values available include the following:

clear Clear session filter.
dintf Destination interface.
dport Destination port.
dst Destination IP address.
duration duration
expire expire
negate Inverse filter.
nport NAT'd source port
nsrc NAT'd source ip address
policy Policy ID.
proto Protocol number.
proto-state Protocol state.
sintf Source interface.
sport Source port.
src Source IP address.
vd Index of virtual domain. -1 matches all.

Even though UDP is a sessionless protocol, the FortiGate unit still keeps track of the following two different states:

  • UDP reply not seen with a value of 0
  • UDP reply seen with a value of 1

The following illustrates FW session states from the session table:

State Meaning
log Session is being logged.
local Session is originated from or destined for local stack.
ext Session is created by a firewall session helper.
may_dirty Session is created by a policy. For example, the session for ftp control channel will have this state but ftp data channel will not. This is also seen when NAT is enabled.
ndr Session will be checked by IPS signature.
nds Session will be checked by IPS anomaly.
br Session is being bridged (TP) mode.

Firewall session setup rate

The number of sessions that can be established in a set period of time is useful information. A session is an end-to-end TCP/IP connection for communication with a limited lifespan. If you record the setup rate during normal operation, when you experience problems you have that setup rate with the current number to see if its very different. While this will not solve your problems, it can be a useful step to help you define your problem.

A reduced firewall session setup rate could be the result of a number of things from a lack of system resources on the FortiGate unit, to reaching the limit of your session count for your VDOM.

To view your session setup rate method 1- CLI

FGT# get sys performance status

CPU states: 0% user 0% system 0% nice 100% idle

Memory states: 10% used

Average network usage: 0 kbps in 1 minute, 0 kbps in 10 minutes,

13 kbps in 30 minutes

Average sessions: 31 sessions in 1 minute, 30 sessions in 10

minutes, 31 sessions in 30 minutes

Average session setup rate: 0.5 sessions per second in last 1

minute, 0 sessions per second in last 10 minutes, 0 sessions per

second in last 30 minutes

Virus caught: 0 total in 1 minute

IPS attacks blocked: 0 total in 1 minute

Uptime: 44 days, 18 hours, 42 minutes

 

The information you are looking for is the Average sessions section, highlighted in the above output. In this example you can see there were 31 sessions in 1 minute, or an average of 0.5 sessions per second. The values for 10 minutes and 30 minutes allow you to take a longer average for a more reliable value if your FortiGate unit is working at maximum capacity. The smallest FortiGate unit can have 1 000 sessions established per second across the unit.

Remember that session setup rate is a global command. If you have multiple VDOMs configured with many sessions in each one, the session setup rate per VDOM will be slower than if there were no VDOMs configured.

Finding object dependencies

An administrator may not be permitted to delete a configuration object if there are other configuration objects that depend on it. This command identifies other objects which depend on or make reference to the configuration object in question. If an error is displayed that an object is in use and cannot be deleted, this command can help identify the source of the problem.

Another use is if you have a virtual interface with objects that depend on it, you need to find and remove those dependencies before you delete that interface.

CLI method

When running multiple VDOMs, this command is run in the Global configuration only and it searches for the named object both in the Global and VDOM configuration most recently used:

diag sys checkused <path.object.mkey>

 

For example, to verify which objects are referred to in a security policy with an ID of 1, enter the command as follows:

diag sys checkused firewall.policy.policyid 1

 

To check what is referred to by interface port1, enter the following command:

diag sys checkused system.interface.name port1

 

To show all the dependencies for an interface, enter the command as follows:

diag sys checkused system.interface.name <interface name>

Sample Output:

entry used by table firewall.address:name '10.98.23.23_host’

entry used by table firewall.address:name 'NAS'

entry used by table firewall.address:name 'all'

entry used by table firewall.address:name 'fortinet.com'

entry used by table firewall.vip:name 'TORRENT_10.0.0.70:6883'

entry used by table firewall.policy:policyid '21'

entry used by table firewall.policy:policyid '14'

entry used by table firewall.policy:policyid '19'

In this example, the interface has dependent objects, including four address objects, one VIP, and three security policies.

Web-based manager method

In the web-based manager, the object dependencies for an interface can be easily checked and removed.

To remove interface object dependencies - web-based manager
  1. Go to System > Interfaces.

The number in the Ref. column is the number of objects that refer to this interface.

  1. Select the number in the Ref. column for the desired interface.

A Window listing the dependencies will appear.

  1. Use these detailed entries to locate and remove object references to this interface.

The trash can icon will change from gray when all object dependencies have been removed.

  1. Remove the interface by selecting the check box for the interface, and select Delete.

Flow trace

To trace the flow of packets through the FortiGate unit, use the following command:

diag debug flow trace start

 

If your network is using IPv4, follow packet flow by setting a flow filter using this command:

diag debug flow filter <option>

 

Filtering options include the following:

addr IPv4 address

clear clear filter

daddr destination IPv4 address

dport destination port

negate inverse IPv4 filter

port port

proto protocol number

saddr source IPv4 address

sport source port

vd index of virtual domain, -1 matches all

 

If your network is using IPv6, follow packet flow by setting a flow filter using this command:

diag debug flow filter6 <option>

 

Filtering options include the following:

addr IPv6 address

clear clear filter

daddr destination IPv6 address

dport destination port

negate inverse IPv6 filter

port port

proto protocol number

saddr source IPv6 address

sport source port

vd index of virtual domain, -1 matches all

 

Enable the output to be displayed to the CLI console using the following command:

diag debug flow show console enable

diag debug flow output is recorded as event log messages and are sent to a FortiCloud or a FortiAnalyzer unit if connected. Do not let this command run longer than necessary since it generates significant amounts of data.

Start flow monitoring with a specific number of packets using this command:

diag debug flow trace start <N>

 

Stop flow tracing at any time using:

diag debug flow trace stop

 

The following is an example of the flow trace for the device at the following IP address: 203.160.224.97

diag debug enable

diag debug flow filter addr 203.160.224.97

diag debug flow show console enable

diag debug flow show function-name enable

diag debug flow trace start 100

Flow trace output example - HTTP

Connect to the web site at the following address to observe the debug flow trace. The display may vary slightly:

https://www.fortinet.com

 

Comment: SYN packet received:

id=20085 trace_id=209 func=resolve_ip_tuple_fast

line=2700 msg="vd-root received a packet(proto=6,

192.168.3.221:1487->203.160.224.97:80) from port5."

 

SYN sent and a new session is allocated:

id=20085 trace_id=209 func=resolve_ip_tuple line=2799

msg="allocate a new session-00000e90"

 

Lookup for next-hop gateway address:

id=20085 trace_id=209 func=vf_ip4_route_input line=1543

msg="find a route: gw-192.168.11.254 via port6"

 

Source NAT, lookup next available port:

id=20085 trace_id=209 func=get_new_addr line=1219

msg="find SNAT: IP-192.168.11.59, port-31925"

direction“

 

Matched security policy. Check to see which policy this session matches:

id=20085 trace_id=209 func=fw_forward_handler line=317

msg="Allowed by Policy-3: SNAT"

 

Apply source NAT:

id=20085 trace_id=209 func=__ip_session_run_tuple

line=1502 msg="SNAT 192.168.3.221->192.168.11.59:31925"

 

SYN ACK received:

id=20085 trace_id=210 func=resolve_ip_tuple_fast line=2700

msg="vd-root received a packet(proto=6, 203.160.224.97:80-

>192.168.11.59:31925) from port6."

 

Found existing session ID. Identified as the reply direction:

id=20085 trace_id=210 func=resolve_ip_tuple_fast line=2727

msg="Find an existing session, id-00000e90, reply direction"

 

Apply destination NAT to inverse source NAT action:

id=20085 trace_id=210 func=__ip_session_run_tuple

line=1516 msg="DNAT 192.168.11.59:31925-

>192.168.3.221:1487"

 

Lookup for next-hop gateway address for reply traffic:

id=20085 trace_id=210 func=vf_ip4_route_input line=1543

msg="find a route: gw-192.168.3.221 via port5"

 

ACK received:

id=20085 trace_id=211 func=resolve_ip_tuple_fast line=2700

msg="vd-root received a packet(proto=6,

192.168.3.221:1487->203.160.224.97:80) from port5."

 

Match existing session in the original direction:

id=20085 trace_id=211 func=resolve_ip_tuple_fast line=2727

msg="Find an existing session, id-00000e90, original

direction"

 

Apply source NAT:

id=20085 trace_id=211 func=__ip_session_run_tuple

line=1502 msg="SNAT 192.168.3.221->192.168.11.59:31925"

 

Receive data from client:

id=20085 trace_id=212 func=resolve_ip_tuple_fast

line=2700 msg="vd-root received a packet(proto=6,

192.168.3.221:1487->203.160.224.97:80) from port5."

 

Match existing session in the original direction:

id=20085 trace_id=212 func=resolve_ip_tuple_fast

line=2727 msg="Find an existing session, id-00000e90,

original direction"

 

Apply source NAT:

id=20085 trace_id=212 func=__ip_session_run_tuple

line=1502 msg="SNAT 192.168.3.221->192.168.11.59:31925"

 

Receive data from server:

id=20085 trace_id=213 func=resolve_ip_tuple_fast

line=2700 msg="vd-root received a packet(proto=6,

203.160.224.97:80->192.168.11.59:31925) from port6."

 

Match existing session in reply direction:

id=20085 trace_id=213 func=resolve_ip_tuple_fast

line=2727 msg="Find an existing session, id-00000e90,

reply direction"

 

Apply destination NAT to inverse source NAT action:

id=20085 trace_id=213 func=__ip_session_run_tuple

line=1516 msg="DNAT 192.168.11.59:31925-

>192.168.3.221:1487"

 

Flow trace output example - IPsec (policy-based)

id=20085 trace_id=1 msg="vd-root received a packet(proto=1, 10.72.55.240:1->10.71.55.10:8) from internal."

id=20085 trace_id=1 msg="allocate a new session-00001cd3"

id=20085 trace_id=1 msg="find a route: gw-66.236.56.230 via wan1"

id=20085 trace_id=1 msg="Allowed by Policy-2: encrypt"

id=20085 trace_id=1 msg="enter IPsec tunnel-RemotePhase1"

id=20085 trace_id=1 msg="encrypted, and send to 15.215.225.22 with source 66.236.56.226"

id=20085 trace_id=1 msg="send to 66.236.56.230 via intf-wan1“

id=20085 trace_id=2 msg="vd-root received a packet (proto=1, 10.72.55.240:1-1071.55.10:8) from internal."

id=20085 trace_id=2 msg="Find an existing session, id-00001cd3, original direction"

id=20085 trace_id=2 msg="enter IPsec ="encrypted, and send to 15.215.225.22 with source 66.236.56.226“ tunnel-RemotePhase1"

id=20085 trace_id=2 msgid=20085 trace_id=2 msg="send to 66.236.56.230 via intf-wan1"

Packet sniffing and packet capture

FortiOS devices can sniff packets using commands in the CLI or capture packets using the web-based manager. The differences between the two methods are not large.

Packet sniffing in the CLI is well suited for spot checking traffic from the CLI, but if you have complex filters to enter it can be a lot of work to enter them each time. You can also save the sniffing output; however, you must log to a file and then analyze the file later by hand.

Packet capture in the web-based manager makes it easy to set up multiple filters at once and just run one or two as you need them. You also have controls to start and stop capturing as you wish. Packet capture output is downloaded to your local computer as a *.pcap file which requires a third party application to read the file, such as Wireshark. This method is useful to send Fortinet support information to help resolve an issue.

Features Packet sniffing Packet capture
Command location CLI web-based manager
Third party software required puTTY to log plaintext output Wireshark to read *.pcap files
Read output in plain text file yes no
Read output as *.pcap file using Wireshark no yes
Easily configure single quick and simple filter yes no
Record packet interface yes no
Configure complex sniffer filters on multiple interface no yes
sniff IPv6 hard easy
sniff non-IP packets no yes
Filter packets by protocol and/or port easy easy
Filter packets by source and/or destination address easy easy

Packet sniffing

Before you start sniffing packets on the CLI, you should be prepared to capture the output to a file — there can be huge amounts of data that you will not be able to see without saving it to a file. One method is to use a terminal program like puTTY to connect to the FortiGate unit’s CLI. Then once the packet sniffing count is reached you can end the session and analyze the output in the file.

Details within packets passing through particular interfaces can be displayed using the packet sniffer with the following command:

diag sniffer packet <interface> <filter> <verbose> <count> <tsformat>

 

The <interface> value is required, with the rest being optional. If not included the default values will be “none”.

For example the simplest valid sniffer command would be:

diag sniffer packet any

 

The <interface> value can be any physical or virtual interface name. Use any to sniff packets on all interfaces.

The <filter> value limits the display of packets using filters, including Berkeley Packet Filtering (BPF) syntax. The <filter> value must be enclosed in quotes.

'[[src|dst] host <host_name_or_IP1>] [[src|dst] host <host_name_or_IP2>] [[arp|ip|ip6|gre|esp|udp|tcp] [port_no]] [[arp|ip|ip6|gre|esp|udp|tcp] [port_no]]‘

 

If a second host is specified in the filter, only the traffic between the two hosts will be displayed. Optionally, you can use logical OR to match only one of the hosts, or match one of multiple protocols or ports. When defining a port, there are up to two parts — protocol and port number.

For example, to display UDP 1812 traffic or TCP 8080 traffic, use the following:

'udp port 1812 or tcp port 8080’

 

To display all IP traffic that has a source of 192.168.1.2 and a destination of 192.168.2.3:

'ip src host 192.168.1.2 and dst host 192.168.2.3’

 

The <verbose> option allows different levels of information to be displayed. The verbose levels include:

1 Print header of packets

2 Print header and data from the IP header of the packets

3 Print header and data from the Ethernet header of the packets

4 Print header of packets with interface name

5 Print header and data from ip of packets with interface name

6 Print header and data from ethernet of packets with interface name

The <count> value indicates the number of packets to sniff before stopping. If this variable is not included, or is set to zero, the sniffer will run until you manually halt it with Ctrl-C.

The <tsformat> value define the format of timestamp. It can be:

a: absolute UTC time, yyyy-mm-dd hh:mm:ss.ms

l: absolute LOCAL time, yyyy-mm-dd hh:mm:ss.ms

otherwise: relative to the start of sniffing, ss.ms

Packet capture

FortiOS 5.2 includes packet capture to the web-based manager. The FortiGate unit must have a disk and then capture-packet feature can be enabled from the CLI within the firewall policy as below:

config firewall policy

edit <id>

set capture-packet enable

end

 

To configure packet capture filters, go to System > Network > Packet Capture.

When you add a packet capture filter, enter the following information and select OK.

Interface Select the interface to sniff from the dropdown menu.

You must select one interface. You cannot change the interface without deleting the filter and creating a new one, unlike the other fields.
Max Packets to Capture Enter the number of packets to capture before the filter stops.

This number cannot be zero. You can halt the capturing before this number is reached.
Enable Filters Select this option to specify your filter fields
Host(s) Enter one or more hosts IP address

Separate multiple hosts with commas. Enter a range using a dash without spaces, for example 172.16.1.5-172.16.1.15 or enter a subnet.
Port(s) Enter one or more ports to capture on the selected interface.

Separate multiple ports with commas. Enter a range using a dash without spaces, for example 88-90
VLAN(s) Enter one or more vlans (if there is any).

Separate multiple vlans with commas.
Protocol Enter one or more protocol. Separate multiple protocol with commas. Enter a range using a dash without spaces, for example 1-6, 17, 21-25
Include IPv6 packets Select this option if you are troubleshooting IPv6 networking, or if your network uses IPv6. Otherwise, leave it disabled.
Capture Non-IP packets The protocols available in the list are all IP based except for ICMP (ping). To capture non-IP based packets select this feature. Some examples of non-IP packets include IPsec, IGMP, ARP, and as mentioned ICMP.

If you select a filter and go back to edit it, you have the added option of starting and stopping packet capture in the edit window, or downloading the captured packets. You can also see the filter status and the number of packets captured.

You can also select the filter and select Start to start capturing packets. While the filter is running, you will see the number of captured packets increasing until it reaches the max packet count or you select Stop. While the filter is running you cannot download the output file.

When the packet capture is complete, you can select Download to send the packet capture filter captured packets to your local computer as a *.pcap file. To read this file format, you will need to use Wireshark  or a similar third party application. Using this tool you will have extensive analytics available to you and the full contents of the packets that were captured.

NPU based interfaces

Many Fortinet products contain network processors such as NP1, NP2, and NP4 network processors. Therefore offloading requirements, vary by network processor model.

When using the NPU-based interfaces, only the initial session setup will be seen through the diag debug flow command. If the session is correctly programmed into the ASIC (fastpath), the debug flow command will no longer see the packets arriving at the CPU. If the NPU functionality is disabled, the CPU will see all the packets, however, this should only be used for troubleshooting purposes.

First, obtain the NP4 id and the port numbers with the following command:

diag npu np4 list

Sample output:

ID Model Slot Interface

0 On-board port1 fabric1 fabric3 fabric5

1 On-board fabric2 port2 base2 fabric4

 

Run the following commands:

diag npu np4 fastpaf th disable <dev_id>

 

(where dev_id is the NP4 number)

Then, run this command:

diag npu np4 fastpath-sniffer enable port1

Sample output:

NP4 Fast Path Sniffer on port1 enabled

 

This will cause all traffic on port1 of NP4 to be sent to the CPU meaning a standard sniffer trace can be taken and other diag commands should work if it was a standard CPU driven port.

These commands are only for the newer NP4 interfaces.

Debug command

Debug output provides continuous, real-time event information. Debugging output continues until it is explicitly stopped or until the unit is rebooted. Debugging output can affect system performance and will be continually generated even though output might not be displayed in the CLI console.

Debug information displayed in the console will scroll in the console display and may prevent CLI commands from being entered, for example, the command to disable the debug display. To turn off debugging output as the display is scrolling by, press the á key to recall the recent diag debug command, press backspace, and type “0”, followed by Enter.

Debug output display is enabled using the following command:

diag debug enable

 

When finished examining the debug output, disable it using:

diag debug disable

 

Once enabled, indicate the debug information that is required using this command:

diag debug <option> <level>

 

Debug command options include the following:

enable Enable debug output.
disable Disable debug output.
info Show active debug level settings.
reset Reset all debug level to default.
report Report for tech support.
crashlog Crash log info.
config-error-log Configure error log info.
sql-log-error SQL log database error info.
application application.
kernel kernel.
remote-extender remote-extender.
console console.
cli Debug CLI.
cmdb-trace Trace CLI.
rating Display rating info.
authd Authentication daemon.
fsso-polling FSSO active directory poll module.
flow Trace packet flow in kernel.
urlfilter urlfilter.
admin Admin user.

The debug level can be set at the end of the command. Typical values are 2 and 3, for example:

diag debug application DHCPS 2

diag debug application spamfilter 2

 

Fortinet support will advise which debugging level to use.

Timestamps can be enabled to the debug output using the following command:

diag debug console timestamp enable

Debug output example

This example shows the IKE negotiation for a secure logging connection from a FortiGate unit to a FortiAnalyzer system.

diag debug reset

diag vpn ike log-filter src-addr4 192.168.11.2

diag debug enable

Sample Output:

FGh_FtiLog1: IPsec SA connect 0 192.168.11.2->192.168.10.201:500, natt_mode=0 rekey=0 phase2=FGh_FtiLog1

FGh_FtiLog1: using existing connection, dpd_fail=0

FGh_FtiLog1: found phase2 FGh_FtiLog1

FGh_FtiLog1: IPsec SA connect 0 192.168.11.2 -> 192.168.10.201:500 negotiating

FGh_FtiLog1: overriding selector 225.30.5.8 with 192.168.11.2

FGh_FtiLog1: initiator quick-mode set pfs=1536...

FGh_FtiLog1: try to negotiate with 1800 life seconds.

FGh_FtiLog1: initiate an SA with selectors: 192.168.11.2/0.0.0.0->192.168.10.201, ports=0/0, protocol=0/0

Send IKE Packet(quick_outI1):192.168.11.2:500(if0) -> 192.168.10.201:500, len=348

Initiator: sent 192.168.10.201 quick mode message #1 (OK)

FGh_FtiLog1: set retransmit: st=168, timeout=6.

 

In this example:

192.168.11.2->192.168.10.201:500 Source and Destination gateway IP address
dpd_fail=0 Found existing Phase 1
pfs=1536... Create new Phase 2 tunnel

The execute tac report command

exec tac report is an execute command that runs an exhaustive series of diagnostic commands. It runs commands that are only needed if you are using certain features like HA, VPN tunnels, or a modem. The report takes a few minutes to complete due to the amount of output generated. If you have your CLI output logged to a file, you can run this command to familiarize yourself with the CLI commands involved.

When you call Fortinet Customer Support, you will be asked to provide information about your unit and its current state using the output from this CLI command.

Other commands

ARP table

To view the ARP cache, use the following command:

get sys arp

 

To view the ARP cache in the system, use this command:

diag ip arp list

Sample output:

index=14 ifname=internal 224.0.0.5 01:00:5e:00:00:05 state=00000040 use=72203 confirm=78203 update=72203 ref=1

index=13 ifname=dmz 192.168.3.100 state=00000020 use=1843 confirm=650179 update=644179 ref=2 ? VIP

index=13 ifname=dmz 192.168.3.109 02:09:0f:78:69:ff state=00000004 use=71743 confirm=75743 update=75743 ref=1

index=14 ifname=internal 192.168.11.56 00:1c:23:10:f8:20 state=00000004 use=10532 confirm=10532 update=12658 ref=4

 

To remove the ARP cache, use this command:

execute clear system arp table

 

To remove a single ARP entry, use:

diag ip arp delete <interface name> <IP address>

 

To add static ARP entries, use the following command:

config system arp-table

Time and date settings

Check time and date settings for log message timestamp synchronization (the Fortinet support group may request this) and for certificates that have a time requirement to check for validity. Use the following commands:

execute time

current time is: 12:40:48

last ntp sync:Thu Mar 16 12:00:21 2006

execute date

current date is: 2006-03-16

 

To force synchronization with an NTP server, toggle the following command:

Config system ntp

set ntpsync enable/disable

end

 

If all devices have the same time, it helps to correlate log entries from different devices.

IP address

There may be times when you want to verify the IP addresses assigned to the FortiGate unit interfaces are what you expect them to be. This is easily accomplished from the CLI using the following command.

diag ip address list

 

The output from this command lists the IP address and mask if available, the index of the interface (a sort of ID number) and the devname is the name of the interface. While physical interface names are set, virtual interface names can vary. Listing all the virtual interface names is a good use of this command. For vsys_ha and vsys_fgfm, the IP addresses are the local host — these are internally used virtual interfaces.

# diag ip address list

IP=10.31.101.100->10.31.101.100/255.255.255.0 index=3 devname=internal

IP=172.20.120.122->172.20.120.122/255.255.255.0 index=5 devname=wan1

IP=127.0.0.1->127.0.0.1/255.0.0.0 index=8 devname=root

IP=127.0.0.1->127.0.0.1/255.0.0.0 index=11 devname=vsys_ha

IP=127.0.0.1->127.0.0.1/255.0.0.0 index=13 devname=vsys_fgfm

FortiOS ports

In the TCP and UDP stacks, there are 65 535 ports available for applications to use when communicating with each other. Many of these ports are commonly known to be associated with specific applications or protocols. These known ports can be useful when troubleshooting your network.

Use the following ports while troubleshooting the FortiGate device:

Port(s) Functionality
UDP 53 DNS lookup, RBL lookup
UDP 53 or UDP 8888 FortiGuard Antispam or Web Filtering rating lookup
UDP 53 (default) or UDP 8888 and UDP 1027 or UDP 1031 FDN Server List - source and destination port numbers vary by originating or reply traffic. See the article “How do I troubleshoot performance issues when FortiGuard Web Filtering is enabled?” in the Knowledge Base.
UDP 123 NTP Synchronization
UDP 162 SNMP Traps
UDP 514 SYSLOG - All FortiOS versions can use syslog to send log messages to remote syslog servers. FortiOS v2.80 and v3.0 can also view logs stored remotely on a FortiAnalyzer unit.
TCP 22 Configuration backup to FortiManager unit or FortiGuard Analysis and Management Service.
TCP 25 SMTP alert email, encrypted virus sample auto-submit
TCP 389 or TCP 636 LDAP or PKI authentication
TCP 443 FortiGuard Antivirus or IPS update - When requesting updates from a FortiManager unit instead of directly from the FDN, this port must be reconfigured as TCP 8890.
TCP 443 FortiGuard Analysis and Management Service
TCP 514 FortiGuard Analysis and Management Service log transmission (OFTP)
TCP 541 SSL Management Tunnel to FortiGuard Analysis and Management Service (FortiOS v3.0 MR6 or later)
TCP 514 Quarantine, remote access to logs and reports on a FortiAnalyzer unit, device registration with FortiAnalyzer units (OFTP)
TCP 1812 RADIUS authentication
TCP 8000 and TCP 8002 FSSO
TCP 10151 FortiGuard Analysis and Management Service contract validation

FortiAnalyzer/FortiManager ports

If you have a FortiAnalyzer unit or FortiManager unit on your network you may need to use the following ports for troubleshooting network traffic.

Functionality Port(s)
DNS lookup UDP 53
NTP synchronization UDP 123
Windows share UDP 137-138
SNMP traps UDP 162
Syslog, log forwarding UDP 514
Log and report upload TCP 21 or TCP 22
SMTP alert email TCP 25
User name LDAP queries for reports TCP 389 or TCP 636
RVS update TCP 443
RADIUS authentication TCP 1812
Log aggregation client TCP 3000

FortiGuard troubleshooting

The FortiGuard service provides updates to Antivirus, Antispam, IPS, Webfiltering, and more. The FortiGuard Distribution System (FDS) involves a number of servers across the world that provide updates to your FortiGate unit. Problems can occur both with connection to FDS, and its configuration on your local FortiGate unit. Some of the more common troubleshooting methods are listed here including

Troubleshooting process for FortiGuard updates

The following process are the logical steps to take when troubleshooting FortiGuard update problems. This includes antivirus (AV), intrusion protection services (IPS), antispam (AS), and web filtering (WB).

  1. Does the device have a valid licence that includes these services?
    Each device requires a valid FortiGuard license to access updates for some or all of these services. You can verify the support contract status for your devices at the Fortinet Support website — https://support.fortinet.com/.
  2. If the device is part of an HA cluster, do all members of the cluster have the same level of support?
    As with the previous step, you can verify the support contract status for all the devices in your HA cluster at the Fortinet Support website.
  3. Have services been enabled on the device?
    To see the FortiGuard information and status for a device, in the web-based manager go to System > Config > FortiGuard. On that page you can verify the status of each component, and if required enable each service. If there are problems, see the FortiGuard section of the FortiOS Handbook.
  4. Is the device able to communicate with FortiGuard servers?
    At System > Config > FortiGuard you can also attempt to update AV and IPS, or test the availability of WF and AS default and alternate ports. If there are problems, see the FortiGuard section of the FortiOS Handbook.
  5. Is there proper routing to reach the FortiGuard servers?
    Ensure there is a static or dynamic route that enables your ForitGate unit to reach the FortiGuard servers. Usually a generic default route to the internet is enough, but you may need to verify this if your network is complex.
  6. Are there issues with DNS?
    An easy way to test this is to attempt a traceroute from behind the FortiGate unit to an external network using the FQDN for a location. If the traceroute FQDN name does not resolve, you have general DNS problems.
  7. Is there anything upstream that might be blocking FortiGuard traffic, either on the network or ISP side?
    Many firewalls block all ports by default, and often ISPs block ports that are low. There may be a firewall between the FortiGate unit and the FortiGuard servers that is blocking the traffic. FortiGuard uses port 53 by default, so if it is being blocked you need to either open a hole for it, or change the port it is using.
  8. Is there an issue with source ports?
    It is possible that ports used to contact FortiGuard are being changed before reaching FortiGuard or on the return trip before reaching your FortiGate unit. A possible solution for this is to use a fixed-port at NATd firewalls to ensure the port remains the same. Packet sniffing can be used to find more information on what is happening with ports.
  9. Are there security policies that include antivirus?
    If no security policies include antivirus, the antivirus databse will not be updated. If antivirus is included, only the database type used will be updated.

FortiGuard server settings

Your local FortiGate unit connects to remote FortiGuard servers get updates to FortiGuard information such as new viruses that may have been found or other new threats. This section demonstrates ways to display information about FortiGuard server information on your FortiGate unit, and how to use that information and update it to fix potential problems.

Displaying the server list

The get webfilter status or diag debug rating command shows the list of FDS servers the FortiGate unit is using to send web filtering requests. Rating requests are only sent to the server on the top of the list in normal operation. Each server is probed for Round Trip Time (RTT) every two minutes.

You can optionally add a refresh rate to the end of this command and that will determine how often the server list will be refreshed.

Rating may not be enabled on your FortiGate unit.

get webfilter status

 

Sample Output:

Locale : english

License : Contract

Expiration : Thu Oct 9 02:00:00 2011

-=- Server List (Mon Feb 18 12:55:48 2008) -=-

  IP Weight RTT Flags TZ Packets CurrLost TotalLost
  a.b.c.d 0 1 DI 2 1926879 0 11176
  10.1.101.1 10 329   1 10263 0 633
  10.2.102.2 20 169   0 16105 0 80
  10.3.103.3 20 182   0 6741 0 776
  10.4.104.4 20 184   0 5249 0 987
  10.5.105.5 25 181   0 12072 0 178

Output Details

The Server List includes the IP addresses of alternate servers if the first entry cannot be reached. In this example the IP addresses are not public addresses

The following flags in get webfilter status indicate the server status:

  • D - the server was found through the DNS lookup of the hostname. If the hostname returns more than one IP address, all of them will be flagged with D and will be used first for INIT requests before falling back to the other servers.
  • I - the server to which the last INIT request was sent.
  • F - the server has not responded to requests and is considered to have failed.
  • T - the server is currently being timed.
  • S - means that rating requests can be sent to the server. The flag is set for a server only in two cases:

  1. The server exists in the servers list received from the Fortimanager or any other INIT server.
  2. The servers list received from the FortiManager is empty so the Fortimanager itself would be the only server known by Fortigate thus it should be used as the rating server.

The servers that are not currently serving will be pushed down to the bottom list (under the available serving servers, and on top of the failed servers) in order for the load-balance-servers feature in the config system fortiguard to work properly.

Sorting the server list

The server list is sorted first by weight. The server with the smallest RTT is put at the top of the list, regardless of weight. When a packet is lost (there has been no response in 2 seconds), it will be resent to the next server in the list. Therefore, the top position in the list is selected based on RTT while the other list positions are based on weight.

Calculating weight

The weight for each server increases with failed packets and decreases with successful packets. To lower the possibility of using a remote server, the weight is not allowed to dip below a base weight, calculated as the difference in hours between the FortiGate unit and the server times 10. The further away the server is, the higher its base weight and the lower in the list it will appear.

 

Copyright © 2018 Fortinet, Inc. All Rights Reserved. | Terms of Service | Privacy Policy