Fine-tuning & best practices : Improving performance
 
Improving performance
When you configure your FortiWeb appliance and its features, there are many settings and practices that can yield better performance.
System performance
• Delete or disable unused policies. FortiWeb allocates memory with each server policy, regardless of whether it is actually in active use. Configuring extra policies unnecessarily consumes memory and decreases performance.
• To reduce latency associated with DNS queries, use a DNS server on your local network as your primary DNS. See “Configuring DNS settings”.
• If your network’s devices support them, you can create one or more VLAN interfaces. VLANs reduce the size of a broadcast domain and the amount of broadcast traffic received by network hosts, which improves network performance. See “Adding VLAN subinterfaces”.
• If you have enabled the server health check feature as part of a server pool and one of the pool members is down for an extended period, you can improve the performance of your FortiWeb appliance by disabling the physical server, rather than allowing the server health check to continue checking for the server's responsiveness. See “Configuring server up/down checks”.
• Use the least intensive, earliest possible scan to deflect attacks. See “Sequence of scans”.
• Use Period Block if possible as the Action setting for DoS protection rules. This setting allows FortiWeb to conserve scanning resources that are under heavy demand during a DoS or DDoS attack.
Antivirus performance
• Disable scanning of BZIP2 if it is not necessary.
• Reduce the scanning buffer to the minimum necessary.
• Reduce the number of redundant levels of compression that FortiWeb will scan. Normally, people will not put a ZIP file within a ZIP file, because it is inconvenient to open and does not offer significant compression ratio improvements. Nested compression is usually used by viruses to bypass antivirus scanners.
Regular expression performance tips
• Use a simple string instead if possible. Generally, regular expressions should only be used when defining all matching text requires a complex pattern. Regular expressions such as:
^.*/index\.html$
are usually more computationally intensive than a literal string comparison such as:
/index.html
• Reduce evaluation complexity.
 
Short regular expressions can sometimes be more complex to compute. Don’t look at the number of characters in the regular expression. Instead, think of both the usual and worst possible case in the match string: the maximum number of characters that must be compared to the pattern before a match can be verified or not.
The usual case will tell you the average CPU and RAM load. The worst case will tell you if your regular expression could sometimes cause potential hang-like conditions, temporarily blocking traffic throughput until it finishes evaluating.
 
If the worst possible match string is short and not complex to match, the regular expression may not be worth your time to optimize.
For example, when using auto-learning to discover if street addresses are a valid input, scanning for postal codes or state abbreviations instead may dramatically improve performance. A pattern to fully match all possible street addresses is significantly more complex, involving many more computations, and the most difficult addresses to verify might be complex enough to impact traffic throughput.
 
If missed matches are an acceptable performance trade-off (for example, if matching 99% of cases is efficient, but matching 100% of cases would require deep recursion), or if you do not need to match the whole text, remove the unnecessary part of the regular expression.
For example, if a phone number always resembles 555-5555, your regular expression would not have to accommodate cases where a space separates the numbers, or it is prefixed by a country code. This is less comprehensive, but also less CPU-intensive.
• Avoid backtracking (i.e. revisiting the match string after failing to match part of the pattern). Backtracking occurs when regular expression features use recursion (definite or indefinite). This can increase execution time exponentially. Examples include the following:
• Avoid nested parentheses with indefinite repeats such as:
^((a+)b+)*
which can take a very long time to evaluate, especially if a long string does not match, but this cannot be determined until the very last character is evaluated.
In the above example, both the + and * indicate matches that repeat potentially infinitely, forcing the regular expression engine to continue until it finds the longest possible match (or runs out of RAM; see “Killing system-intensive processes”). Using both in a nested set of parentheses compounds the problem.
• Minimize capture groups and back-references such as:
(/a)(/b)/(c)
$0$1\?user=$2
To use back-references, FortiWeb must keep the text that matched the capture groups in memory, which increases RAM consumption.
• Order matters if using alternate match patterns (i.e. multiple patterns are concatenated with a pipe ( | )). Put rare patterns last. If you put less likely patterns first, most times FortiWeb will be evaluating the string multiple times — not once — before it finds a match. This significantly decreases performance.
When comparing single characters, use character classes such as:
[abc]
instead of alternative matches like
(a|b|c)
Match character by character, not word by word. If words begin with the same characters, it is not efficient to evaluate the beginning of the match string multiple times — once for each possible word.
For example, to match the words “the”, “then”, “this”, and “these”, this expression is easy to read, but inefficient because it evaluates the first two characters (“th”) up to 4 times:
\b(this|the|then|these)\b
While harder to read, this expression improves performance, evaluating “th” once, and will match the most common word in English (“the”) before considering less probable words:
\bth(e(n|se)|is)\b
• Reduce nested quantifiers such as:
(abc){1,6}
(abc)+
Worst-case evaluations do not increase computation time linearly, but exponentially. When such an expression is compiled, it also consumes much more RAM. Use the smallest possible repetition, or an alternative expression.
• Avoid Unicode character properties such as /p{Nd} if you can use a character class instead. Due to the huge numbers and complexity of potential matches in Unicode, these can be dramatically slower.
• Avoid look-ahead match conditions such as:
?=abcdefge
?!abcdefge
To do this, FortiWeb must make additional computations — in the example above, 8 in the best case scenario, an immediate match. FortiWeb also must keep the originally consumed match string in memory while it does this, which increases RAM consumption.
Logging performance
• If you have a FortiAnalyzer, store FortiWeb’s logs on the FortiAnalyzer to avoid resource usage associated with writing logs to FortiWeb’s own hard disks. See “Configuring log destinations”.
• If you do not need a traffic log, disable it to reduce the use of system resources. See “Enabling log types, packet payload retention, & resource shortage alerts”.
• Reduce repetitive log messages. Use the alert email settings, as shown in Figure 87, to define the interval that emails are sent if the same condition persists following the initial occurrence. See “Configuring email settings”.
Figure 87: Log&Report > Log Policy > Email Policy
• Avoid recording log messages using low severity thresholds, such as information or notification, to the local hard disk for an extended period of time. Excessive logging frequency saps system resources and can cause undue wear on the hard disk and may cause premature failure. See “Configuring log destinations”.
Report performance
Generating reports can be resource intensive. To avoid performance impacts, consider scheduling report generation during times with low traffic volume, such as at night and on weekends. See Figure 88 and “Scheduling reports”.
Figure 88: Log&Report > Report Config > Report Config
Keep in mind that most reports are based upon log messages. All caveats regarding log performance also apply.
Auto-learning performance
• Each URL in an auto-learning report includes the right-click menu option Stop Learning. If a URL is dynamic or hard to predict effectively and may generate inaccurate data, you can improve performance by pausing or stopping auto-learning for that URL. See “Pausing auto-learning for a URL”.
• Once you have collected enough auto-learning data for generating protection profiles, consider turning off the auto-learning function to save resources. To do so, deselect the auto-learning profile in applicable server policies. See “How operation mode affects server policy behavior”.
• Use less computationally intensive data types and suspicious URLs, and disable unneeded ones, where possible. See “Regular expression performance tips”.
• Reduce the list of predefined data type groups to include just those the FortiWeb appliance is likely to encounter when gathering data for an auto-learning report. By pruning the list, you reduce the resources used to recognize data types, freeing them to improve the throughput of the FortiWeb appliance. See “Grouping predefined data types”.
Figure 89: Auto Learn > Predefined Pattern > Data Type Group
• When configuring a suspicious URL pattern, clear one or more web server type options if you do not operate all three web servers, as shown in Figure 90. By pruning the list, you reduce the resources used by the FortiWeb appliance when applying the rule. See “Grouping all suspicious request URLs”.
Figure 90: Auto Learn > Predefined Pattern > Suspicious URL
• When you configure a signature set as part of a web protection profile, consider limiting the scope and application of the Information Disclosure options shown in Figure 91. (Click the blue arrow next to Information Disclosure to see the list.)
Do you need to watch for all information types? If not, disable them to increase performance. Disable signatures that do not apply to your web servers. For example, if your web server does not run Adobe ColdFusion, you could disable CF Source Code Leakage to omit that scan and improve performance. See “Specifying URLs allowed to initiate sessions”.
Figure 91: Disabling unnecessary server information disclosure signatures in Web Protection > Known Attacks > Signatures
The Information Disclosure feature can potentially require the FortiWeb appliance to rewrite the header of every request from a server, resulting in reduced performance. Fortinet recommends enabling this feature only to help you identify information disclosure through logging, and until you can reconfigure the server to omit such sensitive information. Clear the All / None check box to disable the feature.
• If you use the web anti-defacement feature, tune your configuration to avoid backing up overly large files. See Figure 92 and “Anti-defacement”.
Figure 92: Omitting large files from the backup in Web Anti-Defacement > Web Anti-Defacement >Web Site with Anti-Defacement
Unless you need to back up large files, reduce the setting for the Skip Files Larger Than option from the default of 10 240 KB.
Use the Skip Files With These Extensions option to exclude specific types of large files, such as compressed files and video clips.
Vulnerability scan performance
Vulnerability scan performance depends on the speed and reliability of your network. It also can be impacted by your configuration. See “Delay Between Each Request”.
Packet capture performance
Packet capture can be useful for troubleshooting but can be resource intensive. (See “Packet capture”.) To minimize the performance impact on your FortiWeb appliance, use packet capture only during periods of minimal traffic. Use a local console CLI connection rather than a Telnet or SSH CLI connection, and be sure to stop the command when you are finished.