Protection settings can be configured manually or with assistance from auto-learning.
Auto-learning can teach you a great deal about the threats your web assets face. It also helps you to understand your web applications’ structures and how end-users use them. Most importantly, though, auto-learning can help you to quickly tailor FortiWeb’s configuration to suit your web applications.
For data centers, colocation centers, and complex web applications, auto-learning-assisted configuration can save significant amounts of time compared to purely manual configuration. However, auto-learning is also resource-intensive and can decrease performance while gathering data. For strategies on minimizing the impact to your network, see Running auto-learning and Regular expression performance tips. |
Auto-learning discovers the URLs and other characteristics of HTTP and/or HTTPS sessions by observing traffic that is passing to your web servers. To learn about whether the request is legitimate or a potential attack attempt, it performs the following tasks:
401 Unauthorized
or 500 Internal Server Error
By learning from your traffic, the FortiWeb appliance can suggest appropriate configurations, and help you to quickly generate profiles designed specifically for your unique traffic.
When web applications have dynamic URLs or unusual parameter styles, you must adapt auto-learning to recognize them.
By default, auto-learning assumes that your web applications use the most common URL structure:
?
). They do not follow a hash ( # ) or other separator character.&
). They are not separated by a semi-colon ( ; ) or other separator character.All paths before the question mark ( ?
) are static — they do not change based upon input, blending the path with parameters (sometimes called a dynamic URL).
For example, the page at:
/app/main
always has that same path. After a person logs in, the page’s URL doesn’t become:
/app/marco/main
or
/app#deepa
For another example, the URL does not dynamically reflect inventory, such as:
/app/sprockets/widget1024894
Some web applications, however, embed parameters within the path structure of the URL, or use unusual or non-uniform parameter separator characters. If you do not configure URL replacers for such applications, it can cause your FortiWeb appliance to gather auto-learning data incorrectly. This can cause the following symptoms:
For example, with Microsoft Outlook Web App (OWA), the user’s login name could be embedded within the path structure of the URL, such as:
/owa/tom/index.html
/owa/mary/index.html
instead of suffixed as a parameter, such as:
/owa/index.html?username=tom
/owa/index.html?username=mary
Auto-learning would continue to create new URLs as new users are added to OWA. Auto-learning would also expend extra resources learning about URLs and parameters that are actually the same. Additionally, auto-learning may not be able to fully learn the application structure, as each user may not request the same URLs.
To solve this, you would create a URL replacer that recognizes the user name within the OWA URL as if it were a standard, suffixed parameter value so that auto-learning can function properly.
When using auto-learning, you must define how to interpret dynamic URLs and URLs that include parameters in non-standard ways, such as with different parameter separators (;
or #
, for example) or by embedding the parameter within the URL’s path structure.
In the web UI, these interpreter plug-ins are called “URL replacers.”
URL replacers match the URL as it appears in the HTTP header of the client’s request (using the regular expression in URL Path) and interpret it into this standard URL formulation:
New URL?
New Param=
Param Change
For example, if the URL is:
/application/value
and the URL replacer settings are:
Setting name | Value |
---|---|
Type | Custom-Defined |
URL Path | (/application)/([^/]+)
|
New URL | $0
|
Param Change | $1
|
New Param | setting
|
$0
holds this part of the matched URL:
/application
and $1
holds this part of the matched URL:
value
so then the URL will be understood by auto-learning, and displayed in the report, as:
/application?setting=value
Need a refresher on regular expressions? See Regular expression syntax, What are back-references?, and Cookbook regular expressions. You can also use the examples in this section, such as Example: URL interpreter for WordPress. |
1. Go to Auto Learn > Application Templates > URL Replacer.
To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Autolearn Configuration category. For details, see Permissions.
2. Click Create New.
3. Configure these settings:
Name | Type a unique name that can be referenced by other parts of the configuration. Do not use spaces or special characters. The maximum length is 35 characters. |
Type |
Select either:
|
4. If you selected Predefined in Type, also configure this setting:
5. If you selected Custom-Defined in Type, configure these settings:
URL Path |
Type a regular expression, such as The pattern does not require a slash ( / ). However, it must at least match URLs that begin with a slash as they appear in the HTTP header, such as For examples, see Example: URL interpreter for WordPress. To test the regular expression against sample text, click the >> (test) icon. This opens the Regular Expression Validator window where you can fine-tune the expression (see Regular expression syntax, What are back-references? and Cookbook regular expressions) Note: If this URL replacer will be used sequentially in its set of URL replacers, instead of being mutually exclusive, this regular expression should match the URL produced by the previous interpreter, not the original URL from the request. |
New URL |
Type either a literal URL, such as Note: Back-references can only refer to capture groups (parts of the expression surrounded with parentheses) within the same URL replacer. Back-references cannot refer to capture groups in other URL replacers. |
Param Change | Type either the parameter’s literal value, such as user1 , or a back-reference (such as $0 ) defining how the value will be interpreted. |
New Param |
Type either the parameter’s literal name, such as Note: Back-references can only refer to capture groups (parts of the expression surrounded with parentheses) within the same URL replacer. Back-references cannot refer to capture groups in other URL replacers. |
6. Click OK.
7. Group the URL replacers in an application policy (see Grouping URL interpreters).
8. Select the application policy in one or more auto-learning profiles (see Configuring an auto-learning profile).
9. Select the auto-learning profiles in server policies (see Configuring a server policy).
The HTTP request URL from a client is:
/app/login.jsp;jsessionid=xxx;p1=111;p2=123?p3=5555&p4=66aaaaa
which uses semi-colons as parameter separators ( ; ) in the URL, a behavior typical to JSP applications. You would create a URL replacer to recognize the JSP application’s parameters: the semi-colons.
Setting name | Value |
---|---|
Type | Predefined |
Application Type | JSP |
The predefined JSP interpreter plug-in will interpret the URL as:
/app/login.jsp?p4=66aaaaa&p1=111&p2=123&p3=5555
When a client sends requests to Microsoft Outlook Web App (OWA), many of its URLs use structures like this:
/exchange/jane.doe/memo.EML
/exchange/qinlu/2012/1.html
These have user name parameters embedded in the URL. In order for auto-learning to recognize the parameters, you must either:
A custom URL replacer for those URLs could look like this:
URL interpreter | |
---|---|
Setting name | Value |
Name | OWAusername1
|
Type | Custom-Defined |
URL Path | (/exchange/)([^/]+)/(.*)
|
New URL | $0$2
|
Param Change | $1
|
New Param | username1
|
Then the URLs would be recognized by auto-learning as if OWA used a more conventional parameter structure like this:
/exchange/index.html?username1=tom
/exchange/memo.EML?username1=jane.doe
/exchange/2012/1.html?username1=qinlu
Notably, OWA can also include other parameters in the URL, such as a mail folder’s name. Also, OWA can include the user name and folder in more than one way. Therefore multiple URL interpreters are required to match all possible URL structures. In addition to the first URL replacer, you would also configure the following URL replacers and group them into a single set (an auto-learning “application policy”) in order to recognize all possible URLs.
Sample URL | /exchange/archive-folders/2011
|
---|---|
URL interpreter | |
Setting name | Value |
Name | OWAfoldername1
|
Type | Custom-Defined |
URL Path | (/exchange/)([^/]+/)(.*)
|
New URL | $0
|
Param Change | $1$2
|
New Param | folder1
|
Results | /exchange/?folder1=archive-folders/2011
|
Sample URL | /exchange/jane.doe
|
---|---|
URL interpreter | |
Setting name | Value |
Name | OWAusername2
|
Type | Custom-Defined |
URL Path | (/exchange/)([^/]+\.[^/]+)
|
New URL | $0
|
Param Change | $1
|
New Param | username2
|
Results | /exchange/?username2=jane.doe
|
Sample URL | /public/imap-share-folders/memos
|
---|---|
URL interpreter | |
Setting name | Value |
Name | OWAfoldername2
|
Type | Custom-Defined |
URL Path | (/public/)([^/]+/)(.*)
|
New URL | $0
|
Param Change | $1$2
|
New Param | folder2
|
Results | /public/?folder2=imap-share-folders/memos
|
If the HTTP request URL from a client is a slash-delimited chain of multiple parameters, like either of these:
/index/province/ontario/city/ottawa/street/moodie
then the format is either of these:
/wordpress/value1/value2/value3
/index/param1/value1/param2/value2/param3/value3
In this URL format, there are 3 parameter values (with or without their names) in the URL:
param1
param2
param3
Because each interpreter can only extract a single parameter, you would create 3 URL interpreters, and group them into a set where they are used sequentially — a chain. Each interpreter would use the interpreted output of the previous one as its input, until all parameters had been extracted, at which point the last interpreter would output both the last parameter and the final interpreted URL. FortiWeb would then append parameters back onto the interpreted URL in the standard structure before storing them in the auto-learning data set.
This configuration requires that for every request:
If parameter order or existence vary, this URL interpreter will not work. Requests will not match the URL interpreter set if either
then the regular expression would be too flexible: auto-learning might mistakenly match and learn some of |
Setting name | Value |
---|---|
Name | slash-parameter3
|
Type | Custom-Defined |
URL Path | /index/param1/(.*)/param2/(.*)/param3/(.*)/
|
New URL | /index/param1/$0/param2/$1/
|
Param Change | $2
|
New Param | param3
|
Setting name | Value |
---|---|
Name | slash-parameter2
|
Type | Custom-Defined |
URL Path | /index/param1/(.*)/param2/(.*)/
|
New URL | /index/param1/$0/
|
Param Change | $1
|
New Param | param2
|
Setting name | Value |
---|---|
Name | slash-parameter1
|
Type | Custom-Defined |
URL Path | /index/param1/(.*)/
|
New URL | /index
|
Param Change | $0
|
New Param | param1
|
Until you add the URL interpreters to a group, FortiWeb doesn’t know the sequential order.
These URL interpreters will not function correctly if they are not used in that order, because each interpreter’s input is the output from the previous one. So you must set the priorities correctly when referencing each of those interpreters in the set of URL interpreters (Grouping URL interpreters). |
Setting name | Value |
---|---|
Priority | 0
|
Type | URL REPLACER |
Plugin Name | slash-parameter3
|
Setting name | Value |
---|---|
Priority | 1
|
Type | URL REPLACER |
Plugin Name | slash-parameter2
|
Setting name | Value |
---|---|
Priority | 2
|
Type | URL REPLACER |
Plugin Name | slash-parameter1
|
Then the URL will be interpreted by auto-learning as if the application used a more conventional and easily understood URL/parameter structure:
/index?param1=value1¶m2=value2¶m3=value3
In order to use URL interpreters with an auto-learning profile, you must group URL replacers into sets.
Sets can be:
1. Before you create an application policy, first create the URL replacers that it will include (see Configuring URL interpreters).
2. Go to Auto Learn > Application Templates > Application Policy.
To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Autolearn Configuration category. For details, see Permissions.
3. Click Create New.
A dialog appears.
4. In Name, type a name that can be referenced by other parts of the configuration. Do not use spaces or special characters. The maximum length is 35 characters.
5. Click OK.
6. Click Create New.
A dialog appears.
7. From Plugin Name, select an existing URL replacer from the drop-down list.
Rule order affects URL replacer matching and behavior. FortiWeb appliances evaluate URLs for a matching URL replacer starting with the smallest ID number (greatest priority) rule in the list, and continue towards the largest number in the list. |
8. Click OK.
9. Repeat the previous steps for each URL replacer you want added to the policy.
10. Select the application policy in an auto-learning profile (see Configuring an auto-learning profile).
11. Select the auto-learning profiles in server policies (see Configuring a server policy).
FortiWeb appliances recognize the data types of parameters by matching them with regular expressions. These regular expressions are categorized as either:
When you install FortiWeb, it already has some data type regular expressions that are predefined — default signatures for common data types so that you do not need to write them yourself. Initial ones are included with the FortiWeb firmware. If your FortiWeb is connected to FortiGuard Security Service updates, it can periodically download updates to its predefined data types. This will provide new and enhanced data types without any effort on your part. Simply use the new signatures in parts of the configuration where they are useful to you.
Predefined data type patterns cannot be used directly. Instead, they must be grouped before they can be used in other areas of the configuration. For details, see Grouping predefined data types.
To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Server Policy Configuration category. For details, see Permissions.
A data type group defines a set of predefined data types (see Predefined data types) that can be used in an auto-learning profile.
For example, if you include the Email data type in the data type group, auto-learning profiles that use the data type group might discover that your web applications use a parameter named username
whose value is an email address.
The predefined data type group, named predefine-data-type-group, cannot be edited or deleted.
1. Go to Auto Learn > Predefined Pattern > Data Type Group.
To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Server Policy Configuration category. For details, see Permissions.
2. Click Create New.
A dialog appears.
3. In Name, type a unique name that can be referenced by other parts of the configuration. Do not use spaces or special characters. The maximum length is 35 characters.
4. In Type, mark the check box of each predefined data type that you want to include in the set, such as Email or Canadian Social Insurance Number.
To examine the regular expressions for each data type, see Predefined data types.
5. Click OK.
6. To use a data type group, select it when configuring either an auto-learning profile (see Configuring an auto-learning profile) or input rule (see Validating parameters (“input rules”)).
FortiWeb appliances can recognize known attacks by comparing each request to a signature. How, then, does it recognize requests that aren’t known to be an attack, or aren’t always an attack, but might be?
FortiWeb uses several methods for this:
Web applications’ administrative URLs often should not be accessible by clients on the Internet, and therefore any request for those URLs from source IP addresses on the Internet may represent an attempt to scout your web servers in advance of an attack. (Exceptions include hosting providers, whose clients may span the globe and often configure their own web applications.) Administrative requests from the Internet are therefore suspicious: the host may have been compromised by a rootkit, or its administrative login credentials may have been stolen via spyware, phishing, or social engineering.
FortiWeb appliances can compare each request URL with regular expressions that define known administrative URLs, and log and/or block these requests.
Regular expressions for suspicious requests by URL are categorized as:
Both types can be grouped into a set that can be used in auto-learning profiles.
Predefined regular expressions can be used by auto-learning to detect requests that are suspicious because they are for a URL that provides administrative access to the web server, servlet, or web application, such as:
/admin.php
/conf/Catalina/localhost/admin.xml
or access to its back-end cache, data files, or Berkeley databases, such as:
/local/notesdata
Normally, requests for these URLs should only originate from a trusted network such as your management computers, not from the Internet. (Exceptions include hosting providers, whose clients around the globe configure their own web applications.) Therefore these requests are a good candidate for URL access control rules.
Many signatures exist for popular web servers and applications such as Apache, nginx IIS, Tomcat, and Subversion. Known suspicious request URLs can be updated. See Connecting to FortiGuard services.
To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Server Policy Configuration category. For details, see Permissions.
To augment FortiWeb’s predefined list of suspicious request URLs, you can configure your own.
1. Go to Auto Learn > Custom Pattern > Suspicious URL Rule.
To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Server Policy Configuration category. For details, see Permissions.
2. Click Create New.
A dialog appears.
3. In Name, type a unique name that can be referenced by other parts of the configuration. Do not use spaces or special characters. The maximum length is 35 characters.
4. In URL Expression, enter a regular expression that defines this suspicious URL, such as ^/my_admin_panel.jsp
.
To test the regular expression against sample text, click the >> (test) icon. This opens the Regular Expression Validator window where you can fine-tune the expression (see Regular expression syntax and Cookbook regular expressions).
5. Click OK.
6. Group custom suspicious URL patterns (see Grouping custom suspicious request URLs).
7. Group custom and predefined suspicious URL groups together (see Grouping all suspicious request URLs).
8. Select the supergroup when configuring an auto-learning profile (see Configuring an auto-learning profile).
Before you can use them, you must first group custom and predefined suspicious URLs.
1. Before you can create a custom suspicious URL rule, you must first define one or more custom suspicious URLs (see Configuring custom suspicious request URLs).
2. Go to Auto Learn > Custom Pattern > Suspicious URL Policy.
To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Server Policy Configuration category. For details, see Permissions.
3. Click Create New.
A dialog appears.
4. In Name, type a unique name that can be referenced by other parts of the configuration. Do not use spaces or special characters. The maximum length is 35 characters.
5. Click OK.
6. Click Create New to add an entry to the set.
A dialog appears.
7. From Suspicious URL Name, select the name of a custom suspicious URL rule.
8. Click OK.
9. Repeat the previous steps for each custom suspicious URL rule you want added to the policy.
10. Group custom and predefined suspicious URL groups together (see Grouping all suspicious request URLs).
11. Select the supergroup when configuring an auto-learning profile (see Configuring an auto-learning profile).
Auto Learn > Predefined Pattern > Suspicious URL groups both custom and predefined suspicious URLs together so that they can be selected in an auto-learning profile.
1. Before grouping all suspicious URL patterns, you must first group any custom suspicious URL groups that you want to include. For details, see Grouping custom suspicious request URLs.
2. Go to Auto Learn > Predefined Pattern > Suspicious URL.
To access this part of the web UI, your administrator's account access profile must have Read and Write permission to items in the Server Policy Configuration category. For details, see Permissions.
3. Click Create New.
Alternatively, to clone an existing pattern as the basis for a new group, mark the check box next to it, then click the Clone icon.
A dialog appears.
4. In Name, type a unique name that can be referenced by other parts of the configuration. Do not use spaces or special characters. The maximum length is 35 characters.
5. In Server Type, enable one or more of the predefined, web server-specific suspicious URL sets that you want to detect.
To view detailed descriptions of the types of patterns that each suspicious URL type will detect, see Predefined suspicious request URLs.
If you know that your network’ does not rely on one or more of the listed web server types, disable scans for suspicious access to their administrative URLs in order to improve performance. |
6. From the Custom Suspicious Policy drop-down list, select a group of custom suspicious URLs, that you have configured, if any.
7. Click OK.
8. To use a suspicious URL pattern, select it when configuring an auto-learning profile (see Configuring an auto-learning profile).
Auto-learning profiles are selected in a server policy in conjunction with an inline or offline protection profile. Auto-learning profiles gather data for the auto-learning report from any attacks and parameters that FortiWeb detects.
You cannot edit or delete Default Auto Learn Profile, the predefined auto-learning profile. If you do not want to configure your own auto-learning profile, or are not sure how to, you can use this profile. Alternatively, use it as a starting point by cloning it and then modifying the clone.
Default Auto Learn Profile assumes that you want to learn about all parameters, and allow web crawlers from the search engines Google, Yahoo!, Baidu, and MSN/Bing.
Default Auto Learn Profile uses a predefined data type group, a predefined suspicious URL pattern, and other settings that populate an auto-learning report with a complete data set. It does not use attack signatures that could cause false positives.
You can also use an auto-learning report to generate a new auto-learning profile based on existing data. For details, see Generating a profile from auto-learning data. |
1. Before you create an auto-learning profile, configure the following components:
2. Go to Auto Learn > Auto Learn Profile > Auto Learn Profile.
To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Autolearn Configuration category. For details, see Permissions.
3. Click Create New.
A dialog appears.
4. Configure these settings:
Setting name | Description |
---|---|
Name | Type a unique name that can be referenced by other parts of the configuration. Do not use spaces or special characters. The maximum length is 35 characters. |
Data Type Group |
Select the name of a data type group to use, if any. Auto-learning learns about the names, length, and required presence of these types of parameters in HTTP requests. For details, see Grouping predefined data types. |
Suspicious URL |
Select the name of a suspicious URL pattern to use, if any. Auto-learning considers HTTP requests for these URLs as either malicious vulnerability scanning, data harvesting (a type of web scraping), or administrative login attacks. For details, see Grouping all suspicious request URLs. |
Server Protection Threshold |
Enter a percentage of detected attacks, relative to total hits, that FortiWeb interprets as a false positive for the entire web host. When you use auto-learning to generate a protection profile (see Blocking known attacks & data leaks), FortiWeb disables attack signatures that meet or exceed this overall threshold. For example, if all normal HTTP requests, for whatever reason, sometimes match an attack signature, and therefore do not represent a genuine attack attempt, you could adjust this threshold to reflect the percentage of normal requests that match the attack signature for the overall protected web host. If an average of 99% of requests to the web host match the attack signature, but are actually harmless, you could adjust this setting to Note: This percentage does not have to be greater than Server Protection Exception Threshold. |
Server Protection Exception Threshold |
Enter a percentage of detected attacks, relative to total hits, that FortiWeb interprets as a false positive for specific URLs. When you use auto-learning to generate a protection profile, FortiWeb disables attack signatures that meet or exceed this threshold on specific URLs. For example, if normal HTTP requests to some URLS, for whatever reason, match an attack signature, and therefore do not represent a genuine attack attempt, you could adjust this threshold to reflect the percentage of normal requests that match the attack signature for those specific URLs. If an average of 50% of the requests to some URLs match an attack signature, but are actually harmless, you could adjust this setting to Note: This percentage does not have to be less than Server Protection Threshold. |
Application Policy |
Select a URL interpreter set to use, if any. If the web application embeds parameters in the URL or uses non-standard parameter separators, include an auto-learning adaptor to define how auto-learning should find parameters in the URL. For details, see How to adapt auto-learning to dynamic URLs & unusual parameters. |
5. Click OK.
6. In a server policy, select the auto-learning profile with its protection profile in Web Protection Profile and WAF Auto Learn Profile (see Configuring a server policy). If you do not want to change all Action settings to Alert in each of the protection profile’s components, also enable Monitor Mode.
7. To ensure that the appliance can learn about HTTP/HTTPS requests’ usual page order and other session-related attacks and features, enable the Session Management option in the protection profile.
8. Continue with Running auto-learning.
After you have configured and applied auto-learning profiles, you can use them to collect data for an auto-learning report, and to suggest a configuration.
1. Enable the server policy where you have selected the auto-learning policy for WAF Auto Learn Profile .
2. Route traffic to or through the FortiWeb appliance, depending on your operation mode.
3. Wait for the FortiWeb appliance to gather data.
Time required varies by the rate of legitimate hits for each URL, the parameters that are included with each hit, and the percentage of hits that are attack attempts detected by attack signatures. You can gauge traffic volumes and hits using the Policy Summary widget (see Real Time Monitor widget).
You can pause auto-learning’s data gathering if necessary (see Pausing auto-learning for a URL).
4. Gauge progress by periodically reviewing the auto-learning report, which FortiWeb keeps up-to-date during auto-learning (see Viewing auto-learning reports and Generating a profile from auto-learning data). If parameters are missing, auto-learning is not complete.
5. If there is an unusual number of attacks, there are false positives, or if some auto-learning data is incorrect, you can do one of the following:
6. Continue with Generating a profile from auto-learning data.
Dynamic URLs that you have not configured to be interpreted by a URL replacer cause:
To solve these problems, stop auto-learning for those URLs (right-click them in the auto-learning report and select Stop Learning), then configure a URL replacer. For details, see How to adapt auto-learning to dynamic URLs & unusual parameters.
If you decide later that the URLs were not, in fact, dynamic, you can resume auto-learning: right-click the URL in the auto-learning report, then select Start Learning. Otherwise, for dynamic URLs, you can delete split auto-learning data (see Removing old auto-learning data).
Auto Learn > Auto Learn Report > Auto Learn Report displays the list of reports that the FortiWeb appliance has automatically generated from information gathered by auto-learning profiles.
Primarily, you use auto-learning reports to determine whether or not the auto-learning feature has collected sufficient data to end the auto-learning phase of your installation, and transition to purely applying your security policies (see Generating a profile from auto-learning data).
Sometimes, such as when you change the web applications that are installed on your web servers, you may want to run additional phases of auto-learning. To create a fresh auto-learning report, new protection profiles, or both, you can reset the auto-learning report and delete its data. For details, see Removing old auto-learning data. |
Reports from auto-learning profile data can also provide information about your web servers’ traffic.
Whitelisted items are not be included in auto-learning reports. See Configuring the global object white list. |
Alternatively, for information on normal network traffic, you can use the data analytics feature. See Viewing web site statistics. |
To view auto-learning reports, the Adobe Flash Player browser plug-in is required. |
1. Go to Auto Learn > Auto Learn Report > Auto Learn Report.
To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Autolearn Configuration category. For details, see Permissions.
2. Mark the check box for the report you want to see.
3. Click View.
The report appears, with two panes:
If a report contains multiple pages of results, click the arrows at the bottom of the page to move forward or backwards through the pages of results.
To view report data, click the expand icon ( + ) next to items in the navigation tree and click items to see applicable information. Different tree levels provide different report data.
If URL rewriting is configured, the tree’s URL is the one requested by the client, not the one to which it was rewritten before passing on. |
If the tree contains many URLs that are actually forms of the same URL, or includes sessions IDs, such as: /app/login.asp;jsessionid=xxx;p1=111;p2=123?p3=5555&p4=66aaaaa the web application may use dynamic URLs or unusual parameter separators, and require a URL interpreter for auto-learning to function normally. For details, see How to adapt auto-learning to dynamic URLs & unusual parameters |
You can change the display and content of data using the context menu. To do so, right-click the name of an item in the navigation tree, then select a pop-up menu option:
Setting name | Description |
---|---|
Refresh the Tree | Select to update the display in the navigation pane. If hosts or URLs have been discovered since you last loaded the auto-learning report web page, this will update the tree to reflect those new discoveries. |
Filter the Tree | Select to show or hide HTTP sessions in the report by their HTTP request method and/or other attributes. A pop-up dialog appears. See Filtering an auto-learning report. |
Expand Current Node |
Select to expand the item and all of its subitems. This option has no effect when right-clicking the name of the auto-learning profile. |
Stop Learning |
Select this option if you have determined that the item is a dynamic URL. For details, see Pausing auto-learning for a URL. If you have erroneously categorized the URL as dynamic, to resume learning, right-click the URL again and select Start Learning. |
Clean Data |
Select to remove auto-learning’s statistical data for this item. This may be useful if either:
|
If you select Filter the Tree, a dialog appears.
Depending on its level in the navigation tree, an item may be either a server policy observing multiple hosts, a single host, a common part of a path contained in multiple URLs, or a single requested file. Depending on the part of the navigation tree that you select, the auto-learning report displays:
To show only specific nodes in the URL tree and hide the rest (that is, “filter”), select which attributes that a node or its subnode must satisfy in order to be included in the report’s statistics.
For example, to include only statistics for parts of the URL tree pertaining to HTTP POST
requests to Java server pages (JSP files), you would enter .jsp
in the Search field under URL and enable POST under HTTP Method, disabling in order to filter out all other HTTP methods.
If auto-learning is using a URL interpreter to understand the structure of your application’s URLs, search for the interpreted URL as it appears in the report’s navigation tree, not the real URL as it appears in the HTTP request. |
Tabs, statistics and charts appear on the report display (right-hand) pane. Their appearance varies depending on which level you selected in the navigation tree.
The report display pane contains several feature buttons above the report.
Setting name | Description | |
---|---|---|
Refresh | Click to update the report display to reflect statistics, if any, that have been gathered since you loaded the auto-learning report web page. | |
Generate Config | Click to generate a web protection profile from the auto-learning profile. For instructions, see Generating a profile from auto-learning data. | |
Generate PDF | Click to download a PDF copy of the report. A pop-up dialog appears. Type a file name for the PDF, then click OK. |
The Overview tab provides a statistical summary for all sessions established with the host during the use of the auto-learning profile, or since its auto-learning data was last cleared, whichever is shorter. The contents and buttons of the Overview tab change depending on the level in the navigation tree.
Setting name | Description |
---|---|
Edit Protected Servers |
Click to open a dialog where you can select or deselect IP addresses and/or domain names that will be members of the protected host names group for the generated profile. This button appears only when you select the policy in the navigation pane. |
Edit URL Page |
Click to open a dialog where you can specify that the currently selected URL will be allowed, and whether it will be regarded as a start page for the generated profile. You can also select which action to take if there is a rule violation:
This button appears only when you select a URL in the navigation pane. |
Hits Count |
Click the link to go to the Visits tab. This row appears in the Item column of the Overview table. |
Attack Count |
Click the link to go to the Attacks tab. This row appears in the Item column of the Overview table. |
The Attacks tab provides statistics in both tabular and graphical format on HTTP sessions that contained one of the types of attacks that the web protection profile was configured to detect.
Sometimes, auto-learning reports may contain fewer attacks than you see in the FortiWeb appliance’s attack logs. For details, see About the attack count. |
Depending on the level of the item selected in the navigation pane, the Action and Enable columns may appear. Using these settings, you can override the FortiWeb’s statistically suggested attack protection settings.
To display a pop-up list of an attack type’s protection profile settings estimated from current auto-learning data, click the Detail icon. The dialog that appears may vary by the attack type. You can use it to manually override the estimated settings.
1. From the drop-down list in the Type column, select either:
2. If you selected Custom from Type, from each drop-down list in the Custom column, select one of these options:
If the URL is not susceptible to a specific type of attack, select Off to improve performance. |
3. In the row for each attack type where you have set the drop-down list to Custom, click the Detail icon.
A dialog appears which lists the individual attack signatures for that attack category.
4. For each signature that you want to manually enable, mark its Status check box.
You must mark the Status check box of every signature that you want to enable. Failure to select any signatures will effectively disable attack prevention, even though you have selected On from the Enable drop-down lists for the attack category. |
5. Click OK.
6. From each drop-down list in the Action column, select one of the following options:
403 Forbidden
error message and generate an alert and/or log message.If FortiWeb is deployed behind a NAT load balancer, when using Period Block, you must also define an X-header that indicates the original client’s IP (see Defining your proxies, clients, & X-headers). Failure to do so may cause FortiWeb to block all connections when it detects a violation of this type. |
Sometimes, auto-learning reports may contain fewer attacks than you see in the FortiWeb appliance’s attack logs.
In some cases, the count is low because the attack was attempted, but was targeted towards a URL that did not actually exist on the server (that is, it resulted in an HTTP 404 File Not Found
reply code). Because the URL did not exist, the auto-learning report does not include it in its tree of requested URLs. In other words, the attack was not counted in the report because it did not result in an actual page hit.
The Visits tab displays the following statistics:
Buttons above the tables and charts allow you to edit the profile that auto-learning generates from the Visits tab statistics.
Setting name | Description | |
---|---|---|
Edit Allow Method |
Allows you to specify whether an HTTP request method is allowed in the generated profile using one of the following values:
Available only when a policy is selected in the navigation tree. |
|
Edit Exception Method |
Allows you to specify whether an HTTP request method is an exception in the generated profile using one of the following values:
Available only when individual URL is selected in the navigation tree. |
|
Edit URL Access (In the Most hit URL table and chart section) |
Click this button to open a dialog where you can select which pages will be included in a URL access rule whose Action is Pass (i.e. allow the request and do not generate an attack log message). To include the URL, click and drag it from the column named Available on the right into the column on the left, named URL Access rules with action 'Pass'. Essentially, auto-learning’s assumption in this case is that most page hits are legitimate, so that URLs that are frequently hit should be normally accessible. This button appears only when you select the policy in the navigation pane. |
|
Edit Start Page |
Click this button to open a dialog where you can select which pages will be included in a URL access rule whose Action is Pass (i.e. allow the request and do not generate an attack log message). To include the URL, click and drag it from the column named Available on the right into the column on the left, named URL Access rules with action 'Pass'. This button appears only when you select the policy in the navigation pane. |
|
Edit URL Access (In the Least hit URL table and chart section) |
Click this button to open a dialog where you can select which pages will be included in a URL access rule whose Action is Alert & Deny (i.e. block the request and generate an alert email and/or attack log message). To include the URL, click and drag it from the column named Available on the right into the column on the left, named URL Access rules with action 'Alert & Deny'. Essentially, auto-learning’s assumption in this case is that most page hits are legitimate, so that URLs that are not frequently hit possibly could be a back door or other hidden URL, and therefore should not be accessible. This button appears only when you select the policy in the navigation pane. |
|
Edit URL Access (In the Suspicious URL table and chart section) |
Click this button to open a dialog where you can select which pages will be included in a URL access rule whose Action is Alert & Deny (i.e. block the request and generate an alert email and/or attack log message). To include the URL, click and drag it from the column named Available on the right into the column on the left, named URL Access rules with action 'Alert & Deny'. Essentially, auto-learning’s assumption in this case is that administrative URLs should not be accessible to the general public on the Internet, so that requests for these URLs could be a potential attack or scouting attempt, and should be blocked. This button appears only when you select the policy in the navigation pane. |
|
Edit Content Type (In the Most hit IP table section) |
Allows you to specify which content types FortiWeb includes in any Advanced Protection custom rule it generates using Most hit IP table data. By default, the following content types are selected:
The custom rule FortiWeb generates is designed to detect and prevent web scraping (content scraping) activity. For more information, see Most hit IP table and web scraping detection. |
The Most hit IP table displays the data that FortiWeb uses to automatically generate Advanced Protection custom rules that target web scraping (also called content scraping, web harvesting, or web data extraction). Web scraping is an automated process for collecting information from the web. In many cases, web scraping is performed with the intention of re-using the content without authorization.
For efficiency, web scrapers scan web sites quickly, which generates a file request rate that is noticeably higher than non-automated traffic. However, web scrapers also target dynamic web site content, represented by content types such as XML, soap/XML, JSON, and text/plain, rather than static content like graphics files.
Therefore, for its web scraping custom rule data, FortiWeb collects statistics for both the rate of requests for files (hit rate) and the type of content requested. By creating a rule that accounts for the content type, FortiWeb can provide targeted protection against web scraping in addition to its DoS prevention features, which focus on rate alone.
The Most hit IP table displays the following information:
To determine the most-hit IP addresses, the auto-learning feature divides the auto-learning period into five-minute observation periods. It records the total number of hits that individual IP addresses receive during each five-minute period. The Visits tab displays information about the ten observation periods that had the highest number of hits, including the source IP address and details about the content types of files.
When you generate a profile using the auto-learning report, FortiWeb generates an Advanced Protection custom rule for each selected row in the Most hit IP table. (The first row is selected by default.) FortiWeb converts the data in the row to a baseline maximum hit rate for a specific IP address and content type. When web scraping activity generates a higher hit rate, it triggers the rule action.
Each generated custom rule contains the following filters:
Content Type — Matches requests for files of the specified type.
By default, the following content types are selected:
Use Edit Content Type to customize the values that FortiWeb uses in the filter.
For example, Most hit IP table contains an item with the following values. The values represent hit statistics during an observation period that was among the top ten:
Source IP | Content type | Count | Percentage |
---|---|---|---|
10.200.0.1 | text/html | 44 | 81.48% |
unrecognized content-type | 10 | 18.52% | |
Total | 54 | 100% |
This item generates an Advanced Protection custom rule with a Content Type filter that matches text/html content (one of the default types) and an Occurrence filter with the following values:
Setting | Value | Description |
---|---|---|
Occurrence | 44 |
The number of times clients requested this type of file from the source IP during this top-ten observation period. If the Most hit IP table has statistics for more than one of the selected content types, the value is the total count for all the content types. |
Within | 300 | The length of the observation period. The auto-learning feature collects hits by source IP data using a 5-minute (300 second) observation period. |
Enable Percentage Matching | Selected | Specifies that the filter matches when the number of hits of the specified content types, expressed as a percentage of the total number of hits, exceeds the value of Percentage of Hits. |
Percentage of Hits | 81 |
The number of times clients requested the specified type of file from the source IP, expressed as a percentage of the total number of hits for the source IP in the observation period. If the Most hit IP table has statistics for more than one of the selected content types, the value is the total of all content types. |
Traced By | Source IP | Most hit IP table data is based on hits by source IP, although you can create an Occurrence filter based on User. |
For more information about Advanced Protection custom rules, see Combination access control & rate limiting.
The Parameters tab provides tabular statistics on the parameters and their values as they appeared in HTTP requests, as well as any parameters that were extracted from the URL by a URL interpreter.
This tab appears only for items that are leaf nodes in the navigation tree; that is, they represent a single complete URL as it appeared in a real HTTP request, and therefore could have had those exact associated parameters.
The Name column contains the name of the parameter, exactly as it was observed in the parameter or (for parameters extracted by URL replacers) within the URL.
If the Name column contains part of a URL or the parameter’s value instead of its name, verify the regular expression and back references used in your URL replacer. |
Percentages in the Type Match and Required columns indicate how likely the parameter with that name is of that exact data type, and whether or not the web application requires that input for that URL. The Min. Length and Max. Length columns indicate the likely valid range of length for that input’s value. The Avg. Length column indicates the average length for that input’s value. Together, the columns provide information on what is likely the correct configuration of a profile for that URL.
For example, if Max. Length is 255 but Min. Length is 63 and Avg. Length is 64, before generating a protection profile, you may want to investigate to determine whether 255 is indeed an appropriate maximum input length, since it deviates so much from the norm. In this case, the intended minimum and maximum length might really be 63, but a single malicious observed input had a maximum length of 255.
By default, when you generate a protection profile from auto-learning data, FortiWeb will use these statistics to estimate appropriate input rules. However, if auto-learning suggestions are not appropriate, you can manually override these estimates by using the Set icon and Custom check box before generating a protection profile. For details, see To configure a profile using auto-learning data.
The Cookies tab provides tabular statistics on the name, value, expiry date, and associated URL (path) of each cookie crumb that appeared in HTTP requests.
Cookies that you see in this table can be protected by enabling Cookie Poisoning.
This tab appears only for hosts that use cookies, and for items that are leaf nodes in the navigation tree; that is, they represent a single complete URL as it appeared in a real HTTP request, and therefore could have had those exact cookies.
When viewing a report generated from auto-learning data, you can generate an inline protection profile or an offline protection profile suitable for the HTTP sessions observed. If some observed sessions are not indicative of typical traffic and you do not want to include elements in the generated profile, or you want to select an action other than the default for a type of observed attack, you can selectively change the action for that type of attack.
In addition to the generated profile itself, the FortiWeb appliance also generates all rules and other auxiliary configurations that the profile requires.
For example, FortiWeb observes HTTP PUT
requests that require a password and a user name that is an email address. When it generates a profile, it also uses the data types and maximum lengths of the arguments observed in the HTTP sessions to generate the required parameter validation rules and input rules.
You can edit the generated profiles and auxiliary configurations or use them as the starting point for additional configuration.
1. Go to Auto Learn > Auto Learn Report > Auto Learn Report.
To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Autolearn Configuration category. For details, see Permissions.
2. Mark the check box in the row that corresponds to the auto-learning profile whose data you want to view.
3. Click View.
The report appears.
4. Review the configuration suggestions from auto-learning.
If you want to adjust the behavior of the profile and components to generate, in the left-hand pane, click the expand icon ( + ) next to items to expand the tree, then click the name of the single URL whose protection you want to manually configure.
Buttons and drop-down lists in the report display pane may vary. For most URLs, they enable you to adjust the profile that FortiWeb generates.
Auto-learning suggests an appropriate configuration based upon the traffic that it observed. If a suggestion is not appropriate, you can manually override it.
Configure these settings:
Setting name | Description | |
---|---|---|
Overview tab | ||
Edit Protected Servers |
Click to open a pop-up dialog. Enable or disable the IP addresses and/or domain names that will be members of the generated protected host names group. For details, see Defining your protected/allowed HTTP “Host:” header names. This appears only if you have selected the name of the auto-learning profile in the navigation pane. |
|
Edit URL Page |
Click to open a pop-up dialog. Enable or disable whether the currently selected URL will be included in start pages and white/black IP list rules in the generated profile. This appears only if you have selected a URL in the navigation pane. For more information on those rule types, see Specifying URLs allowed to initiate sessions and Access control. |
|
Attacks Tab | ||
Action and Enable |
Select from the Enable drop-down list to enable or disable detection of each type of attack, and select from Action which action that the generated profile will take. The availability of these lists varies with the level of the item selected in the navigation pane. For details, see the actions in Configuring a protection profile for inline topologies or Configuring a protection profile for an out-of-band topology or asynchronous mode of operation. |
|
Visits Tab | ||
Edit Allow Method |
Click to open a pop-up dialog. Change the Status option to select which HTTP request methods to allow in the generated profile. This appears only if you have selected a profile in the navigation pane. For details, see Configuring a protection profile for inline topologies and Configuring a protection profile for an out-of-band topology or asynchronous mode of operation. |
|
Edit URL Access |
Click to open a pop-up dialog. This appears only if you have selected a profile in the navigation pane. For details, see Access control. |
|
Edit Start Page |
Click to open a pop-up dialog. This appears only if you have selected a profile in the navigation pane. For details, see Specifying allowed HTTP methods. |
|
Edit Exception Method |
Click to open a pop-up dialog. This appears only if you have selected a URL in the navigation pane. For details, see Configuring allowed method exceptions. |
|
Most hit IP table: Edit Content Type |
Click to edit the values that FortiWeb adds to the Content Type filter in an automatically generated Advanced Protection custom rule. This rule is designed to detect web scraping (content scraping) activity. Available only if a policy or host is selected in the navigation pane. For more information, see Most hit IP table and web scraping detection. |
|
Most hit IP table: row selection button |
Selects the data that FortiWeb uses to create an Occurrence filter in an Advanced Protection custom rule in the generated profile. This rule is designed to detect web scraping activity. Available only if a policy or host is selected in the navigation pane. For more information, see Most hit IP table and web scraping detection. |
|
Parameters tab | ||
Set |
Type the data type and maximum length of the parameter, and indicate whether or not the parameter is required input. These settings will appear in the generated parameter validation rule and input rules. For details, see Validating parameters (“input rules”) and Preventing zero-day attacks. Caution: Before you leave the page, mark the Custom check boxes for rows where you have clicked this icon. Failure to do so will cause FortiWeb appliance to discard your settings when you leave the page. |
|
Custom | Before you click Set or leave the page, enable this option for each row whose manual settings you want to save. |
5. Above the display pane, click Generate Config.
A pop-up dialog appears.
6. In Profile Name, type a name prefix, such as generated-profile
.
The FortiWeb appliance adds a dash ( -
) to the profile name followed by a number indicating the year, month, day, and time on which the profile was generated in order to indicate the data on which the profile was based.
7. From Profile Type, select which type of web profile you want to generate, either Inline (to generate an inline protection profile) or Offline (to generate an offline protection profile).
8. Click OK.
The generated profile appears in either:
Adjust configuration items used by the generated profile, such as input rules, when necessary. Generated configuration items are based on auto-learning data current at the time that the profile is generated. Data may have changed while you were reviewing the auto-learning report, and/or after you have generated the profiles. |
If you do not configure any settings, by default, the FortiWeb appliance generates a profile that allows the HTTP GET
method and any other methods whose usage exceeded the threshold, and adds the remaining methods to an allowed method exception. It also creates start page rules and trusted IP rules for the most commonly requested URLs, and blacklist IP addresses that commonly requested suspicious URLs. Attack signatures are disabled or exceptions added according to your configurations in Server Protection Threshold and Server Protection Exception Threshold.
9. Continue with Transitioning out of the auto-learning phase.
As your web servers change, you may periodically want to run auto-learning for them on a smaller scale.
For example, perhaps you will install or update a web application or web server, resulting in new structures and different vulnerabilities.
However, for most day-today use, auto-learning should be disabled and your protection profiles fully applied.
1. To apply a profile generated by auto-learning, select it in Web Protection Profile in a server policy (see Configuring a server policy).
2. If, during auto-learning, any Action in the protection profile or its auxiliary components was set to Alert & Deny or Alert & Erase, verify that those same actions are applied in the protection profile that you generated from auto-learning data. (Incomplete session data due to those actions may have caused auto-learning to be unable to detect those attack types.)
3. If necessary, either:
4. Modify the policy to select your newly generated profile in Web Protection Profile.
5. To validate the configuration, test it (see Testing your installation.)
6. When you are done collecting auto-learning data and generating your configuration, to improve performance, disable auto-learning by deselecting the auto-learning profile in WAF Auto Learn Profile in all server policies.
7. Disable Monitor Mode.
There are many reasons why you may want to delete old auto-learning data.
You can delete old data. Reports and any profiles generated from the auto-learning profile will then include only subsequently gathered data.
Alternatively, you can remove auto-learning data by, when the auto-learning profile’s report is open, right-clicking the node in the left-hand pane, then selecting Clean Data |
1. Go to Auto Learn > Auto Learn Report > Auto Learn Report.
To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Autolearn Configuration category. For details, see Permissions.
2. Either:
3. Click Clean Data.