You are here: How to set up your FortiWeb > Auto-learning

Auto-learning

Protection settings can be configured manually or with assistance from auto-learning.

Auto-learning can teach you a great deal about the threats your web assets face. It also helps you to understand your web applications’ structures and how end-users use them. Most importantly, though, auto-learning can help you to quickly tailor FortiWeb’s configuration to suit your web applications.

For data centers, colocation centers, and complex web applications, auto-learning-assisted configuration can save significant amounts of time compared to purely manual configuration. However, auto-learning is also resource-intensive and can decrease performance while gathering data. For strategies on minimizing the impact to your network, see Running auto-learning and Regular expression performance tips.

Auto-learning discovers the URLs and other characteristics of HTTP and/or HTTPS sessions by observing traffic that is passing to your web servers. To learn about whether the request is legitimate or a potential attack attempt, it performs the following tasks:

By learning from your traffic, the FortiWeb appliance can suggest appropriate configurations, and help you to quickly generate profiles designed specifically for your unique traffic.

See also

IPv6 support

How to adapt auto-learning to dynamic URLs & unusual parameters

When web applications have dynamic URLs or unusual parameter styles, you must adapt auto-learning to recognize them.

By default, auto-learning assumes that your web applications use the most common URL structure:

Some web applications, however, embed parameters within the path structure of the URL, or use unusual or non-uniform parameter separator characters. If you do not configure URL replacers for such applications, it can cause your FortiWeb appliance to gather auto-learning data incorrectly. This can cause the following symptoms:

For example, with Microsoft Outlook Web App (OWA), the user’s login name could be embedded within the path structure of the URL, such as:

/owa/tom/index.html

/owa/mary/index.html

instead of suffixed as a parameter, such as:

/owa/index.html?username=tom

/owa/index.html?username=mary

Auto-learning would continue to create new URLs as new users are added to OWA. Auto-learning would also expend extra resources learning about URLs and parameters that are actually the same. Additionally, auto-learning may not be able to fully learn the application structure, as each user may not request the same URLs.

To solve this, you would create a URL replacer that recognizes the user name within the OWA URL as if it were a standard, suffixed parameter value so that auto-learning can function properly.

See also

Configuring URL interpreters

When using auto-learning, you must define how to interpret dynamic URLs and URLs that include parameters in non-standard ways, such as with different parameter separators (; or #, for example) or by embedding the parameter within the URL’s path structure.

In the web UI, these interpreter plug-ins are called “URL replacers.”

URL replacers match the URL as it appears in the HTTP header of the client’s request (using the regular expression in URL Path) and interpret it into this standard URL formulation:

New URL?New Param=Param Change

For example, if the URL is:

/application/value

and the URL replacer settings are:

Setting name Value
Type Custom-Defined
URL Path (/application)/([^/]+)
New URL $0
Param Change $1
New Param setting

$0 holds this part of the matched URL:

/application

and $1 holds this part of the matched URL:

value

so then the URL will be understood by auto-learning, and displayed in the report, as:

/application?setting=value

Need a refresher on regular expressions? See Regular expression syntax, What are back-references?, and Cookbook regular expressions. You can also use the examples in this section, such as Example: URL interpreter for WordPress.
To create a URL interpreter

1.  Go to Auto Learn > Application Templates > URL Replacer.

To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Autolearn Configuration category. For details, see Permissions.

2.  Click Create New.

3.  Configure these settings:

Name Type a unique name that can be referenced by other parts of the configuration. Do not use spaces or special characters. The maximum length is 35 characters.
Type

Select either:

4.  If you selected Predefined in Type, also configure this setting:

Application Type

Select one of the predefined URL interpreter plug-ins for well-known web applications:

  • JSP — Use the URL replacer designed for Java server pages (JSP) web applications, where parameters are often separated by semi-colons ( ; ).
  • OWA — User the URL replacer designed for default URLs in Microsoft Outlook Web App (OWA), where user name and directory parameters are often embedded within the URL:

    (^/public/)(.*)

    (^/exchange/)([^/]+)/*(([^/]+)/(.*))*

5.  If you selected Custom-Defined in Type, configure these settings:

URL Path

Type a regular expression, such as (^/[^/]+)/(.*), matching all and only the URLs to which the URL replacer should apply. The maximum length is 255 characters.

The pattern does not require a slash ( / ). However, it must at least match URLs that begin with a slash as they appear in the HTTP header, such as /index.html. Do not include the domain name, such as www.example.com.

For examples, see Example: URL interpreter for WordPress.

To test the regular expression against sample text, click the >> (test) icon. This opens the Regular Expression Validator window where you can fine-tune the expression (see Regular expression syntax, What are back-references? and Cookbook regular expressions)

Note: If this URL replacer will be used sequentially in its set of URL replacers, instead of being mutually exclusive, this regular expression should match the URL produced by the previous interpreter, not the original URL from the request.

New URL

Type either a literal URL, such as /index.html, or a regular expression with a back-reference (such as $1) defining how the URL will be interpreted. The maximum length is 255 characters.

Note: Back-references can only refer to capture groups (parts of the expression surrounded with parentheses) within the same URL replacer. Back-references cannot refer to capture groups in other URL replacers.

Param Change Type either the parameter’s literal value, such as user1, or a back-reference (such as $0) defining how the value will be interpreted.
New Param

Type either the parameter’s literal name, such as username, or a back-reference (such as $2) defining how the parameter’s name will be interpreted in the auto-learning report. The maximum length is 255 characters.

Note: Back-references can only refer to capture groups (parts of the expression surrounded with parentheses) within the same URL replacer. Back-references cannot refer to capture groups in other URL replacers.

6.  Click OK.

7.  Group the URL replacers in an application policy (see Grouping URL interpreters).

8.  Select the application policy in one or more auto-learning profiles (see Configuring an auto-learning profile).

9.  Select the auto-learning profiles in server policies (see Configuring a server policy).

See also
Example: URL interpreter for a JSP application

The HTTP request URL from a client is:

/app/login.jsp;jsessionid=xxx;p1=111;p2=123?p3=5555&p4=66aaaaa

which uses semi-colons as parameter separators ( ; ) in the URL, a behavior typical to JSP applications. You would create a URL replacer to recognize the JSP application’s parameters: the semi-colons.

Example: URL replacer for JSP applications
Setting name Value
Type Predefined
Application Type JSP

The predefined JSP interpreter plug-in will interpret the URL as:

/app/login.jsp?p4=66aaaaa&p1=111&p2=123&p3=5555

See also
Example: URL interpreter for Microsoft Outlook Web App 2007

When a client sends requests to Microsoft Outlook Web App (OWA), many of its URLs use structures like this:

/exchange/tom/index.html

/exchange/jane.doe/memo.EML

/exchange/qinlu/2012/1.html

These have user name parameters embedded in the URL. In order for auto-learning to recognize the parameters, you must either:

A custom URL replacer for those URLs could look like this:

Example: URL replacer for Microsoft Outlook Web App — User name structure #1

URL interpreter
Setting name Value
Name OWAusername1
Type Custom-Defined
URL Path (/exchange/)([^/]+)/(.*)
New URL $0$2
Param Change $1
New Param username1

Then the URLs would be recognized by auto-learning as if OWA used a more conventional parameter structure like this:

/exchange/index.html?username1=tom

/exchange/memo.EML?username1=jane.doe

/exchange/2012/1.html?username1=qinlu

Notably, OWA can also include other parameters in the URL, such as a mail folder’s name. Also, OWA can include the user name and folder in more than one way. Therefore multiple URL interpreters are required to match all possible URL structures. In addition to the first URL replacer, you would also configure the following URL replacers and group them into a single set (an auto-learning “application policy”) in order to recognize all possible URLs.

Example: URL replacer for Microsoft Outlook Web App — Folder name structure #1

Sample URL /exchange/archive-folders/2011
URL interpreter
Setting name Value
Name OWAfoldername1
Type Custom-Defined
URL Path (/exchange/)([^/]+/)(.*)
New URL $0
Param Change $1$2
New Param folder1
Results /exchange/?folder1=archive-folders/2011
Example: URL replacer for Microsoft Outlook Web App — User name structure #2

Sample URL /exchange/jane.doe
URL interpreter
Setting name Value
Name OWAusername2
Type Custom-Defined
URL Path (/exchange/)([^/]+\.[^/]+)
New URL $0
Param Change $1
New Param username2
Results /exchange/?username2=jane.doe
Example: URL replacer Microsoft Outlook Web App — Folder name structure #2

Sample URL /public/imap-share-folders/memos
URL interpreter
Setting name Value
Name OWAfoldername2
Type Custom-Defined
URL Path (/public/)([^/]+/)(.*)
New URL $0
Param Change $1$2
New Param folder2
Results /public/?folder2=imap-share-folders/memos
See also
Example: URL interpreter for WordPress

If the HTTP request URL from a client is a slash-delimited chain of multiple parameters, like either of these:

/wordpress/2012/06/05

/index/province/ontario/city/ottawa/street/moodie

then the format is either of these:

/wordpress/value1/value2/value3

/index/param1/value1/param2/value2/param3/value3

In this URL format, there are 3 parameter values (with or without their names) in the URL:

Because each interpreter can only extract a single parameter, you would create 3 URL interpreters, and group them into a set where they are used sequentially — a chain. Each interpreter would use the interpreted output of the previous one as its input, until all parameters had been extracted, at which point the last interpreter would output both the last parameter and the final interpreted URL. FortiWeb would then append parameters back onto the interpreted URL in the standard structure before storing them in the auto-learning data set.

Analysis of a request URL into its interpretation by a chain of URL interpreters

This configuration requires that for every request:

  • the web application includes parameters in the same sequential order, and
  • all parameters are always present

If parameter order or existence vary, this URL interpreter will not work. Requests will not match the URL interpreter set if either param2 or param3 come first, or if any of the parameters are missing. On the opposite end of the spectrum, if the URL interpreter used regular expression capture groups such as (.*) to match anything in any order, i.e.:

/index/(.*)/(.*)/(.*)/(.*)/(.*)/(.*)/

then the regular expression would be too flexible: auto-learning might mistakenly match and learn some of param3’s possible values for param2, and so on.

Example: URL replacer 1 for slash-separated parameters

Setting name Value
Name slash-parameter3
Type Custom-Defined
URL Path /index/param1/(.*)/param2/(.*)/param3/(.*)/
New URL /index/param1/$0/param2/$1/
Param Change $2
New Param param3
Example: URL replacer 2 for slash-separated parameters

Setting name Value
Name slash-parameter2
Type Custom-Defined
URL Path /index/param1/(.*)/param2/(.*)/
New URL /index/param1/$0/
Param Change $1
New Param param2
Example: URL replacer 3 for slash-separated parameters

Setting name Value
Name slash-parameter1
Type Custom-Defined
URL Path /index/param1/(.*)/
New URL /index
Param Change $0
New Param param1

Until you add the URL interpreters to a group, FortiWeb doesn’t know the sequential order.

These URL interpreters will not function correctly if they are not used in that order, because each interpreter’s input is the output from the previous one. So you must set the priorities correctly when referencing each of those interpreters in the set of URL interpreters (Grouping URL interpreters).

Example: URL replacer group for slash-separated parameters — entry 1
Setting name Value
Priority 0
Type URL REPLACER
Plugin Name slash-parameter3
Example: URL replacer group for slash-separated parameters — entry 2
Setting name Value
Priority 1
Type URL REPLACER
Plugin Name slash-parameter2
Example: URL replacer group for slash-separated parameters — entry 3
Setting name Value
Priority 2
Type URL REPLACER
Plugin Name slash-parameter1

Then the URL will be interpreted by auto-learning as if the application used a more conventional and easily understood URL/parameter structure:

/index?param1=value1&param2=value2&param3=value3

See also

Grouping URL interpreters

In order to use URL interpreters with an auto-learning profile, you must group URL replacers into sets.

Sets can be:

To create a custom application policy

1.  Before you create an application policy, first create the URL replacers that it will include (see Configuring URL interpreters).

2.  Go to Auto Learn > Application Templates > Application Policy.

To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Autolearn Configuration category. For details, see Permissions.

3.  Click Create New.

A dialog appears.

4.  In Name, type a name that can be referenced by other parts of the configuration. Do not use spaces or special characters. The maximum length is 35 characters.

5.  Click OK.

6.  Click Create New.

A dialog appears.


7.  From Plugin Name, select an existing URL replacer from the drop-down list.

Rule order affects URL replacer matching and behavior. FortiWeb appliances evaluate URLs for a matching URL replacer starting with the smallest ID number (greatest priority) rule in the list, and continue towards the largest number in the list.

  • If no rule matches, parameters in the URL will not be interpreted.
  • If multiple rules match, the output (New URL) from earlier URL replacers will be used as the input (URL Path) for the next URL replacer, resulting in a chain of multiple interpreted parameters.

8.  Click OK.

9.  Repeat the previous steps for each URL replacer you want added to the policy.

10.  Select the application policy in an auto-learning profile (see Configuring an auto-learning profile).

11.  Select the auto-learning profiles in server policies (see Configuring a server policy).

See also

Recognizing data types

FortiWeb appliances recognize the data types of parameters by matching them with regular expressions. These regular expressions are categorized as either:

See also

Predefined data types

When you install FortiWeb, it already has some data type regular expressions that are predefined — default signatures for common data types so that you do not need to write them yourself. Initial ones are included with the FortiWeb firmware. If your FortiWeb is connected to FortiGuard Security Service updates, it can periodically download updates to its predefined data types. This will provide new and enhanced data types without any effort on your part. Simply use the new signatures in parts of the configuration where they are useful to you.

Predefined data type patterns cannot be used directly. Instead, they must be grouped before they can be used in other areas of the configuration. For details, see Grouping predefined data types.

To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Server Policy Configuration category. For details, see Permissions.

Setting name Description
Pattern The regular expression used to detect the presence of the data type. Parameter values must match the regular expression in order for an auto-learning profile to successfully detect the data type, or for an input rule to allow the input.
Description A description of what the data type is. It may include examples of values that match the regular expression.
 

Select the blue arrow beside a pattern to expand the entry and display the individual rules contained in the entry.

Displays the name of the data type.

  • Address — Canadian postal codes and United States ZIP code and ZIP + 4 codes.
  • Canadian Postal Code — Canadian postal codes such as K2H 7B8 or k2h7b8. Does not match hyphenations such as K2H-7B8.
  • Canadian Province Name and Abbrev. — Modern and older names and abbreviations of Canadian provinces in English, as well as some abbreviations in French, such as Quebec, PEI, Sask, and Nunavut. Does not detect province names in French, such as Québec.
  • Canadian Social Insurance Number — Canadian Social Insurance Numbers (SIN) such as 123-456-789.
  • Chinese Postal Code — Chinese postal codes such as 610000.
  • Country Name and Abbrev. — Country names, codes, and abbreviations as they are known in English, such as CA, Cote d’Ivoire, Brazil, Russian Federation, and Brunei.
  • Credit Card Number — American Express, Carte Blanche, Diners Club, enRoute, Japan Credit Bureau (JCB), Master Card, Novus, and Visa credit card numbers.
  • Date/Time — Dates and times in various formats such as +13:45 for time zone offsets, 1:01 AM, 1am, 23:01:01, and 01.01.30 AM for times, and 31.01.2009, 31/01/2009, 01/31/2000, 2009-01-3, 31-01-2009, 1-31-2009, 01 Jan 2009, 01 JAN 2009, 20-Jan-2009 and February 29, 2009 for dates.
  • Denmark Postal Code — Danish postal code (“postnumre”) such as DK-1499 and dk-1000. Does not match codes that are not prefixed by “DK-”, nor numbers that do not belong to the range of valid codes, such as 123456 or dk 12.
  • Email — Email addresses such as
    admin@example.com
  • GPA — A student’s grade point average, such as 3.5, based upon the 0.0-to-4.0 point system, where an “A” is worth 4 points and an “F” is worth 0 points. Does not match GPAs weighted on the 5 point scale for honors, IB, or AP courses, such as 4.1. The exception is 5.5, which it will match.
  • GUID — A globally unique identifier used to identify partition types in the hard disk’s master boot record (MBR), such as BFDB4D31-3E35-4DAB-AFCA-5E6E5C8F61EA. Partition types are relevant on computers which boot via EFI, using the MBR, instead of an older-style BIOS.
Name
  • Indian Vehicle Number — An Indian Vehicle Registration Number, such as mh 12 bj 1780.
  • IP Address — A public or private IPv4 address, such as 10.0.0.1. Does not match IPv6 addresses.
  • Italian Mobile Phone — Italian mobile phone numbers with the prefix for international calls, such as +393471234567, or without, such as 3381234567. Does not match numbers with a dash or space after the area code, nor VoIP or land lines.
  • Kuwait Civil ID — Personal identification number for Kuwait, such as 273032401586. Must begin with 1, 2, or 3, and follow all other number patterns for valid civil IDs.
  • Level 1 Password — A string of at least 6 characters, with one or more each of lower-case characters, upper-case characters, and digits, such as aBc123. Level 1 passwords are “weak” passwords, generally easier to crack than level 2 passwords.
  • Level 2 Password — A string of at least 8 characters, with one or more each of lower-case characters, upper-case characters, digits, and special characters, such as aBc123$%. Level 2 passwords are moderately strong.
  • Markup/Code — HTML comments, wiki code, hexadecimal HTML color codes, quoted strings in VBScript and ANSI SQL, SQL statements, and RTF bookmarks such as:
    • #00ccff, <!‑‑A comment.‑‑>
    • [link url="http://example.com/url?var=A&var2=B"]
    • SELECT * FROM TABLE
    • {\*\bkmkstart TagAmountText}
    Does not match ANSI escape codes. They are detected as strings.
  • Microsoft Product Key — An alphanumeric key for activation of Microsoft software, such as ABC12-34DEF-GH567-IJK89-LM0NP. Does not match keys which are non-hyphenated, nor where letters are not capitalized.
  • Netherlands Postal Code — Netherlands postal codes (“postcodes”) such as 3000 AA or 3000AA. Does not match postal codes written in lower-case letters, such as 3000aa.
  • NINO — A United Kingdom National Insurance Number (NINO), such as AB123456D. Does not match NINOs written in lower-case letters, such as ab123456d.
  • Numbers — Numbers in various monetary, scientific, decimal, comma-separated value (CSV), and other formats such as 123, +1.23, $1,234,567.89, 1'235.140, and -123.45e-6. Does not detect some types, such as hexadecimal numbers (which are instead detected as strings or code), and US Social Security Numbers (which are instead detected as strings).
 
  • Personal Name — A person’s full or abbreviated name in English. It can contain punctuation, such as A.J Schwartz, Jean-Pierre Ferko, or Jane O’Donnell. Does not match names written in other languages, such as Renée Wächter or 林美
  • Phone — Australian, United States, and Indian telephone numbers in various formats such as (123)456-7890, 1.123.456.7890, 0732105432, and +919847444225.
  • Quebec Postal CodePostal codes written in the style sometimes used by Quebecers, with hyphens between the two parts, such as h2j-3c4 or H2J-3C4.
  • StringsAny string of characters, including all other data types, such as alphanumeric words, credit card numbers, United States social security numbers (SSN), UK vehicle registration numbers, ANSI escape codes, and hexadecimal numbers in formats such as user1, 123-45-6789, ABC 123 A, 4125632152365, [32mHello, and 8ECCA04F.
  • Swedish Personal Number — Personal identification number (“personnummer”) for Sweden, such as 19811116-7845. Must be hyphenated. Does not match PINs for persons whose age is 100 or greater.
  • Swedish Postal CodePostal codes (“postnummer”) for Sweden, with or without spaces or hyphens, such as S 751 70, s75170, or S-751-70. Requires the initial S or s letter. Does not match invalid postal codes such as ones that begin with a 0, or ones that do not begin with the letter S or s.
  • UAE Land Phone Telephone number for the United Arab Emirates, such as 04 - 3452499 or 04 3452499. Does not match phone numbers beginning with 01 or 08.
  • UK Bank Sort CodeBank sort codes for the United Kingdom, such as 09‑01‑29. Must be hyphenated.
  • Unix Device Name — Standard Linux or UNIX non-loopback wired Ethernet network interface names, such as eth0. Does not match names for any other type of device, such as lo, hdda, or ppp.
 
  • URIUniform resource identifiers (URI) such as:
    http://www.example.com
    ftp://ftp.example.com
    mailto:admin@example.com
  • US Social Security Number — United States Social Security Numbers (SSN) such as 123-45-6789.
  • US State Name and Abbrev. — United States state names and modern postal abbreviations such as HI and Wyoming. Does not detect older postal abbreviations ending with periods ( . ), such as Fl. or Wyo.
  • US Street Address — United States city and street address, possibly including an apartment or suite number. City and street may be either separated with a space or written on two lines according to US postal conventions, such as:
    123 Main Street Suite #101
    Honolulu, HI 10001
    Does not match:
  • ZIP + 4 codes that include spaces, or do not have a hyphen (e.g. “10001 - 1111” or “10001 1111”)
  • city abbreviations of 2 characters (e.g. “NY” instead of “NYC”)
  • Washington D.C. addresses
  • US ZIP Code — United States ZIP code and ZIP + 4 codes such as 34285‑3210.
  • Windows File Name — A valid windows file name, such as Untitled.txt. Does not match file extensions, or file names without their extensions.
See also

Grouping predefined data types

A data type group defines a set of predefined data types (see Predefined data types) that can be used in an auto-learning profile.

For example, if you include the Email data type in the data type group, auto-learning profiles that use the data type group might discover that your web applications use a parameter named username whose value is an email address.

The predefined data type group, named predefine-data-type-group, cannot be edited or deleted.

To configure a predefined data type group

1.  Go to Auto Learn > Predefined Pattern > Data Type Group.

To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Server Policy Configuration category. For details, see Permissions.

2.  Click Create New.

A dialog appears.

3.  In Name, type a unique name that can be referenced by other parts of the configuration. Do not use spaces or special characters. The maximum length is 35 characters.

4.  In Type, mark the check box of each predefined data type that you want to include in the set, such as Email or Canadian Social Insurance Number.

If you know that your network’s HTTP sessions do not include a specific data type, omit it from the data type group to improve performance. The FortiWeb appliance will not expend resources scanning traffic for that data type.

To examine the regular expressions for each data type, see Predefined data types.

5.  Click OK.

6.  To use a data type group, select it when configuring either an auto-learning profile (see Configuring an auto-learning profile) or input rule (see Validating parameters (“input rules”)).

See also

Recognizing suspicious requests

FortiWeb appliances can recognize known attacks by comparing each request to a signature. How, then, does it recognize requests that aren’t known to be an attack, or aren’t always an attack, but might be?

FortiWeb uses several methods for this:

Web applications’ administrative URLs often should not be accessible by clients on the Internet, and therefore any request for those URLs from source IP addresses on the Internet may represent an attempt to scout your web servers in advance of an attack. (Exceptions include hosting providers, whose clients may span the globe and often configure their own web applications.) Administrative requests from the Internet are therefore suspicious: the host may have been compromised by a rootkit, or its administrative login credentials may have been stolen via spyware, phishing, or social engineering.

FortiWeb appliances can compare each request URL with regular expressions that define known administrative URLs, and log and/or block these requests.

Regular expressions for suspicious requests by URL are categorized as:

Both types can be grouped into a set that can be used in auto-learning profiles.

See also

Predefined suspicious request URLs

Predefined regular expressions can be used by auto-learning to detect requests that are suspicious because they are for a URL that provides administrative access to the web server, servlet, or web application, such as:

/admin.php

/conf/Catalina/localhost/admin.xml

or access to its back-end cache, data files, or Berkeley databases, such as:

/local/notesdata

Normally, requests for these URLs should only originate from a trusted network such as your management computers, not from the Internet. (Exceptions include hosting providers, whose clients around the globe configure their own web applications.) Therefore these requests are a good candidate for URL access control rules.

Many signatures exist for popular web servers and applications such as Apache, nginx IIS, Tomcat, and Subversion. Known suspicious request URLs can be updated. See Connecting to FortiGuard services.

To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Server Policy Configuration category. For details, see Permissions.

Auto Learn > Predefined Pattern > URL Pattern (image cropped)

Setting name Description
Name

The name of the predefined suspicious URL pattern set.

To display the patterns it contains, click the blue arrow next to the name.

Pattern When you click a blue arrow to expand a suspicious URL pattern, this column displays the regular expression used to detect the presence of the suspicious URL in a client’s request.
Description When you click a blue arrow to expand a data type, this column displays a description of the URLs matched by this pattern, such as Apache web server administrative web UI files or IBM Lotus Domino data.
See also

Configuring custom suspicious request URLs

To augment FortiWeb’s predefined list of suspicious request URLs, you can configure your own.

To create a custom suspicious request URL pattern

1.  Go to Auto Learn > Custom Pattern > Suspicious URL Rule.

To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Server Policy Configuration category. For details, see Permissions.

2.  Click Create New.

A dialog appears.

3.  In Name, type a unique name that can be referenced by other parts of the configuration. Do not use spaces or special characters. The maximum length is 35 characters.

4.  In URL Expression, enter a regular expression that defines this suspicious URL, such as ^/my_admin_panel.jsp.

To test the regular expression against sample text, click the >> (test) icon. This opens the Regular Expression Validator window where you can fine-tune the expression (see Regular expression syntax and Cookbook regular expressions).

5.  Click OK.

6.  Group custom suspicious URL patterns (see Grouping custom suspicious request URLs).

7.  Group custom and predefined suspicious URL groups together (see Grouping all suspicious request URLs).

8.  Select the supergroup when configuring an auto-learning profile (see Configuring an auto-learning profile).

See also

Grouping custom suspicious request URLs

Before you can use them, you must first group custom and predefined suspicious URLs.

To configure a custom suspicious URL policy

1.  Before you can create a custom suspicious URL rule, you must first define one or more custom suspicious URLs (see Configuring custom suspicious request URLs).

2.  Go to Auto Learn > Custom Pattern > Suspicious URL Policy.

To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Server Policy Configuration category. For details, see Permissions.

3.  Click Create New.

A dialog appears.

4.  In Name, type a unique name that can be referenced by other parts of the configuration. Do not use spaces or special characters. The maximum length is 35 characters.

5.  Click OK.

6.  Click Create New to add an entry to the set.

A dialog appears.

7.  From Suspicious URL Name, select the name of a custom suspicious URL rule.

8.  Click OK.

9.  Repeat the previous steps for each custom suspicious URL rule you want added to the policy.

10.  Group custom and predefined suspicious URL groups together (see Grouping all suspicious request URLs).

11.  Select the supergroup when configuring an auto-learning profile (see Configuring an auto-learning profile).

See also

Grouping all suspicious request URLs

Auto Learn > Predefined Pattern > Suspicious URL groups both custom and predefined suspicious URLs together so that they can be selected in an auto-learning profile.

To configure a suspicious URL pattern group

1.  Before grouping all suspicious URL patterns, you must first group any custom suspicious URL groups that you want to include. For details, see Grouping custom suspicious request URLs.

2.  Go to Auto Learn > Predefined Pattern > Suspicious URL.

To access this part of the web UI, your administrator's account access profile must have Read and Write permission to items in the Server Policy Configuration category. For details, see Permissions.

3.  Click Create New.

Alternatively, to clone an existing pattern as the basis for a new group, mark the check box next to it, then click the Clone icon.

A dialog appears.


4.  In Name, type a unique name that can be referenced by other parts of the configuration. Do not use spaces or special characters. The maximum length is 35 characters.

5.  In Server Type, enable one or more of the predefined, web server-specific suspicious URL sets that you want to detect.

To view detailed descriptions of the types of patterns that each suspicious URL type will detect, see Predefined suspicious request URLs.

If you know that your network’ does not rely on one or more of the listed web server types, disable scans for suspicious access to their administrative URLs in order to improve performance.

6.  From the Custom Suspicious Policy drop-down list, select a group of custom suspicious URLs, that you have configured, if any.

7.  Click OK.

8.  To use a suspicious URL pattern, select it when configuring an auto-learning profile (see Configuring an auto-learning profile).

See also

Configuring an auto-learning profile

Auto-learning profiles are selected in a server policy in conjunction with an inline or offline protection profile. Auto-learning profiles gather data for the auto-learning report from any attacks and parameters that FortiWeb detects.

You cannot edit or delete Default Auto Learn Profile, the predefined auto-learning profile. If you do not want to configure your own auto-learning profile, or are not sure how to, you can use this profile. Alternatively, use it as a starting point by cloning it and then modifying the clone.

Default Auto Learn Profile assumes that you want to learn about all parameters, and allow web crawlers from the search engines Google, Yahoo!, Baidu, and MSN/Bing.

Default Auto Learn Profile uses a predefined data type group, a predefined suspicious URL pattern, and other settings that populate an auto-learning report with a complete data set. It does not use attack signatures that could cause false positives.

To configure an auto-learning profile
You can also use an auto-learning report to generate a new auto-learning profile based on existing data. For details, see Generating a profile from auto-learning data.

1.  Before you create an auto-learning profile, configure the following components:

2.  Go to Auto Learn > Auto Learn Profile > Auto Learn Profile.

To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Autolearn Configuration category. For details, see Permissions.

3.  Click Create New.

A dialog appears.

4.  Configure these settings:

Setting name Description
Name Type a unique name that can be referenced by other parts of the configuration. Do not use spaces or special characters. The maximum length is 35 characters.
Data Type Group

Select the name of a data type group to use, if any.

Auto-learning learns about the names, length, and required presence of these types of parameters in HTTP requests. For details, see Grouping predefined data types.

Suspicious URL

Select the name of a suspicious URL pattern to use, if any.

Auto-learning considers HTTP requests for these URLs as either malicious vulnerability scanning, data harvesting (a type of web scraping), or administrative login attacks. For details, see Grouping all suspicious request URLs.

Server Protection Threshold

Enter a percentage of detected attacks, relative to total hits, that FortiWeb interprets as a false positive for the entire web host.

When you use auto-learning to generate a protection profile (see Blocking known attacks & data leaks), FortiWeb disables attack signatures that meet or exceed this overall threshold.

For example, if all normal HTTP requests, for whatever reason, sometimes match an attack signature, and therefore do not represent a genuine attack attempt, you could adjust this threshold to reflect the percentage of normal requests that match the attack signature for the overall protected web host. If an average of 99% of requests to the web host match the attack signature, but are actually harmless, you could adjust this setting to 99. When requests to this web site meet the threshold, FortiWeb disables scanning for this attack signature for the entire web site.

Note: This percentage does not have to be greater than Server Protection Exception Threshold.

Server Protection Exception Threshold

Enter a percentage of detected attacks, relative to total hits, that FortiWeb interprets as a false positive for specific URLs.

When you use auto-learning to generate a protection profile, FortiWeb disables attack signatures that meet or exceed this threshold on specific URLs.

For example, if normal HTTP requests to some URLS, for whatever reason, match an attack signature, and therefore do not represent a genuine attack attempt, you could adjust this threshold to reflect the percentage of normal requests that match the attack signature for those specific URLs. If an average of 50% of the requests to some URLs match an attack signature, but are actually harmless, you could adjust this setting to 50. Other URLs on the web host, where the signature is not disabled, would still be subject to scanning by the attack signature.

Note: This percentage does not have to be less than Server Protection Threshold.

Application Policy

Select a URL interpreter set to use, if any.

If the web application embeds parameters in the URL or uses non-standard parameter separators, include an auto-learning adaptor to define how auto-learning should find parameters in the URL. For details, see How to adapt auto-learning to dynamic URLs & unusual parameters.

5.  Click OK.

6.  In a server policy, select the auto-learning profile with its protection profile in Web Protection Profile and WAF Auto Learn Profile (see Configuring a server policy). If you do not want to change all Action settings to Alert in each of the protection profile’s components, also enable Monitor Mode.

Auto-learning is resource-intensive, and can decrease performance. If performance becomes unacceptable, consider selecting the auto-learning profile in only a few policies at a time.

Alternatively or in addition, briefly run a first phase of auto-learning, then disable features which are obviously unnecessary according to auto-learning data, and begin a second, more lightweight phase of auto-learning.

7.  To ensure that the appliance can learn about HTTP/HTTPS requests’ usual page order and other session-related attacks and features, enable the Session Management option in the protection profile.

8.  Continue with Running auto-learning.

See also

Running auto-learning

After you have configured and applied auto-learning profiles, you can use them to collect data for an auto-learning report, and to suggest a configuration.

To form configuration suggestions using auto-learning

1.  Enable the server policy where you have selected the auto-learning policy for WAF Auto Learn Profile .

2.  Route traffic to or through the FortiWeb appliance, depending on your operation mode.

For best results, do not use incomplete or unrealistic traffic.

To minimize performance impacts, consider running an initial phase of auto-learning while your FortiWeb is operating in offline protection mode before you transition to your final choice of operation mode.

3.  Wait for the FortiWeb appliance to gather data.

To quickly reduce risk of attack while auto-learning is in progress, in the protection profile and its components, for attacks and disclosures that you are sure cannot be false positives, set the Action to Alert & Deny or Alert & Erase.

Time required varies by the rate of legitimate hits for each URL, the parameters that are included with each hit, and the percentage of hits that are attack attempts detected by attack signatures. You can gauge traffic volumes and hits using the Policy Summary widget (see Real Time Monitor widget).

For faster results, from an external IP, connect to the web site and access all URLs that a legitimate client would. Provide valid parameters. This activity populates auto-learning data with an initial, realistic set.

To improve performance during auto-learning, run it in a few phases.

For example, after an initial short phase of auto-learning, generate a protection profile with the most obvious attack settings. Then delete the auto-learning data, revise the protection profile to omit auto-learning for the settings that you have already discovered, and start the next phase of auto-learning.

Alternatively or additionally, you can run auto-learning on only a few policies at a time.

You can pause auto-learning’s data gathering if necessary (see Pausing auto-learning for a URL).

4.  Gauge progress by periodically reviewing the auto-learning report, which FortiWeb keeps up-to-date during auto-learning (see Viewing auto-learning reports and Generating a profile from auto-learning data). If parameters are missing, auto-learning is not complete.

Auto-learning considers URLs up to approximately 128 characters long (assuming single-byte character encoding, after FortiWeb has decoded any nested hexadecimal or other URL encoding — therefore, the limit is somewhat dynamic). If the URL is longer than that buffer size, auto-learning cannot learn it, and therefore ignores it. No event log is generated.

In those cases, you must manually configure FortiWeb protection settings for the URL, rather than discovering recommended protection settings via auto-learning. However, you may be able to re-use the settings recommended for other, shorter URLs by auto-learning.

For example, if auto-learning discovers an email address parameter, it probably should have the same input constraints regardless of which URL uses it.

5.  If there is an unusual number of attacks, there are false positives, or if some auto-learning data is incorrect, you can do one of the following:

6.  Continue with Generating a profile from auto-learning data.

Pausing auto-learning for a URL

Dynamic URLs that you have not configured to be interpreted by a URL replacer cause:

To solve these problems, stop auto-learning for those URLs (right-click them in the auto-learning report and select Stop Learning), then configure a URL replacer. For details, see How to adapt auto-learning to dynamic URLs & unusual parameters.

If you decide later that the URLs were not, in fact, dynamic, you can resume auto-learning: right-click the URL in the auto-learning report, then select Start Learning. Otherwise, for dynamic URLs, you can delete split auto-learning data (see Removing old auto-learning data).

See also

Viewing auto-learning reports

Auto Learn > Auto Learn Report > Auto Learn Report displays the list of reports that the FortiWeb appliance has automatically generated from information gathered by auto-learning profiles.

Primarily, you use auto-learning reports to determine whether or not the auto-learning feature has collected sufficient data to end the auto-learning phase of your installation, and transition to purely applying your security policies (see Generating a profile from auto-learning data).

Sometimes, such as when you change the web applications that are installed on your web servers, you may want to run additional phases of auto-learning.

To create a fresh auto-learning report, new protection profiles, or both, you can reset the auto-learning report and delete its data. For details, see Removing old auto-learning data.

Reports from auto-learning profile data can also provide information about your web servers’ traffic.

Whitelisted items are not be included in auto-learning reports. See Configuring the global object white list.

 

Alternatively, for information on normal network traffic, you can use the data analytics feature. See Viewing web site statistics.
To view a report generated from auto-learning data
To view auto-learning reports, the Adobe Flash Player browser plug-in is required.

1.  Go to Auto Learn > Auto Learn Report > Auto Learn Report.

To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Autolearn Configuration category. For details, see Permissions.

2.  Mark the check box for the report you want to see.

3.  Click View.

The report appears, with two panes:

If a report contains multiple pages of results, click the arrows at the bottom of the page to move forward or backwards through the pages of results.

Parts of auto-learning reports

See also

Using the report navigation pane

To view report data, click the expand icon ( + ) next to items in the navigation tree and click items to see applicable information. Different tree levels provide different report data.

Parts of the report navigation pane

If URL rewriting is configured, the tree’s URL is the one requested by the client, not the one to which it was rewritten before passing on.

 

If the tree contains many URLs that are actually forms of the same URL, or includes sessions IDs, such as:

/app/login.asp;jsessionid=xxx;p1=111;p2=123?p3=5555&p4=66aaaaa

the web application may use dynamic URLs or unusual parameter separators, and require a URL interpreter for auto-learning to function normally. For details, see How to adapt auto-learning to dynamic URLs & unusual parameters

You can change the display and content of data using the context menu. To do so, right-click the name of an item in the navigation tree, then select a pop-up menu option:

Setting name Description
Refresh the Tree Select to update the display in the navigation pane. If hosts or URLs have been discovered since you last loaded the auto-learning report web page, this will update the tree to reflect those new discoveries.
Filter the Tree Select to show or hide HTTP sessions in the report by their HTTP request method and/or other attributes. A pop-up dialog appears. See Filtering an auto-learning report.
Expand Current Node

Select to expand the item and all of its subitems.

This option has no effect when right-clicking the name of the auto-learning profile.

Stop Learning

Select this option if you have determined that the item is a dynamic URL. For details, see Pausing auto-learning for a URL.

If you have erroneously categorized the URL as dynamic, to resume learning, right-click the URL again and select Start Learning.

Clean Data

Select to remove auto-learning’s statistical data for this item. This may be useful if either:

  • You want to clear the data set to begin fresh for a new phase of auto-learning.
  • You know that the inputs required by a specific URL have changed since you initially began learning about a web site’s parameters. This could happen when you upgrade a web application.
  • The item was an instance of a dynamic URL, and you did not apply a matching URL interpreter, and therefore the data was corrupted.

See Removing old auto-learning data.

If you select Filter the Tree, a dialog appears.

Filtering an auto-learning report

Depending on its level in the navigation tree, an item may be either a server policy observing multiple hosts, a single host, a common part of a path contained in multiple URLs, or a single requested file. Depending on the part of the navigation tree that you select, the auto-learning report displays:

To show only specific nodes in the URL tree and hide the rest (that is, “filter”), select which attributes that a node or its subnode must satisfy in order to be included in the report’s statistics.

For example, to include only statistics for parts of the URL tree pertaining to HTTP POST requests to Java server pages (JSP files), you would enter .jsp in the Search field under URL and enable POST under HTTP Method, disabling in order to filter out all other HTTP methods.

If auto-learning is using a URL interpreter to understand the structure of your application’s URLs, search for the interpreted URL as it appears in the report’s navigation tree, not the real URL as it appears in the HTTP request.
See also

Using the report display pane

Tabs, statistics and charts appear on the report display (right-hand) pane. Their appearance varies depending on which level you selected in the navigation tree.

The report display pane contains several feature buttons above the report.

Buttons at the top of the auto-learning report’s display pane

Setting name Description
Refresh Click to update the report display to reflect statistics, if any, that have been gathered since you loaded the auto-learning report web page.
Generate Config Click to generate a web protection profile from the auto-learning profile. For instructions, see Generating a profile from auto-learning data.
Generate PDF Click to download a PDF copy of the report. A pop-up dialog appears. Type a file name for the PDF, then click OK.
Overview tab

The Overview tab provides a statistical summary for all sessions established with the host during the use of the auto-learning profile, or since its auto-learning data was last cleared, whichever is shorter. The contents and buttons of the Overview tab change depending on the level in the navigation tree.

Auto-learning report Overview tab

Setting name Description
Edit Protected Servers

Click to open a dialog where you can select or deselect IP addresses and/or domain names that will be members of the protected host names group for the generated profile.

This button appears only when you select the policy in the navigation pane.

Edit URL Page

Click to open a dialog where you can specify that the currently selected URL will be allowed, and whether it will be regarded as a start page for the generated profile. You can also select which action to take if there is a rule violation:

  • Alert & DenyBlock the request (reset the connection) and generate an alert email and/or log message.

    You can customize the web page that FortiWeb returns to the client with the HTTP status code. See Customizing error and authentication pages (replacement messages).

  • Continue — Continue by evaluating any subsequent rules defined in the web protection profile (see Sequence of scans). If no other rules are violated, allow the request. If multiple rules are violated, a single request will generate multiple attack log messages and/or alert email.
  • PassAllow the request. Do not generate an alert email and/or log message.

This button appears only when you select a URL in the navigation pane.

Hits Count

Click the link to go to the Visits tab.

This row appears in the Item column of the Overview table.

Attack Count

Click the link to go to the Attacks tab.

This row appears in the Item column of the Overview table.

Attacks tab

The Attacks tab provides statistics in both tabular and graphical format on HTTP sessions that contained one of the types of attacks that the web protection profile was configured to detect.

Sometimes, auto-learning reports may contain fewer attacks than you see in the FortiWeb appliance’s attack logs. For details, see About the attack count.
Auto-learning report Attacks tab

Depending on the level of the item selected in the navigation pane, the Action and Enable columns may appear. Using these settings, you can override the FortiWeb’s statistically suggested attack protection settings.

To display a pop-up list of an attack type’s protection profile settings estimated from current auto-learning data, click the Detail icon. The dialog that appears may vary by the attack type. You can use it to manually override the estimated settings.

To override configuration suggested by auto-learning for a specific attack type

1.  From the drop-down list in the Type column, select either:

2.  If you selected Custom from Type, from each drop-down list in the Custom column, select one of these options:

If the URL is not susceptible to a specific type of attack, select Off to improve performance.
Auto-learning report Attacks tab — Manually enabling attack signatures

3.  In the row for each attack type where you have set the drop-down list to Custom, click the Detail icon.

A dialog appears which lists the individual attack signatures for that attack category.

4.  For each signature that you want to manually enable, mark its Status check box.

You must mark the Status check box of every signature that you want to enable. Failure to select any signatures will effectively disable attack prevention, even though you have selected On from the Enable drop-down lists for the attack category.

5.  Click OK.

6.  From each drop-down list in the Action column, select one of the following options:

If FortiWeb is deployed behind a NAT load balancer, when using Period Block, you must also define an X-header that indicates the original client’s IP (see Defining your proxies, clients, & X-headers). Failure to do so may cause FortiWeb to block all connections when it detects a violation of this type.
About the attack count

Sometimes, auto-learning reports may contain fewer attacks than you see in the FortiWeb appliance’s attack logs.

In some cases, the count is low because the attack was attempted, but was targeted towards a URL that did not actually exist on the server (that is, it resulted in an HTTP 404 File Not Found reply code). Because the URL did not exist, the auto-learning report does not include it in its tree of requested URLs. In other words, the attack was not counted in the report because it did not result in an actual page hit.

Visits tab

The Visits tab displays the following statistics:

Buttons above the tables and charts allow you to edit the profile that auto-learning generates from the Visits tab statistics.

Auto-learning report Visits tab (image truncated)

Setting name Description
Edit Allow Method

Allows you to specify whether an HTTP request method is allowed in the generated profile using one of the following values:

  • On — Enable the method in the generated profile.
  • Off — Disable the method in the generated profile.
  • Default — When you generate a protection profile, FortiWeb automatically determines whether to enable or disable the HTTP method in the profile based on current auto-learning data.

Available only when a policy is selected in the navigation tree.

Edit Exception Method

Allows you to specify whether an HTTP request method is an exception in the generated profile using one of the following values:

  • On — Enable the method for the URL in the generated profile.
  • Off — Disable the method for the URL in the generated profile.
  • Default — When you generate a protection profile, FortiWeb automatically determines whether to enable or disable the HTTP method in the profile based on current auto-learning data. FortiWeb will use whichever setting is indicated by the current auto-learning data.

Available only when individual URL is selected in the navigation tree.

Edit URL Access

(In the Most hit URL table and chart section)

Click this button to open a dialog where you can select which pages will be included in a URL access rule whose Action is Pass (i.e. allow the request and do not generate an attack log message). To include the URL, click and drag it from the column named Available on the right into the column on the left, named URL Access rules with action 'Pass'.

Essentially, auto-learning’s assumption in this case is that most page hits are legitimate, so that URLs that are frequently hit should be normally accessible.

This button appears only when you select the policy in the navigation pane.

Edit Start Page

Click this button to open a dialog where you can select which pages will be included in a URL access rule whose Action is Pass (i.e. allow the request and do not generate an attack log message). To include the URL, click and drag it from the column named Available on the right into the column on the left, named URL Access rules with action 'Pass'.

This button appears only when you select the policy in the navigation pane.

Edit URL Access

(In the Least hit URL table and chart section)

Click this button to open a dialog where you can select which pages will be included in a URL access rule whose Action is Alert & Deny (i.e. block the request and generate an alert email and/or attack log message). To include the URL, click and drag it from the column named Available on the right into the column on the left, named URL Access rules with action 'Alert & Deny'.

Essentially, auto-learning’s assumption in this case is that most page hits are legitimate, so that URLs that are not frequently hit possibly could be a back door or other hidden URL, and therefore should not be accessible.

This button appears only when you select the policy in the navigation pane.

Edit URL Access

(In the Suspicious URL table and chart section)

Click this button to open a dialog where you can select which pages will be included in a URL access rule whose Action is Alert & Deny (i.e. block the request and generate an alert email and/or attack log message). To include the URL, click and drag it from the column named Available on the right into the column on the left, named URL Access rules with action 'Alert & Deny'.

Essentially, auto-learning’s assumption in this case is that administrative URLs should not be accessible to the general public on the Internet, so that requests for these URLs could be a potential attack or scouting attempt, and should be blocked.

This button appears only when you select the policy in the navigation pane.

Edit Content Type

(In the Most hit IP table section)

Allows you to specify which content types FortiWeb includes in any Advanced Protection custom rule it generates using Most hit IP table data.

By default, the following content types are selected:

  • application/soap+xml
  • application/xml(or)text/xml
  • text/html
  • text/plain
  • application/json

The custom rule FortiWeb generates is designed to detect and prevent web scraping (content scraping) activity.

For more information, see Most hit IP table and web scraping detection.

Most hit IP table and web scraping detection

The Most hit IP table displays the data that FortiWeb uses to automatically generate Advanced Protection custom rules that target web scraping (also called content scraping, web harvesting, or web data extraction). Web scraping is an automated process for collecting information from the web. In many cases, web scraping is performed with the intention of re-using the content without authorization.

For efficiency, web scrapers scan web sites quickly, which generates a file request rate that is noticeably higher than non-automated traffic. However, web scrapers also target dynamic web site content, represented by content types such as XML, soap/XML, JSON, and text/plain, rather than static content like graphics files.

Therefore, for its web scraping custom rule data, FortiWeb collects statistics for both the rate of requests for files (hit rate) and the type of content requested. By creating a rule that accounts for the content type, FortiWeb can provide targeted protection against web scraping in addition to its DoS prevention features, which focus on rate alone.

The Most hit IP table displays the following information:

To determine the most-hit IP addresses, the auto-learning feature divides the auto-learning period into five-minute observation periods. It records the total number of hits that individual IP addresses receive during each five-minute period. The Visits tab displays information about the ten observation periods that had the highest number of hits, including the source IP address and details about the content types of files.

When you generate a profile using the auto-learning report, FortiWeb generates an Advanced Protection custom rule for each selected row in the Most hit IP table. (The first row is selected by default.) FortiWeb converts the data in the row to a baseline maximum hit rate for a specific IP address and content type. When web scraping activity generates a higher hit rate, it triggers the rule action.

Each generated custom rule contains the following filters:

For example, Most hit IP table contains an item with the following values. The values represent hit statistics during an observation period that was among the top ten:

Source IP Content type Count Percentage
10.200.0.1 text/html 44 81.48%
unrecognized content-type 10 18.52%
Total 54 100%

This item generates an Advanced Protection custom rule with a Content Type filter that matches text/html content (one of the default types) and an Occurrence filter with the following values:

Setting Value Description
Occurrence 44

The number of times clients requested this type of file from the source IP during this top-ten observation period.

If the Most hit IP table has statistics for more than one of the selected content types, the value is the total count for all the content types.

Within 300 The length of the observation period. The auto-learning feature collects hits by source IP data using a 5-minute (300 second) observation period.
Enable Percentage Matching Selected Specifies that the filter matches when the number of hits of the specified content types, expressed as a percentage of the total number of hits, exceeds the value of Percentage of Hits.
Percentage of Hits 81

The number of times clients requested the specified type of file from the source IP, expressed as a percentage of the total number of hits for the source IP in the observation period.

If the Most hit IP table has statistics for more than one of the selected content types, the value is the total of all content types.

Traced By Source IP Most hit IP table data is based on hits by source IP, although you can create an Occurrence filter based on User.

For more information about Advanced Protection custom rules, see Combination access control & rate limiting.

Parameters tab

The Parameters tab provides tabular statistics on the parameters and their values as they appeared in HTTP requests, as well as any parameters that were extracted from the URL by a URL interpreter.

Auto-learning report Parameter tab

This tab appears only for items that are leaf nodes in the navigation tree; that is, they represent a single complete URL as it appeared in a real HTTP request, and therefore could have had those exact associated parameters.

The Name column contains the name of the parameter, exactly as it was observed in the parameter or (for parameters extracted by URL replacers) within the URL.

If the Name column contains part of a URL or the parameter’s value instead of its name, verify the regular expression and back references used in your URL replacer.

Percentages in the Type Match and Required columns indicate how likely the parameter with that name is of that exact data type, and whether or not the web application requires that input for that URL. The Min. Length and Max. Length columns indicate the likely valid range of length for that input’s value. The Avg. Length column indicates the average length for that input’s value. Together, the columns provide information on what is likely the correct configuration of a profile for that URL.

For example, if Max. Length is 255 but Min. Length is 63 and Avg. Length is 64, before generating a protection profile, you may want to investigate to determine whether 255 is indeed an appropriate maximum input length, since it deviates so much from the norm. In this case, the intended minimum and maximum length might really be 63, but a single malicious observed input had a maximum length of 255.

By default, when you generate a protection profile from auto-learning data, FortiWeb will use these statistics to estimate appropriate input rules. However, if auto-learning suggestions are not appropriate, you can manually override these estimates by using the Set icon and Custom check box before generating a protection profile. For details, see To configure a profile using auto-learning data.

Cookies tab

The Cookies tab provides tabular statistics on the name, value, expiry date, and associated URL (path) of each cookie crumb that appeared in HTTP requests.

Cookies that you see in this table can be protected by enabling Cookie Poisoning.

Auto-learning report Cookies tab

This tab appears only for hosts that use cookies, and for items that are leaf nodes in the navigation tree; that is, they represent a single complete URL as it appeared in a real HTTP request, and therefore could have had those exact cookies.

See also

Generating a profile from auto-learning data

When viewing a report generated from auto-learning data, you can generate an inline protection profile or an offline protection profile suitable for the HTTP sessions observed. If some observed sessions are not indicative of typical traffic and you do not want to include elements in the generated profile, or you want to select an action other than the default for a type of observed attack, you can selectively change the action for that type of attack.

In addition to the generated profile itself, the FortiWeb appliance also generates all rules and other auxiliary configurations that the profile requires.

For example, FortiWeb observes HTTP PUT requests that require a password and a user name that is an email address. When it generates a profile, it also uses the data types and maximum lengths of the arguments observed in the HTTP sessions to generate the required parameter validation rules and input rules.

You can edit the generated profiles and auxiliary configurations or use them as the starting point for additional configuration.

To configure a profile using auto-learning data

1.  Go to Auto Learn > Auto Learn Report > Auto Learn Report.

To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Autolearn Configuration category. For details, see Permissions.

2.  Mark the check box in the row that corresponds to the auto-learning profile whose data you want to view.

3.  Click View.

The report appears.

4.  Review the configuration suggestions from auto-learning.

If you want to adjust the behavior of the profile and components to generate, in the left-hand pane, click the expand icon ( + ) next to items to expand the tree, then click the name of the single URL whose protection you want to manually configure.

Buttons and drop-down lists in the report display pane may vary. For most URLs, they enable you to adjust the profile that FortiWeb generates.

Auto-learning suggests an appropriate configuration based upon the traffic that it observed. If a suggestion is not appropriate, you can manually override it.

Configure these settings:

Setting name Description
Overview tab  
  Edit Protected Servers

Click to open a pop-up dialog. Enable or disable the IP addresses and/or domain names that will be members of the generated protected host names group. For details, see Defining your protected/allowed HTTP “Host:” header names.

This appears only if you have selected the name of the auto-learning profile in the navigation pane.

  Edit URL Page

Click to open a pop-up dialog. Enable or disable whether the currently selected URL will be included in start pages and white/black IP list rules in the generated profile. This appears only if you have selected a URL in the navigation pane.

For more information on those rule types, see Specifying URLs allowed to initiate sessions and Access control.

Attacks Tab  
  Action and Enable

Select from the Enable drop-down list to enable or disable detection of each type of attack, and select from Action which action that the generated profile will take. The availability of these lists varies with the level of the item selected in the navigation pane.

For details, see the actions in Configuring a protection profile for inline topologies or Configuring a protection profile for an out-of-band topology or asynchronous mode of operation.

Visits Tab  
  Edit Allow Method

Click to open a pop-up dialog. Change the Status option to select which HTTP request methods to allow in the generated profile. This appears only if you have selected a profile in the navigation pane.

For details, see Configuring a protection profile for inline topologies and Configuring a protection profile for an out-of-band topology or asynchronous mode of operation.

  Edit URL Access

Click to open a pop-up dialog. This appears only if you have selected a profile in the navigation pane.

For details, see Access control.

  Edit Start Page

Click to open a pop-up dialog. This appears only if you have selected a profile in the navigation pane.

For details, see Specifying allowed HTTP methods.

  Edit Exception Method

Click to open a pop-up dialog. This appears only if you have selected a URL in the navigation pane.

For details, see Configuring allowed method exceptions.

  Most hit IP table: Edit Content Type

Click to edit the values that FortiWeb adds to the Content Type filter in an automatically generated Advanced Protection custom rule. This rule is designed to detect web scraping (content scraping) activity.

Available only if a policy or host is selected in the navigation pane.

For more information, see Most hit IP table and web scraping detection.

  Most hit IP table: row selection button

Selects the data that FortiWeb uses to create an Occurrence filter in an Advanced Protection custom rule in the generated profile. This rule is designed to detect web scraping activity.

Available only if a policy or host is selected in the navigation pane.

For more information, see Most hit IP table and web scraping detection.

Parameters tab  
  Set

Type the data type and maximum length of the parameter, and indicate whether or not the parameter is required input. These settings will appear in the generated parameter validation rule and input rules. For details, see Validating parameters (“input rules”) and Preventing zero-day attacks.

Caution: Before you leave the page, mark the Custom check boxes for rows where you have clicked this icon. Failure to do so will cause FortiWeb appliance to discard your settings when you leave the page.

  Custom Before you click Set or leave the page, enable this option for each row whose manual settings you want to save.

5.  Above the display pane, click Generate Config.

A pop-up dialog appears.

6.  In Profile Name, type a name prefix, such as generated-profile.

The FortiWeb appliance adds a dash ( - ) to the profile name followed by a number indicating the year, month, day, and time on which the profile was generated in order to indicate the data on which the profile was based.

7.  From Profile Type, select which type of web profile you want to generate, either Inline (to generate an inline protection profile) or Offline (to generate an offline protection profile).

8.  Click OK.

The generated profile appears in either:

Adjust configuration items used by the generated profile, such as input rules, when necessary. Generated configuration items are based on auto-learning data current at the time that the profile is generated. Data may have changed while you were reviewing the auto-learning report, and/or after you have generated the profiles.

If you do not configure any settings, by default, the FortiWeb appliance generates a profile that allows the HTTP GET method and any other methods whose usage exceeded the threshold, and adds the remaining methods to an allowed method exception. It also creates start page rules and trusted IP rules for the most commonly requested URLs, and blacklist IP addresses that commonly requested suspicious URLs. Attack signatures are disabled or exceptions added according to your configurations in Server Protection Threshold and Server Protection Exception Threshold.

9.  Continue with Transitioning out of the auto-learning phase.

Transitioning out of the auto-learning phase

As your web servers change, you may periodically want to run auto-learning for them on a smaller scale.

For example, perhaps you will install or update a web application or web server, resulting in new structures and different vulnerabilities.

However, for most day-today use, auto-learning should be disabled and your protection profiles fully applied.

To transition to day-to-day use

1.  To apply a profile generated by auto-learning, select it in Web Protection Profile in a server policy (see Configuring a server policy).

2.  If, during auto-learning, any Action in the protection profile or its auxiliary components was set to Alert & Deny or Alert & Erase, verify that those same actions are applied in the protection profile that you generated from auto-learning data. (Incomplete session data due to those actions may have caused auto-learning to be unable to detect those attack types.)

3.  If necessary, either:

4.  Modify the policy to select your newly generated profile in Web Protection Profile.

5.  To validate the configuration, test it (see Testing your installation.)

6.  When you are done collecting auto-learning data and generating your configuration, to improve performance, disable auto-learning by deselecting the auto-learning profile in WAF Auto Learn Profile in all server policies.

7.  Disable Monitor Mode.

See also

Removing old auto-learning data

There are many reasons why you may want to delete old auto-learning data.

You can delete old data. Reports and any profiles generated from the auto-learning profile will then include only subsequently gathered data.

To delete auto-learning data
Alternatively, you can remove auto-learning data by, when the auto-learning profile’s report is open, right-clicking the node in the left-hand pane, then selecting Clean Data

1.  Go to Auto Learn > Auto Learn Report > Auto Learn Report.

To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Autolearn Configuration category. For details, see Permissions.

2.  Either:

3.  Click Clean Data.

See also