How to adapt auto-learning to dynamic URLs & unusual parameters

When web applications have dynamic URLs or unusual parameter styles, you must adapt auto-learning to recognize them.

By default, auto-learning assumes that your web applications use the most common URL structure:

• All parameters follow after a question mark ( ? ). They do not follow a hash ( # ) or other separator character.
• If there are multiple name-value pairs, each pair is separated by an ampersand ( & ). They are not separated by a semi-colon ( ; ) or other separator character.

• All paths before the question mark ( ? ) are static — they do not change based upon input, blending the path with parameters (sometimes called a dynamic URL).

  For example, the page at:

  /app/main

  always has that same path. After a person logs in, the page’s URL doesn’t become:

  /app/marco/main

  or

  /app#deepa

  For another example, the URL does not dynamically reflect inventory, such as:

  /app/sprockets/widget1024894

Some web applications, however, embed parameters within the path structure of the URL, or use unusual or non-uniform parameter separator characters. If you do not configure URL replacers for such applications, it can cause your FortiWeb appliance to gather auto-learning data incorrectly. This can cause the following symptoms:

• Auto-learning reports do not contain a correct URL structure.
• URL or parameter learning is endless.
• When you generate a protection profile from auto-learning, it contains many more URLs than actually exist, because auto-learning cannot predict that the URL is actually dynamic.
• Parameter data is not complete, despite the fact that the FortiWeb appliance has seen traffic containing the parameter.

For example, with Microsoft Outlook Web App (OWA), the user’s login name could be embedded within the path structure of the URL, such as:

/owa/tom/index.html

/owa/mary/index.html

instead of suffixed as a parameter, such as:

/owa/index.html?username=tom

/owa/index.html?username=mary

Auto-learning would continue to create new URLs as new users are added to OWA. Auto-learning would also expend extra resources learning about URLs and parameters that are actually the same. Additionally, auto-learning may not be able to fully learn the application structure, as each user may not request the same URLs.

To solve this, you would create a URL replacer that recognizes the user name within the OWA URL as if it were a standard, suffixed parameter value so that auto-learning can function properly.

See also

• Configuring URL interpreters
• Grouping URL interpreters
• Configuring an auto-learning profile
• Regular expression syntax

Configuring URL interpreters

When using auto-learning, you must define how to interpret dynamic URLs and URLs that include parameters in non-standard ways, such as with different parameter separators (; or #, for example) or by embedding the parameter within the URL’s path structure.

In the web UI, these interpreter plug-ins are called “URL replacers.”

URL replacers match the URL as it appears in the HTTP header of the client’s request (using the regular expression in URL Path) and interpret it into this standard URL formulation:

New URL?New Param=Param Change

For example, if the URL is:

/application/value

and the URL replacer settings are:

$0 holds this part of the matched URL:

/application

and $1 holds this part of the matched URL:

value

so then the URL will be understood by auto-learning, and displayed in the report, as:

/application?setting=value

Need a refresher on regular expressions? See Regular expression syntax, What are back-references?, and Cookbook regular expressions. You can also use the examples in this section, such as Example: URL interpreter for WordPress.

To create a URL interpreter

1.  Go to Auto Learn > Application Templates > URL Replacer.

To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Autolearn Configuration category. For details, see Permissions.

2.  Click Create New.

3.  Configure these settings:

4.  If you selected Predefined in Type, also configure this setting:

5.  If you selected Custom-Defined in Type, configure these settings:

6.  Click OK.

7.  Group the URL replacers in an application policy (see Grouping URL interpreters).

8.  Select the application policy in one or more auto-learning profiles (see Configuring an auto-learning profile).

9.  Select the auto-learning profiles in server policies (see Configuring a server policy).

See also

• Regular expression syntax
• Example: URL interpreter for a JSP application
• Example: URL interpreter for Microsoft Outlook Web App 2007
• Example: URL interpreter for WordPress

Example: URL interpreter for a JSP application

The HTTP request URL from a client is:

/app/login.jsp;jsessionid=xxx;p1=111;p2=123?p3=5555&p4=66aaaaa

which uses semi-colons as parameter separators ( ; ) in the URL, a behavior typical to JSP applications. You would create a URL replacer to recognize the JSP application’s parameters: the semi-colons.

Example: URL replacer for JSP applications

The predefined JSP interpreter plug-in will interpret the URL as:

/app/login.jsp?p4=66aaaaa&p1=111&p2=123&p3=5555

See also

• Regular expression syntax
• Example: URL interpreter for Microsoft Outlook Web App 2007
• Example: URL interpreter for WordPress

Example: URL interpreter for Microsoft Outlook Web App 2007

When a client sends requests to Microsoft Outlook Web App (OWA), many of its URLs use structures like this:

/exchange/tom/index.html

/exchange/jane.doe/memo.EML

/exchange/qinlu/2012/1.html

These have user name parameters embedded in the URL. In order for auto-learning to recognize the parameters, you must either:

• Set Type to Predefined and Application Type to OWA. This predefined auto-learning URL interpreter will match and recognize parameters in all default URLs.
• Create your own custom URL interpreters.

A custom URL replacer for those URLs could look like this:

Example: URL replacer for Microsoft Outlook Web App — User name structure #1

Then the URLs would be recognized by auto-learning as if OWA used a more conventional parameter structure like this:

/exchange/index.html?username1=tom

/exchange/memo.EML?username1=jane.doe

/exchange/2012/1.html?username1=qinlu

Notably, OWA can also include other parameters in the URL, such as a mail folder’s name. Also, OWA can include the user name and folder in more than one way. Therefore multiple URL interpreters are required to match all possible URL structures. In addition to the first URL replacer, you would also configure the following URL replacers and group them into a single set (an auto-learning “application policy”) in order to recognize all possible URLs.

Example: URL replacer for Microsoft Outlook Web App — Folder name structure #1

Example: URL replacer for Microsoft Outlook Web App — User name structure #2

Example: URL replacer Microsoft Outlook Web App — Folder name structure #2

See also

• Regular expression syntax
• Example: URL interpreter for a JSP application
• Example: URL interpreter for WordPress

Example: URL interpreter for WordPress

If the HTTP request URL from a client is a slash-delimited chain of multiple parameters, like either of these:

/wordpress/2012/06/05

/index/province/ontario/city/ottawa/street/moodie

then the format is either of these:

/wordpress/value1/value2/value3

/index/param1/value1/param2/value2/param3/value3

In this URL format, there are 3 parameter values (with or without their names) in the URL:

• param1
• param2
• param3

Because each interpreter can only extract a single parameter, you would create 3 URL interpreters, and group them into a set where they are used sequentially — a chain. Each interpreter would use the interpreted output of the previous one as its input, until all parameters had been extracted, at which point the last interpreter would output both the last parameter and the final interpreted URL. FortiWeb would then append parameters back onto the interpreted URL in the standard structure before storing them in the auto-learning data set.

Analysis of a request URL into its interpretation by a chain of URL interpreters

This configuration requires that for every request: the web application includes parameters in the same sequential order, and all parameters are always present If parameter order or existence vary, this URL interpreter will not work. Requests will not match the URL interpreter set if either param2 or param3 come first, or if any of the parameters are missing. On the opposite end of the spectrum, if the URL interpreter used regular expression capture groups such as (.*) to match anything in any order, i.e.: /index/(.*)/(.*)/(.*)/(.*)/(.*)/(.*)/ then the regular expression would be too flexible: auto-learning might mistakenly match and learn some of param3’s possible values for param2, and so on.

Example: URL replacer 1 for slash-separated parameters

Example: URL replacer 2 for slash-separated parameters

Example: URL replacer 3 for slash-separated parameters

Until you add the URL interpreters to a group, FortiWeb doesn’t know the sequential order.

These URL interpreters will not function correctly if they are not used in that order, because each interpreter’s input is the output from the previous one. So you must set the priorities correctly when referencing each of those interpreters in the set of URL interpreters (Grouping URL interpreters).

Example: URL replacer group for slash-separated parameters — entry 1

Example: URL replacer group for slash-separated parameters — entry 2

Example: URL replacer group for slash-separated parameters — entry 3

Then the URL will be interpreted by auto-learning as if the application used a more conventional and easily understood URL/parameter structure:

/index?param1=value1&param2=value2&param3=value3

See also

• Grouping URL interpreters
• Configuring an auto-learning profile
• Regular expression syntax
• Example: URL interpreter for a JSP application
• Example: URL interpreter for Microsoft Outlook Web App 2007

Grouping URL interpreters

In order to use URL interpreters with an auto-learning profile, you must group URL replacers into sets.

Sets can be:

• mutually exclusive, where the set contains expressions for all possible URL structures, but only one of the URL replacers will match a given request’s URL
• sequential, where the set contains expressions to interpret multiple parameters in a single given URL; each interpreter’s URL input is the URL output of the previous interpreter, and they each parse the URL until all parameters have been extracted; sequential order of interpreters is determined by the URL interpreter’s Priority in the set

To create a custom application policy

1.  Before you create an application policy, first create the URL replacers that it will include (see Configuring URL interpreters).

2.  Go to Auto Learn > Application Templates > Application Policy.

To access this part of the web UI, your administrator’s account access profile must have Read and Write permission to items in the Autolearn Configuration category. For details, see Permissions.

3.  Click Create New.

A dialog appears.

4.  In Name, type a name that can be referenced by other parts of the configuration. Do not use spaces or special characters. The maximum length is 35 characters.

5.  Click OK.

6.  Click Create New.

A dialog appears.

7.  From Plugin Name, select an existing URL replacer from the drop-down list.

Rule order affects URL replacer matching and behavior. FortiWeb appliances evaluate URLs for a matching URL replacer starting with the smallest ID number (greatest priority) rule in the list, and continue towards the largest number in the list. If no rule matches, parameters in the URL will not be interpreted. If multiple rules match, the output (New URL) from earlier URL replacers will be used as the input (URL Path) for the next URL replacer, resulting in a chain of multiple interpreted parameters.

8.  Click OK.

9.  Repeat the previous steps for each URL replacer you want added to the policy.

10.  Select the application policy in an auto-learning profile (see Configuring an auto-learning profile).

11.  Select the auto-learning profiles in server policies (see Configuring a server policy).

See also

• Configuring URL interpreters
• Example: URL interpreter for Microsoft Outlook Web App 2007
• Example: URL interpreter for WordPress
• Configuring an auto-learning profile

Open topic with navigation