Searches Using Pre-computed Results

If you want to run the same search again and again, or you want to run certain pre-defined searches over a large time window, then the search time can be reduced by setting up pre-computation.

note icon

It is important that search filters, group by, and display parameters and display filters do not change. Otherwise, the pre-computation results will be invalid.

To use this feature, you must complete these steps:

  1. Select a Report and turn on pre-computation.
  2. Select the Pre-computed result option when running the search.

The following sections provide more information about the pre-computation feature and how to use it.

Usage Notes

  1. Currently, pre-computation only works with FortiSIEM EventDB. Elasticsearch and HDFS are not supported.
  2. Pre-computation is currently supported for Aggregated queries with COUNT, SUM, AVG, MAX, and MIN operators. Raw event queries and nested searches are not supported.
  3. If you run a query with pre-computed results, but the search interval is wider than the available pre-computed results, then the results are returned for the pre-computed time interval only. Currently, FortiSIEM does not run a separate search for the missing time window and stitch together the two search results.
  4. Pre-computation begins at hourly/daily boundaries. For example, if you set up hourly pre-computation at 2:34 PM, then the first pre-computation will begin slightly after 3:00 PM for the time interval 2:00 PM – 3:00 PM.
  5. FortiSIEM does not semantically compare search filters, group by, and display parameters and display filters for two searches. Thus, pre-computed results cannot be used for a cloned search.
  6. Pre-computation is set up at a report level and not a report bundle level.
  7. For the Service provider case, you must effectively have the same role in all Organizations to be able to use pre-computed results. Examples are
    1. Full Admin for all Organizations.
    2. Help desk user for one Organization and Read only user for another Organization. Note that both of these roles have empty data conditions and hence are effectively the same role from a pre-computation perspective.

Setting Up Pre-computation

Only a Super Global user having the Full Admin role can set up pre-computation. This is because only such a user can see all the roles. A Full Admin user for a specific organization cannot set up pre-computation. Follow these steps to set up pre-computation.

  1. Log in to FortiSIEM as a Super Global Full Admin user.
  2. Go to Admin > Resources > Reports.
  3. Select a Report, Click More and Select Pre-compute.
  4. Enter the pre-compute options:
    1. Select the Enable option to enable pre-computation. If you do not select Enable, then the definition will be there, but pre-computation will stop and all older results will be deleted.
    2. Carefully select the Organization and the Roles for whom queries will be pre-computed. These selections determine when a user query can use pre-computed results. See Impact of Organization and Roles for more detail.
    3. Select Pre-computation frequency. A lower frequency provides more accuracy at the expense of more system load and storage. Choose the highest frequency you can accept.
    4. Select the Age in number of days. Pre-computed results older than this age will be deleted.
    5. Check the Pre-compute history option if you want the system to automatically run and fill up data from earlier time intervals.
  5. Click OK.

The system will begin pre-computation on the hour or day boundary. For example, if you set up hourly pre-computation at 2:34 PM, then the first pre-computation will begin slightly after 3:00 PM for the time interval 2:00 PM – 3:00 PM. As another example, if you set up daily pre-computation at 10:00 PM, then the first pre-computation will begin slightly after 12:00 AM midnight for the previous day.

Impact of Organization and Roles

A query definition does not enforce Organization and Role restrictions. When you run a query, you are forced to choose one or more Organizations. The data conditions for your role definition are automatically applied. For example, if you run a Top Event Type query as a Full Admin user for Org1 and Org2, then you get All Event Types for Org1 and Org2. However if you run as a Network Admin for Org2 then you only get Network Event Types for Org2. Your Organization and Role assignments have an effect on the query results as they change the query filters.

If you set up pre-computation for an Organization and a set of Roles, then only the users belonging to the same Organization and having exactly the same Role can use pre-computed results. The only exception is for a Full Admin user who can use any pre-computed result in a query. The examples in the following table illustrate this point.

Pre-computation
Definition

 

 

Organization

Role

Who can use pre-computed results

Who cannot use pre-computed results

All Orgs Combined

Full Admin

Super-global users that are Full Admin for all Organizations or have roles without data conditions in some organizations.

Other users, for example, Super Global Network Admins for Org1 and Full Admin for Org2

All Orgs Combined

Network Admin

Super-global users that have the Network Admin role in All Organizations.

 

All Orgs Combined

Network Admin, Server Admin

Super-global users that are both Network Admin and Server Admin in All Organizations.

If the user is a Network Admin for Org1 and Server Admin for Org2.

Org1

Full Admin

Full Admin or users with no data constraints belonging to Org1 can use pre-compute results.

Other users, for example, Org1 Network Admins cannot use these pre-computed results.

Org1

Network Admin

Network Admin users belonging to Org1 can use pre-compute results.

 

Org1

Network Admin, Server Admin

Users belonging to BOTH Network Admin and Server Admin and belonging to Org1.

If the user belongs to only one role, for example Network Admin only, then the user cannot use pre-computed results.

Viewing Pre-computed Results

Once pre-computation is defined, FortiSIEM will pre-compute on the hour or day boundary.

To see the time slots of pre-computed results:

  1. Select a Report.
  2. Click Pre-compute > Results.
  3. Click Refresh to get the latest results.
    1. Time Range From and Time Range To represent the Query Time Window.
    2. Organization and Roles relate to the query conditions.
    3. Finish Time specifies when the pre-computed query finished.

To see the content of a pre-computed result:

  1. Select one row and click View Results.
  2. You will be taken to the ANALYTICS tab with the query conditions already provided. You can see the results. The query name will display (Pre-computed) appended to the end of the name.
  3. Note that because you are running a pre-computed query, you are allowed to perform only these two operations:
    1. Change the time range by clicking the Query Filter search bar and selecting a different Time range.
    2. Load another pre-computed (Organization, Role) combination for the same query by going to the Query Filter search bar and selecting a different (Organization, Role) from the Pre-compute Settings menu.

    All of the other possible choices, such as Query Filters, Organization Drip down, and Query Group By and Display Fields, are grayed out and unavailable.

  4. If you want to stay in this page and change other conditions, click Query Filter search bar and deselect Pre-compute Settings.

Running a GUI Search on Pre-computed Results

You can run a search from the GUI on pre-computed results from the ANALYTICS page or the RESOURCE page.

From the ANALYTICS Page
  1. Load a Report and click >.
  2. If the Report has been pre-computed, then the system will ask you to choose whether you want to use pre-computed results.
    1. If you do not want to use pre-computed results, then check Use pre-compute for and click OK. The query will run by searching the database.
    2. If you want to use pre-computed results, then check Use pre-compute for and select the Organization/Role combination from the drop-down list and click OK. The query will run based on pre-computed results.
  3. Note that because you are running a pre-computed query, you are allowed to perform only these two operations:.
    1. Change the time range by clicking the Query Filter search bar and selecting a different Time range.
    2. Load another pre-computed (Organization, Role) combination for the same query by going to the Query Filter search bar and selecting a different (Organization, Role) from the Pre-compute Settings menu.

    All of the other possible choices, such as Query Filters, Organization Drip down, and Query Group By and Display Fields, are grayed out and unavailable.

  4. If you want to stay in this page and change other conditions, click the Query Filter search bar and deselect Pre-compute Settings.
From the RESOURCE Page
  1. Select a Report and click Run. A dialog box will open.
  2. Select the Organization for which you want to run the report. The query result will contain the selected organizations. Note that based on the selected organizations, the pre-compute options below will change.
  3. Select Report Time Range.
  4. Select the pre-compute option if available from the menu.
  5. Click Run.
  6. You will be taken to the ANALYTICS tab with the query conditions already provided. You can see the results. The Query name will display (Pre-computed) appended at the end of the name.
  7. Note that because you are running a pre-computed query, you are allowed to perform only these two operations:.
    1. Change the time range by clicking the Query Filter search bar and selecting a different Time range.
    2. Load another pre-computed (Organization, Role) combination for the same query by going to the Query Filter search bar and selecting a different (Organization, Role) from the Pre-compute Settings menu.

    All of the other possible choices, such as Query Filters, Organization Drip down, and Query Group By and Display Fields, are grayed out and unavailable.

  8. If you want to stay in this page and change other conditions, click the Query Filter search bar and deselect Pre-compute Settings.

Scheduling a Report Based on Pre-computed Results

  1. Go to the RESOURCE page.
  2. Select a Report and click More > Schedule. A dialog box will open.
  3. Select the Organization for which you want to run the report. Note that based on the selected organizations, the pre-compute options below will change.
    1. If you select Combine all selected Organizations into one Report, then the report will contain data from all organizations. You also have the option to select the organizations that you want to include. For pre-compute to work, you must select All Organizations.
    2. If you select Generate separate Report for each selected Organization, then a separate report will be sent out for each selected organization, Data between organizations will not be mixed in the same report. For pre-compute to work, you must select these Organizations to be pre-computed.
  4. Select Report Time Range.
  5. If you want data to be pre-computed, then select Pre-compute settings from the menu. You can select multiple entries for step 3b above.
  6. Click Next and enter values for the rest of the options in the dialog box.
  7. Click OK.

The system will run the report based on a schedule. If pre-compute settings are specified then the report results will be based on pre-computed data.

Running a Report Bundle Based on Pre-computed Results

A Report Bundle consists of one or more reports. One or more reports may be set to be pre-computed. If you run the Report Bundle, then the reports set up for pre-computation will have pre-computed results, while other reports will run normally (without pre-computation).

  1. Go to Resource > Reports > Report Bundle.
  2. Select a Report Bundle.
  3. On top left, select Export Report Bundle.
  4. In Pre-compute Settings, select the Organization and Role combination. Each Report can be pre-computed for multiple Organization and Role combinations. When scheduling a report bundle, a common Organization and Role combination must be chosen that is applicable for ALL pre-computed reports. When Pre-Computation is defined, Filters cannot be selected.
  5. Select other setting as usual.
  6. Click OK to run the Report Bundle.

Scheduling a Report Bundle Based on Pre-computed Results

A Report Bundle consists of one or more reports. One or more reports may be set to be pre-computed. If you schedule the Report Bundle, then the reports set up for pre-computation will have pre-computed results, while other reports will run normally (without pre-computation).

  1. Go to Resource > Reports > Report Bundle.
  2. Select a Report Bundle.
  3. On top left, select Schedule Report Bundle.
  4. Click + to create a schedule.
  5. In Pre-compute Settings, select the Organization and Role combination. Each Report can be pre-computed for multiple Organization and Role combinations. When scheduling a report bundle, a common Organization and Role combination must be chosen that is applicable for ALL pre-computed reports. When Pre-Computation is defined, Filters cannot be selected.
  6. Select other setting as usual.
  7. Click OK to schedule the Report Bundle.