Configuring Storage

Overview

FortiSIEM provides a wide array of event storage options. Upon arrival in FortiSIEM, events are stored in the Online event database. The user can define retention policies for this database. When the Online event database becomes full, FortiSIEM will move the events to the Archive Event database. Similarly, the user can define retention policies for the Archive Event database. When the Archive becomes full, events are discarded.

The Online event database can be one of the following:

  • FortiSIEM EventDB
    • On local disk for All-in-one installation
    • On NFS for cluster installation
  • Elasticsearch
    • Native installation
    • AWS OpenSearch (Previously known as AWS Elasticsearch)

The Archive event database can be one of the following:

  • FortiSIEM EventDB on NFS
  • HDFS

Note the various installation documents for 3rd party databases, for example.

In this release, the following combinations are supported:

Event DB    Retention
Online Archive Online Archive
FortiSIEM EventDB
(local or NFS)		 FortiSIEM EventDB
(NFS)		 Policy-based and Space-based Policy-based and Space-based
Elasticsearch  FortiSIEM EventDB (NFS) Space-based Policy-based and Space-based
Elasticsearch  HDFS Space-based Space-based

Configuring Online Event Database on Local Disk

This section describes how to configure the Online Event database on local disk. Use this option when you have an all-in-one system, with only the Supervisor and no Worker nodes deployed.

Setting Up the Database

  1. Go to ADMIN > Setup > Storage.
  2. Click Online > Local Disk.
  3. Enter the following parameters :

    SettingsGuidelines
    Disk Name[Required] Local disk name.
    During FortiSIEM installation, you can add a 'Local' data disk of appropriate size as the 4th or 5th disk (Refer to your specific Installation Guide in the FortiSIEM Document Library). Use the command fdisk -l to find the disk name.

    If you want to configure Local Disk for the physical 2000F or 3500F appliances, enter "hardware" in this field. This prompts a script to run that will configure local storage.

  4. Click Test.
  5. If the test succeeds, click Save.

Setting Up Retention

When Online database becomes full, then events have to be deleted to make room for new events. This can be Space-based or Policy-based.

Setting Up Space-Based Retention

Space-based retention is based on two thresholds defined in the phoenix_config.txt file on the Supervisor node.

[BEGIN phDataPurger]

online_low_space_action_threshold_GB=10

online_low_space_warning_threshold_GB=20

[END]

When the Online Event database size in GB falls below the value of online_low_space_action_threshold_GB, events are deleted until the available size in GB goes slightly above the online_low_space_action_threshold_GB value. If Archive is defined, then the events are archived. Otherwise, they are purged.

If you want to change these values, then change them on the Supervisor and restart phDataManager and phDataPurger modules.

Setting Up Policy-Based Retention

Policies can be used to enforce which types of event data remains in the Online event database.

For information on how to create policies, see Creating Online Event Retention Policy. Note: This is a CPU, I/O, and memory-intensive operation. For best performance, try to write as few retention policies as possible.

How Space-Based and Policy-Based Retention Work Together

  1. First, Policy-based retention policies are applied.
  2. If the available space is still below the value of online_low_space_action_threshold_GB, then Space-based policies are enforced.

Viewing Online Data

For more information, see Viewing Online Event Data Usage.

Configuring Online Event Database on NFS

The following sections describe how to configure the Online database on NFS.

Setting Up the Database

You must choose this option when you have multiple Workers deployed and you plan to use FortiSIEM EventDB.

The NFS Storage should be configured as NFS version 3 with these options: “rw,sync,no_root_squash”.

  1. Go to ADMIN > Setup > Storage.
  2. Click Online > NFS
  3. Enter the following parameters :

    SettingsGuidelines
    Server IP/Host[Required] the IP address/Host name of the NFS server
    Exported Directory[Required] the file path on the NFS Server which will be mounted

  4. Click Test.
  5. If the test succeeds, click Save.

Setting Up Retention

When the Online database becomes full, then events must be deleted to make room for new events. This can be Space-based or Policy-based.

Setting Up Space-Based Retention

Space-based retention is based on two thresholds defined in the phoenix_config.txt file on the Supervisor node.

[BEGIN phDataPurger]

online_low_space_action_threshold_GB=10

online_low_space_warning_threshold_GB=20

[END]

When the Online Event database size in GB falls below the value of online_low_space_action_threshold_GB, events are deleted until the available size in GB goes slightly above the online_low_space_action_threshold_GB value. If Archive is defined, then the events are archived. Otherwise, they are purged.

If you want to change these values, then change them on the Supervisor and restart the phDataManager and phDataPurger modules.

Setting Up Policy-Based Retention

Policies can be used to enforce which types of event data stays in the Online event database.

For information on how to create policies, see Creating Online Event Retention Policy. Note: This is a CPU, I/O, and memory-intensive operation. For best performance, try to write as few retention policies as possible.

How Space-Based and Policy-Based Retention Work Together

  1. First, Policy-based retention policies are applied.
  2. If the available space is still below the online_low_space_action_threshold_GB, then Space-based policies are enforced.

Viewing Online Data

For more information, see Viewing Online Event Data Usage.

Configuring Online Event Database on Elasticsearch

The following sections describe how to set up the Online database on Elasticsearch:

Setting Up the Database

There are three options for setting up the database:

Native Elasticsearch Using REST API

Use this option when you want FortiSIEM to use the REST API Client to communicate with Elasticsearch.

  1. Go to ADMIN > Setup > Storage.
  2. Click Online > Elasticsearch and for ES Service Type, select Native.
  3. Enter the following parameters:

    SettingsGuidelines
    Endpoint

    [Required]

    • URL:  IP address or DNS name of the Elasticsearch cluster Coordinating node. The IP/Host must contain http or https.

    • Ingest: Select if the URL endpoint will be used to handle pipeline processing.

    • Query: Select if the URL endpoint will be used to query Elasticsearch.

      Note: Ingest and Query can both be selected for an endpoint URL.

    Click + to add more URL fields to configure any additional Elasticsearch cluster Coordinating nodes.

    Click - to remove any existing URL fields.

    Port[Required] The port number
    User Name [Optional] User name
    Password[Optional] Password associated with the user
    Shard Allocation
    • Fixed -Enter the number of Shards and Replicas.
    • Dynamic-Dynamically shards data using the Elasticsearch rollover API. Enter the number of Starting Shards and Replicas.
    Per Org IndexSelect to create an index for each organization.
    Event Attribute Template
    • Default-Select if you wish to cover FortiSIEM Event attributes covering all event attribute types.
    • Custom-Select if you wish to cover specific FortiSIEM Event attributes. A reduced list of Event Attributes can improve Elasticsearch performance. After selecting Custom, click Select, and select your CSV file with your FortiSIEM Event attributes.

      For information on the listElasticEventAttributes.sh tool that gathers Event Attributes so you can build a custom event attribute template, see Administrator Tools.

     

    		 

  4. Click Test.
  5. If the test succeeds, click Save.

AWS OpenSearch (Previously known as AWS Elasticsearch) Using REST API

Use this option when you have FortiSIEM deployed in AWS Cloud and you want to use AWS OpenSearch (Previously known as AWS Elasticsearch).

  1. Go to ADMIN > Setup > Storage.
  2. Click Online > Elasticsearch and for ES Service Type, select Amazon.
  3. Enter the following parameters:

    SettingsGuidelines
    Endpoint

    [Required]

    • URL:  IP address or DNS name of the Elasticsearch cluster Coordinating node. The IP/Host must contain http or https.

    • Ingest: Select if the URL endpoint will be used to handle pipeline processing.

    • Query: Select if the URL endpoint will be used to query Elasticsearch.

      Note: Ingest and Query can both be selected for an endpoint URL.

    Click + to add more URL fields.

    Click - to remove any existing URL fields.

    Port[Required] The port number
    Access Key ID[Required] Provide your AWS access key id.
    Secret Key[Required] Provide your AWS secret key.
    Shard Allocation
    • Fixed -Enter the number of Shards and Replicas.
    • Dynamic-Dynamically shards data using the Elasticsearch rollover API. Enter the number of Starting Shards and Replicas.
    Per Org IndexSelect to create an index for each organization.
    Event Attribute Template
    • Default-Select if you wish to cover FortiSIEM Event attributes covering all event attribute types.
    • Custom-Select if you wish to cover specific FortiSIEM Event attributes. A reduced list of Event Attributes can improve Elasticsearch performance. After selecting Custom, click Select, and select your CSV file with your FortiSIEM Event attributes.

      For information on the listElasticEventAttributes.sh tool that gathers Event Attributes so you can build a custom event attribute template, see Administrator Tools.

     

    		 

  4. Click Test.
  5. If the test succeeds, click Save.

Elastic Cloud Using REST API

  1. Go to ADMIN > Setup > Storage.
  2. Click Online > Elasticsearch and and for ES Service Type, select Elastic Cloud.
  3. Enter the following parameters:
    SettingsGuidelines
    Endpoint

    [Required]

    • URL:  IP address or DNS name of the Elasticsearch cluster Coordinating node. The IP/Host must contain http or https.

    • Ingest: Select if the URL endpoint will be used to handle pipeline processing.

    • Query: Select if the URL endpoint will be used to query Elasticsearch.

      Note: Ingest and Query can both be selected for an endpoint URL.

    Click + to add more URL fields to configure any additional Elasticsearch cluster Coordinating nodes.

    Click - to remove any existing URL fields.

    Port[Required] The port number
    User Name [Optional] User name
    Password[Optional] Password associated with the user
    Shard Allocation
    • Fixed -Enter the number of Shards and Replicas.
    • Dynamic-Dynamically shards data using the Elasticsearch rollover API. Enter the number of Starting Shards and Replicas.
    Per Org IndexSelect to create an index for each organization.
    Event Attribute Template
    • Default-Select if you wish to cover FortiSIEM Event attributes covering all event attribute types.
    • Custom-Select if you wish to cover specific FortiSIEM Event attributes. A reduced list of Event Attributes can improve Elasticsearch performance. After selecting Custom, click Select, and select your CSV file with your FortiSIEM Event attributes.

      For information on the listElasticEventAttributes.sh tool that gathers Event Attributes so you can build a custom event attribute template, see Administrator Tools.

     

  4. Click Test.
  5. If the test succeeds, click Save.

Setting Up Space-Based / Age-Based Retention

Depending on whether you use Native Elasticsearch, AWS OpenSearch (Previously known as AWS Elasticsearch), or ElasticCloud, Elasticsearch is installed using Hot (required), Warm (optional), and Cold (optional, availability depends on Elasticsearch type) nodes and Index Lifecycle Management (ILM) (availability depends on Elasticsearch type). Similarly, the space is managed by Hot, Warm, Cold node thresholds and time age duration, whichever occurs first, if ILM is available. See What's New for the latest information on elasticsearch retention threshold compatibility. For steps, see here.

  • When the Hot node cluster storage capacity falls below the lower threshold or meets the time age duration, then:
    • if Warm nodes are defined, the events are moved to Warm nodes,
    • else if Warm nodes are not defined, but Cold nodes are defined, the events are moved to Cold nodes,
    • else, if Archive is defined then they are archived,
    • otherwise, events are purged

    This is done until storage capacity exceeds the upper threshold.

  • If Warm nodes are defined and the Warm node cluster storage capacity falls below lower threshold or meets the time age duration, then:
    • if Cold nodes are defined, the events are moved to Cold nodes,
    • else if Cold nodes are not defined, and Archive is defined, then they are archived,
    • otherwise, events are purged

    This is done until storage capacity exceeds the upper threshold.

  • If Cold nodes are defined and the Cold node cluster storage capacity falls below lower threshold, then:
    • if Archive is defined, then they are archived,
    • otherwise, events are purged

    This is done until storage capacity exceeds the upper threshold

Viewing Online Data

For more information, see Viewing Online Event Data Usage.

Configuring Archive Event Database on NFS

The following sections describe how to set up the Archive database on NFS:

Setting Up the Database

You must choose this option when you have multiple Workers deployed and you plan to use FortiSIEM EventDB.

The NFS Storage should be configured as NFS version 3 with these options: “rw,sync,no_root_squash”.

  1. Go to ADMIN > Setup > Storage.
  2. Click Archive > NFS,
  3. Enter the following parameters:

    SettingsGuidelines
    Server IP/Host[Required] the IP address/Host name of the NFS server
    Exported Directory[Required] the file path on the NFS Server which will be mounted
    Real Time Archive(Optional) event data is written to NFS archive at the same time it is written to online storage, when enabled. Click the checkbox to enable/disable.
    Note: You must click Save in step 5 in order for the Real Time Archive setting to take effect. It is strongly recommended you confirm that the test works, in step 4 before saving.    		 

  4. Click Test.
  5. If the test succeeds, click Save.

Setting Up Retention

When the Archive database becomes full, then events must be deleted to make room for new events. This can be Space-based or Policy-based.

Space-Based Retention

Space-based retention is based on two thresholds defined in phoenix_config.txt file on the Supervisor node.

[BEGIN phDataPurger]

archive_low_space_action_threshold_GB=10

archive_low_space_warning_threshold_GB=20

[END]

When the Archive Event database size in GB falls below the value of archive_low_space_action_threshold_GB, events are purged until the available size in GB goes slightly above the value set for archive_low_space_action_threshold_GB.

If you want to change these values, then change them on the Supervisor and restart the phDataManager and phDataPurger modules.

Policy-Based Retention

Policies can be used to enforce which types of event data remain in the Archive event database.

For information on how to create policies, see Creating Offline (Archive) Retention Policy. Note - This is a CPU, I/O, and memory-intensive operation. For best performance, try to write as few retention policies as possible.

How Space-Based and Policy-Based Retention Work Together

  1. First, Policy-based retention policies are applied.
  2. If the available space is still below archive_low_space_action_threshold_GB, then Space-based policies are enforced.

Viewing Archive Data

For more information, see Viewing Archive Data.

Configuring Archive Event Database on HDFS

The following sections describe how to set up the Archive database on HDFS:

Setting Up the Database

HDFS provides a more scalable event archive option - both in terms of performance and storage.

  1. Go to ADMIN > Setup > Storage.
  2. Click Archive > HDFS.
  3. Enter the following parameters:

    SettingsGuidelines
    Real Time Archive

    (Optional) event data is written to HDFS archive at the same time it is written to online storage, when enabled. Click the checkbox to enable/disable.

    Note: You must click Save in step 5 in order for the Real Time Archive setting to take effect. It is strongly recommended you confirm that the test works, in step 4 before saving.

    Spark Master Node
    IP/HostIP or Host name of the Spark cluster Master node.
    PortTCP port number for FortiSIEM to communicate to Spark Master node.
    HDFS Name Node
    IP/Host IP or Host name of HDFS Name node. This is the machine which stores the HDFS metadata: the directory tree of all files in the file system, and tracks the files across the cluster.
    PortTCP port number for FortiSIEM to communicate to HDFS Name node.

  4. Click Test.
  5. If the test succeeds, click Save.

Setting Up Space-Based Retention

When the HDFS database becomes full, events have to be deleted to make room for new events.

This is set by Archive Thresholds defined in the GUI. Go to ADMIN > Settings > Database > Online Settings. Change the Low and High settings, as needed.

When the HDFS database size in GB rises above the value of archive_low_space_action_threshold_GB, events are purged until the available size in GB goes slightly above the value set for archive_low_space_action_threshold_GB.

Viewing Archive Data

For more information, see Viewing Archive Data.

Changing Event Storage Options

It is highly recommended to chose a specific event storage option and retain it. However, it is possible to switch to a different storage type.

Note: In all cases of changing storage type, the old event data is not migrated to the new storage. Contact FortiSIEM Support if this is needed - some special cases may be supported.

For the following three cases, simply choose the new storage type from ADMIN > Setup > Storage.

  • Local to Elasticsearch
  • NFS to Elasticsearch
  • Elasticsearch to Local

The following four storage change cases need special considerations:

Elasticsearch to NFS

  1. Log in to FortiSIEM GUI.
  2. Select and delete the existing Workers from ADMIN > License > Nodes > Delete.
  3. Go to ADMIN > Setup > Storage and update the Storage type as NFS server
  4. Go to ADMIN > License > Nodes and Add the recently deleted Workers in step #2.

Local to NFS

  1. SSH to the Supervisor and stop FortiSIEM processes by running:
    phtools --stop all
  2. Unmount /data by running:
    umount /data
  3. Validate that /data is unmounted by running:
    df –h
  4. Edit /etc/fstab and remove /data mount location.
  5. Log in to FortiSIEM GUI, go to ADMIN > Setup > Storage and update the Storage type as NFS server.

NFS to Local

  1. SSH to the Supervisor and stop FortiSIEM processes by running:
    phtools --stop all
  2. Unmount /data by running:
    umount /data
  3. Validate that /data is unmounted by running:
    df –h
  4. Edit /etc/fstab and remove /data mount location.
  5. Connect the new disk to Supervisor VM.
  6. Log in to FortiSIEM GUI, go to ADMIN > Setup > Storage and update the Storage type as Local Disk.

Changing NFS Server IP

If you are running a FortiSIEM Cluster using NFS and want to change the IP address of the NFS Server, then take the following steps.

Step 1: Temporarily Change the Event Storage Type from EventDB on NFS to EventDB on Local

  1. Go to ADMIN > License > Nodes and remove all the Worker nodes.
  2. SSH to the Supervisor and stop FortiSIEM processes by running:
    phtools --stop all
  3. Unmount /data by running:
    umount /data
  4. Validate that /data is unmounted by running:
    df –h
  5. Edit /etc/fstab and remove /data mount location.
  6. Attach new local disk to the Supervisor. It is recommended that it is 50~80GB.
  7. Go to ADMIN > Setup > Storage > Online.
  8. Change the storage type to Local Disk and add the local disk's partition to the Disk Name field. (e.g. /dev/sde).
  9. Click Test to confirm.
  10. Click Save.

Step 2: Change the NFS Server IP Address

This is a standard system administrator operation. Change the NFS Server IP address.

Step 3: Change the Event Storage Type Back to EventDB on NFS

  1. SSH to the Supervisor and stop FortiSIEM processes by running:
    phtools --stop all
  2. Unmount /data by running:
    umount /data
  3. Validate that /data is unmounted by running:
    df –h
  4. Edit /etc/fstab and remove /data mount location.
  5. Go to ADMIN > Setup > Storage > Online.
  6. Change the storage type to NFS.
  7. In the Server field, with IP selected, enter the new IP address of the NFS server.
  8. In the Exported Directory field, enter the correct NFS folder's path.
  9. Click Test to confirm.
  10. Click Save.
  11. Go to ADMIN > License > Nodes and add back all the Worker nodes.

Disk Space Management

When the Online storage is nearly full, events must either be archived or purged to make room for new events. Similarly, when the Archive storage is nearly full, events are purged to make room for new events from Online storage. This strategy keeps FortiSIEM running continuously.

This section provides details for the various storage options.

Online Event Database on Local Disk or on NFS

There are two parameters in the phoenix_config.txt file on the Supervisor node that determine the operations. They appear under the phDataPurger section.

[BEGIN phDataPurger]

- online_low_space_action_threshold_GB (default 10GB)

- online_low_space_warning_threshold_GB (default 20GB)

[END]

When Online disk space reaches the low threshold (online_low_space_action_threshold_GB) value, then events are archived (if archive directory is set) or purged. This operation continues until the Online disk space reaches the online_low_space_warning_threshold_GB value. This check is done hourly.

You can change these parameters to suit your environment and they will be preserved after upgrade. You must restart phDataPurger module to pick up your changes.

Online Event Database on Elasticsearch

Log in to the FortiSIEM GUI and go to ADMIN > Settings > Online Settings. If Elasticsearch is chosen as Online storage, depending on your elasticsearch type, and whether you have archive configured, the following choices will be available in the GUI.

  • Hot Node - Low Threshold (default 25%), High Threshold (35%), Age 90 days
  • Warm Node - Low Threshold (default 20%), High Threshold (30%), Age 90 days
  • Cold Node - Low Threshold (default 20%), High Threshold (30%)
  • Archive - Low Threshold (default 10%), High Threshold (20%)

When Hot Node disk free space reaches the Low Threshold value, events are moved until the Hot Node disk free space reaches the High Threshold value. Event destination can be one of the following:

  • Warm Node
  • Cold Node - if Warm Nodes are not defined
  • Archive - if prior nodes (Warm, Cold) are not defined
  • Purged - if neither Warm Node, Cold Node, nor Archive is defined

When Warm Node disk free space reaches the Low Threshold value, events are moved to Cold node. If Cold node is not defined, events are moved to Archive or purged (if Archive is not defined) until Warm disk free space reaches High Threshold.

When Cold Node disk free space reaches the Low Threshold value, events are moved to Archive or purged (if Archive is not defined), until Cold disk free space reaches High Threshold.

Archive Event Database on NFS

There are two parameters in the phoenix_config.txt file on the Supervisor node that determine when events are deleted. They appear under the phDataPurger section:

[BEGIN phDataPurger]

- archive_low_space_action_threshold_GB (default 10GB)

- archive_low_space_warning_threshold_GB (default 20GB)

[END]

When the Archive disk space reaches the low threshold (archive_low_space_action_threshold_GB) value, events are purged until the Archive disk space reaches the high threshold (online_low_space_warning_threshold_GB) value. This check is done hourly.

You can change these parameters to suit your environment and they will be preserved after upgrade. You must restart phDataPurger module to pick up your changes.

Archive Event Database on HDFS

This is set by configuring the Archive Threshold fields in the GUI at ADMIN > Settings > Database > Online Settings. Elasticsearch must be configured as online storage, and HDFS as offline storage in order for the Archive Threshold option/field to appear in the configuration. This is the only way to purge data from HDFS. For more information on configuring thresholds, see Setting Elasticsearch Retention Threshold.