Configuring Storage

• Overview
• Configuring Online Event Database on Local Disk
• Configuring Online Event Database on NFS
• Configuring Online Event Database on Elasticsearch
• Configuring Archive Event Database on NFS
• Configuring Archive Event Database on HDFS
• Changing Event Storage Options

Overview

FortiSIEM provides a wide array of event storage options. Upon arrival in FortiSIEM, events are stored in the Online event database. The user can define retention policies for this database. When the Online event database becomes full, FortiSIEM will move the events to the Archive Event database. Similarly, the user can define retention policies for the Archive Event database. When the Archive becomes full, events are discarded.

The Online event database can be one of the following:

• FortiSIEM EventDB
  • On local disk for All-in-one installation
  • On NFS for cluster installation
• Elasticsearch
  • Native installation
  • AWS Elasticsearch

The Archive event database can be one of the following:

• FortiSIEM EventDB on NFS
• HDFS

Note the various installation documents for 3rd party databases, for example.

• Elasticsearch Storage Guide
• NFS Storage Guide

In this release, the following combinations are supported:

Event DB		Retention
Online	Archive	Online	Archive
FortiSIEM EventDB (local or NFS)	FortiSIEM EventDB (NFS)	Policy-based and Space-based	Policy-based and Space-based
Elasticsearch	FortiSIEM EventDB (NFS)	Space-based	Policy-based and Space-based
Elasticsearch	HDFS	Space-based	Space-based

Configuring Online Event Database on Local Disk

• Setting Up the Database
• Setting Up Retention
• Viewing Online Data

This section describes how to configure the Online Event database on local disk. Use this option when you have an all-in-one system, with only the Supervisor and no Worker nodes deployed.

Setting Up the Database

1. Go to ADMIN > Setup > Storage.
2. Click Online > Local Disk.
3. Enter the following parameters :

   Settings	Guidelines
   Disk Name	[Required] Local disk name. During FortiSIEM installation, you can add a 'Local' data disk of appropriate size as the 4th disk. Use the command fdisk -l to find the disk name. If you want to configure Local Disk for the physical 2000F or 3500F appliances, enter "hardware" in this field. This prompts a script to run that will configure local storage.

4. Click Test.
5. If the test succeeds, click Save.

Setting Up Retention

When Online database becomes full, then events have to be deleted to make room for new events. This can be Space-based or Policy-based.

• Setting Up Space-Based Retention
• Setting Up Policy-Based Retention
• How Space- and Policy-Based Retention Work Together

Setting Up Space-Based Retention

Space-based retention is based on two thresholds defined in the phoenix_config.txt file on the Supervisor node.

[BEGIN phDataPurger]

online_low_space_action_threshold_GB=10

online_low_space_warning_threshold_GB=20

[END]

When the Online Event database size in GB falls below the value of online_low_space_action_threshold_GB, events are deleted until the available size in GB goes slightly above the online_low_space_action_threshold_GB value. If Archive is defined, then the events are archived. Otherwise, they are purged.

If you want to change these values, then change them on the Supervisor and restart phDataManager and phDataPurger modules.

Setting Up Policy-Based Retention

Policies can be used to enforce which types of event data remains in the Online event database.

For information on how to create policies, see Creating Online Event Retention Policy. Note: This is a CPU, I/O, and memory-intensive operation. For best performance, try to write as few retention policies as possible.

How Space- and Policy-Based Retention Work Together

1. First, Policy-based retention policies are applied.
2. If the available space is still below the value of online_low_space_action_threshold_GB, then Space-based policies are enforced.

Viewing Online Data

For more information, see Viewing Online Event Data Usage.

Configuring Online Event Database on NFS

The following sections describe how to configure the Online database on NFS.

• Setting Up the Database
• Setting Up Retention
• Viewing Online Data

Setting Up the Database

You must choose this option when you have multiple Workers deployed and you plan to use FortiSIEM EventDB.

1. Go to ADMIN > Setup > Storage.
2. Click Online > NFS
3. Enter the following parameters :

   Settings	Guidelines
   Server IP/Host	[Required] the IP address/Host name of the NFS server
   Exported Directory	[Required] the file path on the NFS Server which will be mounted

4. Click Test.
5. If the test succeeds, click Save.

Setting Up Retention

When the Online database becomes full, then events must be deleted to make room for new events. This can be Space-based or Policy-based.

• Setting Up Space-Based Retention
• Setting Up Policy-Based Retention
• How Space- and Policy-Based Retention Work Together

Setting Up Space-Based Retention

Space-based retention is based on two thresholds defined in the phoenix_config.txt file on the Supervisor node.

[BEGIN phDataPurger]

online_low_space_action_threshold_GB=10

online_low_space_warning_threshold_GB=20

[END]

When the Online Event database size in GB falls below the value of online_low_space_action_threshold_GB, events are deleted until the available size in GB goes slightly above the online_low_space_action_threshold_GB value. If Archive is defined, then the events are archived. Otherwise, they are purged.

If you want to change these values, then change them on the Supervisor and restart the phDataManager and phDataPurger modules.

Setting Up Policy-Based Retention

Policies can be used to enforce which types of event data stays in the Online event database.

For information on how to create policies, see Creating Online Event Retention Policy. Note: This is a CPU, I/O, and memory-intensive operation. For best performance, try to write as few retention policies as possible.

How Space- and Policy-Based Retention Work Together

1. First, Policy-based retention policies are applied.
2. If the available space is still below the online_low_space_action_threshold_GB, then Space-based policies are enforced.

Viewing Online Data

For more information, see Viewing Online Event Data Usage.

Configuring Online Event Database on Elasticsearch

The following sections describe how to set up the Online database on Elasticsearch:

• Setting Up the Database
• Setting Up Space-Based Retention
• Viewing Online Data

Setting Up the Database

There are three options for setting up the database:

• Native Elasticsearch Using REST API
• Native Elasticsearch Using Java Transport Client
• AWS Elasticsearch Using REST API

Native Elasticsearch Using REST API

Use this option when you want FortiSIEM to use the REST API Client to communicate with Elasticsearch.

1. Go to ADMIN > Setup > Storage.
2. Click Online > Elasticsearch and choose Client as Rest API and AWS = No.
3. Enter the following parameters:

   Settings	Guidelines
   ES Service Type	Set to Native
   URL	[Required] IP address or DNS name of the Elasticsearch cluster Coordinating node. The IP/Host must contain https.
   Port	[Required] The port number
   User Name	[Optional] User name
   Password	[Optional] Password associated with the user
   Shard Allocation	Fixed -Enter the number of Shards and Replicas Dynamic-Dynamically shards data using the Elasticsearch rollover API
   Per Organization Index	Select to create an index for each organization

4. Click Test
5. If the test succeeds, click Save

Native Elasticsearch Using Java Transport Client

Use this option when you want FortiSIEM to use Java Transport Client to communicate with Elasticsearch. This is an outdated option.

1. Go to ADMIN > Setup > Storage.
2. Click Online > Elasticsearch and choose Client as Java Transport.
3. Enter the following parameters:

   Settings	Guidelines
   ES Service Type	Set to Native
   Cluster Name	[Required] Name of the Elasticsearch Cluster
   Cluster IP/Host	[Required] IP address or DNS name of the Elasticsearch cluster Coordinating node. The IP/Host must contain http.
   HTTP Port	[Required] HTTP port number
   Java Port	[Required] Java port number
   User Name	[Optional] User name
   Password	[Optional] Password associated with the user
   Shard Allocation	Fixed - Enter the number of Shards and Replicas Dynamic - Dynamically shards data using the Elasticsearch rollover API
   Per Organization Index	Select to create an index for each organization

4. Click Test.
5. If the test succeeds, click Save.

AWS Elasticsearch Using REST API

Use this option when you have FortiSIEM deployed in AWS Cloud and you want to use AWS Elasticsearch.

1. Go to ADMIN > Setup > Storage
2. Click Online > Elasticsearch and choose Client as REST API and AWS = Yes.
3. Enter the following parameters:

   Settings	Guidelines
   ES Service Type	Set to Amazon
   URL	[Required] IP address or DNS name of the Elasticsearch cluster Coordinating node. The IP/Host must contain https.
   Port	[Required] The port number
   User Name	[Optional] User name
   Password	[Optional] Password associated with the user
   Shard Allocation	Fixed -Enter the number of Shards and Replicas Dynamic-Dynamically shards data using the Elasticsearch rollover API
   Per Organization Index	Select to create an index for each organization

4. Click Test.
5. If the test succeeds, click Save.

Setting Up Space-Based Retention

Elasticsearch is installed using Hot (required) and Warm (optional) nodes. The space is managed by Hot and Warm node thresholds defined in Setting Elasticsearch Retention Threshold.

• When the Hot node cluster storage capacity falls below the lower threshold, then:
  • if Warm nodes are defined, the events are moved to Warm nodes,
  • else, if Archive is defined then they are archived,
  • otherwise, events are purged

  This is done until storage capacity exceeds the upper threshold.

• If Warm nodes are defined and the Warm node cluster storage capacity falls below lower threshold, then:
  • if Archive is defined, then they are archived,
  • otherwise, events are purged

  This is done until storage capacity exceeds the upper threshold

Viewing Online Data

For more information, see Viewing Online Event Data Usage.

Configuring Archive Event Database on NFS

The following sections describe how to set up the Archive database on NFS:

• Setting Up the Database
• Setting Up Retention
• Viewing Archive Data

Setting Up the Database

You must choose this option when you have multiple Workers deployed and you plan to use FortiSIEM EventDB.

1. Go to ADMIN > Setup > Storage,
2. Click Archive > NFS,
3. Enter the following parameters:

   Settings	Guidelines
   Server IP/Host	[Required] the IP address/Host name of the NFS server
   Exported Directory	[Required] the file path on the NFS Server which will be mounted

4. Click Test.
5. If the test succeeds, click Save.

Setting Up Retention

When the Archive database becomes full, then events must be deleted to make room for new events. This can be Space-based or Policy-based.

• Space-Based Retention
• Policy-Based Retention
• How Space- and Policy-Based Retention Work Together

Space-Based Retention

Space-based retention is based on two thresholds defined in phoenix_config.txt file on the Supervisor node.

[BEGIN phDataPurger]

archive_low_space_action_threshold_GB=10

archive_low_space_warning_threshold_GB=20

[END]

When the Archive Event database size in GB falls below the value of archive_low_space_action_threshold_GB, events are purged until the available size in GB goes slightly above the value set for archive_low_space_action_threshold_GB.

If you want to change these values, then change them on the Supervisor and restart the phDataManager and phDataPurger modules.

Policy-Based Retention

Policies can be used to enforce which types of event data remain in the Archive event database.

For information on how to create policies, see Creating Offline (Archive) Retention Policy. Note - This is a CPU, I/O, and memory-intensive operation. For best performance, try to write as few retention policies as possible.

How Space- and Policy-Based Retention Work Together

1. First, Policy-based retention policies are applied.
2. If the available space is still below archive_low_space_action_threshold_GB, then Space-based policies are enforced.

Viewing Archive Data

For more information, see Viewing Archive Data.

Configuring Archive Event Database on HDFS

The following sections describe how to set up the Archive database on HDFS:

• Setting Up the Database
• Setting Up Space-Based Retention
• Viewing Archive Data

Setting Up the Database

HDFS provides a more scalable event archive option - both in terms of performance and storage.

1. Go to ADMIN > Setup > Storage.
2. Click Archive > HDFS.
3. Enter the following parameters:

   Settings	Guidelines
   Spark Master Node
   IP/Host	IP or Host name of the Spark cluster Master node.
   Port	TCP port number for FortiSIEM to communicate to Spark Master node.
   HDFS Name Node
   IP/Host	IP or Host name of HDFS Name node. This is the machine which stores the HDFS metadata: the directory tree of all files in the file system, and tracks the files across the cluster.
   Port	TCP port number for FortiSIEM to communicate to HDFS Name node.

4. Click Test.
5. If the test succeeds, click Save.

Setting Up Space-Based Retention

When the HDFS database becomes full, events have to be deleted to make room for new events.

This is set by Archive Thresholds defined in the GUI. Go to ADMIN > Settings > Database > Archive Data. Change the Low and High settings, as needed.

When the HDFS database size in GB rises above the value of archive_low_space_action_threshold_GB, events are purged until the available size in GB goes slightly above the value set for archive_low_space_action_threshold_GB.

Viewing Archive Data

For more information, see Viewing Archive Data.

Changing Event Storage Options

It is highly recommended to chose a specific event storage option and retain it. However, it is possible to switch to a different storage type.

Note: In all cases of changing storage type, the old event data is not migrated to the new storage. Contact FortiSIEM Support if this is needed - some special cases may be supported.

For the following three cases, simply choose the new storage type from ADMIN > Setup > Storage.

• Local to Elasticsearch
• NFS to Elasticsearch
• Elasticsearch to Local

The following four storage change cases need special considerations:

• Elasticsearch to NFS
• Local to NFS
• NFS to Local
• NFS to Elasticsearch to NFS

Elasticsearch to NFS

1. Log in to FortiSIEM GUI.
2. Select and delete the existing Workers from ADMIN > License > Nodes > Delete.
3. Go to ADMIN > Setup > Storage and update the Storage type as NFS server
4. Go to ADMIN > License > Nodes and Add the recently deleted Workers in step #2.

Local to NFS

1. SSH to the Supervisor and stop FortiSIEM processes by running:
   phtools --stop all
2. Unmount /data by running:
   umount /data
3. Validate that /data is unmounted by running:
   df –h
4. Edit /etc/fstab and remove /data mount location.
5. Log in to FortiSIEM GUI, go to ADMIN > Setup > Storage and update the Storage type as NFS server.

NFS to Local

1. SSH to the Supervisor and stop FortiSIEM processes by running:
   phtools --stop all
2. Unmount /data by running:
   umount /data
3. Validate that /data is unmounted by running:
   df –h
4. Edit /etc/fstab and remove /data mount location.
5. Connect the new disk to Supervisor VM.
6. Log in to FortiSIEM GUI, go to ADMIN > Setup > Storage and update the Storage type as Local Disk.

NFS to Elasticsearch to NFS

1. SSH to the Supervisor and stop FortiSIEM processes by running:
   phtools --stop all
2. Unmount /data by running:
   umount /data
3. Validate that /data is unmounted by running:
   df –h
4. Edit /etc/fstab and remove /data mount location.
5. Repeat steps #1 to #4 on all Workers.
6. Log in to FortiSIEM GUI, select and delete all the existing Workers from ADMIN > License > Nodes > Delete.
7. Go to ADMIN > Setup > Storage and update the Storage type as appropriate.
8. Go to ADMIN > License > Nodes and add all recently deleted Workers in step #6.