Configuring Elasticsearch Based Deployment

This section covers the following topics:

Elasticsearch Configuration Overview

 

FortiSIEM supports 3 Elasticsearch deployments

  • Native Elasticsearch – You deploy your own Elasticsearch (Case 1)

  • AWS Opensearch (previously known as AWS Elasticsearch) (Case 2)

  • Elastic Cloud (Case 1)

Creating Elasticsearch Online Storage

 

This assumes that you have already deployed Elasticsearch or have an AWS Opensearch or Elastic Cloud account.

Case 1: To configure Native Elasticsearch or Elastic Cloud, follow these steps:

  1. Go to ADMIN > Setup > Storage.

  2. Click Online, and from the Event Database drop-down list, select Elasticsearch, and from the ES Service Type drop-down list, select Native or Elastic Cloud depending on your Elasticsearch set up.

  3. Enter the following parameters.

    1. Org Storage: This is relevant for FortiSIEM Multi-tenant deployments. Select one of the following from the drop-down list.

      1. All Orgs in One Index – In this option, events from all Organizations are mixed in every Elasticsearch index. This is the most cost-effective option as Elasticsearch does not scale when there are many Organizations with high events per sec, and each Organization being in a separate index may lead to an excessive number of indices (note that Elasticsearch has been observed to have approximately 15K index limit per cluster).

      2. Each Org in its own Index – In this option, events from each Organization is in its own Elasticsearch index. This is a flexible option that provides event isolation among Organizations, but Elasticsearch does not scale when there are many Organizations with high events per sec and each Organization being in a separate index may lead to an excessive number of indices.

      3. Custom Org Assignment – In this option, Organizations can be grouped into Groups (maximum 15 allowed). Organizations belonging to the same group have their events in the same index. This is a balanced approach that provides some amount of event isolation, but does not let the number of indices grow excessively. To create and deploy a custom Organization to Group Mapping, follow these steps:

        1. Click Edit.

        2. In the follow up dialog, click Add.

        3. In the Mapping table, select an Organization in the left column and select the mapped Group in the right column. The 15 specific Groups are numbered 50,001-50,015. Any Organization that is not explicitly mapped, is mapped to the default Group numbered 50,000. A common use case, map 15 of your important customers to the specific groups and the rest to the default groups. Currently, the number of groups (15) is fixed and cannot be changed.

        4. Click Deploy.

    2. Endpoint: Click Edit and enter the following information:

      1. URL: Enter Elasticsearch Coordinator node URL.

      2. Ingest/Query checkbox: If this Coordinator node is to be used for Ingesting Events then check Ingest. If this Coordinator node is going to be used for Querying events, then check the Query flag. If you have multiple Coordinator nodes, then click + and select the URL and Ingest/Query flags. This flexibility enables FortiSIEM to separate a set of Coordinator nodes for Query and Ingest functionalities.

    3. Port: The TCP port for the URL above (set to HTTPS/443 by default)

    4. User Name: Enter the username for basic authentication to be used with the URL

    5. Password: Enter the password for basic authentication to be used with the URL

    6. Shard Allocation:

      1. If you set it to Fixed, then you enter the number of fixed Shards, FortiSIEM will not create new shards, even if a Shard reaches its size limit during event surge. You can set the Shard Allocation to Fixed only if you know your system well.

      2. If you set it to Dynamic, then you enter the number of fixed Starting Shards (default 5) and FortiSIEM will dynamically adjust the number of shards based on event rate. This is the recommended method.

    7. Replicas: If you set it to N, then there will be N+1 copies of every index in Elasticsearch. The most common value is Replicas = 1. A higher number of replicas can increase query speed and resiliency against failures, but may slow down event ingest and will use more storage space.

    8. Event Attribute Template: This defines how FortiSIEM Event Attributes are mapped to Elasticsearch Event Attribute Types. This mapping is used to store events in Elasticsearch. If you set it to Default, then FortiSIEM will use the default mapping. The default mapping maps all (currently 2000+) FortiSIEM Event Attributes and can be a large file. Since this mapping is stored in every index, the global Elasticsearch state also becomes large. It is possible to use a smaller file by including only the FortiSIEM Event Attributes used in your environment. In that case, set this field to Custom and enter the custom mapping file.

  4. Click Test.

  5. If the test succeeds, click Deploy.

 

Case 2: To configure AWS Opensearch, follow these steps:

  1. Go to ADMIN > Setup > Storage.

  2. Click Online, and from the Event Database drop-down list, select Elasticsearch, and from the ES Service Type drop-down list, select Amazon

  3. Enter the following parameters.

    1. Endpoint: Click Edit and the enter the following information:

      1. URL: Enter the AWS Opensearch URL.

      2. Ingest/Query checkbox: If this endpoint is to be used for Ingesting Events, then check Ingest. If this endpoint is going to be used for Querying events, then check the Query flag. If you have multiple endpoints, then click + and select the URL and Ingest/Query flags. This flexibility enables to separate a set of endpoints for Query and Ingest functionalities.

    2. Port: The TCP port for the URL above (set to HTTPS/443 by default).

    3. Access Key ID: Enter the Access Key ID for use with this endpoint.

    4. Secret Key: Enter the Secret Key to be used with this endpoint.

    5. Shard Allocation:

      1. If you set it to Fixed, then you enter the number of fixed Shards, FortiSIEM will not create new shards, even if a Shard reaches its size limit during event surge. You can set the Shard Allocation to Fixed only if you know your system well.

      2. If you set it to Dynamic, then you enter the number of fixed Starting Shards (default 5) and FortiSIEM will dynamically adjust the number of shards based on event rate. This is the recommended method.

    6. Replicas: If you set it to N, then there will be N+1 copies of every index in Elasticsearch. The most common value is Replicas = 1. A higher number of replicas can increase query speed and resiliency against failures, but may slow down event ingest and will use more storage space.

    7. Event Attribute Template: This defines how FortiSIEM Event Attributes are mapped to Elasticsearch Event Attribute Types. This mapping is used to store events in Elasticsearch. If you set it to Default, then FortiSIEM will use the default mapping. The default mapping maps all (currently 2000+) FortiSIEM Event Attributes and can be a large file. Since this mapping is stored in every index, the global Elasticsearch state also becomes large. It is possible to use a smaller file by including only the FortiSIEM Event Attributes used in your environment. In that case, set this field to Custom and enter the custom mapping file.

  4. Click Test.

  5. If the test succeeds, click Deploy.

Creating Archive for Elasticsearch Based Deployments

 

There are 3 archive options

Configuring HDFS Archive from Elasticsearch

In this option, FortiSIEM HDFSMgr process creates Spark jobs to directly pull events from Elasticsearch and store in HDFS. Follow these steps.

  1. Go to ADMIN > Setup > Storage.

  2. Click Archive, and select HDFS.

  3. Enter the following parameters:

    1. Uncheck Real Time Archive.

    2. For Spark Master Node:

      1. Select IP or Host and enter the IP address or Host name of the Spark Cluster Master node.

      2. Set Port to the TCP port number for FortiSIEM to communicate to the Spark Master node.

    3. For HDFS Name Node:

      1. Select IP or Host and enter the IP address or Host name of the HDFS Name node. This is the machine which stores the HDFS metadata: the directory tree of all files in the file system, and tracks the files across the cluster.

      2. Set Port to the TCP port number for FortiSIEM to communicate to the HDFS Name node.

  4. Click Test.

  5. If the test succeeds, click Deploy.

Configuring Real-time HDFS Archive from FortiSIEM

In this option, FortiSIEM HDFSMgr process creates Spark jobs to pull events from FortiSIEM Supervisor and Worker nodes. Follow these steps.

  1. Go to ADMIN > Setup > Storage.

  2. Click Archive, and select HDFS.

  3. Enter the following parameters:

    1. Check Real Time Archive. Set a Start time (in the future) when the real time archive should begin

    2. For Spark Master Node:

      1. Select IP or Host and enter the IP address or Host name of the Spark Cluster Master node.

      2. Set Port to the TCP port number for FortiSIEM to communicate to Spark Master node.

    3. For HDFS Name Node:

      1. Select IP or Host and enter the IP address or Host name of the HDFS Name node. This is the machine which stores the HDFS metadata: the directory tree of all files in the file system, and tracks the files across the cluster.

      2. Set Port to the TCP port number for FortiSIEM to communicate to HDFS Name node.

  4. Click Test.

  5. If the test succeeds, click Deploy.

Configuring Real-time Archive to NFS

In this option, FortiSIEM Supervisor and Worker nodes store events in NFS managed by FortiSIEM EventDB. This happens while events are getting inserted into Elasticsearch. This approach has no impact in Elasticsearch performance, but events are stored in both Elasticsearch and EventDB and managed independently. Follow these steps.

  1. Go to ADMIN > Setup > Storage.

  2. Click Archive, and select NFS.

  3. Enter the following parameters:

    1. IP/Host: [Required] Select IP or Host and enter the IP address/Host name of the NFS server.

    2. Exported Directory: [Required] Enter the file path on the NFS Server which will be mounted.

  4. Click Test.

  5. If the test succeeds, click Deploy.