Configuring Elasticsearch Buffer

When there is a huge number of events, the Elasticsearch server may become overloaded with processing them and cannot accept all uploads during this time.

A solution to this issue is to save events to a local disk when events cannot be uploaded. FortiSIEM allows you to configure a disk based buffer to save these events until Elasticsearch is available again.

Fortinet recommends that a dedicated disk be used for this purpose. At a minimum, ensure it does not use shared NFS, which could cause latency. It is also strongly recommended that if you wish to use this feature on Workers, that all Workers be configured. If a Worker is not configured, it will be blocked, and consequently, will not accept Collector event uploads. In this situation, the Collector will fail to upload to the unconfigured Worker, and will attempt to connect to another Worker that accepts uploads.

To configure, access the phoenix_config file. If the buffer path is defined, the feature is enabled.

[BEGIN Elasticsearch]

log_buffer_per_customer_path=/eventbuffer #empty means disabled.

log_buffer_per_customer_reserved_disk_space=1 # GB

[END]

Next, in the phoenix_config file, in the Elastisearch section, modify index_max_retry to configure the number of times FortiSIEM will attempt to retry uploads. If the retry time is set to 0, FortiSIEM will never drop events and after the connection between FortiSIEM and Elasticsearch has recovered, all events will be uploaded.

After defining the event buffer path, restart phDataManager.

This feature can be enabled on the Supervisor and Workers.