Creating Retention Policy

The life cycle of an event in FortiSIEM begins in the Online event database, before moving to the Archive data store. Online event data resides on faster, but expensive storage. Archive data resides on relatively slower, cheaper and higher capacity storage. You can set up retention policies to specify which events are retained, and for how long, in the online and archive event databases.

ClickHouse Event Retention

This section covers how events retention is managed for ClickHouse based deployments. The deployment possibilities are provided in the following table.

FortiSIEM Deployment

Online Storage

Archive Storage
Non-AWS Hot and Warm tiers Real-time Archive on NFS or use a large Warm tier for Archive purposes.
AWS Hot and Warm tiers AWS S3

How ClickHouse Event Retention Works

Case 1: Regular non-AWS Deployments

For Online storage, event retention is managed using two mechanisms:

  • Space based retention: If free Hot tier disk utilization is less than 10%, then oldest events are moved to the Warm tier until free Hot tier disk utilization is more than 20%. If Warm tier is not defined, then those events are purged. When events are move or purged, FortiSIEM goes through each retention policy (90 days, 180 days, ...) and the oldest data within each policy is moved/purged.

  • Time based retention: You can specify Online event retention policies to specify the duration for which certain events need to be retained. The policies can take event attributes such as Organization, Reporting Device and Event Type as input. See Creating Online Event Retention Policy. During the retention period, the events can be in Hot or Warm storage depending on the Space based retention, e.g., if Hot tier becomes full, then the event may move to Warm tier, etc... After retention period expires, these events are purged from Online storage.

For Archive storage, events are pushed in real-time to Archive storage as they arrive. Therefore, events can stay in both Online and Archive storage and their retention are managed independently.

  • Space based retention: If free Archive disk utilization is less than 10GB, then oldest events are purged until free Archive disk utilization is more than 20GB. These parameters are defined in phoenix_config.txt file.

  • Organization based retention: You can write Archive retention policies to specify the duration for which events for each Organization need to be retained. See Creating Archive Event Retention Policy.

Case 2: AWS Deployments

For AWS deployments, the Online and Archive S3 are managed together.

  • Space based retention:

    • If free Hot tier disk utilization is less than 10%, then oldest events are moved to the Warm tier until free Hot tier disk utilization is more than 20%. If Warm tier is not defined, then those events are moved to S3. When FortiSIEM moves or removes, FortiSIEM goes through each retention policy (90 days, 180 days, ...) and within each policy, FortiSIEM removes/moves the oldest data.

    • Similarly, for Warm tier, if free Warm tier disk utilization is less than 10%, then oldest events are moved to Archive until free Warm tier disk utilization is more than 20%. When FortiSIEM moves or removes, FortiSIEM goes through each retention policy (90 days, 180 days, ...) and within each policy, FortiSIEM removes/moves the oldest data.

  • Time based retention: You can specify Online event retention policies to specify the duration for which certain events need to be retained. The policies can take event attributes such as Organization, reporting Device and Event Type as input. See Creating Online Event Retention Policy. During the retention period, the events can be in Hot or Warm or Archive storage depending on the Space based retention, e.g., if Hot tier becomes full then the event may move to Warm tier, etc... After retention period expires, these events are purged.

Creating ClickHouse Online Event Retention Policy

Online event retention policies specify which events are retained, and for how long, in the online event database. Take the following steps to create an Online Event retention policy for ClickHouse.

  1. Go to ADMIN > Settings > Database > Retention Policy.

  2. Under Online Retention Policy, click New.

  3. Select Enabled if the policy has to be enforced immediately.

  4. Choose the Organizations for which the policy must be applied (for service provider installations). Select All if it should apply to all organizations.

  5. Choose the Reporting Devices to apply this policy using the edit icon and click Save.

  6. Choose the Event Type or event type groups to apply this policy and click Save.

  7. Select the Retention Period from the drop-down list (3 Months, 6 Months, 1 Year, 3 Years, 5 Years, 10 Years, Forever (50 Years). Each month is 30 days.

  8. Enter any Description related to the policy.

  9. Click Save.

  10. When done, click Apply.

Implementation Notes:

  1. Any time the retention policy on ClickHouse environment is changed, you must click Apply to push the retention policy.

  2. Retention policies are evaluated based on Rank. A lower rank policy is evaluated first and first match is applied.

  3. All events matching a retention policy are retained for the duration specified by the Retention Period specified in the policy.

  4. For the events that do not match with any existing retention policy, the default value for Retention Days is 18, 250 (50 years)

Creating ClickHouse Archive Event Retention Policy

These policies specify which events are retained, and for how long, in the archive.

  1. Go to ADMIN > Settings > Database > Retention Policy.

  2. Under Offline Retention Policy, click New to create a new policy.

  3. Select the Organization this policy applies to.

  4. Enter the Time Period in days for archive retention.

  5. Click Save.

Implementation Notes:

  1. Policies are enforced only at the end of the day.

  2. FortiSIEM will attempt to retain the events in the archive according to the policies. However, if the low storage threshold is hit (10GB, by default), then the events which occurred earliest in the day are purged.

FortiSIEM EventDB Event Retention

This section covers how events retention is managed for EventDB based deployments.

How EventDB Event Retention Works

For Online storage, event retention is managed using two mechanisms:

  • Space based retention: If free online disk utilization is less than 10GB, then oldest events are moved to the Archive until free online disk utilization is more than 20GB. If Archive is not defined, then those events are purged.

  • Policy based retention: You can specify Online event retention policies to specify the duration for which certain events need to be retained in online storage. The policies can take event attributes such as Organization, Reporting Device and Event Type as input. See Creating Online Event Retention Policy. If an event has remained in the online EventDB for the time period in the event retention policy, then the event is moved to the Archive at the end of the day.

For Archive storage, event retention is managed using two mechanisms:

  • Space based retention: If free archive disk utilization is less than 10GB, then oldest events are purged until free online disk utilization is more than 20GB.

  • Policy based retention: You can specify Archive event retention policies to specify the duration for specific Organizations. See Creating Archive Event Retention Policy. If an event has remained in the archive EventDB for the time period in the event retention policy, then the event is purged at the end of the day.

Creating EventDB Online Event Retention Policy

  1. Go to ADMIN > Settings > Database > Retention Policy.

  2. Under Online Retention Policy, click New.

  3. Select Enabled if the policy needs to be applied.

  4. Choose the Organizations for which the policy must be applied (for service provider installations). Select All if it should apply to all organizations.

  5. Choose the Reporting Devices to apply this policy using the edit icon and click Save.

  6. Choose the Event Type or event type groups to apply this policy and click Save.

  7. Enter or select the Time Period in days that the event data specified by the conditions (Organizations, Reporting Devices and Event Type) should be held in the online storage before it is moved to archive or purged.

  8. Enter any Description related to the policy.

  9. Click Save.

Implementation Notes:

  • If an event has remained in the online event database for the time period in the event retention policy, then the event is moved to the archive at the end of the day.

  • If an event does not match any online event retention policy, then it remains in the online event database until the low storage threshold (10GB, by default) is reached. The event is then moved to the archive.

  • If the archive mount point is defined, then ALL events are moved from online to archive. Nothing is purged.

  • If the archive is not reachable after multiple retries, then FortiSIEM is forced to purge the event because there is nowhere to store the event.

  • FortiSIEM will attempt to retain the events in the online event database according to the policies. However, if the low storage threshold is hit (10GB, by default), then the events from the earliest day are moved to archive.

  • Implementing an online event policy requires selectively deleting specific events from the database and then re-indexing the database for the affected days. This is expensive in terms of time and performance. Therefore, do not define excessively fine-grained retention policies, because this will affect database performance.

  • Policies are enforced only at the end of day – this means that events are deleted and re-indexed only at the end of the day. This minimizes the impact on database performance because the database usage should be low at that time.

  • Policies are enforced by FortiSIEM only from the date just before the retention period. For example, if the retention period for a policy is 10 days, and today is 12/19/2022, then FortiSIEM will automatically enforce the policy for events with event receive time starting from 12/18/2022. For processing older dates, Fortinet recommends customers to use the EnforceRetentionPolicy tool as follows:

    • EnforceRetentionPolicy <DATES>, where DATES is a comma-separated list of dates or date-range on which to enforce the policy. DATES is specified as the number of days since the UNIX epoch began: 1970-01-01. A date-range can specified by two dates inclusively separated by "-".

      For example, run the command EnforceRetentionPolicy 16230,16233-16235 to enforce retention policies on these dates: 6/8/2014 and from 6/11/2014 to 6/13/2014.

    • Run the tool as admin user.

Creating EventDB Archive Event Retention Policy

These policies specify which events are retained, and for how long, in the archive.

  1. Go to ADMIN > Settings > Database > Retention Policy.

  2. Under Offline Retention Policy, click New to create a new policy.

  3. Select the Organization this policy applies to.

  4. Enter the Time Period in days for archive retention.

  5. Click Save.

Implementation Notes:

  1. Policies are enforced only at the end of the day.

  2. If an event has remained in the archive for the duration specified in the event retention policy, then the event is purged at the end of the day.

Elasticsearch Event Retention

This section covers how events retention is managed for Elasticsearch based deployments. The deployment possibilities are:

FortiSIEM Deployment

Online Storage

Archive Storage
On-premises Elasticsearch – Option 1 Hot and Warm tiers HDFS archive from Elasticsearch
On-premises Elasticsearch – Option 2 Hot and Warm tiers Real-time HDFS archive from FortiSIEM
On-premises Elasticsearch – Option 3 Hot and Warm tiers Real-time Archive to NFS
Elastic Cloud and AWS Elasticsearch Hot and Warm tiers Not available

How Elasticsearch Event Retention Works

Elasticsearch online events storage is managed by the following thresholds:

  • Hot Node

    • Free Space Threshold: When the Hot node cluster disk free space falls below Low Threshold, then events are moved to Warm nodes until the Hot node cluster disk free space reaches High Threshold. If Warm node is not defined, then events are Archived. If Archive is not defined or real time archive option is chosen, then events are purged.

    • Age Limit: Maximum number of days after which events are moved to Warm nodes. If Warm node is not defined, then events are Archived. If Archive is not defined or real time archive option is chosen, then events are purged.

  • Warm Node

    • Free Space Threshold: When the Warm node cluster disk free space falls below Low Threshold, then events are Archived. If Archive is not defined or real time archive option is chosen, then events are purged.

    • Age Limit: Maximum number of days after which events are moved to Archive. If Archive is not defined or real time archive option is chosen, then events are purged.

These thresholds are defined in Configuring Elasticsearch Retention Threshold.

For archive you can choose either HDFS or EventDB on NFS.

  • HDFS archive from Elasticsearch: In this option, FortiSIEM HDFSMgr process creates Spark jobs to directly pull events from Elasticsearch and store in HDFS. This option may result in extra load on Elasticsearch as events have to read and then deleted from Elasticsearch while events are getting inserted. In this option, archive disk is managed by threshold, that is when low threshold is reached, then events are purged until the high threshold is reached – see Configuring HDFS Archive Threshold.

  • Real-time HDFS archive from FortiSIEM: In this option, FortiSIEM HDFSMgr process creates Spark jobs to pull events from FortiSIEM Supervisor and Worker nodes. This happens while events are getting inserted into Elasticsearch. This approach has no impact in Elasticsearch performance, but events are stored in both Elasticsearch and HDFS and managed independently. Note that HDFS has better event storage compression properties. In this option, archive disk is managed by threshold, that is when low threshold is reached, then events are purged until the high threshold is reached – see Configuring HDFS Archive Threshold.

  • Real time archive to NFS: In this option, FortiSIEM Supervisor and Worker nodes store events in NFS managed by FortiSIEM EventDB. This happens while events are getting inserted into Elasticsearch. This approach has no impact in Elasticsearch performance, but events are stored in both Elasticsearch and EventDB and managed independently. Note that EventDB has better event storage compression properties. In this option, archive disk is managed by policies– see Creating Archive Event Retention Policy.

Configuring Elasticsearch Retention Threshold

Complete these steps to configure Native Elasticsearch free space and age retention threshold:

  1. Go to ADMIN > Settings > Database > Online Settings.

  2. Select the low percentage threshold, high percentage threshold, and age under:

    1. Hot Node - Free Space Threshold - Events are moved to Warm nodes based on the first occurrence of one of the following:

      • When the Hot node cluster disk free space falls below Low value, then events are moved to Warm nodes until the Hot node cluster disk free space reaches High value.

      • If the time duration limit set under Hot Age (the Warm age phase) is met, all events under this limit are moved to Warm nodes.

    2. Warm Node - Free Space Threshold - Events are moved to Warm nodes based on the first occurrence of one of the following:

      • When the Warm node cluster disk free space falls below Low value, then events are moved to Cold nodes until the Warm node cluster disk free space reaches High value.

      • If the time duration limit set under Warm Age (the Cold age phase) is met, all events under this limit are moved to Cold nodes.

        Note: In the fsiem_ilm_policy, the cold age phase is reflected as a sum of the warm age phase and cold age phase UI values.

Configuring HDFS Archive Threshold

Complete these steps to configure the HDFS retention threshold:

  1. Go to ADMIN > Settings > Database > Archive Data.

  2. Select the low and high percentage thresholds under Archive Threshold. If HDFS disk utilization falls below Low value, then events are purged until disk utilization reaches High value.

Creating Elasticsearch Archive Event Retention Policy

These policies specify which events are retained, and for how long, in the archive.

  1. Go to ADMIN > Settings > Database > Retention Policy.

  2. Under Offline Retention Policy, click New to create a new policy.

  3. Select the Organization this policy applies to.

  4. Enter the Time Period in days for archive retention.

  5. Click Save.

Implementation Notes:

  1. Policies are enforced only at the end of the day.

  2. If an event has remained in the archive for the duration specified in the event retention policy, then the event is purged at the end of the day.