Creating Retention Policy

The life cycle of an event in FortiSIEM begins in the online event database, before moving to the Archive data store. You can set up retention policies to specify which events are retained, and for how long, in the online event database and the archive.

Creating Online Event Retention Policy

Online event retention policies specify which events are retained, and for how long, in the online event database.

Note: This is applicable only for NFS and Local Storage.

  1. Go to ADMIN > Settings > Database > Retention.
  2. Under Online Retention Policy, click New.
  3. Select Enabled if the policy has to be enforced immediately.
  4. Choose the Organizations for which the policy must be applied (for service provider installations). Select All if it should apply to all organizations. 
  5. Choose the Reporting Devices to apply this policy using the edit icon and click Save. 
  6. Choose the Event Type or event type groups to apply this policy and click Save.  
  7. Enter or select the Time Period in days that the event data specified by the conditions (Organizations, Reporting Devices and Event Type) should be held in the online storage before it is moved to archive or purged.
  8. Enter any Description related to the policy.
  9. Click Save.

Consider the following when implementing online event retention policies:

  • Implementing an online event policy requires selectively deleting specific events from the database and then re-indexing the database for the affected days. This is expensive in terms of time and performance. Therefore, do not define excessively fine-grained retention policies, because this will affect database performance.
  • Policies are enforced only at the end of day – this means that events are deleted and re-indexed only at the end of the day. This minimizes the impact on database performance, because the database usage should be low at that time.
  • Policies are enforced only from the day the policy is first created. It can be expensive to automatically apply retention policies on potentially large amount of historical events. It is advisable to manually enforce the retention policies by running the command: EnforceRetentionpolicy <DATES>, where DATES is a comma-separated list of dates or date-range on which to enforce the policy. DATES is specified as the number of days since the UNIX epoch began: 1970-01-01. A date-range is the range specified by two dates inclusively separated by "-". For example, run the command EnforceRetentionpolicy 16230,16233-16235 to imply "enforce retention policies" on the online event database on these dates: 6/8/2014 and from 6/11/2014 to 6/13/2014.
  • FortiSIEM will attempt to retain the events in the online event database according to the policies. However, if the low storage threshold is hit (20GB, by default), then the events from the earliest day are moved to archive.
  • If an event has remained in the online event database for the time period in the event retention policy, then the event is moved to the archive at the end of the day.
  • If an event does not match any online event retention policy, then it remains in the online event database until the low storage threshold (20GB, by default) is reached. The event is then moved to the archive.
  • If the archive mount point is defined, then ALL events are moved from online to archive. Nothing is purged.
  • If the archive is not reachable after multiple retries, then FortiSIEM is forced to purge the event because there is nowhere to store the event.

Creating Offline (Archive) Retention Policy

These policies specify which events are retained, and for how long, in the archive.  

  1. Go to ADMIN > Settings > Database > Retention.
  2. Under Offline Retention Policy, click New to create a new policy.
  3. Select the Organization this policy applies to.
  4. Enter the Time Period in days for archive retention.
  5. Click Save.

Consider the following when implementing offline (archive) event retention policies:

  • Policies are enforced only at the end of the day.
  • FortiSIEM will attempt to retain the events in the archive according to the policies. However, if the low storage threshold is hit (20GB, by default), then the events which occurred earliest in the day are purged.
  • Policies are enforced only from the day the policy is written. It can be expensive to automatically apply retention policies on potentially large amounts of historical events. It is advisable to manually enforce the retention policies by running the command: TestDiskUChecker purge <archive mount point> <orgId> <StartPurgeDateEpoch> where archive mount point is the full path to the location where data is stores, orgID is the ID of the organization and StartPurgeEpoch is the number of days since the UNIX epoch began.
  • If an event has remained in the archive for the time period in the event retention policy, then the event is purged at the end of the day.
  • If an event does not match any archive retention policies, then it stays in the archive until the low storage threshold (20GB, by default) is reached. It is then purged.