ClickHouse Configuration

This section covers how to configure a ClickHouse topology of Keeper and Data/Query nodes.

Before beginning, make sure that:

  1. You have gone through ClickHouse Operation Overview, FortiSIEM Reference Architecture Using ClickHouse Guide, and the ClickHouse Sizing Guide, with the last two guides located in the FortiSIEM Documentation Library.
  2. You have identified the FortiSIEM nodes that are going to be ClickHouse Keeper nodes and ClickHouse Data nodes.
  3. You have configured appropriate disks on the FortiSIEM nodes appropriate for their role.

  4. It is important to understand that after adding a new shard to an existing cluster, ClickHouse does not automatically rebalance the data among existing shards and new shard. The user must manually rebalance as described in the Scaling FortiSIEM within ClickHouse section of the Reference Architecture, with detailed steps in Rebalancing Shards.

Take the following steps.

  1. Navigate to ADMIN > Settings > Database > ClickHouse Config.
  2. Configure ClickHouse Keeper Cluster.
    1. Click on + and add a Worker.
    2. Click on - to remove a Worker.
    3. An operation for a ClickHouse Keeper Cluster node, such as adding or removing a node, MUST be done individually, meaning that after an operation is done, a test and deploy action must be performed. For example, if you add a ClickHouse Keeper Cluster node, you must then perform a test and deploy before doing any other operation for another ClickHouse Keeper Cluster node, such as adding another ClickHouse Keeper Cluster node, or removing a ClickHouse Keeper Cluster node. Do NOT perform more than one operation, such as adding or removing a ClickHouse Keeper Cluster node without testing and deploying it, as doing so may cause stability issues.
  3. Configure ClickHouse Server Cluster. You need to know the number of shards.
    1. Click on + and add a shard.
    2. Add Workers to the shard.
      1. Check Data if this Worker is a ClickHouse Data Node. A Data node receives events, processes them and writes to ClickHouse database.
      2. Check Query if this Worker is a ClickHouse Query Node. A Query node stores events replicated from data nodes and participates in Queries. However, it does not process events and triggers incidents.
      3. Check both Data and Query if this Worker is both a ClickHouse Data and Query Node. This is the most common setup.
  4. Once the shards have been created and workers have been added to the shard, then click Test.
  5. If Test succeeds, then click Deploy to push the changes to ClickHouse.

Notes:

  1. If you made changes to the ClickHouse Keeper Cluster, then after Deploy succeeds, phClickHouseMonitor, ClickHouseKeeper and ClickHouseServer processes will restart.
  2. If you made changes to the ClickHouse Cluster, then after Deploy succeeds, phClickHouseMonitor, and ClickHouseServer processes will restart.

For Advanced Configuration Operations, see Advanced Operations in the Appendix.