Importing Events into FortiSIEM

The following tools are provided:

phClickHouseImport Tool

Description: This tool is used to migrate EventDB data into your ClickHouse database.

Note: If importing remotely, then for data migration to succeed, the IP address of the source host to the ClickHouse cluster must be added to the destination host ips.xml file.
Example: If you want to import data from 192.0.2.0 (source host with EventDB) to 192.0.2.227 (destination host with ClickHouse), on the destination host (192.0.2.227), you must add the source host IP address 192.0.2.0 to the destination host's ips.xml file.

~]# cat /etc/clickhouse-server/users.d/ips.xml

<yandex>

  <users>
    <default>
      <networks>
        <ip>127.0.0.1</ip>
        <ip>192.0.2.227</ip>
        <ip>192.0.2.0</ip>
      </networks>
    </default>
  </users>

</yandex>

Usage: phClickHouseImport --src [Source Dir] --srcorgid Organization ID --dstorgid Organization ID --starttime [Start Time] --endtime [End Time] --host [IP Address of the ClickHouse Server that the data will be imported to] --orgid [Organization ID]

Argument

Description

--src [Source Dir]

Provide the source directory that contains the eventDB data. The default path is

/data/eventdb/

If a path is provided, the data path will be created as:

<user input path>

Example: If --src /archive/CUSTOMER2000 is used, the data path will be created as /archive/CUSTOMER2000/
--starttime [Start Time]

Starting time of events to be imported. It must be in the following format: "YYYY-MM-DD hh:mm:ss". The supported time zone is GMT. Make sure to enclose the Start Time with quotation marks.

Example: phClickHouseImport --src /data/eventdb --starttime "2022-01-27 10:10:00" --endtime "2022-02-01 11:10:00"
--endtime [End Time]

The end time of events to be imported. It must be in the following format: "YYYY-MM-DD hh:mm:ss". The supported time zone is GMT. Make sure to enclose the End Time with quotation marks.

Example: phClickHouseImport --src /data/eventdb --starttime "2022-01-27 10:10:00" --endtime "2022-02-01 11:10:00"
--host [IP Address of the ClickHouse Server that the data will be imported to] The IP address of the ClickHouse server that the data will be imported to. If the host IP address is not provided, then localhost is used. The default IP address is 127.0.0.1.
--orgid [Organization ID]

Provide the ID of the organization with the events to be imported. The number can be from 0 to 4294967295. Multiple entries are allowed by adding --orgid [Organization ID] for each entry. Only matched orgid will be migrated.

Example: phClickHouseImport --src /data/eventdb --starttime "2022-01-01 23:00:00" --endtime "2022-02-01 10:00:00" --orgid 1 --orgid 2001 --host 192.0.20.0

--srcorgid [Organization ID] --dstorgid [Organization ID]

The --srcorgid is the organization ID that data will be imported from and the --dstorgid is the target organization ID that the data will be imported into ClickHouse.

Example: phClickHouseImport --src /data/eventdb/ --starttime "2022-01-27 10:10:00" --endtime "2022-02-01 11:10:00" --srcorgid 2005 --dstorgid 2008 --host 192.0.20.0

Notes:

  1. Can be run from Supervisor or Worker.
  2. Can be run as admin user.
  3. phClickHouseImport tool requires FortiSIEM 6.5.0 or higher.
  4. EventDB data needs to be copied to the machine where this tool can run.

Example:

 phClickHouseImport --src /data/eventdb --starttime "2022-01-01 23:00:00" --endtime "2022-02-01 10:00:00" --orgid 1 --orgid 2001 --host 192.0.20.0