Exporting Events to Files
phExportEvent
You can run the phExportEvent tool from a Supervisor or Worker node to export events to CSV files. The file will contain these fields:
-
Customer Id (applicable to SP license)
-
Reporting Device IP
-
Reporting Device Name
-
Event Received Time
-
Raw Message
This code block shows the commands that you can use with phExportEvent, followed by a table that describes
them in more detail.
phExportEvent {--dest DESTINATION_DIR} {--starttime START_TIME | --relstarttime RELATIVE_START_TIME} {--endtime END_TIME | --relendtime RELATIVE_END_TIME} [--dev DEVICE_NAME] [--org ORGANIZATION_NAME] [-t TIME_ZONE]
| pHExportEvent Command | Description |
|---|---|
DESTINATION_
DIR
|
Destination directory where the exported event files are saved. |
START_TIME
|
Starting time of events to be exported. The format is YYYY-MM-DD HH:MM:SS {+|-}
TZ. If TZ is not given, the local time zone of the machine where the script is running will
be used. Example: 2010-03-10 23:00:00 -8 means Pacific Standard Time,
23:00:00 03/10/2010. 2010-07-29 10:20:00 +5:30 means India Standard
Time 10:20:00 07/29/2010.
|
RELATIVE_
START_TIME
|
This must be used together with
where |
END_TIME
|
Ending time of events to be exported. The format is the same as described for START_TIME. |
RELATIVE_END_
TIME
|
This must be used together with START_TIME. Ending time of events to be exported
is relative forward to the start time, specified using START_TIME. The format is
the same that is used for RELATIVE_START_TIME.
|
DEVICE_NAME
|
Provide the host name or IP address of the device with the events to be exported. Use a
comma-separated list to specify multiple IPs or host names, for example, --dev
10.1.1.1,10.10.10.1,router1,router2. Host name is case insensitive. |
ORGANIZATION_
NAME
|
This is used only for Service Provider deployments. Provide the name of the organization with the
events to be exported. To specify multiple organizations, enter a command for each organization, for example, --org "Public Bank" --org "Private
Bank". The organization name is case insensitive.
|
TIME_ZONE
|
Specifies the time zone used to format the event received time in the exported event
files. The format is {+|-}TZ, for example, -8 means Pacific Standard Time,
+5:30 means India Standard Time. |
TestESSplitter
You can run the TestESSplitter tool from a Supervisor or Worker node to export events from ElasticSearch to FortiSIEM eventDB format. It is located in n /opt/phoenix/bin/.
This code block shows the commands you can use with TestESSplitter followed by a table that describes them in more detail.
TestESSplitter <ESBroker> <ESUser> <ESPassword> <IndexName> <destDir> <splitThreads> <logLevel>
Note: For <destDir>, a trailing slash is mandatory. Example: https://<destDir>/.
| TestESSplitter Command | Description |
|---|---|
ESBroker
|
The IP of ElasticSearch Co-ordinator node. |
ESUser
|
The ElasticSearch username for authentication. |
ESPassword
|
The ElasticSearch password for authentication. |
IndexName
|
Provide an Index name. A new Index is created per day. Here is an example index name, fortisiem-event-2021.05.14-2000-000001 where“fortisiem-event-2021.05.14” is the day and “2000” is the Organization ID. To find a list of indexes, run this command:curl -XGET '10.10.2.5:9200/_cat/shards?v'replacing 10.10.2.5 with the IP of a Co-ordinator node. |
destDir
|
Destination directory where the exported events are saved in FortiSIEM eventDB format. |
|
|
Number of threads. |
|
|
INFO or DEBUG level log messages. |
See TestESSplitter Example for an example.
Example Usage
TestESSplitter Example
[root@fsm]# /opt/phoenix/bin/TestESSplitter 10.10.2.5 "" "" fortisiem-event-2021.05.14-2000-000001 /root/output 10 INFO [PH_MODULE_LOG_LEVEL_CHANGE]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=phBaseProcess.cpp,[lineNumber]=675,[oldLogLevel]=2047,[newLogLevel]=424,[phLogDetail]=Module received log level change [PH_MODULE_LOCAL_CONFIG_LOADED]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=phConfigLoader.cpp,[lineNumber]=166,[configName]=global,[phLogDetail]=Module loaded local config successfully [PH_MODULE_LOCAL_CONFIG_LOADED]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=phConfigLoader.cpp,[lineNumber]=166,[configName]=phdatamanager,[phLogDetail]=Module loaded local config successfully [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=phHttpClientPool.cpp,[lineNumber]=46,[phLogDetail]=phHttpClientPool: init hosts/port/auth/header=10.10.2.5/9200/:****/Content-Type: application/json * Trying 10.10.2.5... * TCP_NODELAY set * Connected to 10.10.2.5 (10.10.2.5) port 9200 (#0) > GET / HTTP/1.1 Host: 10.10.2.5:9200 Accept: */* Content-Type: application/json < HTTP/1.1 200 OK < content-type: application/json; charset=UTF-8 < content-length: 530 < * Connection #0 to host 10.10.2.5 left intact [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1732,[phLogDetail]=Elastic init success: http://10.10.2.5:9200/ * Found bundle for host 10.10.2.5: 0x18f0870 [can pipeline] * Re-using existing connection! (#0) with host 10.10.2.5 * Connected to 10.10.2.5 (10.10.2.5) port 9200 (#0) > GET /_cat/indices/fortisiem-event-2021.05.14-2000-000001?h=pri,rep,docs.count HTTP/1.1 Host: 10.10.2.5:9200 Accept: */* Content-Type: application/json … … … … < [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1974,[phLogDetail]=Elastic succeed hits total 66 for index fortisiem-event-2021.05.14-2000-000001 slice 1 max 10 * Connection #0 to host 10.10.2.5 left intact [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1974,[phLogDetail]=Elastic succeed hits total 61 for index fortisiem-event-2021.05.14-2000-000001 slice 8 max 10 [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=2002,[phLogDetail]=Elastic index query completed 0 seconds [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=2002,[phLogDetail]=Elastic index query completed 0 seconds < HTTP/1.1 200 OK < content-type: application/json; charset=UTF-8 < content-length: 47737 < * Connection #0 to host 10.10.2.5 left intact [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1974,[phLogDetail]=Elastic succeed hits total 53 for index fortisiem-event-2021.05.14-2000-000001 slice 3 max 10 < HTTP/1.1 200 OK < content-type: application/json; charset=UTF-8 < content-length: 47178 < * Connection #0 to host 10.10.2.5 left intact < HTTP/1.1 200 OK < content-type: application/json; charset=UTF-8 < content-length: 41910 < * Connection #0 to host 10.10.2.5 left intact < HTTP/1.1 200 OK < content-type: application/json; charset=UTF-8 < content-length: 53258 < * Connection #0 to host 10.10.2.5 left intact < HTTP/1.1 200 OK < content-type: application/json; charset=UTF-8 < content-length: 60587 < * Connection #0 to host 10.10.2.5 left intact [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1974,[phLogDetail]=Elastic succeed hits total 59 for index fortisiem-event-2021.05.14-2000-000001 slice 4 max 10 [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1974,[phLogDetail]=Elastic succeed hits total 53 for index fortisiem-event-2021.05.14-2000-000001 slice 7 max 10 [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1974,[phLogDetail]=Elastic succeed hits total 68 for index fortisiem-event-2021.05.14-2000-000001 slice 6 max 10 [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=1974,[phLogDetail]=Elastic succeed hits total 46 for index fortisiem-event-2021.05.14-2000-000001 slice 2 max 10 [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=2002,[phLogDetail]=Elastic index query completed 0 seconds [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=2002,[phLogDetail]=Elastic index query completed 0 seconds [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=2002,[phLogDetail]=Elastic index query completed 0 seconds [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=2002,[phLogDetail]=Elastic index query completed 0 seconds [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=EventLoader.cpp,[lineNumber]=2002,[phLogDetail]=Elastic index query completed 0 seconds [PH_GENERIC_INFO]:[eventSeverity]=LM_INFO,[procName]=<unknown>,[fileName]=TestESSplitter.cpp,[lineNumber]=82,[phLogDetail]=Events processed for split: 559 3.15
The result will be eventDB structured directories and files.
[root@fsm]# ls -l /root/output/ total 0 drwx------ 3 root root 22 May 14 15:25 CUSTOMER_2000 [root@fsm]# ls -l /root/output/CUSTOMER_2000/ total 0 drwx------ 3 root root 19 May 14 15:25 internal [root@fsm]# ls -l /root/output/CUSTOMER_2000/internal/ total 0 drwx------ 3 root root 37 May 14 15:25 18761 [root@fsm]# ls -l /root/output/CUSTOMER_2000/internal/18761/ total 4 drwx------ 12 root root 4096 May 14 15:25 450264-450287-168428094 [root@fsm]# ls -l /root/output/CUSTOMER_2000/internal/18761/450264-450287-168428094/ total 0 drwx------ 3 root root 18 May 14 15:25 seg-1-0-48-1620951010-1620971132 drwx------ 3 root root 18 May 14 15:25 seg-1-1-70-1620950470-1620971172 drwx------ 3 root root 18 May 14 15:25 seg-1-2-35-1620950916-1620971172 drwx------ 3 root root 18 May 14 15:25 seg-1-3-66-1620951819-1620969371 drwx------ 3 root root 18 May 14 15:25 seg-1-4-61-1620950830-1620970642 drwx------ 3 root root 18 May 14 15:25 seg-1-5-59-1620950830-1620971132 drwx------ 3 root root 18 May 14 15:25 seg-1-6-53-1620950482-1620970632 drwx------ 3 root root 18 May 14 15:25 seg-1-7-46-1620951278-1620971182 drwx------ 3 root root 18 May 14 15:25 seg-1-8-53-1620950470-1620970452 drwx------ 3 root root 18 May 14 15:25 seg-1-9-68-1620950650-1620971132
Copyright © 2024 Fortinet, Inc. All Rights Reserved. | Terms of Service | Privacy Policy
