Parser Inbuilt Functions

The following parser inbuilt functions are available:

Split JSON Event

This function allows you to take an array of JSON events, and create an individual log event for each object in the array.

Purpose

Some log events may arrive to FortiSIEM which are actually a list of multiple distinct log objects. This is typically the case with JSON API logs, where a single JSON response from some API contains a batch of log events held somewhere within the JSON structure.

 

How it Works

You build a parser that catches the monolithic event. You parse the portion of the event containing the array of individual events to a temporary variable.

You pass the JSON to the splitJsonEvent function.

  • This function will generate an individual log event for each log object in the array of events.

  • This function will prepend the optional defined log header string and optional trailing string to the message.

  • If drop original event set to true, the SIEM will drop the monolithic event from being processed.

The net result is you will get multiple individual log events for the SIEM to parse vs just the monolithic event.

Note: You can either have the same parser handle both the monolithic event and individual events (Consistent log header for event format recognizer). Alternatively, you can write a parser for the monolithic event, and have a separate parser for the individual event (Easier to do but requires multiple parsers)

 

Example

Example: <setEventAttribute attr="_resultCount">splitJsonEvent($_json, "alerts", "CustomHeader", "Optional Custom Trailer", "true")

Arg1: Variable pointing to the JSON structure, e.g. $_json which contains the following: "{"alerts":[{"event1"},{"event2"},{"event3"}]}"

Arg2: JSON Array Key Path: The JSON path to the array of events.

  • If the structure is exactly a JSON array, such as [{"event1"},{"event2"},{"event3"}], you can leave this empty e.g. ""

  • If the array of individual events is nested under a given JSON key, you specify the key under which it exists. We specify "alerts" in the below example because the JSON array is nested.

Arg3: String header to prepend to the individual message. This must cleanly reference the log source so you can write a parser to parse this log.

Arg4: String trailer to append to the individual message. This is mostly unnecessary, but in some edge cases you may want this. In must cases specify "" here.

Arg5: Drop Original Event bool flag. If true, the original monolithic event is dropped after splitting the events. If false, the original event is kept in FortiSIEM along with the individual split events.

 

For a Full Parser example, navigate to ADMIN > Settings > Device Support > Parsers, and search for "SAPEnterpriseThreatDetectionParser".

The SAP Parser reference is outlined below with comments.

Scenario

User is ingesting SAP Enterprise Threat logs into FortiSIEM using the HTTPS POST method of ingesting JSON formatted logs. See Ingesting JSON Formatted Events Received via HTTP(S) POST.

 

The ingested monolithic log looks like this: 

[
   "PH_DEV_MON_CUSTOM_JSON"
]:[
   "reptVendor"
]"=SAP",
[
   "reptModel"
]"=ETD",
[
   "reptDevName"
]"=sap.edt.device",
[
   "reptDevIpAddr"
]=192.0.20.0
[
   "json"
]"="
{ "alerts": [ { "Version": "1.0", "AlertCreationTimestamp": "2021-09-21T07:45:45.420Z", "AlertId": 812125, "AlertSeverity": "MEDIUM", "AlertStatus": "FORWARDED", "AlertSource": { "EventLogType": "SystemLog", "SystemIdActor": "System25" }, "AlertSystemIds": [ "System25" ], "HostNames": [ "lab1" ], "Category": "Log Failure", "PatternId": "624F0A8BB948854F997942AC0EDB2102", "PatternType": "FLAB", "PatternName": "Low Log Amount per system", "PatternNameSpace": "http://demo", "PatternDescription": "", "MinTimestamp": "2021-09-21T07:44:39.522Z", "MaxTimestamp": "2021-09-21T07:44:39.522Z", "Text": "Measurement 48 exceeded threshold 50 for ('Event, Log Type' = 'SystemLog' / 'System ID, Actor' = 'System25')", "Score": 50, "UiLink": "http://192.0.20.0:80/sap/hana/uis/clients/ushell-app/shells/fiori/FioriLaunchpad.html?siteId=sap.secmon.ui.mobile.launchpad|ETDLaunchpad#AlertDetails-show?alert=62037185B38D5449999D57F465F1BBF6", "TriggeringEvents": [ { "Id": "C60C48EF7B4E4D848F45F1904820C2CC", "Timestamp": "2021-09-21T07:44:39.522Z", "TechnicalLogEntryType": "SM21_D01", "TechnicalNumber": "481537", "TechnicalTimestampOfInsertion": "2021-09-21T07:44:40.681Z", "CorrelationId": "120CD18982991EEA9AB8F87143C8DE88", "CorrelationSubId": "00000000000000000000000000000000", "EventCode": "D01", "EventSemantic": "Executable, Run, Cancel", "EventLogType": "SystemLog", "EventMessage": "LMDB_UPLOAD_ERRORS 011", "EventSeverityCode": "50", "EventSourceId": "198.51.100.0", "EventSourceType": "IP Address", "GenericRiskLevel": "3", "NetworkHostnameActor": "test-1", "NetworkHostnameInitiator": "0.0.0.0", "NetworkHostnameReporter": "test-1", "NetworkIPAddressInitiator": "0.0.0.0", "ServiceExecutableName": "00016", "ServiceExecutableType": "B", "ServiceInstanceName": "lab1_BLK_00", "ServiceProgramName": "RLMDB_UPLOAD_BACKGROUND", "ServiceTransactionName": "S000", "SystemIdActor": "System25", "SystemGroupIdActor": "BLN", "SystemGroupIdInitiator": "BLN", "SystemGroupIdIntermediary": "BLN", "SystemIdReporter": "System25", "SystemGroupIdReporter": "BLN", "SystemGroupIdTarget": "BLN", "SystemTypeActor": "ABBA", "SystemGroupTypeActor": "SAP", "SystemGroupTypeInitiator": "SAP", "SystemGroupTypeIntermediary": "SAP", "SystemTypeReporter": "ABBA", "SystemGroupTypeReporter": "SAP", "SystemGroupTypeTarget": "SAP", "UsernameDomainNameActing": "System25", "UsernameDomainTypeActing": "ABBA", "UserPseudonymActing": "BLK_BTC_SMP", "EventName": "ExecutableRunCancel", "EventNamespace": "http://sap.com/secmon", "TechnicalTimestampInteger": "1632210279522" }, { "Id": "765451B9772A4FA580059A7BAD7D149C", "Timestamp": "2021-09-21T07:44:39.522Z", "TechnicalLogEntryType": "SM21_E0A", "TechnicalNumber": "481536", "TechnicalTimestampOfInsertion": "2021-09-21T07:44:40.681Z", "CorrelationId": "120CD18982991EEA9AB8F87143C8DE88", "CorrelationSubId": "00000000000000000000000000000000", "EventCode": "E0A", "EventLogType": "SystemLog", "EventMessage": "&aRLMDB_UPLOAD_DISPLAY_LOG&b00000000000011498140", "EventSeverityCode": "9", "EventSourceId": "198.51.100.0", "EventSourceType": "IP Address", "GenericRiskLevel": "0", "NetworkHostnameActor": "test-1", "NetworkHostnameInitiator": "0.0.0.0", "NetworkHostnameReporter": "test-1", "NetworkIPAddressInitiator": "0.0.0.0", "ServiceExecutableName": "00016", "ServiceExecutableType": "B", "ServiceInstanceName": "lab1_BLK_00", "ServiceProgramName": "RLMDB_UPLOAD_BACKGROUND", "ServiceTransactionName": "S000", "SystemIdActor": "System25", "SystemGroupIdActor": "BLN", "SystemGroupIdInitiator": "BLN", "SystemGroupIdIntermediary": "BLN", "SystemIdReporter": "System25", "SystemGroupIdReporter": "BLN", "SystemGroupIdTarget": "BLN", "SystemTypeActor": "ABBA", "SystemGroupTypeActor": "SAP", "SystemGroupTypeInitiator": "SAP", "SystemGroupTypeIntermediary": "SAP", "SystemTypeReporter": "ABBA", "SystemGroupTypeReporter": "SAP", "SystemGroupTypeTarget": "SAP", "UsernameDomainNameActing": "System25", "UsernameDomainTypeActing": "ABBA", "UserPseudonymActing": "BLK_BTC_SMP", "TechnicalTimestampInteger": "1632210279522" } ] } ] }

The parser processes the log header, and stores the data after the "[json]=" key into a variable called "_json".

Upon close inspection, we identify multiple distinct log events (several alerts) under a JSON key called "alerts".

To generate a new log event for each single alert, we pass this JSON to our split json function.

<setEventAttribute attr="_resultCount">splitJsonEvent($_json, "alerts", "[PH_DEV_MON_CUSTOM_JSON]:[reptVendor]=SAP,[reptModel]=ETD,SAP_Individual_Event,json=", "", "true")</setEventAttribute>

The _resultCount doesn't do much other than return an integer number of alerts we processed.

If we wanted to, we could do something with this, but mostly this variable is unused.

The arguments are explained above, but repeated below, the info is:

Arg1: our $_json var containing our JSON portion of the log.

Arg2: "alerts" - The JSON key holding the array of alert objects. The function will resolve here and treat each object as a new event.

Arg3: "[PH_DEV_MON_CUSTOM_JSON]:[reptVendor]=SAP,[reptModel]=ETD,SAP_Individual_Event,json=" We want the same parser handling the monolithic event to handle the individual events, so we attach this very unique log header.

Arg4: "" - Optional trailing string to append to log, in most cases we don't need this.

Arg5: "true" - String value of true or false to indicate if we should drop (not store or process) the monolithic event. In most cases after processing you don't want the original.

 

The individual events when triggered will look like this:

[PH_DEV_MON_CUSTOM_JSON]:[reptVendor]=SAP,[reptModel]=ETD,SAP_Individual_Event,json={"Id":"C60C48EF7B4E4D848F45F1904820C2CC","Timestamp":"2021-09-21T07:44:39.522Z","TechnicalLogEntryType":"SM21_D01","TechnicalNumber":"481537","TechnicalTimestampOfInsertion":"2021-09-21T07:44:40.681Z","CorrelationId":"120CD18982991EEA9AB8F87143C8DE88","CorrelationSubId":"00000000000000000000000000000000","EventCode":"D01","EventSemantic":"Executable,Run,Cancel","EventLogType":"SystemLog","EventMessage":"LMDB_UPLOAD_ERRORS011","EventSeverityCode":"50","EventSourceId":"198.51.100.0","EventSourceType":"IPAddress","GenericRiskLevel":"3","NetworkHostnameActor":"test-1","NetworkHostnameInitiator":"0.0.0.0","NetworkHostnameReporter":"test-1","NetworkIPAddressInitiator":"0.0.0.0","ServiceExecutableName":"00016","ServiceExecutableType":"B","ServiceInstanceName":"lab1_BLK_00","ServiceProgramName":"RLMDB_UPLOAD_BACKGROUND","ServiceTransactionName":"S000","SystemIdActor":"System25","SystemGroupIdActor":"BLN","SystemGroupIdInitiator":"BLN","SystemGroupIdIntermediary":"BLN","SystemIdReporter":"System25","SystemGroupIdReporter":"BLN","SystemGroupIdTarget":"BLN","SystemTypeActor":"ABBA","SystemGroupTypeActor":"SAP","SystemGroupTypeInitiator":"SAP","SystemGroupTypeIntermediary":"SAP","SystemTypeReporter":"ABBA","SystemGroupTypeReporter":"SAP","SystemGroupTypeTarget":"SAP","UsernameDomainNameActing":"System25","UsernameDomainTypeActing":"ABBA","UserPseudonymActing":"BLK_BTC_SMP","EventName":"ExecutableRunCancel","EventNamespace":"http://sap.com/secmon","TechnicalTimestampInteger":"1632210279522"}

 

Notice that the contents of the JSON body of the log is the first alert object in the original monolithic event.

We can now parse the individual event via a matching parser (either using the same parser as the original event, or by using a different parser).

Combining Two or More Strings to Produce a Final String

This is accomplished by the combineMsgId function.

<setEventAttribute attr="eventType">combineMsgId("string-", $_evIdPrefix, "-", $_evIdSuffix)</setEventAttribute>
_evIdPrefix: prefix
_evIdSuffix: suffix
eventType: string-prefix-suffix

Strings can only be wrapped by double quotes " but not single quotes '.

Normalize MAC Address

This is accomplished by the normalizeMAC function.

<setEventAttribute attr="hostMACAddr">normalizeMAC($_mac)</setEventAttribute>

Compare Interface Security Level

This is accomplished by the compIntfSecVal function.

<setEventAttribute attr="_result">compIntfSecVal($srcIntf, $destIntf)</setEventAttribute>

Compare the Security Level of srcIntf and destIntf. The result may be "LESS", "GREATER" or "EQUAL".

Convert Hex Number to Decimal Number

This is accomplished by the convertHexStrToInt function.

<setEventAttribute attr="ipConnId">convertHexStrToInt($_ipConnId)</setEventAttribute>

Convert TCP/UDP Protocol String to Port Number

This is accomplished by the following convertStrToIntIpPort function.

<setEventAttribute attr="destIpPort">convertStrToIntIpPort($_dport)</setEventAttribute>

Convert Protocol String to Number

This is accomplished by the following convertStrToIntIpProto function.

<setEventAttribute attr="ipProto">convertStrToIntIpProto($_proStr)</setEventAttribute>

Convert Decimal IP to String

This is accomplished by the following convertIpDecimalToStr function.

<setEventAttribute attr="srcIpAddr">convertIpDecimalToStr($_srcIpAddr)</setEventAttribute>

Convert Host Name to IP

This is accomplished by the following convertHostNameToIp function.

<setEventAttribute attr="srcIpAddr">convertHostNameToIp($_saddr)</setEventAttribute>

Add Two Numbers

This is accomplished by the following add function.

<setEventAttribute attr="totBytes">add($sentBytes, $recvBytes)</setEventAttribute>

Divide Two Numbers

This is accomplished by the following divide function.

<setEventAttribute attr="memUtil">divide($_usedMem, $_totalMem)</setEventAttribute>

Scale

This is accomplished by the following scale function.

<setEventAttribute attr="durationMSec">scale($_durationSec, 1000)</setEventAttribute>

Calculate Micro Seconds

This is accomplished by the following calculateMSec function.

<setEventAttribute attr="durationMSec">calculateMSec($_duration)</setEventAttribute>
_duration: 00:00:15
durationMSec: 15000

Extract Host from FQDN

This is accomplished by the following extractHostFromFQDN function.

<setEventAttribute attr="hostName">extractHostFromFQDN($_fqdn)</setEventAttribute>
_fqdn: host.abc.net
hostName: host

If _fqdn contains dot, get the string before the first dot; otherwise, get the whole string.

Replace String by Regular Expression

This is accomplished by the following replaceStringByRegex function.

<setEventAttribute attr="eventType">replaceStringByRegex($_eventType, "\s+", "_")</setEventAttribute>
_eventType: Event Type
eventType: Event_Type

Replace String in String

This is accomplished by the following replaceStrInStr function.

<setEventAttribute attr="computer">replaceStrInStr($_computer, "\\", "")</setEventAttribute>

Resolve DNS Name

This is accomplished by the following resolveDNSName function.

<setEventAttribute attr="destIpAddr">resolveDNSName($destName)</setEventAttribute>

Shift Time Sec

This is accomplished by the following shiftTimeSec function.

<setEventAttribute attr="logonTime">shiftTimeSec($_mon, $_day, $_year, $_time, $_durationSec)</setEventAttribute>
_mon: 1
_day: 1
_year: 2000
_time: 01:00:10
_durationSec: 10
logonTime: 01:00:00 01/01/2000

To DateTime

This is accomplished by the following toDateTime function.

<setEventAttribute attr="deviceTime">toDateTime($_mon, $_day, $_year, $_time)</setEventAttribute>
<setEventAttribute attr="deviceTime">toDateTime($_mon, $_day, $_time)
</setEventAttribute>

Trim Attribute

This is accomplished by the following trimAttribute function.

<setEventAttribute attr="destName">trimAttribute($destName, ".")
</setEventAttribute>

Trim leading and trailing dots in destName.

Get Severity from Syslog Priority

This is accomplished by the following getEventSeverityFromSyslogPriority function.

<setEventAttribute attr="eventSeverity">getEventSeverityFromSyslogPriority($_pri)</setEventAttribute>
_pri: 52
eventSeverity: 5

Set severity by syslog priority. The bottom 3 bits of the priority indicates the severity.

http://en.wikipedia.org/wiki/Syslog

To Unix Time (with Time Zone)

This is accomplished by the following toUnixTime function.

<setEventAttribute attr="deviceTime">toUnixTime($_deviceTime)</setEventAttribute>
_deviceTime: 20130509073221.932817-000

Decode Base64

This is accomplished by the following decodeBase64 function.

<setEventAttribute attr="httpFullRequest">decodeBase64($_msg)</setEventAttribute>

Unzip String

This is accomplished by the following unzip function.

<setEventAttribute attr="msg">unzip($_msg)</setEventAttribute>

Calculate Latency

This is accomplished by the following calculateLatency function.

<setEventAttribute attr="_latency">calculateLatency($_evtRecvTime, $deviceTime)</setEventAttribute>

Calculate the latency. If _evtRecvTime is later than deviceTime, return the latency in seconds. Otherwise, return 0.