Event Parser Specification

Custom Parser XML Specification Template

The basic template for a custom parser XML specification includes five sections. Click the name of any section for more information.

Section Description
Parser File Adding, editing, or cloning a parser file.
Device or Application Type Specification The type of device or application associated with the parser.
Event Format Recognizer Specification Patterns that determine whether an event will be parsed by this parser.
Pattern Definition Specification Defines the parsing patterns that are iterated over by the parsing instructions.
Parsing Instructions Specification Instructions on how to parse events that match the format recognizer patterns.

 

Custom Parser XML Specification Template 

  <patternDefinitions> </patternDefinitions>
  <eventFormatRecognizer> </eventFormatRecognizer>
  <parsingInstructions> </parsingInstructions>

Parser File

This section provides steps to create, edit, or clone a parser file.

Create a Parser File

To create a parser, take the following steps:

  1. Navigate to ADMIN > Device Support > Parsers.

  2. Click New.

  3. From the Add Event Parser Definition window, take the following steps:

    1. In the Name field, enter the name of the parser.

    2. In the Device Type drop-down list, select the appropriate device.

    3. In the main field, provide your parser XML.

    4. The following options are also available:

      Parser XML Button Description
      Validate Click to check your XML code syntax.
      Test Click to test your XML code.
      Reformat Click to format your XML code.

      Enable

      Click the Enable checkbox to enable the parser/XML code.
      Clear XML Click to remove the existing XML code.
      Previous Click to go to the prior XML code page.
      Next Click to go to the next XML code page.

    5. When done, click Save.

Edit a Parser File

You are only allowed to edit a custom parser file. To edit an existing custom parser, take the following steps:

  1. From ADMIN > Device Support > Parsers, select a custom parser.

  2. Click Edit.

  3. From the Edit Event Parser Definition window, you can make changes to the following fields: 

    1. In the Name field, make any changes to the name of the parser.

    2. In the Device Type drop-down list, make any changes to the device type.

    3. In the main field, make any changes to your parser XML.
      See the table in Create a Parser for available options.

  4. When done, click Save.

Clone a Parser File

To clone an existing parser, take the following steps:

  1. From ADMIN > Device Support > Parsers, select a parser.

  2. Click Clone.

  3. From the Add Event Parser Definition window, you can make changes to the following fields: 

    1. In the Name field, make any changes to the name of the parser.

    2. In the Device Type drop-down list, make any changes to the device type.

    3. In the main field, make any changes to your parser XML.
      See the table in Create a Parser for available options.

  4. When done, click Save.

Device or Application Type Specification

This section specifies the device or the application to which this parser applies. The device and application definitions enable FortiSIEM to detect the device and application type for a host from the received events. This is called log-based discovery in FortiSIEM. Once a received event is successfully parsed by this file, a CMDB entry is created with the device and application set from this file. FortiSIEM discovery may further refine the device.

There are two separate subsections for device and application. In each section, vendor, model and version can be specified, but version is not typically needed.

Set Version to Any

In the examples in this topic, <Version> is set to ANY because events are generally not tied to a particular version of a device or software. You could of course set this to a specific version number if you only wanted this parser to apply to a specific version of an application or device.

Vendor and Model Must Match the FortiSIEM Version

<Vendor> and <Model> entries must match the spelling and capitalization in the CMDB.

Examples of Specifications for Types of Device and Applications

Hardware Appliances

In this case, the type of event being parsed specifies the device type, for example Cisco IOS, Cisco ASA, etc.

To add a device type, see Adding a Device or Application.

Software Operating Systems that Specify the Device Type

In this case, the type of events being parsed specifies the device type, for example Microsoft Windows etc. In this case the device type section looks like:

<deviceType>
  <Vendor>Microsoft</Vendor>
  <Model>Windows</Model>
  <Version>ANY</Version>
</deviceType>
Applications that Specify Both Device Type and Application

In this case, the events being parsed specify the device and application types because Microsoft SQL Server can only run on Microsoft Windows OS.

<deviceType>
  <Vendor>Microsoft</Vendor>
  <Model>Windows</Model>
  <Version>ANY</Version>
</deviceType>
<appType>
  <Vendor>Microsoft</Vendor>
  <Model>SQL Server</Model>
  <Version>ANY</Version>
  <Name> Microsoft SQL Server</Name>
</appType>
Applications that Specify the Application Type but Not the Device Type

Consider the example of an Oracle database server, which can run on both Windows and Linux operating systems. In this case, the device type is set to Generic but the application is specific. FortiSIEM depends on discovery to identify the device type.

<deviceType>
  <Vendor>Generic</Vendor>
  <Model>Generic</Model>
  <Version>ANY</Version>
</deviceType>
<appType>
  <Vendor>Oracle</Vendor>
  <Model>Database Server</Model>
  <Version>ANY</Version>
  <Name>Oracle Database Server</Name>
</appType>

Format Recognizer Specification

In many cases, events associated with a device or application will contain a unique pattern. You can enter a regular expression in the Format Recognizer section of the parser XML file to search for this pattern, which if found, will then parse the events according to the parser instructions. After the first match, the event source IP to parser file map is cached, and only that parser file is used for all events from that source IP. A notable exception is when events from disparate sources are received via a syslog server, but that case is handled differently.

While not a required part of the parser specification, a format recognizer can speed up event parsing, especially when one parsing pattern file among many pattern files must be chosen. Only one pattern check can determine whether the parsing file must be used or not. The other less efficient option would be to examine patterns in every file. At the same time, the format recognizer must be carefully chosen so that it is not so broad to misclassify events into wrong files, and at the same time, not so narrow that it fails at classifying the right file.

Order in Which Parsers are Used

FortiSIEM parser processes the files in the specific order listed in the file parserOrder.csv.

Format Recognizer Syntax

The specification for the format recognizer section is:

<eventFormatRecognizer><!\[CDATA\[regexpattern\]\]></eventFormatRecognizer>

In the regexpattern block, a pattern can be directly specified using regex or a previously defined pattern (in the pattern definition section in this file or in the GeneralPatternDefinitions.xml file) can be referenced.

Example Format Recognizers

Cisco IOS

All Cisco IOS events have a %module name pattern.

<patternDefinitions>
  <pattern name="patCiscoIOSMod" list="begin"><!\[CDATA\[FW|SEC|SEC_LOGIN|SYS|SNMP|\]\]></pattern>
  <pattern name="patCiscoIOSMod" list="continue"><!\[CDATA\[LINK|SPANTREE|LINEPROTO|DTP|PARSER|\]\]></pattern>
  <pattern name="patCiscoIOSMod" list="end"><!\[CDATA\[CDP|DHCPD|CONTROLLER|PORT_SECURITY-SP\]\]></pattern>
</patternDefinitions>
<eventFormatRecognizer><!\[CDATA\[:%<:patCiscoIOSMod>-<:gPatInt>-<:patStrEndColon>:\]\]></eventFormatRecogniz
er>
Cisco ASA

All Cisco ASA events have the pattern ASA-severity-id pattern, for example ASA-5-12345.

<eventFormatRecognizer><!\[CDATA\[ASA-\\d-\\d+\]\]></eventFormatRecognizer>
Palo Alto Networks Log Parser

In this case, there is no unique keyword, so the entire message structure from the beginning to a specific point in the log must be considered.

Event 

<14>May 6 15:51:04 1,2010/05/06 15:51:04,0006C101167,TRAFFIC,start,1,2010/05/06 15:50:58,192.168.28.21,172.16.255.78,::172.16.255.78,172.16.255.78,rule3,,,icmp,vsys1,untrust,untrust,ethernet1/1,ethernet1/1,syslog-172.16.20.152,2010/05/06 15:51:04,600,2,0,0,0,0,0x40,icmp,allow,196,196,196,2,2010/05/06 15:50:58,0,any,0
<eventFormatRecognizer><!\[CDATA\[<:gPatTime>,\\w+,(?:TRAFFIC|THREAT|CONFIG|SYSTEM)\]\]></eventFormatRecognizer>

Pattern Definition Specification

In this section of the parser XML specification, you set the regular expression patterns that that FortiSIEM will iterate through to parse the device logs.

Reusing Pattern Definitions in Multiple Parser Specifications

If you want to use a pattern definition in multiple parser specifications, you must define it in the GeneralPatternDefinitions.xml file. The patterns in the file must have a g prefix, and can be referenced as shown in this example:

<generalPatternDefinitions>
<pattern name="gPatSyslogPRI"><!\[CDATA\[<\\d+>\]\]></pattern>
  <pattern name="gPatMesgBody"><!\[CDATA\[.*\]\]></pattern>
  <pattern name="gPatMonNum"><!\[CDATA\[\\d{1,2}\]\]></pattern>
  <pattern name="gPatDay"><!\[CDATA\[\\d{1,2}\]\]></pattern>
  <pattern name="gPatTime"><!\[CDATA\[\\d{1,2}:\\d{1,2}:\\d{1,2}\]\]></pattern>
  <pattern name="gPatYear"><!\[CDATA\[\\d{2,4}\]\]></pattern>
</generalPatternDefinitions>

Each pattern has a name and the regular expression pattern within the CDATA section. This the basic syntax:

<pattern name="patternName"><!\[CDATA\[pattern\]\]></pattern>

This is an example of a pattern definition:

<patternDefinitions>
  <pattern name="patIpV4Dot"><!\[CDATA\[\\d{1,3}.\\d{1,3}.\\d{1,3}.\\d{1,3}\]\]></pattern>
  <pattern name="patComm"><!\[CDATA\[\[^,\]+\]\]></pattern>
  <pattern name="patUpDown"><!\[CDATA\[up|down\]\]></pattern>
  <pattern name="patStrEndColon"><!\[CDATA\[\[^:\]*\]\]></pattern>
</patternDefinitions>

You can also write a long pattern definition in multiple lines and indicate their order as shown in this example. The value of the list attribute should be begin in first line and end in last line. If there are more than two lines, the attribute should be set to continue for the other lines.

<pattern name="patSolarisMod" list="begin"><!\[CDATA\[sshd|login|\]\]></pattern>
<pattern name="patSolarisMod" list="continue"><!\[CDATA\[inetd|lpstat|\]\]></pattern>
<pattern name="patSolarisMod" list="end"><!\[CDATA\[su|sudo\]\]></pattern>

Parsing Instructions Specification

This section is the heart of the parser, which attempts to recognize patterns in a log message and populate parsed event attributes.

In most cases, parsing involves applying a regular expression to the log, picking up values, and setting them to event attributes. Sometimes the processing is more involved, for example when attributes must be stored as local variables and compared before populating the event attributes. There are three key components that are used in parsing instructions: Event attributes and variables, inbuilt functions that perform operations on event attributes and variables, and switch and choose branching constructs for logical operations. Values can be collected from both unstructured and structured strings in log messages.

Event Attributes and Variables

The dictionary of event attributes are defined in FortiSIEM database and any member not belonging to that list is considered a local variable. For readability, local variables should begin with an underscore (_), although this is not enforced.

Setting an Event Attribute to a Constant
<setEventAttribute attr="eventSeverity">1</setEventAttribute>
Setting an Event Attribute from Another Variable

The $ symbol is used to specify the content of a variable. In the example below, attribute hostMACAddr gets the value stored in the local variable _mac.

<setEventAttribute attr="hostMACAddr">$_mac</setEventAttribute>

Inbuilt Functions

Combining Two or More Strings to Produce a Final String

Use the combineMsgId function to do this. Here _evIdPrefix is the prefix, _evIdSuffix is the suffix, and the output will be string1-_evIdPrefix-_evIdSuffix.

<setEventAttribute attr="eventType">combineMsgId("string1", $_evIdPrefix, "-", $_evIdSuffix)</setEventAttribute>

Strings can only be wrapped by double quotes " but not single quotes '.

Normalize MAC Address

Use the normalizeMAC function to do this. The output will be six groups of two nibbles separated by a colon, for example AA:BB:CC:DD:EE:FF.

<setEventAttribute attr="hostMACAddr">normalizeMAC($_mac)</setEventAttribute>
Compare Interface Security Level

Use the compIntfSecVal function to do this. This primarily applies to Cisco ASA and PIX firewalls. The results returned are:

  • LESS if srcIntf has strictly lower security level than destIntf

  • GREATER if srcIntf has strictly higher security level than destIntf

  • EQUAL if srcIntf and destIntf have identical security levels

<setEventAttribute attr="_result">compIntfSecVal($srcIntf, $destIntf)</setEventAttribute>
Convert Hex Number to Decimal Number

Use the convertHexStrToInt function to do this.

<setEventAttribute attr="ipConnId">convertHexStrToInt($_ipConnId)</setEventAttribute>
Convert TCP/UDP Protocol String to Port Number

Use the convertStrToIntIpPort function to do this.

<setEventAttribute attr="destIpPort">convertStrToIntIpPort($_dport)</setEventAttribute>
Convert Protocol String to Number

Use the convertStrToIntIpProto function to do this.

<setEventAttribute attr="ipProto">convertStrToIntIpProto($_proStr)</setEventAttribute>
Convert Decimal IP to String

Use the converIpDecimalToStr function to do this.

<setEventAttribute attr="srcIpAddr">convertIpDecimalToStr($_srcIpAddr)</setEventAttribute>
Convert Host Name to IP

Use the convertHostNameToIp function to do this.

<setEventAttribute attr="srcIpAddr">convertHostNameToIp($_saddr)</setEventAttribute>
Add Two Numbers

Use the add function to do this.

<setEventAttribute attr="totBytes">add($sentBytes, $recvBytes)</setEventAttribute>
Divide Two Numbers

Use the divide function to do this.

<setEventAttribute attr="memUtil">divide($\_usedMem, $\_totalMem)</setEventAttribute>
Scale Function

Use the scale function to do this.

<setEventAttribute attr="durationMSec">scale($_durationSec, 1000)</setEventAttribute>
Calculate Micro Seconds

Use the calculateMSec function to do this.

<setEventAttribute attr="durationMSec">calculateMSec($_duration)</setEventAttribute>

_duration: 00:00:15
durationMSec: 15000
Extract Host from Fully Qualified Domain Name

Use the extractHostFromFQDN function to do this. If _fqdn contains a period (.) , get the string before the first period. If it does not contain a period, get the entire string.

<setEventAttribute attr="hostName">extractHostFromFQDN($_fqdn)</setEventAttribute>
Replace a String Using a Regular Expression

Use the replaceStringByRegex function to do this.

<setEventAttribute attr="computer">replaceStrInStr($_computer, "\\\", "")</setEventAttribute>
Replace String in String

Use the replaceStrInStr function to do this.

<setEventAttribute attr="computer">replaceStrInStr($_computer, "\\\", "")</setEventAttribute>
Resolve DNS Name

Use the resolveDNSName function to do this. This function converts the DNS name to an IP address.

<setEventAttribute attr="destIpAddr">resolveDNSName($destName)</setEventAttribute>
Shift Time Seconds

Use the shiftTimeSec function to do this.

<setEventAttribute attr="logonTime">shiftTimeSec($_mon, $_day, $_year, $_time, $_durationSec)</setEventAttribute>

_mon: 1
_day: 1
_year: 2000
_time: 01:00:10
_durationSec: 10
logonTime: 01:00:00 01/01/2000
Convert to UNIX Time

Use the toDateTime function to do this.

<setEventAttribute attr="deviceTime">toDateTime($_mon, $_day, $_year, $_time)</setEventAttribute><setEventAttribute attr="deviceTime">toDateTime($_mon, $_day, $_time)</setEventAttribute>
Trim Attribute

Use the trimAttribute function to do this. In this example, it is used to trim the leading and trailing dots in destName.

<setEventAttribute attr="destName">trimAttribute($destName, ".")</setEventAttribute>
Get Severity from Syslog Priority

Use the getEventSeverityFromSyslogPriority function to do this.

Set severity by syslog priority. The bottom 3 bits of the priority indicates the severity. Refer to

https://en.wikipedia.org/wiki/Syslog#Severity_level 

<setEventAttribute attr="eventSeverity">getEventSeverityFromSyslogPriority($_pri)</setEventAttribute>
_pri: 52
eventSeverity: 5
Convert to UNIX Time (with Timezone)

Use the toUnixTime function to do this.

<setEventAttribute attr="deviceTime">toUnixTime($_deviceTime)</setEventAttribute>
_deviceTime: 20130509073221.932817-000
Decode Base64

Use the decodeBase64 function to do this.

<setEventAttribute attr="httpFullRequest">decodeBase64($_msg)</setEventAttribute>
Calculate Latency

Use the calculateLatency function to do this.

Calculate the latency. If _evtRecvTime is later than deviceTime, return the latency in seconds. Otherwise, return 0.

<setEventAttribute attr="_latency">calculateLatency($_evtRecvTime, $deviceTime)</setEventAttribute>
Decode URL

Use the URLDecode function to do this.

<setEventAttribute attr="infoURL">URLDecode($_url)</setEventAttribute>

Branching Constructs

 

  • Choose
    The format is:

    <choose>
  <when test="$AttributeOrVariable1 operator Value1">
    ...
  </when>
  <when test="$AttributeOrVariable2 operator Value2">
    ...
  </when>
  <otherwise>
    ...
  </otherwise>
</choose>

  • Switch
    The format is:


    <switch>
  <case>
    ...
  </case>
  <case>
    ...
  </case>
</switch>

Collecting Values from using prebuilt functions for Structured and Unstructured Logs

Summary: Functions used to simplify data extraction from certain sections of a log event. The usual first step is to separate the log header from the log message body. Identify the event type (usually by message ID if possible), and parse specific attributes based on event type.

Collecting Fields from Structured Strings

Summary: Logs that contain a structured mapping / format such as the below.

There are usually two types of structured strings in device logs:

Common parse methods:

collectAndSetAttrByByJSON

collectAndSetAttrByByKeyValuePair

Collecting Fields from Unstructured Strings

Summary: Logs that contain variable / non consistent formatting of data structure depending on log type. Use a combination of regex parsing, <switch><case></case></switch> or <choose><when></when></choose> statements to break down and parse logs based on a particular format.

Example vendors with unstructured logs: Cisco ASA / Firepower

Common parse methods:

collectAndSetAttrByRegex - Evaluates and maps match groups to variables based on regex inserted given an argument containing a string log message.

Function List:
collectAndSetAttrByJSON

Summary: Used to extract key value pairs from a json variable, in our example $_body is the variable containing a json object.

Note one example to access sub elements where a json key value contains a json array of objects.

<collectAndSetAttrByJSON src="$_body">
  <attrKeyMap attr="domain" key="domain"/>
  <attrKeyMap attr="_eventTime" key="ts"/>
  <attrKeyMap attr="ipConnId" key="uid"/>
  <attrKeyMap attr="hostIpAddr" key="assigned_ip"/>
  <attrKeyMap attr="durationMSec" key="lease_time"/>
  <attrKeyMap attr="seqNum" key="trans_id"/>
  <!-- access json key network_addresses that contains a json array, access first element's ip key, return value -->
  <attrKeyMap attr="hostIpAddr" key="network_addresses[0].ip"/>
</collectAndSetAttrByJSON>
collectAndSetAttrByJsonArray

Summary: Another method to gather data from JSON Arrays, iterate through an array of objects. If an object key matches one type, gather the value of another key.

Example: You have a json array where the key Type can be one of 3 values. Only map attribute x if Type=someValue

Sample event parsable by this function:

"Resources":[{"Type":"AwsAccount","Id":"AWS::::Account:600000000000","Partition":"aws","Region":"us-west-2"}],"Compliance":{"Status":"WARNING"},"WorkflowState":"NEW","RecordState":"ACTIVE"}]]

<collectAndSetAttrByJsonArray src="$_resource" sep=" ">
  <attrKeyMap attr="ec2InstanceId" key="entries.find(Type='AwsEc2Instance', Id)"/>
  <attrKeyMap attr="_ec2IP" key="entries.find(Type='AwsEc2Instance', Details.AwsEc2Instance.IpV4Addresses[0])"/>
  <attrKeyMap attr="user" key="entries.find(Type='AwsIamAccessKey', Details.AwsIamAccessKey.UserName)"/>
</collectAndSetAttrByJsonArray>
collectAndSetAttrByJsonSymbol

Summary: Seen only in GenericJSONParser so far, I believe the purpose of this was to auto map the keys of a json object into temp variables.

<collectAndSetAttrByJsonSymbol src="$_rawmsg">
  <!-- Auto maps to key into a tmp var? -->
</collectAndSetAttrByJsonSymbol>
collectAndSetAttrByKeyValuePair

Certain logs, such as SNMP traps, are structured as Key1 = value1 <separator> Key2 = value2,.... These can be parsed using the collectAndSetAttrByKeyValuePair XML attribute tag with this syntax.

<collectAndSetAttrByKeyValuePair sep="separatorString"src="$inputString">
  <attrKeyMap attr="variableOrEventAttribute1" key="key1"/>
  <attrKeyMap attr="variableOrEventAttribute2" key="key2"/>
</collectAndSetAttrByKeyValuePair>

When a key1 match is found, the entire string following key1 up to the separatorString is parsed out and stored in the attribute variableOrEventAttribute1.

For example, consider this log fragment:

\_body =
SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.60 = Hex-STRING: 07 D8 06 0B 13 15 00 00 2D 07 00	SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.11.0 = Hex-STRING: 00 16 B6 DB 12 22	SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.12.0 = Hex-STRING: 00 21 55 4D 66 B0	SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.13.0 = INTEGER: 36	SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.1.0 = Hex-STRING: 00 1A 1E C0 60 7A	SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.56.0 = INTEGER: 2	SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.17.0 = STRING: "00:1a:1e:c0:60:7a"

The corresponding parser fragment is:

<collectAndSetAttrByKeyValuePair sep="\\t\\\| SNMP" src="$_body">
  <attrKeyMap attr="srcMACAddr" key="SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.11.0 = Hex-STRING: "/>
  <attrKeyMap attr="_destMACAddr" key="SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.12.0 = Hex-STRING: "/>
  <attrKeyMap attr="wlanSSID" key="SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.6.0 = STRING: "/>
  <attrKeyMap attr="wlanRadioId" key="SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.56.0 = INTEGER: "/>
  <attrKeyMap attr="apMac" key="SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.17.0 = STRING: "/>
</collectAndSetAttrByKeyValuePair>

After parsing, the attribute values are set:

Value

Attribute
00 16 B6 DB 12 22 srcMACAddr
00 21 55 4D 66 B0 destMacAddr
2 wlanRadioId
00:1a:1e:c0:60:7a apMac
collectAndSetAttrByKeyValuePairMultiValue

Summary: Seen only in CiscoACSParserPlus.xml so far, designed when a key is repeated multiple times, each with distinct values. Notice Step= is repeated many times.

Each value should be concatenated into a parsable var. --untested

 <181>May 16 08:18:13 dotacs12 CSCOacs_Passed_Authentications 0001575987 3 0 2012-05-16 08:18:13.572 -05:00 0025628800 5201 NOTICE Passed-Authentication: Authentication succeeded, ACSVersion=acs-5.3.0.40-B.839, ConfigVersionId=21, Device IP Address=10.15.1.248, UserName=abc, Protocol=Tacacs, RequestLatency=13, NetworkDeviceName=Default Network Device, Type=Authentication, Action=Login, Privilege-Level=1, Authen-Type=ASCII, Service=Login, User=joeUser, Port=tty1, Remote-Address=10.16.13.251, UserName=joeUser, AcsSessionID=dotacs12/126121712/1427032, AuthenticationIdentityStore=Internal Users, AuthenticationMethod=PAP_ASCII, SelectedAccessService=TACACS Administration, SelectedShellProfile=NetworkAdmins, IdentityGroup=IdentityGroup:All Groups:Network Administrators, Step=13020 , Step=13013 , Step=15008 , Step=15004 , Step=15012 , Step=15041 , Step=15004 , Step=15013 , Step=24210 , Step=24212 , Step=13045 , Step=13015 , Step=13020 , Step=13014 , Step=15037 , Step=15041 , Step=15004 , Step=15013 ,

Example:

<collectAndSetAttrByKeyValuePairMultiValue src="$_body" sep=",">
  <attrKeyMap attr="_step" key="Step="/>
  <attrKeyMap attr="_deviceadmin" key="Device-Administration: "/>
</collectAndSetAttrByKeyValuePairMultiValue>
collectAndSetAttrByPos

<a id="Value"></a>Value List Structured Data

Certain application logs, such as those from Microsoft IIS, are structured as a list of values with a separator. These can be parsed using the collectAndSetAttrByPos XML attribute tag following this syntax.

<collectAndSetAttrByPos sep="separatorString" src="$inputString">
  <attrPosMap attr="variableOrEventAttribute1" pos="offset1"/>
  <attrPosMap attr="variableOrEventAttribute2" pos="offset2"/>
</collectAndSetAttrByPos>

When the position offset1 is encountered, the subsequent values up to the separatorString is stored in variableOrEventAttribute1.

For example, consider this log fragment:

\_body =
W3SVC1 ADS-PRI 192.168.0.10 GET /Document/ACE/index.htm - 80 -
192.168.20.55 HTTP/1.1
Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+en-US;+rv:1.8.1.11)+Gecko/20071
127+Firefox/2.0.0.11 \[http://wwwin/Document/\] wwwin 200 0 0 5750 445 15

The parser fragment is:

<collectAndSetAttrByPos src="$_body" sep="  ">
  <attrPosMap attr="srvInstName" pos="1"/>
  <attrPosMap attr="destName" pos="2"/>
  <attrPosMap attr="relayDevIpAddr" pos="2">
  <attrPosMap attr="destIpAddr" pos="3"/>
  <attrPosMap attr="httpMethod" pos="4"/>
  <attrPosMap attr="uriStem" pos="5"/>
  <attrPosMap attr="uriQuery" pos="6"/>
  <attrPosMap attr="destIpPort" pos="7"/>
  <attrPosMap attr="user" pos="8"/>
  <attrPosMap attr="srcIpAddr" pos="9"/>
  <attrPosMap attr="httpVersion" pos="10"/>
  <attrPosMap attr="httpUserAgent" pos="11"/>
  <attrPosMap attr="httpReferrer" pos="13"/>
  <attrPosMap attr="httpStatusCode" pos="15"/>
  <attrPosMap attr="httpSubStatusCode" pos="16"/>
  <attrPosMap attr="httpWin32Status" pos="17"/>
  <attrPosMap attr="recvBytes" pos="18"/>
  <attrPosMap attr="sentBytes" pos="19"/>
  <attrPosMap attr="durationMSec" pos="20"/>
 </collectAndSetAttrByPos>

For structured strings, techniques in this section are more efficient than in the previous section because the expression is simpler and ONE tag can be used to parse regardless of the order in which the keys or values appear in the string.

collectAndSetAttrByPosWithNestedSep

Summary: Some events will be position separated, and have position separators, or nested separators. In logs that have space separators, but the values themselves contain spaces, they use nested delimiters to treat the value of each position as a literal.

Example Log:

<166>Sep 25 17:39:43 hog (squid-1): 192.168.0.171 33763 example.net 192.168.0.86 3128 204 - - - - [25/Sep/2015:17:39:43 +0100] GET "http://example.net/ping?" HTTP/1.1 200 356 921 "http://example.com/news/england" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36" TCP_MISS:HIER_DIRECT

In the above log, values are space separated, but use quotes to signify the start and end of the value of that position. User Agent is position 9, contains several spaces, which is okay since they are in the nested separator values "user agent data". L2Sep also takes a comma separated list of multiple separator types.

Position 1 (Device Time), uses the option for [] for inner separator

Position 9 (User Agent), uses the option for "" for inner separator

<collectAndSetAttrByPosWithNestedSep src="$_body" L1Sep=" " L2Sep="&quot;&quot;, []">
  <attrPosMap attr="_devTime" pos="1"/>
  <attrPosMap attr="httpMethod" pos="2"/>
  <attrPosMap attr="uriStem" pos="3"/>
  <attrPosMap attr="httpVersion" pos="4"/>
  <attrPosMap attr="httpStatusCode" pos="5"/>
  <attrPosMap attr="recvBytes64" pos="6"/>
  <attrPosMap attr="sentBytes64" pos="7"/>
  <attrPosMap attr="httpReferrer" pos="8"/>
  <attrPosMap attr="httpUserAgent" pos="9"/>
</collectAndSetAttrByPosWithNestedSep>
collectAndSetAttrByPosWithQuotes

Summary: Used to specify an inner separator in addition to outer separator by position. The difference between this and collectAndSetAttrByPosWithNestedSep is that Nested Separator can supply a list of inner separators.

collectAndSetAttrByPosWithNestedSep src="$_body" L1Sep=" " L2Sep="&quot;&quot;,[]" - This allows key="value" or key=[value].

collectAndSetAttrByPosWithQuotes - This can only provide a single argument for the inner separator.

Seen in JuniperSteelBeltAAAParser.xml and a few others.

Example log: CSV separated, with quotes for nested separator

<45>Jul  9 03:20:30 example.com SteelBeltedLog      0       "2008-07-09","03:20:26","SJ-QA-A-CAT-COR","AbcHacker","NT Domain User","User name or credential incorrect","","172.16.10.1"
<collectAndSetAttrByPosWithQuotes src="$_body" sep="," quo="&quot;">
  <attrPosMap attr="_nasNameOrIp" pos="3"/>
  <attrPosMap attr="_user1OrPort" pos="4"/>
  <attrPosMap attr="_user2" pos="5"/>
  <attrPosMap attr="_reasonOrNasIp" pos="6"/>
  <attrPosMap attr="_reasonOrOwnIp" pos="7"/>
  <attrPosMap attr="_someIp" pos="8"/>
</collectAndSetAttrByPosWithQuotes>
collectAndSetAttrByRegex

From a string input source, a regex match is applied and variables are set. The variables can be event attributes or local variables. The input will be a local variable or the default raw message variable. The syntax is:

<collectAndSetAttrByRegex src="$inputString">
  <regex><!\[CDATA\[regexpattern\]\]></regex>
</collectAndSetAttrByRegex>

The regexpattern is specified by a list of variables and sub-patterns embedded within a larger pattern. Each variable and sub-pattern pair are enclosed within angle brackets (<>).

Consider an example in which the local variable _body is set to list 130 permitted eigrp 172.16.34.4(Serial1 ) > 172.16.34.3, 1 packet. From this string we must set the values to local variables and event attributes.

Value

Set To

Type
130 _aclName Local Variable
permitted _action Local Variable
eigrp _proto Local Variable
172.16.34.4 srcIpAddr Event Attribute
Serial1 srcIntfName Event Attribute
172.16.34.3 destIpAddr Event Attribute
1 totPkts Event Attribute

This is achieved by using this XML. Note that you can use both the collectAndSetAttrByRegex and collectFieldsByRegex functions to collect values from fields.

<collectAndSetAttrByRegex src="$_body">
  <regex><!\[CDATA\[list <\_aclName:gPatStr> <\_action:gPatWord> <_proto:gPatWord> <srcIpAddr:gPatIpV4Dot>(<:srcIntfName:gPatWord>) -> <destIpAddr:gPatIpV4Dot>, <totPkts:gPatInt> <:gPatMesgBody>\]\]></regex>
</collectAndSetAttrByRegex>
collectAndSetAttrBySymbol

Summary: Automatically maps The key between symStart and symEnd as a variable that contains the value. Primarily used with log formats built by phAgentManager pollers, which have key values

that match valid programmatic names of event attributes in FortiSIEM, e.g. sentBytes64 which is the correct FortiSIEM event attribute for Sent Bytes uint64.

Sample Event:

Tue Sep 19 18:00:06 2017 AWS_VPC_FLOW_ACCEPT [accountName]=abc@cda.com,[awsRegion]=us-east-2,[groupName]=logGroupName,[streamName]=eni-1780864b-all,[version]=2,[accountId]=658308615768,[srcIntfName]=eni-1780864b,[srcIpPort]=22,[destIpPort]=16931,[ipProto]=6,[sentPkts64]=13,[sentBytes64]=3171,[sentPktsReverse]=14,[sentBytesReverse]=1268,[startTime]=1505808418,[endTime]=1505808460,[status]=OK,[srcAction]=ACCEPT,[destAction]=ACCEPT,[srcIpAddr]=10.0.0.244,[destIpAddr]=10.112.150.78

Example: Map [attribName]=Value to variable attribName for each, separated by ',['

<collectAndSetAttrBySymbol src="$_body" sep=",[" symStart="[" symEnd="]=">
  <excludeAttr>phLogDetail</excludeAttr>
</collectAndSetAttrBySymbol>

Resulting variables:

ipProto == 6

awsRegion == us-east-2

collectAndSetAttrByXPath

Summary: Uses XML XPath notation to place the value of a given XML tag into a variable.

Sample Event:

<13>Nov 09 00:55:09 172.30.58.88 <!-- PHBOX RULE ENGINE --><event name="phRuleIncident"><deviceTime>1409271060</deviceTime><firstSeenTime>1409271060</firstSeenTime><count>1</count><durationMSec>900000</durationMSec><ruleId>1491921</ruleId><ruleName>Server Hardware Critical</ruleName><ruleDescription>Detects a critical server hardware alert.</ruleDescription><eventType>PH_RULE_SERVER_HW_CRITICAL</eventType><eventSeverity>9</eventSeverity><eventSeverityCat>HIGH</eventSeverityCat><phEventCategory>1</phEventCategory><phCustId>1</phCustId><incidentSrc></incidentSrc><incidentTarget>hostName:Host-172.16.22.120, hostIpAddr:172.16.22.120, </incidentTarget><hostIpAddr>172.16.22.120</hostIpAddr><hostName>Host-172.16.22.120</hostName><hwComponentName>RAID 0 vol2 Logical Volume 1 on controller 0- Drives(1e32-?)  - OFFLINE</hwComponentName><hwComponentStatus></hwComponentStatus><incidentDetail>hwComponentName:RAID 0 vol2 Logical Volume 1 on controller 0- Drives(1e32-?)  - OFFLINE, hwComponentStatus:, </incidentDetail><incidentRptIp>172.16.22.120</incidentRptIp><triggerEventLists><triggerEvents subpatName="HwIssueCrit">8264949741156592346</triggerEvents></triggerEventLists><incidentId>14</incidentId></event>

Example: place nested xml value of the event tag into the $_body variable.

<collectAndSetAttrByXPath src="$_body" xpath="/event/*"/>

Documentation: XML path expressions

https://www.w3schools.com/xml/xml_xpath.asp

collectAndSetAttrFromAnotherEvent

Summary: Allows for mapping of correlated events given some variable in each event matches. Example, if two authentication events occur in a chain, and contain a logonID, you can have the SIEM search for the prior event, and retrieve a given attribute from that event to copy into the one you are parsing. This is used for advanced intelligence where a later log event does not contain a needed attribute. Assuming the events have an attribute linking them, e.g. the same user causing the generated audit events.

Sample Log:

<13>Dec 12 10:09:00 ADS-Pri.example.com MSWinEventLog    1       Security        1756    Wed Dec 12 10:08:53 2007        517     Security        SYSTEM  User    Success Audit   ADS-PRI The audit log was cleared               Client User Name: joeUser   Client Domain: ABC          Client Logon ID: (0x0, 0x158E87)

Example: Seen in Windows Event 517 - Audit log cleared. Copy the source IP from Windows event 540 or 528 if the logonIDs match, as event 517 itself does not contain a source IP attribute.

<collectAndSetAttrFromAnotherEvent AnotherEventType="Win-Security-540 OR Win-Security-528">
  <when test="$winLogonId = $AnotherEventType.winLogonId">
    <setEventAttribute attr="srcIpAddr">$AnotherEventType.srcIpAddr</setEventAttribute>
  </when>
</collectAndSetAttrFromAnotherEvent>
collectFieldsByCsvFile

Summary: Allows for a search of a key,value CSV file that replaces the target variable with the value of the key in the CSV file. You can specify which column you want to map.

Example CSV for Windows Logon Failure Codes: /opt/phoenix/data-definition/eventAttrDesc/winLogonFailCode2.csv

0XC000005E,Login failed - There are currently no logon servers available to service the logon request.

0XC0000064,Login failed - User logon with misspelled or bad user account

0XC000006A,Login failed - User logon with misspelled or bad password

0XC000006D,Login failed - This is either due to a bad username or authentication information

0XC000006E,Login failed - Unknown user name or bad password.

 

Structure of CSV File:

col0,col1

Logon Code,Logon Code Description

Example: Take the upper case arg variable $subStatus, if it matches one of the CSV lines, map column 1 to variable description.

<when test="exist subStatus">
  <setEventAttribute attr="_subStatus">toUpper($subStatus)</setEventAttribute>
  <collectFieldsByCsvFile file="/opt/phoenix/data-definition/eventAttrDesc/winLogonFailCode2.csv" key="$_subStatus" reloadInterval="3600">
    <attrKeyMap attr="description" column="1"/>
  </collectFieldsByCsvFile>
</when>
collectFieldsByKeyValuePair

Summary: Near duplicate of collectAndSetAttrByKeyValuePair, allows for kvsep so you don't have to include the key value separator in the key mapping.

Sample Event:

<134>Jul 24 2008 03:29:15: %ASA-6-113005: AAA user authentication Rejected : reason = AAA failure : server = 192.168.0.40 : user = joeUser

Example: Map body values separated by ' : ' and key value separator of ' = '

<collectAndSetAttrByRegex src="$_body">
  <regex><![CDATA[AAA user authentication Rejected :\s*<_detail:gPatMesgBody>]]></regex>
</collectAndSetAttrByRegex>

<collectFieldsByKeyValuePair sep=" : " kvsep=" = " src="$_detail">
  <attrKeyMap attr="user" key="user"/>
  <attrKeyMap attr="srcIpAddr" key="user IP"/>
</collectFieldsByKeyValuePair>
collectFieldsByRegex

Summary: Seems to be an identical construct to: collectAndSetAttrByRegex

Example: Map each key value pair of <someVariable:someRegexMatchGroup> evaluating regex from left to right of the variable $_rawmsg

<collectFieldsByRegex src="$_rawmsg">
  <regex><![CDATA[^<_header:gPatMesgBodyMin>%<_vendor:gPatWord>-<_sev:gPatInt>-<_evtId:gPatInt>:\s+<_body:gPatMesgBody>]]></regex>
</collectFieldsByRegex>
collectFieldsBySNMPTrap

Summary: Currently only seen in FireEyeTrapParser.xml, take an SNMP trap log message, map oid defined under key= to a given FortiSIEM event attribute.

Sample Log: Shortened for brevity

2016-05-26 07:50:28 0.0.0.0TRAP2, SNMP v2c, community R-OEnWinLog$            . Cold Start Trap (0) Uptime: 0:00:00.00            DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (610853754) 70 days, 16:48:57.54  SNMPv2-MIB::snmpTrapOID.0 = OID: SNMPv2-SMI::enterprises.25597.3.0.1            SNMPv2-SMI::enterprises.25597.1.1.2.1.2.1116 = Gauge32: 1116  SNMPv2-SMI::enterprises.25597.1.1.2.1.3.1116 = STRING: "malware-callback"         SNMPv2-SMI::enterprises.25597.1.1.2.1.4.1116 = STRING: "2016-05-26" SNMPv2-SMI::enterprises.25597.1.1.2.1.5.1116 = STRING: "11:46:48+00"  SNMPv2-SMI::enterprises.25597.1.1.2.1.6.1116 = Counter64: 0      SNMPv2-SMI::enterprises.25597.1.1.2.1.7.1116 = IpAddress: 10.1.201.82               SNMPv2-SMI::enterprises.25597.1.1.2.1.8.1116 = IpAddress: 1.1.1.1             SNMPv2-SMI::enterprises.25597.1.1.2.1.9.1116 = STRING: "70:38:ee:91:cc:80"       SNMPv2-SMI::enterprises.25597.1.1.2.1.10.1116 = STRING: "58:49:3b:2d:98:11"    SNMPv2-SMI::enterprises.25597.1.1.2.1.11.1116 = INTEGER: 80    SNMPv2-SMI::enterprises.25597.1.1.2.1.12.1116 = INTEGER: 0                SNMPv2-SMI::enterprises.25597.1.1.2.1.13.1116 = STRING: "tcp"

Example:

<collectFieldsBySNMPTrap src="$_body">
  <attrKeyMap attr="_id" key="SNMPv2-MIB::snmpTrapOID"/>
  <attrKeyMap attr="srcMACAddr" key="SNMPv2-SMI::enterprises.25597.1.1.2.1.9"/>
  <attrKeyMap attr="destMACAddr" key="SNMPv2-SMI::enterprises.25597.1.1.2.1.10"/>
  <attrKeyMap attr="destIpAddr" key="SNMPv2-SMI::enterprises.25597.1.1.2.1.38"/>
</collectFieldsBySNMPTrap>