Home

> Administration > Device support > Working with Parsers

Working with Parsers

Creating a custom parser for device logs involves writing an XML specification for the parser, and using a test event to make sure the logs are parsed correctly.

Prerequisites
  • You should have examples of the logs that you want to parse.
  • You should have created any new device/application types, event attribute types, or event types that you want to use in your XML specification.
  • You should already have written the XML specification for your parser.
  • You should have prepared a test event that you can use to validate the parser.

Parsers are applied in the order they are listed in ADMIN > Device Support > Parser, so it is important to add your custom parser to the list in relation to any other parsers that may be applied to your device logs. If you click Fix Order, this will arrange the parsers with system-defined parsers at the top of the list in their original order, and user-defined parsers at the bottom. Make sure to click Apply to make sure the change in order is picked up by the back-end module.

This section provides the procedure to create Parsers.

Custom Parser XML Specification Template

The basic template for a custom parser XML specification includes five sections. Click on the name of any section for more information.

SectionDescription
Parser Name SpecificationName of the parser file
Device TypeThe type of device or application associated with the parser
Format Recognizer Specification Patterns that determine whether an event will be parsed by this parser
Pattern Definition SpecificationDefines the parsing patterns that are iterated over by the parsing instructions
Parsing Instructions SpecificationInstructions on how to parse events that match the format recognizer patterns

Custom Parser XML Specification Template

<eventParser name="xxx">   <deviceType> </deviceType>   <eventFormatRecognizer> </eventFormatRecognizer>   <patternDefinitions> </patternDefinitions>   <parsingInstructions> </parsingInstructions></eventParser>

Parser Name Specification

This section specifies the name of the parser, which is used only for readability and identifying the device type associated with the parser.

<eventParser name="CiscoIOSParser"></eventParser>

Device or Application Type Specification

This section specifies the device or the application to which this parser applies. The device and application definitions enable FortiSIEM to detect the device and application type for a host from the received events. This is called log-based discovery in FortiSIEM. Once a received event is successfully parsed by this file, a CMDB entry is created with the device and application set from this file. FortiSIEM discovery may further refine the device.

There are two separate subsections for device and application. In each section, vendor, model and version can be specified, but version is not typically needed. 

Set Version to Any. In the examples, <Version> is set to ANY because events are generally not tied to a particular version of a device or software. You could set this to a specific version number if you only wanted this parser to apply to a specific version of an application or device.

Note: Vendor and Model Must Match the FortiSIEM Version - <Vendor> and <Model> entries must match the spelling and capitalization in the CMDB.

Examples of Specifications for Types of Device and Applications

Hardware Appliances

In this  case, the type of event being parsed specifies the device type, for example Cisco IOS, Cisco ASA, etc.

<deviceType>    <Vendor>Cisco</Vendor>    <Model>IOS</Model>    <Version>ANY</Version></deviceType>

Software Operating Systems that Specify the Device Type

In this case, the type of events being parsed specifies the device type, for example Microsoft Windows etc. In this case the device type section looks like

<deviceType>    <Vendor>Microsoft</Vendor>    <Model>Windows</Model>    <Version>ANY</Version></deviceType>

Applications that Specify Both Device Type and Application

In this case, the  events being parsed specify the device and application types because Microsoft SQL Server can only run on Microsoft Windows OS.

<deviceType>    <Vendor>Microsoft</Vendor>    <Model>Windows</Model>    <Version>ANY</Version></deviceType><appType>    <Vendor>Microsoft</Vendor>    <Model>SQL Server</Model>    <Version>ANY</Version>    <Name> Microsoft SQL Server</Name></appType>

Applications that Specify the Application Type but Not the Device Type

Consider the example of an Oracle database server, which can run on both Windows and Linux operating systems. In this case, the device type is set to Generic but the application is specific. FortiSIEM depends on discovery to identify the device type.

<deviceType>    <Vendor>Generic</Vendor>    <Model>Generic</Model>    <Version>ANY</Version></deviceType><appType>    <Vendor>Oracle</Vendor>    <Model>Database Server</Model>    <Version>ANY</Version>    <Name>Oracle Database Server</Name></appType>

Format Recognizer Specification

In many cases, events associated with a device or application will contain a unique pattern. You can enter a regular expression in the Format Recognizer section of the parser XML file to search for this pattern, which, if found, will then parse the events according to the parser instructions. After the first match, the event source IP to parser file map is cached, and only that parser file is used for all events from that source IP. A notable exception is when events from disparate sources are received via a syslog server, but that case is handled differently.

While not a required part of the parser specification, a format recognizer can speed up event parsing, especially when one parsing pattern file among many pattern files must be chosen. Only one pattern check can determine whether the parsing file must be used or not. The other less efficient option would be to examine patterns in every file. At the same time, the format recognizer must be carefully chosen so that it is not so broad to misclassify events into wrong files, and at the same time, not so narrow that it fails at classifying the right file. 

Order in Which Parsers are Used

 FortiSIEM parser processes the files in the specific order listed in the file parserOrder.csv.

Format Recognizer Syntax

The specification for the format recognizer section is:

<eventFormatRecognizer><![CDATA[regexpattern]]></eventFormatRecognizer>

In the regexpattern block, a pattern can be directly specified using regex or a previously defined pattern (in the pattern definition section in this file or in the GeneralPatternDefinitions.xml file) can be referenced.

Example Format Recognizers

Cisco IOS

All Cisco IOS events have a %module name pattern.

<patternDefinitions>    <pattern name="patCiscoIOSMod" list="begin"><![CDATA[FW|SEC|SEC_LOGIN|SYS|SNMP|]]></pattern>    <pattern name="patCiscoIOSMod" list="continue"><![CDATA[LINK|SPANTREE|LINEPROTO|DTP|PARSER|]]></pattern>    <pattern name="patCiscoIOSMod" list="end"><![CDATA[CDP|DHCPD|CONTROLLER|PORT_SECURITY-SP]]></pattern></patternDefinitions><eventFormatRecognizer><![CDATA[: %<:patCiscoIOSMod>-<:gPatInt>-<:patStrEndColon>:]]></eventFormatRecognizer>
Cisco ASA

All Cisco ASA events have the pattern ASA-severity-id pattern, for example ASA-5-12345.

<eventFormatRecognizer><![CDATA[ASA-\d-\d+]]></eventFormatRecognizer>
Palo Alto Networks Log Parser

In this case, there is no unique keyword, so the entire message structure from the beginning to a specific point in the log must be considered.

Event
<14>May 6 15:51:04 1,2010/05/06 15:51:04,0006C101167,TRAFFIC,start,1,2010/05/06 15:50:58,192.168.28.21,172.16.255.78,::172.16.255.78,172.16.255.78,rule3,,,icmp,vsys1,untrust,untrust,ethernet1/1,ethernet1/1,syslog-172.16.20.152,2010/05/06 15:51:04,600,2,0,0,0,0,0x40,icmp,allow,196,196,196,2,2010/05/06 15:50:58,0,any,0

<eventFormatRecognizer><![CDATA[<:gPatTime>,\w+,(?:TRAFFIC|THREAT|CONFIG|SYSTEM)]]></eventFormatRecognizer>

Adding a Parser

Follow the procedure below to add a Parser:

  1. Go to ADMIN > Device Support > Parser.
  2. Select a parser that is above the location in the list where you want to add your parser, and click New.
  3. Enter a Name for the parser. 
  4. Select a Device Type to which the parser should apply.
    If the device type doesn't appear in the menu, you should create a new device type.
  5. Enter a Test Event containing an example of an event that you want to use to validate the parser.
  6. Enter the Parser XML.
  7. Click Validate to validate the XML.
  8. Click Test
    This will send the test event to the parser to make sure it is parsed correctly, and will also test the parsers above and below yours in the list to make sure they continue to parse logs correctly.
  9. If the XML for your parser validates and the test event is correctly parsed, select Enabled.
    If you need to continue working on your parser, you can Save it without selecting Enabled
  10. Enter a Description about the parser.
  11. Click Save.
  12. Click Apply to have the back-end module pick up your parser and begin applying it to device logs. 
    You should now validate that events are being parsed by creating some activity that will cause a log to be generated, and then run a query against the new device IP address and validate the parsed results. 

Cloning New Parsers

You can also clone an existing parser and use it as the basis for creating a new one. Select the parser you want to clone, and click Clone. Modify the parser as required, and make sure you use the Up and Down buttons to place it in the list of parsers at the point at which should be applied. You need to disable the system-defined parser before testing the cloned parser.

Modifying a Parser

Follow the procedure below to modify a Parser:

  1. Select one or more Parsers to edit from the list.
  2. Click the required action:
    • Edit to modify a parser setting.
    • Delete to delete a parser.
  3. Click Save.