Python Threat Feed Framework

In release 7.0.0, FortiSIEM introduces a Python framework for FortiSIEM threat feed integrations.

Traditionally, customers or technical assistance center (TAC) engineers had to use a Java package framework to build threat feed integrations. However, adoption of the Java based framework was extremely low. This feature uses the Python framework to make custom threat feed integrations more accessible, with a well defined structure for data.

In this framework, FortiSIEM calls a Python module based threat feed, passing it a number of arguments necessary for communication with a threat feed. This framework then writes the threat entries to a CSV file which AppServer will parse. The framework is also extendable. If the customer knows the natural ID of the threat feed, the customer can run a script externally from FortiSIEM, providing their username and password for basic authentication to call AppServer’s API to update the threat feed.

Updating a threat feed via Python can be done in two ways.

• FortiSIEM Internal Threat Feed Update: Python integration built into FortiSIEM – Threat feed update schedule passes standard set of arguments to python script, which uses these to obtain threat feed information, and saves it to CSV file for AppServer to process.
• FortiSIEM External Threat Feed Update: Python integration is executed externally from FortiSIEM in some customer environment. Provided customer has access to FortiSIEM credential, Threat feed natural ID, Threat feed URL and credentials. Python script collects threat feed data, and does an HTTP POST to FortiSIEM to push the data to Threat feed via API.

To update a threat feed, you will need to take the following steps.

1. From the new threat feed, obtain the URL endpoint, and credentials, if applicable.
2. Create your custom Python threat feed integration by taking the following steps.
   1. Obtain a copy of the Firehol and/or Anomail (TAXII2.x) threat feed integration script.
   2. Place your Python script in the /opt/phoenix/data-definition/threatfeedIntegrations/ folder on the FortiSIEM Supervisor.
3. From the FortiSIEM Supervisor GUI, navigate to RESOURCES, and from the left pane, select Malware Domains, Malware IPs, Malware Hash, or Malware URLs.
4. In the same left pane, click + to create a new Malware Threat Feed Group.
5. From the Create New Malware Group Window, take the following steps.
   1. In the Group field, enter a name for the Malware Group threat feed.
   2. In the Description field, enter any information you wish to make available about the Malware Group threat feed.
   3. If Malware IPs was selected, from the Value Type drop-down list, select IP or IP Range.
   4. Click Save.
6. Select your malware threat feed group from the left pane, and from the main pane, click on More, and select Update.
7. From the Update Malware window, take the following steps.
   1. Select Update via API.
   2. From the URL row, click the Edit icon.
   3. In the URL field, enter the threat feed endpoint being used by your custom threat feed service.
      Note: Ensure you enter the http:// or https:// prefix.
      
   4. In the User Name field, if required, enter the user name associated with the API access.
      Note: Your script can manipulate these fields. If the threat feed requires an API key, you can simply place that in the password field, and dummy value in the username field.
      
   5. In the Password field, enter the password associated with the user name.
   6. For Plugin Type, select Python.
   7. In the Plugin Class row, click the refresh button next to plugin class to refresh the list of python integrations available. This enumerates all .py files in the /opt/phoenix/data-definition/threatfeedIntegrations/ folder.
   8. Select your Python file from the drop-down list.
      Note: You can click the refresh icon to refresh the drop-down list after copying your .py file to the /opt/phoenix/data-definition/threatfeedIntegrations/ folder.
      
   9. For Data Update, select Full or Incremental.
      

      Option	Description
      Full	Each time a scheduled trigger occurs, your threat feed data will be completely replaced with a new copy. In most cases, select “Full” to prevent accumulation of ineffective/aged threat indicators.
      Incremental	Preserves existing data, and adds to it.

   10. Click Save.
       

       
       

8. To create a schedule, see Specifying a Schedule.
   After the first schedule has been executed, confirm that the entries are populated.

Threat Feed Workflow

In this scenario, FortiSIEM’s threat feed update schedule will gather the threat feed configuration data you provided during setup, and pass those as arguments to your custom python script. If you use Fortinet's provided framework, the threat feed data can be passed to a function which will store the data in the appropriate cache folder to update FortiSIEM.

Internal/Scheduled by FortiSIEM Execution Scenario:

In this scenario, the python script, when called by FortiSIEM, obtains the threat feed data from the remote source, parses the data into a list of dictionary objects, and passes that to a framework function which will save it to a local CSV file on the FortiSIEM Supervisor.

Remember that your custom python script must exist on the Supervisor, and be placed in this folder: /opt/phoenix/data-definition/threatfeedIntegrations/

External Execution Scenario:

For Advanced Users Only: In this scenario, you can download all these python scripts to a remote location (not on FortiSIEM Supervisor), and execute the threat feed update. This however, requires that you securely store all the required arguments needed, and pass them to the script on execution. In addition, you need to provide the script 3 extra arguments (FortiSIEM base URL, a FortiSIEM user account with authorization (org/user), and FortiSIEM password)

In this scenario, the script uses the FortiSIEM public API to POST threat feed data in a defined JSON format.

Note: The naturalId of the threat feed you created is the most difficult item to obtain here, you must obtain it via Postgres in order to run it externally to FortiSIEM.

Listing available threat feeds to update:

select display_name,natural_id from ph_group where type=27;
        display_name         |             natural_id
-----------------------------+-------------------------------------
 Emerging Threat Malware IP  | PH_SYS_EMER_THREAT
 ThreatStream Malware IP     | PH_SYS_THREATSTREAM_BLOCKED_IP
 FortiGuard Malware IP       | PH_SYS_FORTIGUARD_BLOCKED_IP
 TruSTAR Malware IP          | PH_SYS_TRUSTAR_BLOCKED_IP
 ThreatConnect Malware IP    | PH_SYS_THREATCONNECT_BLOCKED_IP
 Dragos Worldview Malware IP | PH_SYS_DRAGOSWORLDVIEW_BLOCKED_IP

Threatfeed Module

The threat feed package is located here: /opt/phoenix/data-definition/threatfeedIntegrations/

Directory Structure

The Threatfeed module directory consists of the following files and folders.

Directory File/Folder	Description
anomaly_threatfeed.py	An example working threat feed file implementing TAXII2.0 and TAXII2.1 threat feed integration. This is actually implemented in the helper modules “threatfeed_integration”.
firehol_threatfeed.py	An example working threat feed file implementing Firehol IP threat feeds.
fsiem_utils/	This is the package directory for the “fsiem_utils” python package.
fsiem_utils/__init__.py	Empty, simply the designated directory for package.
fsiem_utils/threatfeed_integration.py	Helper module defining the classes representing a threat feed integration and data objects.

Overview of Threat Feed Integrations

Required import: from fsiem_utils.threatfeed_integration import

All custom threat feeds inherit from the parent class ThreatfeedIntegration using the following statement:

class MyCustomThreatFeed(ThreatfeedIntegration):

This parent class handles a lot of the internals of handling input arguments and handling format of data that FortiSIEM can understand.

Script Input Arguments

The sample threat feeds have built in handling for the expected script input, so do not deviate from them. You can merely replace your custom class name with the existing example.

• Required Input Arguments: updateType,naturalId,tfType,tfURL
• (Optional) Input Arguments: tfUser,tfPW, appUser = None, appPW = None, appHost = 'https://127.0.0.1')

Data Object Classes

For ease of formatting data to FortiSIEM expected format, Fortinet has a number of helper classes for representation of an IP object, URL object, Domain object,, or Hash object.

Data Object Class Member Functions

<data_obj>.get_dict() – Returns dictionary form of object

Each of the classes representing a threat feed object (IP_entry, URL_entry, Domain_entry, and Hash_entry) have a function called get_dict() that returns the dictionary representation of the object, which is on the format that AppServer expects. This can be dumped to JSON, which will be in AppServer’s public API json format, or it can be dumped to CSV where the field order is preserved.

Class IP_entry()

Creates an object representing a single IP and/or contiguous network range indicator

Example Definition

IP_entry(name="Test IP",low_ip="1.1.1.1",high_ip="2.2.2.2",description="This is ransomware C2 server",date_found=now)

Required Arguments

• name
• low_ip

Optional Arguments

• high_ip
• malware_type
• confidence
• severity
• asn
• org
• country
• description
• date_found – datetime object
• lastSeen – datetime object

Class Domain_entry()

Creates an object representing a single domain indicator.

Example Definition

Domain_entry(domainName="google.com",ip="9.9.9.9")

Required Arguments

• domainName

Optional Arguments

• ip
• reverseLookup
• malware_type
• confidence
• severity
• asn
• org
• country
• description
• date_found – datetime object
• lastSeen – datetime object

Class URL_entry()

Creates an object representing a single URL indicator.

Example Definition

URL_entry(url="google.com/test2.xml",origin="https://google.com",description="Test URL")

Required Arguments

• url

Optional Arguments

• origin
• malware_type
• confidence
• description
• lastSeen – datetime object

Class Hash_entry()

Creates an object representing a single hash indicator.

Example Definition

Hash_entry(name="REvil Hash1",algorithm="SHA256",hashcode="XXXXXXXXX",description="test")

Required Arguments

• names
• algorithm
• hashcode

Optional Arguments

• controller_ip
• malware_type
• confidence
• severity
• asn
• org
• country
• description
• date_found – datetime object
• lastSeen – datetime object

Important Member Variables of the ThreatfeedIntegration Class

If you properly follow the example threat feeds by inheriting from the ThreatfeedIntegration class, these are automatically populated by the scripts input arguments.

        self.updateType = updateType #full or incremental, indicates if threatfeed does a full replace or append
        self.appserverUsername = appUser #Optional if run external to FortiSIEM
        self.appserverPassword = appPW #Optional if run external to FortiSIEM
        self.threatfeed_natural_id = naturalId #Threatfeed natural ID to update
        self.threatfeed_type = tfType #Threatfeed type (ip,site,url,hash,proc)
        self.threatfeed_url = tfURL #Threatfeed source URL
        self.threatfeed_username = tfUser #Threatfeed optional user
        self.threatfeed_password = tfPW #Threatfeed optional pw
        self.appserver_host = appHost #FortiSIEM super IP if script run externally

Important Member Functions of the ThreatfeedIntegration Class

Member Function	Description
getThreatFeedData(self)	You must override this function in your script’s child class. It references the above member variables to authenticate to your given threat feed source, and calls the save function to send to FortiSIEM.
saveThreatFeedData(self,post_data)	Call this function once you have retrieved your threat feed data. It accepts a python list of dictionary objects, which represent either IP, URL, Domain, or Hash objects. You only call this function, you do not override it.

Instructions on Creating a New Python Integration

Take the following steps to create a new python integration.

1. Copy an existing example file.
   For example: 
   cp firehol_threatfeed.py my_example.py
2. Rename the class “FireHolThreatFeed” to your new class name. It is referenced in 3 locations in this file.
3. Override the getThreatFeedData function, as this is where all your work will be done. You will build a list of dictionary objects using the existing integrations as an example.
4. Call saveThreatFeedData(post_data). This will convert the list of dictionary objects into a CSV file (if run locally on Supervisor) or call AppServer save API if run externally to Supervisor. You can review the file example_threatfeed.py to see an example of how to convert data ingested from a threat feed into a representation that the framework can convert to a CSV file.

Python Script

The python script which is user defined e.g. “threatfeed_script_name.py” is called with the input arguments listed in the following table.

Argument Number	Argument	Argument Type	Description
Arg1	updateType	String	Either "full" or "incremental". If script is externally run, calls to the save API can override whether the threat feed does a full replace of data ("full") or an append ("incremental").
Arg2	naturalId	String	String literal of the threat feed (to update)s’ natural ID.
Arg3	tfType	String	Threat feed type, which will be one of the following: ip, hash, site, or url. This allows a single script to add handling for the type of threat feed if the service provides multiple types of threat feeds.
Arg4	tfURL	String	URL endpoint for the threat feed service.
Arg5	tfUser	String	(Optional) Username value for basic authentication to threat feed service.
Arg6	tfPW	String	(Optional)Password value for threat feed basic authentication. In other cases, it can be used for holding the API secret key instead. The script can manipulate these values as desired.
Arg7	appUser	String	(Optional) Username for FortiSIEM in the format (org/user) e.g. super/admin. This is only needed if the script is executed externally from FortiSIEM and not executed via AppServer schedule.
Arg8	appPW	String	(Optional) Password for FortiSIEM user. This is only needed if the script is executed externally from FortiSIEM and not executed via AppServer schedule.
Arg9	appHost	String	(Optional) If script is run externally to FortiSIEM, this can override the app server (Supervisor) URL. It normally defaults to https://127.0.0.1 as it is only executed programmatically on the FortiSIEM Supervisor.

Example Adhoc Script Run Locally on FortiSIEM

Caution: Script must only be run as user "admin", never as root.

python3 /opt/phoenix/data-definition/threatfeedIntegrations/firehol_threatfeed.py -updateType full -naturalId Malware_IPs_FireHol_Cybercrime_Threatfeed_0 -tfType ip -tfURL https://iplists.firehol.org/files/firehol_level1.netset -tfUser guest -tfPW guest

This generates a CSV file for AppServer to consume: /opt/phoenix/cache/MalwareIP/Malware_IPs_FireHol_Cybercrime_Threatfeed_0.csv

Note: When you manually run the script, you’ll notice that it doesn’t actually update in the GUI. This is because normally, FortiSIEM is the one executing the script, and will only look for this CSV file after it executes. You can manually run this to test for errors and ensure the CSV file is populated appropriately. Afterward, you can try an end to end test by configuring FortiSIEM to call your new threat feed script.

Example Adhoc Script Run Externally to FortiSIEM (e.g. remote customer machine) – This assumes you have a FortiSIEM user account and know the threat feed’s natural ID.

python3 /opt/phoenix/data-definition/threatfeedIntegrations/firehol_threatfeed.py -updateType full -naturalId Malware_IPs_FireHol_Cybercrime_Threatfeed_0 -tfType ip -tfURL https://iplists.firehol.org/files/firehol_level1.netset -tfUser guest -tfPW guest -appUser 'super/admin' -appPW ‘xxxxx’ –appHost ‘https://192.168.1.25’

The only difference here is the addition of appUser, appPW, and appHost, indicating to the script that it must call the app server API which is on a remote system using HTTP POST with basic authentication. If run locally, the script will merely generate a CSV file in the /opt/phoenix/cache directory for AppServer to consume (removing the need for app server credentials).

Save Imported Data to Database

Normally the details of the CSV file location, and its format are completely abstracted and handled by the parent class “ThreatfeedIntegration”. The content here is for informational purposes only.

• Python script has to save data in following directories depending on threat feed type.
  • /opt/phoenix/cache/MalwareDomain
  • /opt/phoenix/cache/MalwareIP
  • /opt/phoenix/cache/MalwareHash
  • /opt/phoenix/cache/MalwareUrl
• Supported file type is CSV
• File Name: <natural_id>.csv e.g. Malware_IPs_FireHol_Cybercrime_Threatfeed_0.csv
• CSV structure:
  • IPs: name, malwareType, lowIp, highIp, description, confidence ,severity, org, country, asn, dateFound, lastSeen
  • Domains: domainName, malwareType, description, ipAddr, reverseLookup, asn, confidence, severity, org, country, dateFound, lastSeen
  • Hash: botnetName, algorithm, hash, controllerIp, confidence, country, malwareType, severity, asn, org, description, dateFound, lastSeen
  • URL: url, description, malwareType, confidence, lastSeen

Performance Considerations

There are no performance concerns a this time. When selecting the threat feed update type in the GUI, in almost all cases you want to select “Full”. FortiSIEM has been enhanced to handle massive threat feed lists. Only in exceedingly rare cases would you choose update type of “Incremental”. Doing "Incremental" can create problems as threat lists quickly become exceedingly large (not truncated), and often contain stale, and non-effective indicators.

Custom python scripts can be either configured to do either of the following:

• Incremental (append only) - Updates to the threat feed, new calls are append only, and does not delete and re-enter the same data. Care should be taken to periodically do a full replace on a maintenance schedule to replace stale entries.
• Full - Updates to the threat feed, each call to the script does a complete replace of the threat feed data with the latest copy from the source threat feed source. This should be considered the default option even in really large threat feeds in the hundreds of thousands of entries.

Public APIs

These public APIs are only important if you plan to execute this script outside of FortiSIEM as part of your own custom application. Normally FortiSIEM will hold all these scripts locally and execute them on a schedule, removing the need for these public APIs.

AppServer provides the following two public REST APIs to import or delete malware data. These are only needed if the python script is executed external to FortiSIEM. This allows external systems to integrate with FortiSIEM threat feeds, as opposed to having FortiSIEM execute an update on a schedule.

• Import API
• Delete API

Import API

API Endpoint: POST phoenix/rest/malware/{type}/save

Query Parameters

1. groupNaturalId : String [*required field]
2. updateType : String [full | incremental]

Path Parameters

1. type: String [*required field]
   1. Accepted values for type - [hash, site, url, ip]

POST Body

Example for IP:

        [
        {
            "name":"RansomwareIP",
            "lowIp":"1.1.1.1",
            "highIp":"1.1.1.1"
        },
        {
            "name":"RansomwareIP2",
            "lowIp":"2.1.1.1",
            "highIp":"2.1.1.1"  
        },
        {
            "name":"RansomwareIP3",
            "lowIp":"3.1.1.1",
            "highIp":"3.1.1.1"  
        }
    ]

Example 2 for Domain:

[
{
"domainName":"",
"ip":"",
"name":""
} //This JSON Object keys are defined in ThreatFeedDTO
]

JSON Structure: Array of Objects: ThreatFeedDTO

See the following tables for their ThreatFeedDTO JSON Object Structure.

Malware Domain Threat Feed Parameter	Value Type
domainName	String
ip	String
reverseLookup	String

Malware IP Threat Feed Parameter	Value Type
name	String
lowIp	String
highIp	String

Malwre Hash Threat Feed Parameter	Value Type
botnetName	String
algorithm	String
hash	String
controllerIp	String

Malware Url Threat Feed Parameter	Value Type
url	String
origin	String

Common Malware Threat Feed Parameter	Value Type
active	Boolean
malwareType	String
confidence	String
severity	String
asn	String
org	String
country	String
description	String
dateFound	String
lastSeen	String

Delete API

API Endpoint: POST phoenix/rest/malware/threatfeed/deleteall

Query Parameter

1. groupNaturalId : String [*required field]

Return: Success | Error

Notes about Threat Feed Authentication

The python threat feed GUI still only has the ability to pass basic authentication credentials to the Python script (e.g. username and password). You can place values such as an API key in one of these fields and manipulate that data in your custom threat feed integration. So although the UI only displays an optional user/pw fields, you can still put anything you want in these fields and manipulate accordingly. Note that only the password field is encrypted, so try not to place sensitive data in the username field if overriding them for a different authentication type.

Installing New Python Packages

Using new packages not already installed on the Supervisor need to be manually installed with care. Fortinet uses the requests API which covers the most common use cases. The system python3 install already contains many useful packages such as requests and urllib3.

Here are recommended steps to help keep your system secure.

1. Follow your organization’s change control procedures
2. Backup your FortiSIEM Supervisor prior to making the change
3. Ensure you have a rollback/revert plan
4. Document the time of the change and the purpose
5. Make the change only on the Supervisor appliances