Incoming feed - MISP

This article describes how to configure incoming feeds for a particular feed source. To see how to configure incoming feeds in general, see Configure incoming feeds general options.


Specifications

Transport type

MISP API

Content type

MISP JSON

Ingested data

Ingest MISP events with attributes and MISP objects.

Processed data

Creates entities and related observables from ingested data.

Requirements

  • URL used to access your MISP instance.

  • MISP API key.

Configure the incoming feed

  1. Create or edit an incoming feed.

  2. Under Transport and content, fill out these fields:

    Required fields are marked with an asterisk (*).

    Field

    Description

    Transport type*

    Select MISP API from the drop-down menu.

    Content type*

    Select MISP JSON from the drop-down menu.

    MISP URL*

    Set this to the URL you access your MISP instance at.

    MISP key*

    Set this to your MISP API key.

    Include tags

    Select to include tags when ingesting data from the target MISP instance.

    Prioritize TLP tag

    Select to have ingested entities inherit TLP (Traffic Light Protocol) values from TLP tags set by the feed source instead of inheriting Distribution setting.

    Setting a value for the Override TLP field sets all ingested entities to that TLP value instead of inheriting the TLP value from the feed source.

    Include IDS flag as tag

    Selected by default. When selected, the platform checks if the ‘to_ids’ flag is set to ‘true’ for for incoming MISP attributes, and adds a tag named ‘IDS’ to the resulting entities.

    Reduce lock contention

    Select Reduce lock contention to speed up ingestion.

    Entities will update at random. For more information, see Reduce lock contention below.

    SSL verification

    Selected by default. Select this option to enable SSL for this feed.

    Start ingesting from*

    Ingest data from the feed source starting from this date and time. Use the drop-down calendar to select the date and time you want to start ingesting feed data from.

    End ingestion

    By default, this is set to the date and time the incoming feed is created.

    Set this to the latest date and time for the latest incident the platform should ingest from the feed source.

    SSL cert location

    Used when connecting to a feed source that uses a custom CA. Set this as the path to the SSL certificate to use when authenticating the feed source.

    For more information, see SSL certificates.

    Client cert location

    Used when connecting to a MISP instance that requires a client certificate to authenticate.

    Set the absolute path to the certificate file here. For example:

    /path/to/cert.crt

    For more information, see SSL certificates.

    Client cert location

    Used when connecting to a MISP instance that requires a client certificate to authenticate. The certificate used to authenticate may require a certificate key file.

    Set the absolute path to the key file here. For example:

    /path/to/cert.key

    For more information, see SSL certificates.

    Default request interval

    By default, this is set to 240 hours.

    The Default request interval allows you to control the amount of data each request made to the MISP instance retrieves, by limiting the events retrieved to the specified period.

    For example, setting this to 240 hours allows each request to retrieve a maximum of 10 days worth of events from the MISP instance.

    Reducing this value may help with performance issues related to receiving too many feed packages from this feed.

    Filter by MISP event info

    Enter a regular expression to exclude MISP events from ingestion if the expression matches the content of their “Event Info” field (info).

    For example, enter:

    ^(COVID19\-)

    to exclude all events with “Event Info” fields starting with “COVID19-“.

    Filter by the creating organization’s name

    Enter a regular expression to exclude MISP events that have a creating organization (orgc) that matches this expression.

    For example, enter:

    Iglocska

    to exclude all events that have creating organizations with names that contain the word “Iglocska”.

  3. Store your changes by selecting Save.

Distribution settings

MISP labels events and attributes using distribution settings, which tells us who these events and attributes can be shared with.

The platform maps these distribution settings to TLP values:

TLP value

MISP distribution setting

Not Set

No distribution setting

WHITE

All communities

GREEN

Connected communities

AMBER

This community only

RED

Your organization only

Reduce lock contention

Enable the Reduce lock contention option to speed up ingestion for the MISP feed. This is done by splitting incoming data into smaller packages that are then randomly redistributed among ingestion workers, reducing the likelihood that ingestion is stalled by having several workers attempting to update the same record.

This means that if you start working with your data before ingestion completes, you may be working with incomplete entities. If ingestion fails, run the incoming feed again and let it finish to make sure that you have a complete set of data from the source MISP feed.

SSL certificates

To use an SSL certificate with the platform, it must be:

  • Accessible on the EclecticIQ Platform host.

  • Placed in a location that can be accessed by the eclecticiq user.

  • Owned by eclecticiq:eclecticiq.

To make sure that the platform can access the SSL certificate:

  1. Upload the SSL certificate to a location on the platform host.

  2. On the platform host, open the terminal.

  3. Change ownership of the SSL certificate by running as root in the terminal:

    chown eclecticiq:eclecticiq /path/to/cert.pem

    Where /path/to/cert.pem is the location of the SSL certificate the platform needs to access.

Test the feed

  1. In the top navigation bar, click Data Configuration > Incoming feeds.

  2. Select the feed that you just created, using instructions above.

  3. In Overview, click Download now.

  4. Select the Ingested entities tab and check that entities have been ingested.

Or:

  1. In the top navigation bar, click Intelligence > All intelligence > Browse.

  2. Select the Entities tab.

  3. In the top-left corner, click images/download/attachments/86441305/filter.png .

  4. From the Source drop-down menu, select the incoming feed you have just created.

  5. To filter by entity type, select the entity types to include in the filtered results using the Entity drop-down menu.