Incoming feed - HTTP download#

Note

This article describes how to configure incoming feeds for a particular feed source. To see how to configure incoming feeds in general, see Create and configure incoming feeds.

Specifications

Transport type

HTTP download

Content type

  • Eclectic JSON

  • Email message

  • MISP JSON

  • PDF

  • SpyCloud Breach Data JSON

  • STIX 1.0

  • STIX 1.1

  • STIX 1.1.1

  • STIX 1.2

  • STIX 2.1

  • Text

Ingested data

Structured and unstructured data in JSON, PDF, STIX, and plain text format.

Processed data

Structured, STIX-compliant entities and observables.

Requirements#

Users need a username and a password for their own configuration.

Configure the incoming feed#

  1. Create or edit an incoming feed.

  2. From the Transport type drop-down menu, select HTTP download.

  3. From the Content type drop-down menu, select the content type for the data you want to ingest.
    The content type should match the data source format. This can vary, depending on the intel sources you retrieve the data from.

  4. Select the Accept password protected archives checkbox to specify a global password to open any archives retrieved through the incoming feed.
    If the archives are password-protected, enter it in the Archive password field.
    The specified password acts as a master password, and it is used to try to unlock and access any archives retrieved with the feed.
    Supported archive formats:

    • .rar, .tar, .tar.bz2, .tar.gz, .tar.bz2, .tar.z, .zip.

  5. In the URL field, enter the location/directory/folder on the server or network unit hosting the data source for the feed.

  6. In the Regex pattern field, you can define a regex to include in the incoming feed only the content delivered through the links that match the specified regex pattern.
    Your input needs to be a valid regex pattern.
    Examples of valid regex patterns:

    • .+.json

    • .+.pdf

    • .+.*

    • [^/]all_files_with_this_name_but_different_extensions[^/].[a-z0-9]*
      If you do not enter any regex pattern, the feed fetches the base URL response body, and it assumes that any retrieved content is in the format specified under Content type.
      If you enter a regex pattern, the feed assumes that any retrieved content is in HTML format, regardless of the format specified under Content type.
      Retrieved content is scanned for any links matching the regex pattern.
      Matches are downloaded, assuming that their format corresponds to the one specified under Content type.

  7. In the Basic auth username field, enter a valid user name to authenticate and be granted the necessary authorization to access the data source and to download/ingest data.
    HTTP download uses basic HTTP authentication.

  8. In the Basic auth password field, enter a valid password to authenticate and be granted the necessary authorization to access the data source and to download/ingest data.

  9. Select the Verify connection checkbox if you want to test the connection to verify that it works as expected.
    The HTTP server hosting the data source for the feed may require passing additional HTTP headers in the request.

  10. Under Extra headers, click Add or More to include in the request one or more additional HTTP headers.

  11. To store your changes, click Save; to discard them, click Cancel.