Incoming feed - Mount point download#

Note

This article describes how to configure incoming feeds for a particular feed source. To see how to configure incoming feeds in general, see Create and configure incoming feeds.

Specifications

Transport type

Mount point download

Content type

  • Eclectic JSON

  • Email message

  • MISP JSON

  • PDF

  • SpyCloud Breach Data JSON

  • STIX 1.0

  • STIX 1.1

  • STIX 1.1.1

  • STIX 1.2

  • STIX 2.1

  • Text

Ingested data

Structured and unstructured data in JSON, PDF, STIX, and plain text format.

Processed data

Structured, STIX-compliant entities and observables.

Description

Retrieve and process information from specific data sources available on the (local) network.

Configure the incoming feed#

  1. Create or edit an incoming feed.

  2. From the Transport type drop-down menu, select Mount point download.

  3. From the Content type drop-down menu, select the content type for the data you want to ingest.
    The content type should match the data source format. This can vary, depending on the intel sources you retrieve the data from.

  4. Select the Accept password protected archives checkbox to specify a global password to open any archives retrieved through the incoming feed.
    If the archives are password-protected, enter it in the Archive password field.
    The specified password acts as a master password, and it is used to try to unlock and access any archives retrieved with the feed.
    Supported archive formats: .rar, .tar, .tar.bz2, .tar.gz, .tar.bz2, .tar.z, .zip.

  5. In the Path field, enter the location/directory/folder on the server or network unit hosting the data source for the feed.
    The incoming feed first looks for the specified directory, on a server or a network unit, as the origin to poll data from.

    • If you specify also a regex pattern, the path should point to a directory.

    • If you do not define a regex pattern, the path should be an absolute pointing to a file, and the file format should match the configured content type for the feed.

  6. In the Regex pattern field, you can define a regex to include in the incoming feed any file names matching the specified regex pattern.
    If you include a regex pattern, the incoming feed looks in the directory specified under Path for any files whose name matches the regex pattern, and whose format matches the configured content type for the feed. Your input needs to be a valid regex pattern.
    Examples of valid regex patterns:

    • +.json.

    • .+.pdf.

    • .*.

    • [^/]all_files_with_this_name_but_different_extensions[^/].[a-z0-9].

    Matching files are added to the feed, so that the platform can ingest their content.
    If you do not enter any regex pattern, the feed fetches the base path response body, and it tries to open the base path.

  7. To store your changes, click Save; to discard them, click Cancel.

Note

Explicitly whitelist mount point paths to make them accessible to incoming and to outgoing feeds.
If you do not whitelist the mount point path an incoming or an outgoing feed should access to retrieve data for ingestion or for publication, the feed will not be able to fetch or to publish any content.

The /etc/eclecticiq/platform_settings.py configuration file includes dedicated mount point whitelists for ingestion – incoming feeds – and for dissemination – outgoing feeds.

settings.py (sourced from EIQ platform-backend)

Author

Rutger Prins

Commit

ab323b23ebb93fde6c62b124f6823579957bd1d5

Timestamp

August, 27, 2021 08:57 AM

Full path

eiq/platform/settings.py

Title

Merge branch ‘ext-commons-update-2.10.x’ into ‘release-2.10.x’

Description

Extension Commons update 2.10.x See merge request engineering/platform-backend!6075

 
# Directories that can be accessed from mount point feeds. POLL is for incoming
# feeds, PUSH is for outgoing feeds. Example: ["/mnt/", "/media/"]
MOUNT_POINT_POLL_ALLOWED_DIRECTORIES: Sequence[``str``] ``=`` []
MOUNT_POINT_PUSH_ALLOWED_DIRECTORIES: Sequence[``str``] ``=`` []

  • MOUNT_POINT_POLL_ALLOWED_DIRECTORIES is a list of allowed mount point paths that incoming feeds can access to fetch data from.

  • MOUNT_POINT_PUSH_ALLOWED_DIRECTORIES is a list of allowed mount point paths that outgoing feeds can access to publish data to.

To whitelist a mount point path:

  1. Open the configuration file:

    sudo vi /etc/eclecticiq/platform_settings.py

  2. Look for MOUNT_POINT_POLL_ALLOWED_DIRECTORIES to make network locations accessible to incoming feeds, or for MOUNT_POINT_PUSH_ALLOWED_DIRECTORIES to make network locations accessible to outgoing feeds. Both parameters are lists that take valid directory paths as list elements.
    Each path in the list points to a location that incoming feeds can access to fetch the data to be ingested, or that outgoing feeds can access to publish the content of a feed run.
    Incoming and outgoing feeds can access files and directories inside the specified locations, based on the configured access rights of the available assets and resources.

  3. Add as many paths to each list as necessary, then save the file and exit.
    Example:

    # Whitelist specific dirs; specific file types; everything inside subdirs of a dir
MOUNT_POINT_PUSH_ALLOWED_DIRECTORIES = [ "/mnt/", "/media/", "/media/data/" ]