Incoming feed - Mount point download
This procedure describes how to configure incoming feeds for a particular feed data source, transport type, or content type.
For more information about configuring common options shared across all incoming feeds, see Configure incoming feeds general options.
|
|
Specifications |
|
Transport type |
Mount point download |
|
Content type |
|
|
Ingested data |
Structured and unstructured data in JSON, PDF, STIX, and plain text format. |
|
Processed data |
Structured, STIX-compliant entities and observables. |
|
Description |
Retrieve and process information from specific data sources available on the (local) network. |
Configure the incoming feed
Create or edit an incoming feed.
From the Transport type drop-down menu, select Mount point download.
From the Content type drop-down menu, select the content type for the data you want to ingest.
The content type should match the data source format. This can vary, depending on the intel sources you retrieve the data from.Select the Accept password protected archives checkbox to specify a global password to open any archives retrieved through the incoming feed.
If the archives are password-protected, enter it in the Archive password field.
The specified password acts as a master password, and it is used to try to unlock and access any archives retrieved with the feed.
Supported archive formats: .rar, .tar, .tar.bz2, .tar.gz, .tar.bz2, .tar.z, .zip.In the Path field, enter the location/directory/folder on the server or network unit hosting the data source for the feed.
The incoming feed first looks for the specified directory, on a server or a network unit, as the origin to poll data from.If you specify also a regex pattern, the path should point to a directory.
If you do not define a regex pattern, the path should be an absolute pointing to a file, and the file format should match the configured content type for the feed.
In the Regex pattern field, you can define a regex to include in the incoming feed any file names matching the specified regex pattern.
If you include a regex pattern, the incoming feed looks in the directory specified under Path for any files whose name matches the regex pattern, and whose format matches the configured content type for the feed. Your input needs to be a valid regex pattern.
Examples of valid regex patterns:+.json.
.+.pdf.
.*.
[^/]all_files_with_this_name_but_different_extensions[^/].[a-z0-9].
Matching files are added to the feed, so that the platform can ingest their content.
If you do not enter any regex pattern, the feed fetches the base path response body, and it tries to open the base path.To store your changes, click Save; to discard them, click Cancel.
Explicitly whitelist mount point paths to make them accessible to incoming and to outgoing feeds.
If you do not whitelist the mount point path an incoming or an outgoing feed should access to retrieve data for ingestion or for publication, the feed will not be able to fetch or to publish any content.
The /etc/eclecticiq/platform_settings.py configuration file includes dedicated mount point whitelists for ingestion – incoming feeds – and for dissemination – outgoing feeds.
settings.py (sourced from EIQ platform-backend)
|
Author |
Rutger Prins |
|
Commit |
ab323b23ebb93fde6c62b124f6823579957bd1d5 |
|
Timestamp |
August, 27, 2021 08:57 AM |
|
Full path |
eiq/platform/settings.py |
|
Title |
Merge branch 'ext-commons-update-2.10.x' into 'release-2.10.x' |
|
Description |
Extension Commons update 2.10.x See merge request engineering/platform-backend!6075 |
# Directories that can be accessed from mount point feeds. POLL is for incoming# feeds, PUSH is for outgoing feeds. Example: ["/mnt/", "/media/"]MOUNT_POINT_POLL_ALLOWED_DIRECTORIES: Sequence[str] = []MOUNT_POINT_PUSH_ALLOWED_DIRECTORIES: Sequence[str] = []MOUNT_POINT_POLL_ALLOWED_DIRECTORIES is a list of allowed mount point paths that incoming feeds can access to fetch data from.
MOUNT_POINT_PUSH_ALLOWED_DIRECTORIES is a list of allowed mount point paths that outgoing feeds can access to publish data to.
To whitelist a mount point path:
Open the configuration file:
sudovi/etc/eclecticiq/platform_settings.pyLook for MOUNT_POINT_POLL_ALLOWED_DIRECTORIES to make network locations accessible to incoming feeds, or for for MOUNT_POINT_PUSH_ALLOWED_DIRECTORIES to make network locations accessible to outgoing feeds.
Both parameters are lists that take valid directory paths as list elements.
Each path in the list points to a location that incoming feeds can access to fetch the data to be ingested, or that outgoing feeds can access to publish the content of a feed run.
Incoming and outgoing feeds can access files and directories inside the specified locations, based on the configured access rights of the available assets and resources.Add as many paths to each list as necessary, then save the file and exit.
Example:# Whitelist specific dirs; specific file types; everything inside subdirs of a dirMOUNT_POINT_PUSH_ALLOWED_DIRECTORIES=["/mnt/","/media/","/media/data/"]