Incoming feed - HTTP download
This procedure describes how to configure incoming feeds for a particular feed data source, transport type, or content type.
For more information about configuring common options shared across all incoming feeds, see Configure incoming feeds general options.
|
|
Specifications |
|
Transport type |
HTTP download |
|
Content type |
|
|
Ingested data |
Structured and unstructured data in JSON, PDF, STIX, and plain text format. |
|
Processed data |
Structured, STIX-compliant entities and observables. |
Requirements
Users need a username and a password for their own configuration.
Configure the incoming feed
Create or edit an incoming feed.
From the Transport type drop-down menu, select HTTP download.
From the Content type drop-down menu, select the content type for the data you want to ingest.
The content type should match the data source format. This can vary, depending on the intel sources you retrieve the data from.Select the Accept password protected archives checkbox to specify a global password to open any archives retrieved through the incoming feed.
If the archives are password-protected, enter it in the Archive password field.
The specified password acts as a master password, and it is used to try to unlock and access any archives retrieved with the feed.
Supported archive formats:.rar, .tar, .tar.bz2, .tar.gz, .tar.bz2, .tar.z, .zip.
In the URL field, enter the location/directory/folder on the server or network unit hosting the data source for the feed.
In the Regex pattern field, you can define a regex to include in the incoming feed only the content delivered through the links that match the specified regex pattern.
Your input needs to be a valid regex pattern.
Examples of valid regex patterns:.+\.json
.+\.pdf
.+\.*
[^/]all_files_with_this_name_but_different_extensions[^/].[a-z0-9]*
If you do not enter any regex pattern, the feed fetches the base URL response body, and it assumes that any retrieved content is in the format specified under Content type.
If you enter a regex pattern, the feed assumes that any retrieved content is in HTML format, regardless of the format specified under Content type.
Retrieved content is scanned for any links matching the regex pattern.
Matches are downloaded, assuming that their format corresponds to the one specified under Content type.
In the Basic auth username field, enter a valid user name to authenticate and be granted the necessary authorization to access the data source and to download/ingest data.
HTTP download uses basic HTTP authentication.In the Basic auth password field, enter a valid password to authenticate and be granted the necessary authorization to access the data source and to download/ingest data.
Select the Verify connection checkbox if you want to test the connection to verify that it works as expected.
The HTTP server hosting the data source for the feed may require passing additional HTTP headers in the request.Under Extra headers, click Add or More to include in the request one or more additional HTTP headers.
To store your changes, click Save; to discard them, click Cancel.