Exchanging data between IC instances#

About data exchange between IC instances#

You can exchange data between two platform instances using TAXII feeds. This is what you need to implement this platform-to-platform integration:

A group and a user with access rights to both platform instances.
Basic authentication (optional)
A TAXII inbox or a TAXII poll outgoing feed
A TAXII inbox or a TAXII poll incoming feed
An EclecticIQ JSON content type for both outgoing and incoming feeds
Two platform instances acting as data source/publisher and data destination/recipient, respectively.

For the sake of keeping things explicit and unambiguous, let’s give names to our communicating platform instances:

Let’s call the publishing platform instance Alice. This is the data source of the feed content.
And let’s call the recipient platform instance Barbara. This is the data destination and the consumer of the feed content.

Note

Assign unique names to TAXII feeds: TAXII inbox and TAXII poll feeds in the platform, both incoming and outgoing, must have unique names.

Note

About transport

Both platform instances must exchange data through the same TAXII transport protocol service: either TAXII inbox or TAXII poll.

About content

Although TAXII feed configurations make available more than one content type for the incoming and the outgoing feeds, platform to platform data exchange officially supports only EclecticIQ JSON as a valid data exchange content type.
Regardless of Alice’s outgoing and Barbara’s incoming feeds using either the TAXII inbox or the TAXII poll transport types, they both need to publish data to and retrieve data from the same TAXII collection:
- Alice (publisher) should publish content through a TAXII outgoing feed and a specified Collection name – for a TAXII poll transport type – or a Destination collection name – for a TAXII inbox transport type.
- Barbara (recipient) should ingest content through a TAXII incoming feed, and it should ingest from or poll the same Collection name specified under Collection name or Destination collection name in Alice’s TAXII outgoing feed configuration.

About user access control and permissions

Set the automation user and, where applicable, group you created as the user and the authorized group that are granted access to the feed.
Specify the automation user’s user name and password in the corresponding feed configuration fields.
If you enable basic authentication, make sure the automation role has the required additional permissions.

Check package size limits#

The /etc/eclecticiq/platform_settings.py configuration file includes settings that control and limit package file size for ingestion – incoming feeds and manual file upload – and for dissemination – outgoing feeds.

# Package size limits (in bytes)
MAX_BLOB_SIZE = 20 * 1024 * 1024
MAX_UPLOADED_BLOB_SIZE = MAX_BLOB_SIZE / 2

MAX_BLOB_SIZE defines the max allowed file size for ingested packages through incoming feeds and for published packages through outgoing feeds.

The size is expressed in bytes.

Default value: 20 * 1024 * 1024 (20 MB)
MAX_UPLOADED_BLOB_SIZE defines the max allowed file size for ingested packages through manual file upload.

The size is expressed in bytes.

Default value: MAX_BLOB_SIZE / 2 (10 MB)

You can set custom values for MAX_BLOB_SIZE and MAX_UPLOADED_BLOB_SIZE. For example, you can set MAX_UPLOADED_BLOB_SIZE = MAX_BLOB_SIZE, or MAX_UPLOADED_BLOB_SIZE = 30 * 1024 * 1024 to enable manually uploading large PDF files to the platform.

The maximum limit you can set for file uploads is 100 * 1024 * 1024 (100 MB).

Caution

Changes to these fields may affect ingestion, packaging, and dissemination behavior, as well overall platform performance. For example, setting too large a size may result in ingestion bottlenecks or upload timeouts.

Whitelist mount points#

Explicitly whitelist mount point paths to make them accessible to incoming and to outgoing feeds. If you do not whitelist the mount point path an incoming or an outgoing feed should access to retrieve data for ingestion or for publication, the feed will not be able to fetch or to publish any content.

The /etc/eclecticiq/platform_settings.py configuration file includes dedicated mount point whitelists for ingestion – incoming feeds – and for dissemination – outgoing feeds.

# Directories that can be accessed from mount point feeds. POLL is for incoming
# feeds, PUSH is for outgoing feeds. Example: ["/mnt/", "/media/"]
MOUNT_POINT_POLL_ALLOWED_DIRECTORIES: Sequence[str] = []
MOUNT_POINT_PUSH_ALLOWED_DIRECTORIES: Sequence[str] = []

MOUNT_POINT_POLL_ALLOWED_DIRECTORIES is a list of allowed mount point paths that incoming feeds can access to fetch data from.
MOUNT_POINT_PUSH_ALLOWED_DIRECTORIES is a list of allowed mount point paths that outgoing feeds can access to publish data to.

To whitelist a mount point path:

Open the configuration file:

sudo vi /etc/eclecticiq/platform_settings.py

Look for MOUNT_POINT_POLL_ALLOWED_DIRECTORIES to make network locations accessible to incoming feeds, or for for MOUNT_POINT_PUSH_ALLOWED_DIRECTORIES to make network locations accessible to outgoing feeds.

Both parameters are lists that take valid directory paths as list elements.

Each path in the list points to a location that incoming feeds can access to fetch the data to be ingested, or that outgoing feeds can access to publish the content of a feed run.

Incoming and outgoing feeds can access files and directories inside the specified locations, based on the configured access rights of the available assets and resources.

Add as many paths to each list as necessary, then save the file and exit.

Example:

# Whitelist specific dirs; specific file types; everything inside subdirs of a dir
MOUNT_POINT_PUSH_ALLOWED_DIRECTORIES = [ "/mnt/", "/media/", "/media/data/" ]

About TAXII services#

After configuring a TAXII server, you can set up TAXII services. A TAXII service is a specialized data handler that implements a specific TAXII capability.

The platform supports the following TAXII services:

Service type	Description
Collection management service	You can use a TAXII collection management service to: Request information about TAXII data collections. Subscribe to TAXII data collections. Cancel subscriptions to TAXII data collections. You can poll a TAXII collection to: Obtain overviews of the available content. Check for updated content. Select TAXII feeds and datasets to use as data sources. TAXII data collections — structured TAXII data feeds, and unstructured TAXII datasets — are examples of TAXII inbox and TAXII poll service content. Bindings: TAXII HTTP 1.0 TAXII HTTPS 1.0
Discovery service	You can use a TAXII discovery service to obtain information about the availability and the use of TAXII services such as collection management, inbox, and polling. Bindings: TAXII HTTP 1.0 TAXII HTTPS 1.0
Inbox service	The TAXII inbox service enables data producers to initiate push messages to service consumers. This service can be based on a subscription model, or it can be an unsolicited payload a producer pushes to a consumer. Bindings: TAXII HTTP 1.0 TAXII HTTPS 1.0
Poll service	The TAXII poll service enables consumers to request TAXII data collection content from a TAXII producer, usually through TAXII outgoing feeds. Bindings: TAXII HTTP 1.0 TAXII HTTPS 1.0

View TAXII services#

To access an overview of the existing and configured TAXII services available in the platform :

In the side navigation bar select Settings > STIX and TAXII > TAXII. The TAXII view displays the currently configured TAXII services for the platform.

To sort items by column header:
1. Click the header of the column whose content you want to sort.
2. Click or to sort the content in either ascending or descending order, respectively.
Click the menu icon in the row corresponding to the TAXII service you want to inspect.
From the drop-down menu select View.

Alternatively:

In the TAXII view click anywhere in the row corresponding to the TAXII service whose configuration you want to inspect.

The TAXII settings view displays the current configuration of the specified service.