Release notes 3.3.1#

Product	EclecticIQ Intelligence Center
Release version	3.3.1
Release date	Mar 2024
Time to upgrade	~40 minutes to upgrade an instance with 2.67 million entities, 1.85 million observables. From the previous release Using the installation script For an instance running on one machine
Time to migrate	For an instance with 2.67 million entities, 1.85 million observables: PostgreSQL migration: 13m30s Elasticsearch migration: 18m40s

Highlights #

EclecticIQ Intelligence Center 3.3.1 represents a significant advancement in our Threat Intelligence Platform, blending continuity with innovation.

Building on the enhancements introduced in version 3.2, this release further refines threat intelligence management capabilities. It introduces additional data policy features for refined data retention and improved data export via more flexible CSV support, complementing the previous version’s CSV import enhancements. Improvements in source management give administrators greater flexibility in adapting to evolving collection requirements and ensure intelligence contributions are consistently attributed to the correct source.

CTI analysts will be pleased to hear that this release further enhances our MITRE ATT&CK support, continuing the momentum from release 3.2. We’ve updated the built-in Enterprise framework to the latest 14.1 version and automated the creation of mappings for ingested reports. The introduction of a dedicated search interface for relational queries, and upgraded support for TLP-protocol version 2.0, significantly improve the CTI analyst experience as well.

Finally, this version heralds the debut of our first Generative AI-powered feature, signaling the start of a new long-term initiative to supercharge intelligence operations with AI technology. The beta AI Report Creator enables analysts to generate comprehensive reports efficiently using Generative AI from market leader OpenAI. Administrators can enable this beta feature for their users in the Labs section by simply entering an OpenAI API license key, and we encourage feedback to further refine its capabilities.

We’re excited for you to explore these updates and trust they will enhance your threat intelligence operations.

Important: Upgrade operating system #

Important

EclecticIQ Intelligence Center 3.0.0 and newer requires one of these supported operating systems:

Red Hat Enterprise Linux 8
Rocky Linux 8

You must upgrade to one of the supported operating systems before installing EclecticIQ Intelligence Center 3.0 or newer.

See:

Important: `eclecticiq-extension-commons` package is deprecated #

Caution

Only affects users who develop or customize EclecticIQ Intelligence Center extensions.

eclecticiq-extension-commons is deprecated in release 3.3, and will be removed in release 3.4.

If you have written your own extension, or modified an existing extension, that extension may contain references to the eclecticiq-extension-commons package.

In particular, if your extension:

depends on eclecticiq-extension-commons
imports from extension.common

You must remove or change those references in your extension before upgrading to the upcoming release 3.4. A migration guide will be provided.

Custom extensions will continue to work without modification in release 3.3.

What’s new #

(EclecticIQ Labs) Generate reports with AI #

This release adds Intelligence summary AI generation to EclecticIQ Labs.

Intelligence summary AI generation is an early-access feature that allows you to generate reports from entities using OpenAI’s API. When you enable this feature in EclecticIQ Labs, the option to Generate AI report appears when you work with entities.

Note

To use Intelligence summary AI generation, you must have an OpenAI API key, and purchased credits available for that key.

Go to Settings > EclecticIQ Labs and select Intelligence summary AI generation to enable it.

To use this feature, you can:

Go to Graphs and open a graph containing entities you want to include in your report. Select and right-click those entities, then select Generate AI report and follow the on-screen prompts. Or,
Go to Search and browse > Go to search and browse. Select entities that you want to include in your report. Then, from the toolbar, select Add to > Generate AI report and follow the on-screen prompts.

For more information, see AI | Reports.

Adds UI for relational search #

Previously, searching for entities that had a given relationship to another entity required the use of the relationship search query syntax. This release adds options in the UI to search for entities that have a relationship to a given set of entities.

Now, you can go to Search and browse > Entities and select the Relational query from the query type drop-down menu to the left of the search bar to display the relational query UI.

Here, you can:

Set the source query (left query field) and target query (right query field). Results displayed are entities from the source query that have a relation to entities that match the target query.
Filter results by relation type.

For more information, see Search | Relational.

Support for TLP 2.0 #

This release adds support for TLP 2.0. TLP 1.0 support is retained. Now, when setting TLP values you can select Amber Strict and Clear values. When entities are exported, their TLP 2.0 values are converted to TLP 1.0 values where needed.

By default, when converting to TLP 1.0, Clear becomes White and Amber Strict becomes Red. You can change the default converted value of Amber Strict in Settings > System settings > General > TLP settings.

Currently, exporting entities with TLP 2.0 values is only supported by the EclecticIQ JSON content type.

For more information, see About TLP.

New CSV support #

This release brings further improvements to Intelligence Center support for CSV output, with the addition of Manual CSV export and New Advanced CSV content type for outgoing feeds.

Manual CSV export#

You can now export multiple entities or observables as a CSV file. This is available when you select one or more entities or observables in Search and browse or + > Production.

When exporting selected entities or observables as a CSV file, you can select fields to export.

New Advanced CSV content type for outgoing feeds#

This release adds the Advanced Entities CSV and Advanced Observables CSV content type for outgoing feeds.

Set up outgoing feeds with these content types to pack entities or observables with a customizable set of CSV fields that you configure during feed configuration.

For more information, see Configure content types.

Advanced Entities CSV and Advanced Observables CSV content type lets you select fields to export.#

Updated MITRE ATT&CK support #

This release updates support for MITRE ATT&CK with support for MITRE ATT&CK v14.1, and new automated extraction of MITRE ATT&CK classifications for newly created report entities.

MITRE ATT&CK extraction from reports#

Now, when creating or ingesting reports, Intelligence Center scans the Summary or Analysis fields for each report for any MITRE ATT&CK references, and adds detected classifications to the report.

MITRE ATT&CK updated to v14.1#

This release updates the supported MITRE ATT&CK version to v14.1.

If you have existing entities that have older and revoked classifications, you will still be able to search for them in Intelligence Center.

Known issue

MITRE ATT&CK classifications that have been renamed or relocated (ATT&CK ID has changed) in ATT&CK v14.1 will no longer be searchable by their older names or ID.

Queries (e.g. used in Dynamic Datasets) that depend on an ATT&CK ID or name that has changed by a newer version of MITRE ATT&CK may fail because of this.

New customizable defaults for users #

Users can now set defaults for certain values that are commonly used when working with Intelligence Center. In this release, you can set a default value for: the source assigned when creating entities and observables through the UI or the Public API; the TLP assigned; confidence when creating new entities through the UI.

To set these defaults, select your profile icon from the left navigation menu, then go to Default value preferences.

Users can set defaults for certain values for their workflows.#

For more information, see Manage your own user account.

Retention policies have new options #

More retention period options for Delete entities action#

When using the Delete entities action in Retention policies, you can now select more retention period options.

When setting a scope for a retention policy, you must select a Period and Starting from value. For the Delete entities policy action, Starting from sets the entity field which the retention Period will be applied to. You can now select 3 additional Starting from values: Estimated threat start time, Estimated threat end time, and Estimated observed time.

New Starting from values for retention policy scopes#

For more information, see Create data policies.

Add more than one allowed source when configuring groups #

When creating or editing groups, you must add allowed sources to your group in order to allow members of the group to see the contents of these sources.

This release, you can now add more than one allowed source per TLP color when configuring groups.

To do this, select + Add sources when configuring a group, select a TLP color, then select the Sources field for that entry to start selecting sources to assign to this group.

You can now select multiple allowed sources at a time when configuring groups#

For more information, see Manage groups.

Add allowed sources when creating or editing incoming feed #

In order for a user to be able to see the contents of an incoming feed, that feed must be added as an Allowed source to at least one of the groups that user is a member of.

In this release, you can now configure what groups an incoming feed is an allowed source of from the feed configuration itself.

To do so, when configuring an incoming feed select Show advanced options and go to the Groups section. Select + Add to groups to add this incoming feed to one or more groups as an allowed source.

Note

You must be a member of the groups you want to add or remove an incoming feeds for.

Add incoming feed to a group's allowed sources. — Add incoming feed to a group’s allowed sources.#

For more information, see Create and configure incoming feeds.

Improvements #

Data mapping templates UI now show invalid mappings #

In release 3.1.x and earlier, it was possible to create Advanced CSV incoming feeds that had invalid mappings from CSV fields to EclecticIQ fields.

When upgrading to 3.2.0 and later, these mappings are automatically converted to Data mapping templates. However, this meant that existing invalid mappings were also converted, and caused the data mapping templates UI to not load. This is fixed in this release.

In addition, data mapping templates now show warnings in the UI when a template contains fields with invalid mappings.

Installation playbooks improvements #

EclecticIQ Intelligence Center installation playbooks now configure TLS for PostgreSQL and Redis by default.
Deployments can now enable and configure swap files.

For more information, see Prepare nodes.

Web server configuration is updated with best practices #

NGINX configuration is updated:

Now supports TLS 1.3 connections.
Uses secp384r1 curve for ECDH.

PostgreSQL autovacuum configuration is improved #

PostgreSQL is now configured to start freeing up disk usage for operating system use at a lower threshold.

Fixes #

Fixed issue where invalid data mapping templates would cause the UI to not load.
Fixed an issue where removing a retention policy doesn’t remove the backend record of the policy task.
Fixed issue where entities ingested from STIX 2.1 sources and then exported unmodified as STIX 1.2 XML would not contain TLP classifications in the resulting output.
Fixed issue where autocomplete in the search UI does not show suggestions if a source has a name that contains special characters.
Fixed issue where ingesting STIX 2.1 packages containing custom marking definitions fail.
Fixed issue where unprivileged users creating a report entity through the UI would fail if the report has attachments embedded in the description or analysis fields.
Installation playbooks: fixed issue where postfix was unavailable on worker and celery nodes in large deployments.
TAXII 2.1 fixes

Important

This release also introduces changes and known issues to the TAXII 2.1 server.

For more information, see TAXII 2.1
- Fixed issue where querying TAXII 2.1 endpoints could fail with a 504 Gateway Timeout error.
- Fixed issue where TAXII 2.1 server would encounter performance issues when publishing large collections.
- Fixed issue a sorting issue where querying a TAXII 2.1 collection may not return all objects.
- Fixed issue where querying objects from a TAXII 2.1 collection would not return deterministic results
- Fixed issue where a TAXII 2.1 outgoing feed would not generate a collection ID.
Public API fixes:
- Fixed issue where creating multiple observables with a single request would only result in one observable being created.
- Fixed error messages attempting to access an inaccessible resource would return a vague error message. Now, error messages correctly display the reason why an item is not found for 404 errors, and in the case where an inaccessible resource is actually the result of permissions, returns a 403 instead.
- Fixed incorrect data type in OpenAPI spec for the sources property.
  
  with specific source name (group)Doc change
- Fixed issue where unprivileged users could not create entities with attachments.

Known issues #

TAXII 2.1 known issues

Important

Performance Fixes for TAXII 2.1 in Intelligence Center 3.3.1 introduces changes and known issues to the TAXII 2.1 server.

For more information, see TAXII 2.1
If Intelligence Center has more than one possible user-facing domain name, generated links will only use the one that is configured

When generating links (e.g. CSV exports, outgoing feed URLs), Intelligence Center uses the host name that is configured in Settings > System settings > General > Hostname. However, it is possible that a given Intelligence Center instance may have more than one user-facing domain name. In this case, only the configured Hostname is used to generate links, and may cause users to be unable to follow those links.
Changes are lost if, while creating a new entity, the entity fails to publish

While creating a new entity, if the entity fails to save when selecting Publish, the work-in-progress entity can be lost. To avoid this, select Save draft to save a draft before selecting Publish.
Queries that depend on an ATT&CK ID or name that has changed in v12 may fail

MITRE ATT&CK classifications that have been renamed or relocated (ATT&CK ID has changed) in ATT&CK v12 will no longer be searchable by their older names or ID. Queries (e.g. used in Dynamic Datasets) that depend on an ATT&CK ID or name that has changed in v12 may fail because of this.
TLPs applied to relationship objects are not affected by TLP filters

You can now add TLP colors to relationship objects. However, you cannot use TLP colors with TLP filters yet.
Selecting TLP in entity view to override it does not apply to exports

Edit the entity to change its TLP, or override TLPs at feed level instead.
Certain entities added in 3.0 and newer will cause a STIX 1.2 outgoing feed to fail

Including certain entities in an outgoing feed using the STIX 1.2 content type will cause the feed to fail. Entities affected: Location, Identity, and Malware Analysis.
Certain entities added in 3.0 and newer display an option to export as STIX 1.2, but cannot

Nothing happens when Export > STIX 1.2 is selected for Location, Identity, and Malware Analysis entities. These entity types are not compatible with STIX 1.2 exports.
Exploit Target entities with references can create an invalid STIX 2.1 bundle on export

Exploit Target entities have an optional Vulnerability characteristic where you can set additional information. When an Exploit Target with References set in the Vulnerability characteristic, exporting to STIX 2.1 by default sets the type of these references to CVE, which causes an invalid STIX 2.1 bundle to be created if the set references are not valid CVE-IDs.
STIX 2.1 for outgoing feeds: TLP override and filtering has side-effects

See STIX 2.1 Known issues for a list of known issues.
When deleting content of an incoming feed, deleted observables are not included in the count of deleted objects.
Using STIX 2.1 content type to transmit data from one EclecticIQ Intelligence Center instance to another generates duplicates

When using the STIX 2.1 content type to send intelligence from one EclecticIQ Intelligence Center instance (Instance A) to another (Instance B), any updates to entities on Instance A that has already been sent to Instance B will result in duplicate entities being sent to Instance B instead of updating existing entities there.
When upgrading from 2.14 to 3.0, entities with certain fields that contain null values may cause database migrations to fail

In rare instances when upgrading from EclecticIQ Intelligence Center 2.14 to 3.0, older entities with null values in certain fields that don’t expect it may cause the database migration to fail, due to stricter validation of entity schemas. If this occurs, do not continue. Save the trace log and contact customer support for assistance to remediate.
Delete observable actions in policies may cause policies to run for excessively long periods of time.

As of 2.12.0, Delete observable actions are skipped by default to allow policies to run more reliably.
Elasticsearch 7 encounters “Data too large” errors: See Elasticsearch 7: “Data too large”.
Systemd splits log lines exceeding 2048 characters into 2 or more lines.

As a result, log lines exceeding 2048 characters become invalid JSON, causing Logstash to be unable to parse them correctly.
When more than 1000 entities are loaded on the graph, you cannot load related entities and observables by selecting Load entities, Load observables, or Load entities by observable from the context menu.
When creating groups in the graph, it is not possible to merge multiple groups into one.
If an ingestion process crashes while ingestion is still ongoing, data may not always sync to Elasticsearch.
Users can leverage rules to access groups that act as data sources, even if those users are not members of the groups they access through rules.
Running multiple outgoing feed tasks may cause the Intelligence Center to consume a large amount of memory over time, because certain outgoing feeds such as HTTP download must load the data into memory in order to make it available to feed consumers.

Public API compatibility #

From EclecticIQ Intelligence Center 2.12.0 onward, the public API is packaged together with EclecticIQ Intelligence Center.

The following reference table lists the versions of the public API package and EclecticIQ Intelligence Center versions they are compatible with:

Intelligence Center version(s)	Public API package version(s)	Public API version
2.11 - 2.12	`eclecticiq-extension-api==1.0.*`	v1
2.13.0	`eclecticiq-extension-api==1.*`	v1
2.14.0 and newer	Now follows EclecticIQ Intelligence Center versioning scheme. E.g., EclecticIQ Intelligence Center 2.14 is now compatible with `eclecticiq-extension-api==2.14.*`	v1
3.0.0 and newer	EclecticIQ Intelligence Center 3.0 and newer uses Public API v2. Follows EclecticIQ Intelligence Center versioning scheme. E.g., EclecticIQ Intelligence Center 3.0.2 is compatible with `eclecticiq-extension-api==3.0.`, EclecticIQ Intelligence Center 3.1.0 is compatible with `eclecticiq-extension-api==3.1.`, etc.	v2

Download #

For more information about setting up repositories, refer to the installation documentation for your target operating system.

EclecticIQ Intelligence Center and dependencies for Rocky Linux and RHEL	Platform packages: https://downloads.eclecticiq.com/platform-packages-centos/ Platform dependencies: https://downloads.eclecticiq.com/platform-dependencies-centos-2.9/ Note The Intelligence Center dependencies URL for versions 2.9 and later is `https://downloads.eclecticiq.com/platform-dependencies-centos-2.9/`. It contains packages that are incompatible with versions 2.8 and earlier.
EclecticIQ Intelligence Center extensions	Platform extensions: https://downloads.eclecticiq.com/Extensions/

Upgrade #

The diagram below describes upgrade paths for EcelcticIQ Intelligence Center. See the following for upgrade instructions:

In order to upgrade to EclecticIQ Intelligence Center 3.0, you must:

Be running one of the supported operating systems.

See Important: Upgrade operating system.
Upgrade from EclecticIQ Intelligence Center 2.14.

If you are running an older version of EclecticIQ Intelligence Center, you must upgrade to 2.14 before attempting to upgrade to EclecticIQ Intelligence Center 3.0.

See Install Configure Upgrade.