Release notes 2.11.3

Product	EclecticIQ Intelligence Center
Release version	2.11.3
Release date	18 May 2022
Summary	Patch release
Upgrade impact	Medium
Time to upgrade	~18 minutes to upgrade an instance with 4 million entities. From the previous release Using the installation script For an instance running on one machine Additional ~6 minutes to run pre-upgrade scripts for upgrading from 2.8.x and earlier.
Time to migrate	PostgreSQL database: ~6 minutes per 4 million entities Elasticsearch database: ~1 minute per 4 million entities Neo4j database: ~1 minute per 4 million entities.

Upcoming

• 2.12 uses Python 3.8

  Python 3.6 is scheduled for End-of-Life on 23rd December 2021. To address this, release 2.12 onwards will use Python 3.8.

• Rebranding to EclecticIQ Intelligence Center in documentation

  Rebranding from EclecticIQ Platform to EclecticIQ Intelligence Center in documentation is in progress. You may still see instances of “EclecticIQ Platform” remaining — please bear with us while we update the documentation.

Change in data retention mechanisms

Data retention mechanisms are being worked on to allow you to more effectively manage how long data should be kept on your IC.

This release adds:

• Policy run improvements

  • Data retention policy runs are more stable.

  • Delete observable actions are skipped by default on this release.

    There is a known issue with observable removal in retention policies that is being worked on. Meanwhile, Delete observable actions are skipped by default for all policies in this release. This improves the reliability of policy runs, and allows existing policies to run without modification.

    (Not recommended) You can re-enable Delete observable actions with the DISABLE_OBSERVABLE_RETENTION_POLICIES attribute in platform_settings.py. See Update platform_settings.py.

• Schedule removal of old records in raw data tables

  You now configure the IC to automatically delete old records in the blob and content_block tables from the PostgreSQL database.

  The blob table stores raw data downloaded by running incoming feeds, and the content_block table stores data prepared for consumption through outgoing feeds.

  You can set the number of days to retain records in these tables with the CONTENT_BLOCK_RETENTION and BLOB_RETENTION attributes in platform_settings.py. See Update platform_settings.py.

Security fixes

Tip

To see a detailed list of security issues and their mitigations, go to All security issues and mitigations.

• EIQ-2022-0003

  In the IC UI, drop-down menus that render user-defined item names are vulnerable to stored XSS attacks. Addressed in EIQ-2022-0003. These drop-down menus now correctly sanitize item names.

Known issues

• (Beta) Modern editor can cause UI crash when linking to objects with non-ASCII characters in its title

  You can link to entities or observables in your Reports using both the (Beta) Modern editor and the older editor. However, if the title of that entity or observable contains non-ASCII characters, the UI crashes.

• Elasticsearch 7 encounters “Data too large” errors: See Known issue with Elasticsearch 7: “Data too large”.

• Entity incorrectly warns it is outdated: When viewing an entity, the entity may warn that it is not the latest version when it actually is. This is related to an issue where with attachments that have been depulicated multiple times, causing issues in the final state of the entity.

• When you configure the Intelligence Center databases during a Intelligence Center installation or upgrade, you must specify passwords for the databases.

• Systemd splits log lines exceeding 2048 characters into 2 or more lines.

  As a result, log lines exceeding 2048 characters become invalid JSON, causing Logstash to be unable to parse them correctly.

• When more than 1000 entities are loaded on the graph, you cannot load related entities and observables by selecting Load entities, Load observables, or Load entities by observable from the context menu.

• When creating groups in the graph, it is not possible to merge multiple groups into one.

• If an ingestion process crashes while ingestion is still ongoing, data may not always sync to Elasticsearch.

• Users can leverage rules to access groups that act as data sources, even if those users are not members of the groups they access through rules.

• Running multiple outgoing feed tasks may cause the Intelligence Center to consume a large amount of memory over time, because certain outgoing feeds such as HTTP download must load the data into memory in order to make it available to feed consumers.

New knowledge pack permissions

In 2.11, permissions to install and access knowledge packs have changed from ... configuration-bundles to ... knowledge-packs.

This means that if you’ve previously assigned ... configuration-bundles permissions to users or roles, you have to re-assign the corresponding ... knowledge-packs permissions to allow those users or roles to retain access to knowledge pack features.

Users assigned permissions through the System Admin role are not affected.

The table below lists the changed permissions:

Before	2.11.0 and after
install configuration-bundles	install knowledge-packs
modify configuration-bundles	modify knowledge-packs
read configuration-bundles	read knowledge-packs

For more specific information on knowledge packs, see the documentation.

Known issue with Elasticsearch 7: “Data too large”

Since release 2.9.0, the Intelligence Center comes bundled with Elasticsearch (ES) 7.9.1. ES 7 adds a new real memory circuit breaker that causes ES nodes to respond with a circuit_breaking_exception error when it detects that memory use has reached 95% of the totally available JVM heap.

Because of this change, you may encounter issues related to available memory where previously at the same workloads, ES would appear to run smoothly.

If your plaform is encountering issues related to Elasticsearch responding with a circuit_breaking_exception error, you can do the following to mitigate:

Increase available memory for ES

The circuit_breaking_exception error occurs only when ES detects that you are about to go over a memory use threshold that would cause it to fail.

Increase the amount of memory available to ES, or move it to its own host where it does not compete with the Intelligence Center for resources to keep your ES nodes running.

(Not recommended) Disable the “real memory circuit breaker”

Warning

This may allow ES to reach an out of memory state and fail.

(Not recommended) To disable the “real memory circuit breaker”, set the indices.breaker.total.use_real_memory parameter in your ES configuration to false.

This allows ES to use the ES 6 parent circuit breaker instead, but disables the safety guarantees that the real memory circuit breaker provides.

Download

For more information about setting up repositories, refer to the installation documentation for your target operating system.

EclecticIQ Intelligence Center and dependencies for CentOS and RHEL	Platform packages: https://downloads.eclecticiq.com/platform-packages-centos/ Platform dependencies: https://downloads.eclecticiq.com/platform-dependencies-centos-2.9/ Note The Intelligence Center dependencies URL for versions 2.9 and later is https://downloads.eclecticiq.com/platform-dependencies-centos-2.9/. It contains packages that are incompatible with versions 2.8 and earlier.
EclecticIQ Intelligence Center extensions	Platform extensions: https://downloads.eclecticiq.com/Extensions/

Upgrade

The following diagram describes the upgrade path you should take depending on the Intelligence Center version you are upgrading from.

For example:

• You can upgrade from version 2.9.1 of the Intelligence Center to 2.10.0 directly,

• To upgrade from 2.4.0 to 2.10.0, you must first upgrade to 2.5.0, then upgrade from 2.5.0 to 2.10.0.

When upgrading from 2.8.x and earlier to 2.9.x and later:

• You must run the pre-upgrade script to allow it to work with Elasticsearch 7.9.1.

  • CentOS

  • RHEL

• You must run the pre-upgrade script on the Intelligence Center version you are upgrading from.

  For example, when upgrading from 2.8.0 to 2.10.1, you must run the pre-upgrade script on the Intelligence Center while it is running version 2.8.0.

Upgrade diagram

From 2.5.0, the upgrades paths have been tested using the EclecticIQ Intelligence Center install script compiled by Rundoc.

The script only supports:

• Single machine installs.

• Instances installed using the Intelligence Center install script.

and does not support Intelligence Center instances installed in distributed environments.