Release notes 2.10.1#
Product |
EclecticIQ Platform |
|---|---|
Release version |
2.10.1 |
Release date |
31 August 2021 |
Summary |
Patch release |
Upgrade impact |
Medium |
Time to upgrade |
~18 minutes to upgrade an instance with 4 million entities.
Additional ~6 minutes to run pre-upgrade scripts for upgrading from 2.8.x and earlier. |
Time to migrate |
|
Upcoming#
EclecticIQ Platform to be renamed EclecticIQ Intelligence Center
2.10 is the last release using the EclecticIQ Platform name. As of release 2.11 we will rename the product to EclecticIQ Intelligence Center and update all documentation.
Improvements#
Entity and observable rule query timeouts are now configurable
Manually running entity and observable rules may cause the resulting Elasticsearch query to time out if the rules are complex, or the datastore the queries are run against is large.
To mitigate this:
Default timeout values for entity and observable rules have been increased from 60 to 300 seconds.
Entity and observable rule timeouts can now be configured in
platform_settings.pyby setting these attributes:# Set timeout for manually running # Observable rules to 300 seconds OBSERVABLE_RULE_ES_TIMEOUT = 300 # Set timeout for manually running # Entity rules to 300 seconds ENTITY_RULE_ES_TIMEOUT = 300
Important bug fixes#
Text extracted from PDFs appear incorrectly formatted
When uploading PDFs to create reports, the resulting report entity may appear to incorrectly extract and display the text contained in the original PDF.
This happens when the uploaded PDF has text that contains a “zero width non-joiner” whitespace unicode character. This breaks up the flow of text, and causes the extracted text to be difficult to read. Text extracted from PDFs now have those characters replaced with regular whitespace characters.
Unable to set custom values in certain UI fields
Issue where users could not set custom values in certain UI fields.
Kibana could not load data
Users who analyze their data with Kibana were unable to access certain features such as “Saved Objects” and “Index Patterns” because of an issue with how we were routing Kibana GET requests. These requests are now handled correctly.
UI may be unresponsive and display HTTP 504 errors
Users with a large datastore may experience slow loading of data in some UI elements, and may display HTTP errors instead. Fixed by optimizing underlying queries.
Related observables in Report entities could not be updated
If a Report entity is created by ingesting data, it would contain a list of observables related to that Report. Users would be unable to edit that list of observables because they were missing a “link type”. This has been fixed.
Running data retention policies may cause high memory use
Data retention policy tasks were consuming an excessive amount of memory over time, and may lead to a related issue where high-resource applications that share the same host would stop responding, such as in Known issue with Elasticsearch 7: “Data too large”.
This memory leak has been fixed.
Outgoing feeds with large datasets may fail
When running an outgoing feed for a large dataset, there is a chance that the feed fails with a
KeyError: 'scroll_id'exception because the Elasticsearch scroll context is lost while waiting for the platform to process the search.This has been resolved by fixing a bug in how scroll context is handled.
Notification preferences cannot be saved
Users were unable to save their notification preferences in Notifications
> Settings 
.Feeds that update reports may fail because of attachments deduplicate incorrectly
When a report has its attachments updated, attachments with the same name are deduplicated. This is done by discarding one of the versions of the attachments. In some cases, this results in the feed failing when the existing attachment is discarded and the incoming attachment is not fully processed yet and causes an error.
Outgoing feed using EclecticIQ PDF content type may fail
When attempting to send report entities out through an outgoing feed that uses the EclecticIQ PDF content type, the feed may fail.
Security fixes#
For a summary of recent security issues, see Security issues and mitigations.
User who is Group Admin can make themselves a platform administrator
A user who is assigned as a “Group Admin” for any group can manipulate the private API request for editing their own profile and make themselves a platform administrator. This is addressed in EIQ-2021-0008.
Known issues#
Elasticsearch 7 encounters “Data too large” errors: See Known issue with Elasticsearch 7: “Data too large”.
Entity incorrectly warns it is outdated: When viewing an entity, the entity may warn that it is not the latest version when it actually is. This is related to an issue where with attachments that have been depulicated multiple times, causing issues in the final state of the entity.
When you configure the platform databases during a platform installation or upgrade, you must specify passwords for the databases.
Systemd splits log lines exceeding 2048 characters into 2 or more lines.
As a result, log lines exceeding 2048 characters become invalid JSON, causing Logstash to be unable to parse them correctly.
When more than 1000 entities are loaded on the graph, you cannot load related entities and observables by selecting Load entities, Load observables, or Load entities by observable from the context menu.
When creating groups in the graph, it is not possible to merge multiple groups into one.
If an ingestion process crashes while ingestion is still ongoing, data may not always sync to Elasticsearch.
Users can leverage rules to access groups that act as data sources, even if those users are not members of the groups they access through rules.
Running multiple outgoing feed tasks may cause the platform to consume a large amount of memory over time, because certain outgoing feeds such as HTTP download must load the data into memory in order to make it available to feed consumers.
Known issue with Elasticsearch 7: “Data too large”#
Since release 2.9.0, the platform comes bundled with
Elasticsearch (ES) 7.9.1. ES 7 adds a new real memory
circuit breaker that causes ES nodes to respond with a
circuit_breaking_exception error when it detects that
memory use has reached 95% of the totally available JVM
heap.
Because of this change, you may encounter issues related to available memory where previously at the same workloads, ES would appear to run smoothly.
If your plaform is encountering issues related to
Elasticsearch responding with a
circuit_breaking_exception error, you can do the
following to mitigate:
Increase available memory for ES#
The circuit_breaking_exception error occurs only when ES
detects that you are about to go over a memory use threshold
that would cause it to fail.
Increase the amount of memory available to ES, or move it to its own host where it does not compete with the platform for resources to keep your ES nodes running.
(Not recommended) Disable the “real memory circuit breaker”#
Warning
This may allow ES to reach an out of memory state and fail.
(Not recommended) To disable the “real memory circuit breaker”, set the
indices.breaker.total.use_real_memory parameter in your ES
configuration to false.
This allows ES to use the ES 6 parent circuit breaker instead, but disables the safety guarantees that the real memory circuit breaker provides.
Security issues and mitigations#
To see a detailed list of security issues and their mitigations, go to All security issues and mitigations.
ID |
CVE |
Description |
Severity |
Status |
Affected versions |
|---|---|---|---|---|---|
- |
Users with only |
1 - LOW |
|
2.10.x and earlier. |
|
- |
Users with only |
2 - MEDIUM |
|
2.10.x and earlier. |
|
- |
Users with only |
2 - MEDIUM |
|
2.10.x and earlier. |
|
- |
Users without direct assignment to a listed workspace can view details they should not see. |
1 - LOW |
|
2.10.x and earlier. |
|
- |
Users with only |
2 - MEDIUM |
|
2.10.x and earlier. |
|
- |
Users with only |
2 - MEDIUM |
|
2.10.x and earlier. |
|
- |
Users could create entities in Source Groups indirectly assigned through Groups, instead of only being able to create entities in Groups they are directly assigned to. |
1 - LOW |
|
2.10.0 and earlier. |
|
- |
Users could create entities in Source Groups indirectly assigned through Groups, instead of only being able to create entities in Groups they are directly assigned to. |
1 - LOW |
|
2.9.1 and earlier. |
|
- |
SVG file upload could allow cross-site scripting (XSS) |
2 - MEDIUM |
|
2.9.1 and earlier. |
|
- |
HTML injection through the GUI |
2 - MEDIUM |
|
2.9.1 and earlier. |
|
CairoSVG is vulnerable to regular expression denial of service |
2 - MEDIUM |
|
2.9.1 and earlier. |
||
PySAML2 improper verification of cryptographic signature |
2 - MEDIUM |
|
2.9.1 and earlier. |
||
Pillow is vulnerable to buffer overflows |
2 - MEDIUM |
|
2.9.1 and earlier. |
Download#
For more information about setting up repositories, refer to the installation documentation for your target operating system.
EclecticIQ Platform and dependencies for CentOS and RHEL |
Note The platform dependencies URL for versions 2.9 and later is
|
|---|---|
EclecticIQ Platform extensions |
|
Upgrade#
The following diagram describes the upgrade path you should take depending on the platform version you are upgrading from.
For example:
You can upgrade from version 2.9.1 of the platform to 2.10.0 directly,
To upgrade from 2.4.0 to 2.10.0, you must first upgrade to 2.5.0, then upgrade from 2.5.0 to 2.10.0.
When upgrading from 2.8.x and earlier to 2.9.x and later:
You must run the pre-upgrade script to allow it to work with Elasticsearch 7.9.1.
You must run the pre-upgrade script on the platform version you are upgrading from.
For example, when upgrading from 2.8.0 to 2.10.1, you must run the pre-upgrade script on the platform while it is running version 2.8.0.
Upgrade diagram#
From 2.5.0, the upgrades paths have been tested using the EclecticIQ Platform install script compiled by Rundoc.
The script only supports:
Single machine installs.
Instances installed using the platform install script.
and does not support platform instances installed in distributed environments.