Release notes 2.5.0

Product

EclecticIQ Platform

Release version

2.5.0

Release date

15 Oct 2019

Summary

Minor release

Upgrade impact

Medium

Time to upgrade

~30 minutes to upgrade

  • From the previous release

  • Using the installation script

  • For an instance running on one machine.

Time to migrate

  • PostgreSQL database: ~less than 5 minutes per million entities

  • Elasticsearch database: ~25 minutes per million entities

  • Neo4j database: ~less than 5 minutes per million entities.

EclecticIQ Platform 2.5.0 is a minor release. It contains new features, improvements to existing functionality, as well as bug fixes.

With this release we are improving the platform in many ways, while also making necessary preparations for the future.
EclecticIQ Platform 2.5.0 offers:

  • A completely rebuilt graph engine.

  • Upgrades of all internal databases: PostgreSQL, Elasticsearch , Neo4j, and Redis.

  • An improved user experience, thanks to multiple usability tweaks.

  • New API access tokens to programmatically access and consume the platform services we expose through the public API endpoints.

  • Easier configuration of the extension SDK.

  • Easier installation and maintenance of distributed deployments.

All these changes together make working with EclecticIQ Platform on a daily basis easier, faster, and more efficient.

Check out the new bite-size video from the team for a short rundown of these highlights:

images/download/attachments/86441777/release2.5.0-screen.png


Besides the many enhancements, this release requires updating the following apps to the versions reported below, or to later ones:

  • EclecticIQ IBM Resilient App, version 1.1.4.

  • EclecticIQ IBM QRadar App, version 1.3.3.

  • EclecticIQ Splunk and Splunk Enterprise Security App, version 2.5.0.

  • EclecticIQ Splunk Phantom App, version 1.2.0.

To update these apps, follow the instructions in the relevant app documentation.
For assistance, log a service request, or contact our support team .

Download

Follow the links below to download installable packages for EclecticIQ Platform 2.5.0 and its dependencies.
For more information about setting up repositories, refer to the installation documentation for your target operating system.

EclecticIQ Platform and dependencies
for CentOS and RHEL

EclecticIQ Platform and dependencies
for Ubuntu

EclecticIQ Platform extensions

Upgrade

Upgrade paths from release 2.0.x(.x) to 2.5.0:

images/download/attachments/30736396/eiq-tip-upgrade-paths.png

Dependency upgrades

Dependency

Upgraded from

Upgraded to

Elasticsearch

5.6.15

6.8.3

Kibana

5.6.15

6.8.3

Logstash

5.6.15

6.8.3

Neo4j

3.3.5

3.5.6

PostgreSQL

10.7

11.4

Redis

3.2.3

5.0.5

What's new

New features

  • Automated installation on single machines and distributed environments
    The new native package management system simplifies the installation and configuration procedures: fewer steps, more automation, less hassle.
    We rebuilt the whole platform installation process from the ground up, to address user feedback suggestions for simplification, and to support distributed installations.
    The new installation procedure enables performing distributed installations, where platform components are installed across multiple host machines.

    Due to the variety of environments, hardware and software equipment, and configurations, during installation the default network bindings for all databases are set to listen to all incoming connections.
    While these default settings may work in your environment, make sure you carry out the following actions:

    • Edit the default network bindings for the platform databases, and set them to specific IPs within the network that the databases and the platform can access.

    • Verify that the network that the databases and the platform can access is protected to prevent intrusion, and to detect potentially anomalous activities.



    Database or service

    Path and file

    Field and value

    Notes

    Elasticsearch

    /etc/systemd/system/elasticsearch.service.d/20-eclecticiq.conf

    [Service]
    Environment=BINDING_ADDRESS=0.0.0.0

    Set BINDING_a specific IP address to a specific IP address within the network that Elasticsearch and the platform can access.

    For more information, see Network Settings.

    Neo4j

    /etc/eclecticiq-neo4j/neo4j.conf

    dbms.connectors.default_listen_address=0.0.0.0

    Set dbms.connectors.default_listen_address to a specific IP address within the network that Neo4j and the platform can access.

    For more information, see Configure connectors and dbms.connectors.default_listen_address.

    PostgreSQL

    /etc/eclecticiq-postgres/pg_hba.conf


    TYPE DATABASE USER ADDRESS METHOD
    host  all  all  0.0.0.0/0  password

    Set a specific IP address to a specific IP address within the network that PostgreSQL and the platform can access.

    For more information, see The pg_hba.conf File.

    Redis

    /etc/eclecticiq-redis/redis.conf

    bind 0.0.0.0

    Set bind to one or more specific IP addresses within the network that Redis and the platform can access.

    For more information, see Redis security and redis.conf.

  • Distributed health-check
    From this release, systemd replaces Supervisor as a service and process manager.
    This change affects also how health monitoring works: the focus shifted from checking if a process exists to polling native health and status endpoints, as well as checking health and status at application-level.

  • API tokens
    API tokens enable users to programmatically authenticate and to connect to the platform without passing their user name and password credentials.
    API tokens are opaque, and they enable accessing the platform API without using two-factor authentication each time the script tries to connect.

  • External interface maintenance
    We addressed feedback and maintenance requests from consumers of HTTP APIs and extension SDK.

  • Brand new graph engine
    We took the graph, tore it down, and built it back up. This enabled us to remove the technical debt that was hindering us from adding new functionality to our graph capabilities. Most of the work happened under the hood: the new graph retains almost the same look and feel as its predecessor.
    After this rewrite, the graph keeps its familiar appearance, but its brand new engine enables adding new functionality with high speed and low risk. With this upgrade the following improvements are now available:

    • Improved time selection functionality

    • Unlimited undo/redo functionality

  • External references
    This feature enables filtering external references in the Graph and Neighborhood tabs.

  • Filter by Destination
    We made also it easier to see if you have disseminated intelligence to sharing communities or security controls. By adding a new destination option to the search filters, you no longer need to check your outbound feeds or individual entities one by one. Now you can use this filter to quickly see where intelligence has been disseminated to.

  • Responsive Dashboard
    You can now maximize your browser window and enjoy a dashboard view that spans across your entire screen.

New functionality

What's changed

Improvements

  • Search
    We improved search functionality to enhance the overall user experience.

  • Ingestion performance
    We restructured the ingestion process to make it more scalable.
    On the side, we also tightened and improved STIX validation when automatically adding relations to entities.

  • Entity creation in the detail pane
    The entity editor is now available in full screen mode, as well as in a detail pane.
    Now users can stay in the current view and at the same time they can create intelligence in the editor.

  • Workspaces endpoint
    We improved the overall performance of workspaces by tweaking the internal data flow in the platform.

Deprecations

  • Supervisor
    From this release systemd replaces Supervisor as a service and process manager.
    This improvement consolidates platform service and process management.
    We recommend users upgrade their previous versions of the platform to this release to address the vulnerability described in the EIQ-2018-0018 security advisory.

  • Proxy configuration through the GUI
    From release 2.5.0 , EclecticIQ Platform enables proxy configuration through a dedicated file.
    This operation requires root or sudo-level access to the platform instance through a SSH connection and a terminal.
    Previously, it was possible to configure proxy settings through the web-based GUI.
    For more information, see the updated documentation on proxy configuration for CentOS , RHEL , and Ubuntu .

Breaking changes

  • From release 2.5.0 , dynamic dataset search queries that include the the top-level type key – for example: type="indicator" – no longer work.
    The top-level type key is now replaced by data.type.
    This breaking change is related to the Elasticsearch upgrade, and to changes in the Elasticsearch index mapping structure following the upgrade.

    To repair this breaking change, replace type with data.type in your search queries.
    Search queries that look for data in the type key instead of data.type do not return any errors, and do not return any search results.

Important bug fixes

This section is not an exhaustive list of all the important bug fixes we shipped with this release.

  • Users could remove their own user profile from groups and allowed sources. This would result in these users having no access to the platform resources.
    It is no longer possible for users to remove their own profile from groups and allowed sources.

  • Users could deactivate their own user profile while deactivating multiple users in bulk. This would result in these users having no access to the platform resources.
    It is no longer possible for users to deactivate their own profile.

  • Deleting a rule would generate a JavaScript error in the browser, and the rule would not be correctly deleted.
    It is now possible to delete rules correctly without errors in the browser.

  • Workspace collaborators were not visible.
    Newly added collaborators to a workspace were not displayed. This has now been fixed.

  • Archived workspaces were visible in the workspace overview.
    Both active and archived workspaces were visible in the workspaces overview and in the drop-down menu.
    Now, archiving a workspace hides it in the drop-down navigation, and it is no longer possible to edit the archived workspace settings or to add collaborators.
    Datasets, graphs, entities, and tasks stored in archived workspaces remain searchable still show up in all intelligence, the same way they do now.
    You can always find and unarchive an archived workspace in the All workspaces view.

  • Email notifications for assigned users did not work correctly.
    Users could set up email notifications in the workspace settings. However, they would not receive any notifications.
    After setting up workspace email notifications, users now receive emails to inform them about relevant workspace task actions and events.

  • Entity tags were not synced correctly between two connected platform instances.
    The tags of entities being sent to another platform instance through an outgoing feed are now correctly synced and updated in the target platform instance.

  • Users were able to assign data sources to themselves.
    Users with the permission to modify groups were able to assign sources to themselves.
    Users who can modify groups, and who have non-admin access can no longer assign data sources to themselves.

  • Group admins could not modify their own roles.
    Users with the group admin role could not edit their own group roles. This has now been fixed.

  • The tool helping configure SAML authentication in the platform would return a HTTP 500 server error with no further details to help users.
    SAML key and certificate validation is stricter now. In case of invalid key and certificate files, an error message notifies users about the issue.

  • SAML users last login would not be shown in the user detail page.
    The Last logged in field would either be left empty, or it would display the previous time the user signed in without using SAML authentication. This has now been fixed.

  • EclecticIQ log files would not be set up correctly for log rotation.
    EclecticIQ logs were not rotated. To address this issue, we added a log rotation configuration.

Security issues and mitigation actions

The following table lists known security issues, their severity, and the corresponding mitigation actions.
The state of an issue indicates whether a bug is still open, or if it was fixed in this release.

For more information, see All security issues and mitigation actions for a complete and up-to-date overview of open and fixed security issues.

Known issues

  • When more than 1000 entities are loaded on the graph, it is not possible to load related entities and observables by right-clicking an entity on the graph, and then by selecting Load entities , Load observables , or Load entities by observable .

  • When creating groups in the graph, it is not possible to merge multiple groups to one.

  • In case of an ingestion process crash while ingestion is still ongoing, data is not synced to Elasticsearch .

  • Users can leverage rules to access groups that act as data sources, even if those users are not members of the groups they access through rules.

  • Between consecutive outgoing feed tasks, the platform may increase resource usage. This may result in an excessive memory consumption over time.

Contact

For any questions, and to share your feedback about the documentation, contact us at [email protected] .



^ back to top