Release notes 2.9.0

EclecticIQ Platform 2.9.0 is a minor release. It contains new features, improvements to existing functionality, as well as bug fixes.
What makes this release special is that it touches nearly every aspect of the platform – even the feeds you can access with it.

With this release we’ve reached the end of our development track to upgrade the ingestion engine of the platform.
This last step involved strapping our fast and scalable ingestion process on top of quuz - the task manager we introduced in release 2.7.
As a result, the ingestion workload not only becomes more evenly distributed across workers, but the entire process is more resilient against glitches in your infrastructure, ensuring a more robust and smoother ingestion of data.

You will find this release is packed with updates for many on-going initiatives. We’ve once again improved the platform’s usability, making it easier to search, create content and upload files.
We’re also taking the next step towards interoperability with STIX and TAXI 2.1 by supporting exchange Indicator and ingestion and exchange of Observed Data objects.
Lastly, upgrading the platform and tracking its health status becomes a lot easier with EclecticIQ Platform 2.9.0.

However, the biggest change that accompanies this update is a completely new Intelligence offering, created by our in-house team of threat intelligence analysts.
The platform now comes preconfigured with two brand-new EclecticIQ Intelligence Feeds, curated for primary threats and optimized for CTI operations:

• The Open Sources Feed is bundled with the platform, and it is available to all EclecticIQ Platform customers.

• The Commercial Sources Feed requires an EclecticIQ Intelligence subscription, and it is a cost-effective add-on that augments your threat landscape visibility.

Your Customer Success Manager can tell you all about these feeds and help you gain access.

We hope you enjoy reading these release notes – once again accompanied by short feature videos for your convenience – and watching the quick tour video from the team.

• For more information about upgrading Elasticsearch to version 7.9.1, see Upgrade below.

• For more information about new features and functionality, see What's new below.

• For more information about enhancements and improvements, see What's changed below.

• For more information about bugs we fixed, see Important bug fixes below.

• For more information about security issues we addressed, see Security issues and mitigation actions below.

Download

Follow the links below to download installable packages for EclecticIQ Platform 2.9.0 and its dependencies.
For more information about setting up repositories, refer to the installation documentation for your target operating system.

Upgrade

Upgrade paths from release 2.0.x(.x) to 2.9.0:

EclecticIQ Platform upgrade paths to release 2.9.0

From release 2.5.0, the upgrades paths are tested using the EclecticIQ Platform install script compiled by Rundoc.

The script does not yet support upgrading the platform in a distributed environment.
As of now, it can only upgrade a platform instance that:

• Runs on a single machine.

• Was installed using the platform install script.

What's new

• STIX 2.1: identities, indicators, observed data
  The scope of STIX 2.1 support in the platform is based on the Threat Intelligence Platform (TIP) persona defined in the STIX/TAXII™ 2.0 Interoperability Test Document part 1, version 1.1 and part 2, version 1.0.
  Besides STIX 2.1 indicators, the platform now supports also the following STIX domain objects (SDOs) and STIX cyber-observable objects (SCOs):
  
  

  STIX 2.1 object	STIX 2.1 type	Platform support	From release
  Indicator	SDO	Ingestion (incoming feeds)	2.8.0
  Indicator	SDO	Dissemination (outgoing feeds)	2.9.0
  Observed data	SDO	Ingestion (incoming feeds) Dissemination (outgoing feeds)	2.9.0
  Identity	SDO	Ingestion (incoming feeds) Dissemination (outgoing feeds)	2.9.0
  autonomous-system number	SCO	Ingestion (incoming feeds) Dissemination (outgoing feeds)	2.9.0
  domain-name	SCO	Ingestion (incoming feeds) Dissemination (outgoing feeds)	2.9.0
  email-addr	SCO	Ingestion (incoming feeds) Dissemination (outgoing feeds)	2.9.0
  email-message	SCO	Ingestion (incoming feeds) Dissemination (outgoing feeds)	2.9.0
  file	SCO	Ingestion (incoming feeds) Dissemination (outgoing feeds) File hashes: MD5 SHA-1 SHA-256 SHA-512	2.9.0
  ipv4-addr	SCO	Ingestion (incoming feeds) Dissemination (outgoing feeds)	2.9.0
  ipv6-addr	SCO	Ingestion (incoming feeds) Dissemination (outgoing feeds)	2.9.0
  mac-addr	SCO	Ingestion (incoming feeds) Dissemination (outgoing feeds)	2.9.0
  mutex	SCO	Ingestion (incoming feeds) Dissemination (outgoing feeds)	2.9.0
  network-traffic	SCO	Ingestion (incoming feeds) Dissemination (outgoing feeds)	2.9.0
  software	SCO	Ingestion (incoming feeds) Dissemination (outgoing feeds)	2.9.0
  ssdeep	SCO	Ingestion (incoming feeds) Dissemination (outgoing feeds)	2.9.0
  url	SCO	Ingestion (incoming feeds) Dissemination (outgoing feeds)	2.9.0
  user-account	SCO	Ingestion (incoming feeds) Dissemination (outgoing feeds)	2.9.0
  windows-registry-key	SCO	Ingestion (incoming feeds) Dissemination (outgoing feeds)	2.9.0

  

• Incoming feeds: EclecticIQ Intelligence Feeds
  The platform now displays two preconfigured EclecticIQ Intelligence Feeds: EclecticIQ Open Sources Feed and EclecticIQ Commercial Sources Feed.
  These feeds provide reliable intelligence curated by the EclecticIQ Fusion Center team and are optimized for CTI operations.
  The Open Sources Feed is available for all EclecticIQ Platform customers; the Commercial Sources Feed requires an EclecticIQ Intelligence subscription.
  Configure these feeds with your EclecticIQ Fusion Center credentials before use.
  Contact your customer success manager for more information, or read the documentation.

• Observables: JA3
  The platform now supports JA3 observables.
  The platform ingests, recognizes, and processes JA3 observables in incoming data packages; it supports dissemination using generic content types such as EclecticIQ JSON and CSV.
  JA3 fingerprints TLS handshakes between client and server. This helps identify encrypted communication between a specific client and the server it connects to.
  JA3 strings can point to specific client applications. This can help select applications that are eligible for whitelisting in environments requiring strict security.
  Pairing JA3 and JA3S strings makes it possible to detect malware by identifying how it communicates, regardless of what it communicates to.

  The platform supports the following JA3 observable types:

  Observable type	Hash type	Description
  ja3-full	JA3	The decimal values of the bytes in the client-generated Client Hello packet for the following fields: TLSVersion,Ciphers,Extensions,EllipticCurves,EllipticCurvePointFormats Platform support: STIX 1.2: the platform does not support this hash type in STIX 1.2 format because, the hash type is not defined in the STIX 1.2 data model specification. STIX 2.1: the platform does not support this hash type in STIX 2.1 format, yet.
  ja3-hash	JA3 hash	The MD5 hash of the full JA3 string. Platform support: STIX 1.2: the platform does not support this hash type in STIX 1.2 format, because the hash type is not defined in the STIX 1.2 data model specification. STIX 2.1: the platform does not support this hash type in STIX 2.1 format, yet.
  ja3s-full	JA3S	The decimal values of the bytes in the server-generated Server Hello packet for the following fields: TLSVersion,Cipher,Extensions Platform support: STIX 1.2: the platform does not support this hash type in STIX 1.2 format, because the hash type is not defined in the STIX 1.2 data model specification. STIX 2.1: the platform does not support this hash type in STIX 2.1 format, yet.
  ja3s-hash	JA3S hash	The MD5 hash of the full JA3S string. Platform support: STIX 1.2: the platform does not support this hash type in STIX 1.2 format, because the hash type is not defined in the STIX 1.2 data model specification. STIX 2.1: the platform does not support this hash type in STIX 2.1 format, yet.

• Observables: authentihash, imphash, PE rich header hash, SSDEEP, vhash
  Besides JA3, this release also features support for authentihash, imphash, PE rich header hash, SSDEEP, and vhash observables.
  The platform ingests, recognizes, and processes these hash-type observables in incoming data packages; it supports dissemination using generic content types such as EclecticIQ JSON and CSV.
  The platform supports the following new hash-type observables:

  Observable type	Hash type	Description
  hash-authentihash	Authentihash	The hash represents Microsoft Authenticode hash. Authenticode is a hash signature applied to Microsoft PE (portable executable) files. Authenticode-signed files include digitally signed certificates. Microsoft AppLocker rules use Authenticode hashes to allow or prevent apps from running. Platform support: STIX 1.2: the platform supports this hash type in STIX 1.2 format. STIX 2.1: the platform does not support this hash type in STIX 2.1 format, yet.
  hash-imphash	Imphash	An imphash, or import hash, uniquely fingerprints the library and/or API names that a Windows Portable Executable (PE) file requests from other files and resources. Imphashes represent the entire list of imported resources, as well as their loading order. Malware targeting Windows-based systems invokes and loads external libraries and API functions in a similar way. Therefore, imphashes uniquely fingerprint specific malware, and they help identify these threats. Platform support: STIX 1.2: the platform supports this hash type in STIX 1.2 format. STIX 2.1: the platform does not support this hash type in STIX 2.1 format, yet.
  hash-rich-pe-header	Rich PE header hash	The hash represents a specific rich signature header by Microsoft. The header is added to PE files built and compiled with Microsoft Visual Studio. The header contains information that defines the build environment, such as the components and the tooling, used to build the resulting PE file. Different PE files sharing the same components and built with the same tools share the same rich header. The r ich PE header hash can help identify infrastructure and tools used to build malware targeting Microsoft environments. It can help relate malware, as well as track threat actors. However, threat actors can tamper with the data in the rich header to plant a false flag. Platform support: STIX 1.2: the platform supports this hash type in STIX 1.2 format. STIX 2.1: the platform does not support this hash type in STIX 2.1 format, yet.
  hash-ssdeep	ssdeep	ssdeep enables computing context triggered piecewise hashes (CTPH), also known as fuzzy hashes. This enables comparing files to look for similarities. ssdeep helps identify what two different pieces of malware have in common. It is a useful tool when investigating malware attribution. Platform support: STIX 1.2: the platform supports this hash type in STIX 1.2 format. STIX 2.1: the platform supports this hash type in STIX 2.1 format.
  hash-vhash	Vhash	According to VirusTotal definition, a vhash is "an in-house similarity clustering algorithm value, based on a simple structural feature hash allows you to find similar files". Similarly to other hashes, it fingerprints known pieces of malware to make it easier to identify them. However, threat actors can work around vhashes by recompiling their malware, so that it gets hashed to a new, different vhash. Platform support: STIX 1.2: the platform supports this hash type in STIX 1.2 format. STIX 2.1: the platform does not support this hash type in STIX 2.1 format, yet.

  The formats of these hashes are not distinctive enough.
  Since their format is identical to other hash types such as MD5 or SHA, it is not possible to target them accurately in unstructured text.

  Therefore, the platform can extract these hashes only from structured text data sources.

• Lists: add or remove columns
  It is now possible to add columns to and remove columns from the lists of entities and observables displayed in the Browse, Production, Discovery and Exposure views in the platform.

• Export and import users, groups, and roles
  Platform instances grow and evolve organically over time. Besides the intelligence data corpus they store, they hold a wealth of information that is just as valuable: user account profiles, user groups, platform roles and related permission sets.
  If you plan to install the platform to a new or to a different machine, and if you want it to be accessible by users from an existing platform instance, you can export users, groups, and roles from that instance.
  You can then import them into the new target instance, so that users can sign in and start their in-platform activities as smoothly as possible.

• Reports: export as PDF
  In the platform, it is now possible to export your reports in PDF format, which is more suitable for non-technical audiences.
  

What's changed

Improvements

• Dataset creation workflow: edit details in floating panel
  When creating new datasets or updating them, you had to edit fields on a specific page of the platform.
  This meant that if your workflow required you to work on or consult a different page, you needed to leave your editing work temporarily.
  The edit detail fields are now displayed in a floating pane, which keeps them visible and editable throughout your workflow.
  This also applies to other objects in the platform such as uploads, tasks, workspaces and rules.
  

• File upload: upload dialog in modal window
  The dialog for uploading files to the platform has now been placed in modal window so that it does not interfere with your view of the other files on the page.
  It has also been improved regarding interaction clarity and consistency with the rest of the platform.
  

• Manual file upload: max 100 MB per file
  It is now possible to set custom values to cap the maximum size of the files that users manually upload to the platform.
  The option enables adjusting the size to accommodate uploading files as large as 100 MB.
  For example, analysts may occasionally want to upload a 20 MB or a larger PDF to convert it to a report, and eventually to structured intelligence.

• Preview search results
  Searching within the platform has been optimized even further.
  The platform now displays provisional search results as you type your query.
  That way you can tune your query on the fly and only submit it when you are confident that it will retrieve the information you want.
  
  

• Date display format: locale-specific displayed first
  Previously, dates were displayed in a long form such as "Last Thursday at 7:30 AM". To see the locale-specific format, such as "16/07/2020 7:30 PM", you had to hover your mouse over the displayed date. This has now been reversed. The locale-specific date is displayed first and the long form on hovering.

• Platform notifications: periodic notification cleanup
  The platform generates notifications to inform about the outcome of a number of different events related to feeds, enrichers, workspaces, and user tasks.
  Over time, the amount of generated notifications grows, until it may start affecting the performance of the notification index endpoint.
  Now, a scheduled job automatically checks for old notifications to clean up, before they can impact performance.
  Notification history for platform users is limited to the 1000 most recent notifications.

• Ingestion: faster ingestion thanks to SQL query optimization
  The Entity.group_description field was unused, and it did not add intelligence value.
  Removing it helped speed up ingestion: a SQL query that calculated the field could become slow and expensive.
  Since the field was unnecessary, the specific SQL query became redundant as well.
  As a result, ingestion has become faster and smoother.

• Workspace view: edit icon made consistent with rest of platform
  To edit, archive and delete a workspace, you previously clicked a in the top left of the workspace.
  This icon has been replaced with a to conform with the rest of the platform.

• Keep an eye on incoming data queues and pending packages
  The platform helps system administrators keep a sharper eye on incoming data traffic. Statsite now aggregates metrics to monitor the size of Celery queues, as well as the total number of incoming packages that are pending ingestion.
  System administrators can use the information to assess how to effectively and conveniently scale their infrastructure, depending on incoming load.

Deprecations

Elasticsearch 6.8 end of life is November 20th 2020.
To have access to Elasticsearch product updates and security patches after this date, EclecticIQ Platform 2.9.0 includes an upgrade to Elasticsearch 7.9.1.
As a result, Elasticsearch versions earlier than 6.8.0 and Elasticsearch index versions earlier than 6.0.0 are no longer supported.

For more information about the Elasticsearch upgrade to version 7.9.1, see Prepare upgrading to Elasticsearch 7 for CentOS and Prepare upgrading to Elasticsearch 7 for RHEL.

Deprecated operating systems

As of release 2.9.0, support for the following operating systems is deprecated:

• CentOS 7.8 (2003)

• CentOS 7.7 (1908)

• CentOS 7.6 (1810)

• Red Hat Enterprise Linux 7.8

• Red Hat Enterprise Linux 7.7

• Red Hat Enterprise Linux 7.6

• Ubuntu Server 16.04 Xenial Xerus

CentOS 7.6 (1810), 7.7 (1908), and 7.8 (2003), and Red Hat Enterprise Linux 7.6, 7.7, and 7.8 are compatible with release 2.9.x.
However, they are not supported.

Ubuntu Server is no longer supported.

The following operating systems are supported:

• CentOS 7 latest release – currently CentOS 7.9 (2009)

• Red Hat Enterprise Linux 7 latest release – currently Red Hat Enterprise Linux 7.9

Removed Entity.group_description field

The Entity.group_description field as been removed from the platform entity data structure.

The field was originally implemented to contain the informative description of an ingested STIX XML package.
STIX incoming feeds never required it, and the information the fields hold has no intelligence value.

Removing Entity.group_description does not affect ingested platform entities, and it does not alter the ingestion, processing, and indexing mechanisms of the platform.

Important bug fixes

This section is not an exhaustive list of all the important bug fixes we shipped with this release.

• File names with file extension extracted as observables
  File names including a file extension – for example, malware_sample.zip – would produce a file observable, as well as a domain observable.
  Now the platform correctly handles them as file names, and it extracts only file observables.

• Entities created in the graph using a beta feature are not visible in the timebar
  Users who turned on the beta feature enabling entity creation in the graph would not be able to see in the timebar any entities generated using the beta feature.
  Now these entities are correctly displayed in the timebar: the timebar shows when the entities were created, and the entities are visible when they are within the vertical timebar borders.

• RSS feed fails to download
  A specific RSS feed would fail to download because of a datetime error in the datetime comparison check.
  This issue was successfully addressed by attaching UTC time zone values to naive datetime timestamps.

• Flashpoint reports with formatting issues
  Table included in a Flashpoint report body would be formatted incorrectly, resulting in awkward rendering.
  Now tables in the report body are formatted correctly, and they render as expected.

• PDF attachments: attachments no longer missing in outgoing feeds
  Previously, when a report without attachments referred to a report with attachments in an outgoing feed, the attachments in the latter would sometimes go missing.
  This was because the two reports were sometimes sent as separate packages and the placeholder in the first package was not being replaced correctly when the second package arrived.
  This has now been fixed.

• Read-only users could access GUI options to change maliciousness
  Platform users with read-only access to observables could access the drop-down menu options to change the maliciousness confidence level of observables.
  They would not be able to save their changes; instead, they would receive an error message: "Error loading details for observable".
  Now users with read-only access are not shown GUI options that they do not have access to, based on their permissions.

• Only an admin user can assign allowed sources to a group
  Non-admin platform users would not be able to add incoming feeds or enrichers as allowed data sources to the user groups they belong to, despite these users having the modify-group and read-sources permissions.
  This occurred because incoming feeds an enrichers have no owner. Therefore, only admin users can add them as allowed sources to user groups.
  Now users with the read-sources permission can see lists of incoming feeds and enrichers they have access to.
  If they have also the the modify-group permission, they can correctly add incoming feeds and enrichers as allowed data sources to the user groups they belong to.

• Built-in help documentation not available in the platform as a hosted solution
  The built-in help documentation was not available in the platform as a hosted solution.
  Now the built-in help documentation is available also in the platform as a hosted solution.

Known issues

• When you configure the platform databases during a platform installation or upgrade procedure, you must specify passwords for the databases.
  Choose passwords containing only alphanumeric characters (A-Z, a-z, 0-9).
  Do not include any non-alphanumeric or special characters in the password value.

• Systemd splits log lines exceeding 2048 characters into 2 or more lines.
  As a result, log lines exceeding 2048 characters become invalid JSON.
  Therefore, Logstash is unable to correctly parse them.

• When more than 1000 entities are loaded on the graph, it is not possible to load related entities and observables by right-clicking an entity on the graph, and then by selecting Load entities, Load observables, or Load entities by observable.

• When creating groups in the graph, it is not possible to merge multiple groups to one.

• In case of an ingestion process crash while ingestion is still ongoing, data may not always sync to Elasticsearch.

• Users can leverage rules to access groups that act as data sources, even if those users are not members of the groups they access through rules.

• Between consecutive outgoing feed tasks, the platform may increase resource usage.
  This may produce very high memory consumption over time.

Security issues and mitigation actions

The following table lists known security issues, their severity, and the corresponding mitigation actions.
The state of an issue indicates whether a bug is still open, or if it was fixed in this release.

For more information, see All security issues and mitigation actions for a complete and up-to-date overview of open and fixed security issues.

Contact

For any questions, and to share your feedback about the documentation, contact us at [email protected] .

^ back to top