Release notes 2.2.1

Product

EclecticIQ Platform

Release version

2.2.1

Release date

2018-08-23

Summary

  • Access control for observables is extended to search and graph

  • The feature to label observable relationships is renamed to

  • Bug fixes

Upgrade impact

Medium

Time to upgrade

~30 minutes using the rundoc install script on a single instance

Time to migrate

  • PostgreSQL database (mandatory): ~6 hours per million entities

  • Elasticsearch database (optional): ~1 hours per million entities

Highlights

EclecticIQ Platform 2.2.1 is a maintenance release containing a mix of maintenance bug fixes and features which were released with a patch release (2.2.0.1) including:

  • Access control for observable in search and graph

This release requires a data migration to upgrade the platform to this maintenance release, take care of the following:

  • Back up postgresql, neo4j, elasticsearch.

  • Upgrade the platform.

  • Restore the backed up databases.

Relevant documentation:

About versioning and platform upgrades

Platform versioning

EclecticIQ Platform versioning uses up to 4 digits separated by a dot (.)
Each digit holds semantic meaning based on its position in the version number reference format:

# Format
${int}.${int}.${int}.${int}
 
# Meaning
MAJOR.MINOR.MAINTENANCE.PATCH

Example

Release number

Release type

2.0.0.0

Major product release

2.1.0.0

Minor product release

2.1.3.0

Maintenance product release

2.1.3.6

Patch product release

While the first 3 digits are mandatory, the 4th digit may or may not be included in a release reference.
If a version does not include the 4th digit, it is the same as if the missing digit were zero (0).

Example

Referring to a hypothetical release 2.1.3 is the same as referring to release 2.1.3.0.

Upgrade paths

Starting from release 2.0.0, to upgrade EclecticIQ Platform x.x.x(.x) to the latest maintenance and patch releases, the platform instance to upgrade needs to be:

  • Either on the same major and minor version;

  • Or on the same major version and max one minor version behind.

Instance to upgrade

Eligible upgrade for the instance

MAJOR.MINOR.MAINTENANCE.PATCH

MAJOR.MINOR.LATEST_MAINTENANCE.LATEST_PATCH

MAJOR.MINOR.MAINTENANCE.PATCH

MAJOR.MINOR_+1.LATEST_MAINTENANCE.LATEST_PATCH

Example

Upgrade path from release 2.0.x(.x) to 2.2.1:

Instance to upgrade

Intermediate upgrade

Target upgrade

2.0.0

2.0.1

2.0.2

2.0.2.1

2.1.2.1

2.2.1

Upgrade path from release 2.1.x(.x) to 2.2.1:

Instance to upgrade

Intermediate upgrade

Target upgrade

2.1.0

2.1.1

2.1.2

2.1.2.1

None

2.2.1

Further on in this section you can read a short recap of the main new features shipping with EclecticIQ Platform 2.2.1.


What’s changed

Keylines update
  • KeyLines is the third party product that EclecticIQ Platform uses for the visualization of our graph functionality. Since there is a built in time lock for licenses expiring in 2018, an upgrade to the DRM license key was needed. In order to make this easier, we included the new DRM license key into the existing version 3.3.5. This is new DRM license key is valid until 2021. This change applies to platform release 2.0.0 and later.

Enhancements

In addition to the enhancements contained in the patch release, 2.2.1 release comes with the following:

  • The SAML configuration section in the /etc/eclecticiq/platform_settings.py platform configuration file has the following new parameters:

    • SAML_REQUEST_USE_POST_BINDING: Boolean switch to enable/disable HTTP binding for POST requests.
      To enable HTTP binding for POST requests, set SAML_REQUEST_USE_POST_BINDING = True.

    • SAML_CONFIGURE_MODE: Boolean switch to show/hide the guided SAML configuration page.

To access a page with guidelines and debugging tools to help you configure SAML:

  1. To enable SAML authentication, set the SAML_AUTH_ENABLED parameter to True:

    # Enables SAML authentication
    SAML_AUTH_ENABLED = True
  2. To enter SAML configuration mode, add the SAML_CONFIGURE_MODE parameter to the SAML configuration section, and set it to True:

    # Enables SAML config mode
    SAML_CONFIGURE_MODE = True

    SAML_CONFIGURE_MODE = True enables the SAML configuration mode.
    SAML authentication is not available in the platform while the configuration mode is enabled.
    To exit the configuration mode and to make SAML authentication available, remove SAML_CONFIGURE_MODE from /etc/eclecticiq/platform_settings.py, or set it to False.

  3. Go to https://${your-platform-instance-base-url}/private/saml/configure to display the guided SAML configuration page.
    It includes step-by-step instructions to guide you through all SAML-related parameters and variables.
    It generates the necessary platform metadata, and it enables live configuration updates.
    It also features a login test that prints data returned by the IDP (Identity Provider) to the terminal.

    • If SAML_AUTH_ENABLED is set to True, the Sign in with SAML button on the main sign-in page in the platform GUI is available.
      If SAML is set up and enabled for the platform, users can sign in with it.

    • If SAML_CONFIGURE_MODE is set to True, the Sign in with SAML button on the main sign-in page in the platform GUI is not available.
      If the SAML configuration mode is enabled, SAML is not available to users for sign-in.

    • After successfully setting up the SAML authentication mechanism, remove SAML_CONFIGURE_MODE from /etc/eclecticiq/platform_settings.py, or set it to False.

Guided SAML configuration page example
saml_live_configuration.html.j2 (sourced from EIQ platform-backend)

Author

Andreas

Commit

17a58f9f930d83ee862b731813ff472ea3994a37

Timestamp

February, 14, 2022 11:59 AM

Full path

eiq/platform/templates/saml_live_configuration.html.j2

Title

[SNYK] Upgrade packages and ignore issues with no upgrade path

Description

**Upgrade packages:**<br> `ipython==7.16.0` => `ipython==7.16.3` == no risk <br> `cairosvg==2.4.2`=> `cairosvg==2.5.2` == no risk <br> `jinja2==2.10.1` => `jinja2==2.11.3` == no risk<br> `pillow==7.2.0` => `pillow==8.3.2` == no risk <br> `pygments==2.6.1` => `pygments==2.7.4` == no risk <br> <br> **Snyk Ignore:** <br> _Removed issues that no longer affect our product._<br> Increase ignore time for following issues:<br> snyk:lic:pip:html2text:GPL-3.0 - can't be applied for 2.9<br> SNYK-PYTHON-PIP-609855 - can't upgrade PIP due to incompatibility with credential escaping<br> SNYK-PYTHON-PIP-1278135 - can't upgrade PIP due to incompatibility with credential escaping<br> SNYK-PYTHON-DATEPARSER-1063229 - no fix available<br> SNYK-PYTHON-CELERY-2314953 - fix can't be apply due to incompatibility with python 3.6<br> SNYK-PYTHON-PILLOW-2329135 - fix can't be apply due to incompatibility with python 3.6<br> SNYK-PYTHON-PILLOW-2331905 - fix can't be apply due to incompatibility with python 3.6<br> SNYK-PYTHON-PILLOW-2331907 - fix can't be apply due to incompatibility with python 3.6<br> SNYK-PYTHON-PILLOW-2331901 - fix can't be apply due to incompatibility with python 3.6<br> SNYK-PYTHON-PILLOW-2397241 - fix can't be apply due to incompatibility with python 3.6<br> SNYK-PYTHON-CRYPTOGRAPHY-1070544 - can't apply fix risk accepted SNYK-PYTHON-PYSAML2-1063038 - can't apply fix risk accepted SNYK-PYTHON-PYSAML2-1063039 - can't apply fix risk accepted See merge request engineering/platform-backend!6465

SAML can be configured by exchanging metadata files between the Service
Provider (this EclecticIQ Platform) and the Identity Provider (the SSO
point for users).
 
If an INTERNAL SERVER ERROR occurs during configuration, please
inspect the eiq-backend-web.log for details.
 
1: Set SAML_METADATA_CONTACT_PERSON, SAML_METADATA_ORG for your
organization.
 
2: Make sure the server domain is configured correctly so that it
matches what a user uses. Note that this domain is assumed to use HTTPS
and cannot contain a port number. Go to the UI -> System Settings ->
General to set the domain.
 
3. If required, set the filenames for SAML_ENC_KEY and
SAML_ENC_CERT as the key pair to use on the platform side for
signing and encrypting the authentication request. Also set the
flags SAML_WANT_ASSERT_SIGNED and SAML_WANT_RESPONSE_SIGNED as
needed.
 
4. Upload the Platform Metadata to the IDP.
 
5. Download the IDP Metadata and place it in the Platform file
system. Then set SAML_IDP_METADATA to point to the file, like so:
{"file": "/etc/eclecticiq/saml-idp-metadata.xml"}
Note that a url can also be used:
{"url": "https://idp-service/metadata"}
 
6. Set SAML_IDP_ENTITYID so that it exactly matches the
"entityID" attribute of the "EntityDescriptor" element in the
IDP metadata file.
 
7. Set SAML_SIGN_AUTHN_REQ as required and
SAML_REQUEST_USE_POST_BINDING if the authn request should use
POST instead of Redirect.
 
8. Set SAML_AUTH_ENABLED to true and test the login. If the request is
denied by the IDP this might be caused by invalid or missing signatures,
or using redirect binding while POST binding is required.
 
9. If the IDP accepted the login, login attempt info will show
up. The Platform will try to find or create a user, if it has
the available information. This information must be provided by
the IDP through mapped attributes:
 
- SAML_USER_USERID_ATTR must point to the username field
 
- SAML_USER_EMAIL_ATTR must point to the email field
 
- SAML_USER_ROLES_ATTR can contain zero or more Roles that are
specified in the Platform. Default roles in the platform are
"Threat Analyst", "Team Lead", and "System Admin". Note that
the SAML_CASE_SENSITIVE_MATCHING flag applies to finding
matching roles and groups.
 
- Same for SAML_USER_GROUPS_ATTR. A user must be a member of at
least one group. The default group on a new platform
installation is "Testing Group".
 
- Either SAML_USER_IS_ADMIN_ATTR or SAML_ADMIN_ROLE_GROUP_NAME
can be used to flag users as Platform admin. The
SAML_ADMIN_ROLE_GROUP_NAME should match one of the roles present
in the SAML_USER_ROLES_ATTR.
 
- FIRSTNAME and LASTNAME are optional fields.
 
In the Login Attempt Info section all the data the Platform
received from the IDP is displayed. This can be used for
debugging attribute mapping issues.
 
10. If everything works as intended, copy the "Current Configuration"
at the bottom of this page into platform_settings.py.


Important bug fixes

The following section gives an overview of the most important bug fixes to provide context and scope.

  • Delivery of feed packages works as expected with basic authentication. ( 20978)

  • The Ingested packages tab on the incoming feed detail pane does not hang and freeze if the feed has a very large number (millions) of packages in the ingested queue and running in the background.

  • Pagination works as expected in the Incoming Feeds details page.



Known issues

  • A user can create rules to access data sources that would normally be disallowed for that user. (16142)

  • If a user manually adds a sighting to an entity as a characteristic – in the entity editor, under Characteristics, select Characteristic > Sighting – the manually added sighting is not counted in the Exposure view.

  • If a user changes the title of an entity on the entity detail pane, title field, using the quick edit option , the title change is not reflected on the entity result view table under Browse, Production, Discovery, or Exposure.
    This is due to the Elasticsearch index requiring a few seconds to update, apply the change, and make it available to the UI.

  • If a user changes or resets their password, they do not receive an email message to confirm the action.

  • If a user creates an outgoing feed, runs it at least once to publish content, and then changes the feed transport type from one that pushes content – for example, Send email – to one that polls content – for example, TAXII poll – or vice versa, they receive an error message:
    Feed configuration error error in transport or content type
    It is still possible to edit the feed configuration, but it is not possible to run it successfully any longer.

  • When an entity is loaded on the graph, the graph does not update entity data to reflect any changes applied to the entity while it is loaded on the graph.

  • On the left side navigation bar, the UI button to allow user access to the graph is not available to users without the read-graph permission.

  • In the UI, work-in-progress state and view state may sometimes not remember correctly user input or user actions.
    Therefore, when a user leaves a view and then goes back to it, they may lose any input data or any changes to applied to, for example, filters, sorting, or pagination.

  • A user is not able to delete feeds containing large number (eg. 1 million) of entities.

  • Workspace correlation in Entity Rules is very slow and gets killed with soft timeout limit.

  • In Entity rules, filters only displays from the current screen.

  • The summary of digest is not displayed properly as the text is truncated automatically.

  • Wrong e rror Is displayed when wrong certificate is set in live configuration for the SAML tool .

  • SAML login is displayed in the login screen if values are modified in live configuration.

  • Test login throws an error if SAML_AUTH_ENABLED is not enabled.


Contact

For any questions, and to share your feedback about the documentation, contact us at [email protected] .



^ back to top