Playbooks | 3.x | Changelog#

Current#

Adds:

  • Support for deploying the Agentic AI / MCP (Model Context Protocol) server for IC 3.8 and later. Enabled by default via EIQ_AGENTIC_ENABLED="true". Set to "false" to skip MCP deployment. The MCP server requires AI provider credentials to be functional.

  • Support for Elasticsearch and Kibana 9.x for IC 3.8 and later. IC 3.2–3.7 continue to use ES/Kibana 8.15. Major-version upgrades transit an interim ES 8.19 hop automatically.

  • New playbook pb-kibana-upgrade-data.yml, wired into pb-full-install-upgrade.yml. When upgrading across a Kibana major boundary, it installs and briefly starts an interim Kibana so its saved-object metadata migrates cleanly before the final upgrade. No-op on fresh installs and same-major upgrades.

  • Support for PostgreSQL 18 on IC 3.8 and later. IC 3.2–3.7 continue with PostgreSQL 14. Upgrades transit PG 14 as the interim version.

  • Support for Python 3.14 on IC 3.8 and later (via the eclecticiq-python3.14 package). IC 3.4–3.7 use Python 3.11; earlier versions continue on Python 3.8.

  • ZSWAP compressed swap cache, enabled by default. Configurable via:

    export EIQ_ZSWAP_ENABLED="true"          # default: true
export EIQ_ZSWAP_MAX_POOL_PERCENT="20"   # default: 20

    Requires swap to be enabled. Uses the lzo compressor and zbud zpool (both kernel built-ins on EL8 and EL9). The playbook pre-flights kernel support and fails clearly if ZSWAP is unavailable.

  • Automatic swap file sizing. When EIQ_SWAP_FILE_SIZE is not set, the swap file is now sized to MIN(half system RAM, free disk / 2) at deploy time. Pre-existing swap files are never shrunk.

  • Automatic swap disable on dedicated data-tier nodes. Dedicated Elasticsearch, Redis, and PostgreSQL hosts now automatically deactivate swap and set vm.swappiness=0, matching Elastic’s guidance for data hosts.

  • Dynamic Elasticsearch heap sizing based on system RAM and whether the node is dedicated to Elasticsearch. Removes the need to tune heap size manually on most deployments.

  • Automatic configuration of Kibana xpack encryption keys (encryptedSavedObjects, reporting, security) in /etc/eclecticiq-kibana/kibana.yml. Keys have stable defaults so encrypted saved objects survive re-runs and upgrades.

  • Kibana Fleet plugin is now disabled by default (xpack.fleet.agents.enabled: false), which eliminates hourly agentless-deployment sync errors on self-managed clusters where Fleet is not used.

  • Dedicated MCP server database user and database. The MCP server now connects to PostgreSQL with its own least-privilege role (EIQ_MCP_DB_USER / EIQ_MCP_DB_PASSWORD) instead of sharing the platform credentials. A dedicated eiq_mcp_server database is created for MCP-specific state.

  • Automatic generation of the MCP database password. Where these playbooks manage PostgreSQL, EIQ_MCP_DB_PASSWORD no longer needs to be set: a password is generated on first install and stored in /etc/default/eclecticiq-platform-mcp-server on the application node. Later runs and upgrades read that password back and reuse it, so a working credential is never rotated. EIQ_MCP_DB_PASSWORD still takes precedence when set. Managed database deployments must continue to set it, since the mcp role has to be created on the managed instance; the run now stops early with an explanatory message when it is missing, instead of deploying an MCP server that cannot reach the database.

  • Elasticsearch machine-learning plugin disabled by default (xpack.ml.enabled: false in elasticsearch.yml). Override with EIQ_ES_ML_ENABLED=true if ML features are needed.

  • Kibana systemd memory limits via a drop-in override. Defaults: MemoryHigh=2400M, MemoryMax=3000M. Configurable via EIQ_KIBANA_MEMORY_HIGH and EIQ_KIBANA_MEMORY_MAX.

  • SELinux policy support for IC 3.9 and later. The playbook now installs and activates the IC SELinux policy module when SELinux is enforcing.

  • PostgreSQL TLS hardening: generates a 4096-bit DH parameters file and restricts ssl_ciphers to HIGH-strength suites only (no MEDIUM, LOW, EXP, NULL, or 3DES).

  • MCP server environment file (/etc/default/eclecticiq-platform-mcp-server) is now templated from Ansible, allowing database and AI provider credentials to flow through cleanly.

  • prod-one deployment topology (single-node production) with matching inventory ic-prod-one.yml and group vars.

Fixes:

  • Issue where the post-install IC configuration step (email and server name) would fail on hosts that have a valid TLS certificate issued for an FQDN, because the playbooks make the API calls to localhost. TLS validation is now skipped for these local calls only.

  • Issue on prod-large where dynamic worker/ingestion service scaling could incorrectly conclude that a required systemd unit did not exist, skipping the configuration and restart. The check now uses a robust shell lookup that does not depend on systemctl exit behavior.

  • Rewrote ingestion worker scaling to be declarative and idempotent: the playbook now discovers all known ingestion@ unit files and converges to the desired count, so it produces the right result regardless of prior state or partial reruns.

  • Issue where extension installation would fail because pip freeze now emits editable-install paths after an @ character, breaking downstream parsing of extension names.

  • Issue where pb-stop_services.yml did not stop an interim PostgreSQL cluster left running from a major-version upgrade. The interim PG service is now stopped before the target PG starts, avoiding a port 5432 conflict.

  • Issue where synchronize modules would fail because Ansible attempts to transfer files as the wrong SSH remote user.

    We incorrectly set the default value of set_remote_user to false, breaking compatibility with existing playbook runs. set_remote_user now correctly defaults to true.

  • Issue where Kibana failed to start with FleetEncryptedSavedObjectEncryptionKeyRequired after the ES client configuration play ran. The xpack cleanup regex in client_main.yml was overly broad (^(xpack.*)) and commented out the encryption keys and fleet settings written moments earlier. The regex now excludes encryptedSavedObjects, reporting.encryptionKey, security.encryptionKey, fleet, and ml prefixes.

  • Elasticsearch heap sizing capped at 50% of total system memory on dedicated nodes, matching Elastic’s recommendation and preventing OOM on smaller VMs.

  • ES 9 upgrade index reindexing and Kibana interim-hop reliability improvements.

Changes:

  • Default value of EIQ_IC_VERSION bumped from "3.1" to "3.6" so that fresh installs default to a currently supported release.

  • prod-small inventory restructured for HA (three application nodes).

Internal:

  • Update internal build’s pip configuration for extensions.

  • Extension installer now uses uv where supported (IC 3.4 and later) and falls back to pip when uv is unavailable. Minimum uv version pinned to 0.11.9 to avoid a known resolver bug.

3x-1771234933 (16 Feb 2026)#

Adds:

  • Set EXTENSION_SDK_STASH_REDIS_URL.

  • Support for future 3.7.0 additional migration steps (online migrations).

  • Support for edge case where the SSH user name contains unusual characters.

    Set EIQ_ANSIBLE_SYNCHRONIZE_SET_REMOTE_USER='false' to make the ansible.posix.synchronize task read the SSH user name from SSH configuration instead, and allow unusual characters in the SSH user name which would otherwise cause rsync to fail.

Fixes:

  • Fixes issue where playbooks fail to connect to Elasticsearch via HTTPS, because xpack is not enabled.

    Older versions of the platform do not ship with Elasticsearch’s xpack feature enabled. This means that connecting to Elasticsearch with HTTPS, and attempting to access /_security endpoints fails, preventing us from performing user management via the REST API.

  • Fix RHEL9 OpenSSH / OpenSSL mismatch issue: https://access.redhat.com/solutions/7101587

  • Fix timeout starting worker services via systemd config

  • Fixes issue where Elasticsearch fails to start because ES_TMPDIR is located on a volume mounted with noexec. Now explicitly sets ES_TMPDIR to /usr/share/elasticsearch/tmp.

    See: https://www.elastic.co/docs/deploy-manage/deploy/self-managed/executable-jna-tmpdir

  • Fixes issue where the playbooks starts too many workers on application nodes with high CPU count (>64), exhausting other resources e.g. memory, disk and network I/O. Now internally sets a maximum number of workers to avoid this.

3x-1759151449 (29 Sep 2025)#

Changes:

  • Playbooks now install rsyslog in pb-pre_requisites.yml playbook.

  • Automatic configuration of worker CONCURRENCY values can now be disabled by setting EIQ_DYNAMIC_WORKER_SCALING_ENABLED="False".

    If you have custom or fine-tuned worker CONCURRENCY scaling in your deployment and you want to keep that, set this to "False"

Fixes:

  • Issue where block module in Logstash role was not correctly replacing the block, leading to duplicate lines from multiple task runs.

  • Issue where /etc/systemd/system/statsite.service.d/override.conf file would enforce outdated configuration. Playbooks now reconcile this configuration.

  • Issue where resulting Elasticsearch host list in elasticsearch.yml would not be readable by Elasticsearch.

  • Issue where ‘Set Elasticsearch URLs’ in the Logstash role was inadvertently adding newlines every task run.

  • Issue where es_url_list in group_vars/all/config.yml was not correctly handling cases where any of the EIQ_HOST_ELASTIC_NODE_* environment variables are unset.

  • Now allows specifying EIQ_ES_CUSTOM_CA_CERT_SOURCE_PATH and EIQ_ES_CUSTOM_CA_CERT_PATH to configure the application and Elasticsearch to use a specific CA certificate.

    • EIQ_ES_CUSTOM_CA_CERT_SOURCE_PATH should specify a path on the control node.

    • EIQ_ES_CUSTOM_CA_CERT_PATH specifies the path on managed nodes. The platform and Elasticsearch will be configured to look for the Elasticsearch CA cert at this path.

  • Elasticsearch user is now configurable using the EIQ_ES_ELASTIC_USER environment variable.

  • Issue where EIQ_POSTGRES_MANAGED_DB_CA_CERT_PATH would set the incorrect CA cert path for PG in platform_settings.py.

    Now, you should set EIQ_POSTGRES_MANAGED_DB_CA_CERT_PATH to the CA cert path on the controller, and use EIQ_POSTGRES_MANAGED_DB_CA_CERT_PATH_DEST to set the path to the PG CA cert on managed nodes. The file at EIQ_POSTGRES_MANAGED_DB_CA_CERT_PATH is copied to the path at EIQ_POSTGRES_MANAGED_DB_CA_CERT_PATH_DEST.

Removals:

  • IMPORTANT: Removes/replaces elasticsearch tags in Logstash role and eiq.ic/tasks/statsite.yml, as these tasks are required to configure Logstash and statsite services, even in the case where we skip Elasticsearch installation/configuration.

  • Removes all instances of es_hosts as it is not used.

3x-1754302751 (4th August 2025)#

  • Add EIQ_EXTENSIONS_UPGRADE environment variable. Defaults to "True". When upgrading platform patch versions e.g. 3.4.0 to 3.4.1, set to "False" to skip upgrading extensions. No effect in new platform installs, major or minor platform version upgrades e.g. 3.4.0 to 3.5.0.

  • Adds EIQ_ES_CUSTOM_PORT (default 9200), allowing users to specify a custom port for ES (Elasticsearch) nodes, e.g. when deploying with ECE (Elastic Cloud Enterprise) which recommends a default ES port of 9243.

3x-1753784886 (29th July 2025)#

  • Add root CA to all nodes for trusted PKI

  • Dynamically configure RAM and workers based on available resources (ElasticSearch, Celery, Gunicorn, Ingestion)

  • Remove unused deployment YAML

  • Increase MAX_SUPPORTED_VERSION to 3.6.

Fixes#

  • Logstash shutdown issue

  • RedHat bootstrapping issues with python and subscription-manager

Changed#

  • Playbooks no longer remove java-11-openjdk-headless and tzdata-java on deployments with ELK >= 8.

3x-1747397659 (16th May 2025)#

  • Added support for Red Hat Enterprise Linux 9 (RHEL9)

  • Add new a deployment, prod-small, a three node configuration supporting 100 million entities

  • Added optimizations to the install/upgrade process

Fixes#

  • Migration playbook now stops all platform workers and services before performing PG migration on prod-large deployments

  • Migration playbook now logs actual migration command output to /tmp/eiq-migration.log

3x-1746201062 (2nd May 2025)#

  • Added support to install ollama

  • Added support to configure elasticsearch memory usage

3x-1744032951 (27th March 2025)#

  • Increased MAX_SUPPORTED_VERSION for 3.5 release.

  • You can now configure managed DB skipping DB deployment in nodes.

  • Added support for Rocky 9, Alma 9 Operating Systems.

  • Added a single global playbook pb-full-install-upgrade.yml to install/upgrade IC.

  • Added NOCLOBBER option to allow upgrading without modifying PKI or database configuration.

  • Added support for configuring managed PostgreSQL

3x-1728904914 (14 October 2024)#

  • Suport essential Elastisearch upgrades

3x-1727954075 (03 October 2024)#

  • Centralized all yum/pip repository configurations into environment variables to facilitate installation via mirrored local repositories.

  • Added configurable statsd port.

3x-1727157972 (25 September 2024)#

Changes#

  • Centralized all ENV vars under group_vars/all/*.yml.

  • Updated installations for versions >3.3 to use Elasticsearch version 8.15 instead of 8.14.

3x-1724146991 (20 August 2024)#

Changes#

  • Increased MAX_SUPPORTED_VERSION for the 3.4 release.

3x-1718641237 (17 June 2024)#

Changes#

  • Fix ElasticSearch deployment using Fully-Qualified Domain Names (as opposed to IP addresses).

  • Remove un-needed PG URLs from configuration.

  • Fix statsd and syslog configuration

3x-1718267874 (13 June 2024)#

Changes#

  • Fix PostgreSQL issue during upgrade where master is configured as a replica.

3x-1710502964 (15 Mar 2024)#

Changes#

  • Set default release to 3.3 and update README file.

  • PostgreSQL is configured to accept SSL connections. Both SSL and plaintext connections are accepted on the same port (5432). Intelligence Center and replication use SSL connection by default.

  • Redis is configured to accept SSL connections on port 6680. Port 6679 is still available for plaintext connections for debugging purposes. Intelligence Center and replication use SSL connection by default.

Added#

  • Added .env variables to automatically configure email addresses and server hostname. This is done in a new playbook called pb-configure_intelligence_center.yml, which has to be executed when Intelligence is already running (done automatically by the deploy-from-inventory.sh script).

  • Added BUNDLE_VERSION.env to each ansible bundle, which contains the following values:

BUNDLE_VERSION=<version>
GIT_HASH=<git hash>
MAX_SUPPORTED_VERSION=3.3

Fixes#

  • Fixed Playbooks fail for some users due to “undefined” on RHEL

3x-1702283632 (11 Dec 2023)#

Fixes#

  • Fixed issue in PG14 replication

3x-1701862198 (06 Dec 2023)#

Changes#

  • Set default release to 3.2 and update README file.

Fixes#

  • Fixed issue in Kibana export

  • Fixed issue in Kibana to force kibana restore to overwrite duplicate objects

3x-1700648360 (Nov 2023)#

Breaking changes#

  • Playbook sequence is different. Please read the documentation carefully.

  • Environment variables have been changed. Please update your configuration following the provided env-example file.

Added#

  • Playbooks are provided to upgrade to PostgreSQL 14.

  • Playbooks are provided to upgrade to Elasticsearch 8.

  • Backup & Restore playbooks for Kibana.

  • Allows automatic provisioning of authentication for Elasticsearch (on by default for IC 3.2+).

  • Playbooks automatically provision certificates for Elasticsearch. You may manually replace these certs later.

  • You can now configure network addresses of managed nodes through environment variables or by editing group_vars/all/config.yml, instead of editing inventory files.

    You must still run ansible-playbook with inventory files. E.g.:

    EIQ_HOST_IC_APP_01="10.10.2.95" ansible-playbook -i inventories/ic-test-small pb-pre_requisites.yml -vv

Improvements#

  • ansible-example.cfg has optional minor changes for session reliability.

  • Removed EIQ_IS_UPGRADE environment variable. No longer used. Playbooks now detect if you are performing a fresh install or an upgrade.

  • Removed legacy Elasticsearch playbooks.

Fixes#

  • Numerous fixes to PG clustering.

3x-1690533376 (28 Jul 2023)#

Changed:

  • Playbooks now use .env file instead of .envrc to set environment variables.

Fixed:

  • Fixed issue introduced by new version of java-11-openjdk-headless, where the package drops tzdata-java from its list of dependencies, causing Elasticsearch and Logstash services to fail to start because it depends on it. Playbooks now explicitly install tzdata-java.

3x-1689088308 (11 Jul 2023)#

Fixed:

  • Issue where statsite service would fail to reload as part of playbook run.

3x-1688978882 (10 Jul 2023)#

Added:

  • Support for EclecticIQ Intelligence Center 3.x.

  • Playbook and inventory files for Neo4j removal.

  • Various fixes and improvements to deployment reliability.