Prepare nodes#

This section contains instructions on how to prepare your systems for an IC installation with EclecticIQ Intelligence Center playbooks.

Requirements#

You must have the following before running the playbooks:

  • Credentials for https://downloads.eclecticiq.com.

  • One host to act as an Ansible control node.

  • One or more hosts to act as Ansible managed nodes.

Prepare control and managed nodes#

Before you run the playbooks, follow the instructions in this section to prepare your control and managed nodes.

Control node v.s. Managed nodes

Your control node must be separate from managed nodes.

Your control node is where the ansible-playbook is run, and managed nodes are what ansible-playbook acts upon. Running ansible-playbook directly from managed nodes (instead of using a separate control node) can lead to unexpected behavior during playbook runs.

Prepare control node#

Control node v.s. Managed nodes

Your control node must be separate from managed nodes. See Prepare control and managed nodes.

You need at least one host to act as a control node. This is the host where you run the ansible-playbook command.

Your control node must have:

  • Python 3.8 and newer.

  • SSH access to managed nodes, as a user with sudo privileges.

Download and unarchive playbook package#

Tip

You may need to navigate to https://downloads.eclecticiq.com/ on your browser and log in before attempting to open the URLs below.

Download the latest EclecticIQ Intelligence Center installation playbooks from https://downloads.eclecticiq.com/platform/ansible/2.x and unarchive it.

You must

  • Use your https://downloads.eclecticiq.com credentials (<EIQ_USER>:<EIQ_PASSWORD>)

export PLAYBOOK_PACKAGE="eiq-ic-playbooks-2x-<timestamp>.zip"

curl -O -u '<EIQ_USER>:<EIQ_PASSWORD>' https://downloads.eclecticiq.com/platform/ansible/2.x/$PLAYBOOK_PACKAGE

# Example: curl -O -u user%40eclecticiq.com:sUp3rs3cr3tP4ssw0rd https://...

unzip $PLAYBOOK_PACKAGE

Tip

Save and version control your playbook configuration.

These playbook configurations can be useful for documenting your deployments, and are used for future EclecticIQ Intelligence Center upgrades.

Do not store credentials in version control.

Prepare Python environment#

  1. Install Python 3.8 or newer.

  2. Navigate to the EclecticIQ Intelligence Center installation playbooks directory.

    cd ~/path/to/ic-playbooks

    (Recommended) Version control your playbook configuration

    Save and version control your playbook configuration.

    These playbook configurations can be useful for documenting your deployments, and are used for future EclecticIQ Intelligence Center upgrades.

    Do not store credentials in version control.

  3. (Optional) Create and activate a Python virtual environment:

    python3 -m venv .venv
source .venv/bin/activate

  4. Install Python dependencies:

    python3 -m pip install -r python-requirements.txt

Set up secrets#

Create an .envrc file and set the following environment variables. You can use .envrc-example as a template.

For upgrades

If you are setting up the .envrc file to upgrade an existing EclecticIQ Intelligence Center instance, retrieve these values from the /etc/eclecticiq/platform_settings.py file on your existing instance.

# Credentials for https://downloads.eclecticiq.com
export EIQ_REPO_USERNAME='user@example'
export EIQ_REPO_PASSWORD='super_secret_example_password'

# Set passwords for services 
export EIQ_DB_PASSWORD='super_secret_example_password'
export EIQ_NEO4J_PASSWORD='super_secret_example_password'
export EIQ_NEO4JBATCHER_PASSWORD='super_secret_example_password'
export EIQ_REDIS_PASSWORD='super_secret_example_password'

# This variable must be defined if PG replication is being configured for 11host only
export PG_REPLICA_USER_PASSWORD="${EIQ_DB_PASSWORD}"

# IC secret. Set this to a random string.
# You can generate this with $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1)
export EIQ_SECRET_KEY='2ziYbm7ZsbwzXtwQt4AfeEfY3hvCrHzv'

# Set credentials for initial user on EclecticIQ Intelligence Center
export EIQ_ADMIN_USERNAME='admin'
export EIQ_ADMIN_EMAIL='[email protected]'

# Password requirements: at least 10 char with on uppercase, one lowercase, one number and 1 symbol
# We use some sort of password strength validation
export EIQ_ADMIN_PASSWORD='super_secret_example_passw0rd#'

Once done, load these values from the .envrc file by running:

source .envrc

Tip

For information on how to use Ansible with remote users that require passwords for sudo and SSH keys, see Appendix.

Prepare inventory file#

Select and modify the inventory file that suits your deployment architecture.

This package provides pre-configured inventory files for the following deployments:

Deployment

Inventory file

1 node

inventories/ic-1host-test.yml

2 node

inventories/ic-2host-test.yml

4 node

inventories/ic-4host-test.yml

6 node

inventories/ic-6host-prod.yml

11 node

inventories/ic-11host-prod.yml

Tip

Use at least 6 nodes when deploying for production.

Edit that inventory file, and change only the ansible_host values so that they point to the correct target managed node. This is the inventory file that you must invoke when running the ansible-playbook command.

For example, for a 6 node deployment, modify the inventories/ic-6host-prod.yml file:

---
ic6host: &6host
  hosts:
    application-node:
      ansible_host: 10.10.2.11

    pg-node-01:
      ansible_host: 10.10.2.12

    elastic-node-01:
      eiq_ic_elasticsearch__node_name: elastic01
      ansible_host: 10.10.2.13
    elastic-node-02:
      eiq_ic_elasticsearch__node_name: elastic02
      ansible_host: 10.10.2.14
    elastic-node-03:
      eiq_ic_elasticsearch__node_name: elastic03
      ansible_host: 10.10.2.15

    neo4j-node-01:
      ansible_host: 10.10.2.16

// ...

When you run the ansible-playbook command, you must include this inventory file:

ansible-playbook -i inventories/ic-6host-prod.yml <playbook_name.yml>

Prepare ansible.cfg file#

Create an ansible.cfg file at the project root and set the following values.

Use ansible-example.cfg as a template.

[defaults]
interpreter_python=/usr/bin/python3
roles_path=roles
collections_path=collections
remote_user=ec2-user
private_key_file=~/.ssh/path_to_your_key

Tip

For more configuration options, see Ansible: Ansible Configuration Settings

Parameter

Example

Description

interpreter_python

/usr/bin/python3

(Optional) Path to Python interpreter on control node.

roles_path

roles

Path to Ansible roles. This package provides required roles in the roles/ directory. Avoid changing.

collections_path

collections

Path to Ansible collections. Thsi package provides required collections in the collections/ directory. Avoid changing.

remote_user

ec2-user

User name on managed nodes. This user must have sudo privileges. Control node must have SSH access to this user account on managed nodes.

private_key_file

~/.ssh/path_to_your_key

SSH private key file on control node. This should provide SSH access to a user account with sudo privileges on managed nodes.

Set IC version to install#

Set EclecticIQ Intelligence Center version to install by modifying group_vars/all/config-all.yml. In that file, change only these values:

Parameter

Description

eiq_ic_version

Specify an IC version to install. You must write this as "<MAJOR>.<MINOR>" only. For example, to install IC 2.12.1, set this to "2.12".

Default: "2.13"

Tip

The playbooks will always install the latest version of EclecticIQ Intelligence Center within the specified minor version.

eiq_ic_api_version

Specify a version of EclecticIQ Intelligence Center public API package to install.

This must match the EclecticIQ Intelligence Center being installed/upgraded to.

The following is a list of IC versions and their compatible eiq_ic_api_version values:

  • 2.11 - 2.12: 1.0.*

  • 2.13: 1.1.*

  • 2.14: 2.14.*

The following table lists other parameters in the config-all.yml file. Do not change these values unless otherwise instructed.

Parameter

Description

eiq_python_version

Default: "3.8"

data_dir

Data directory on managed nodes. Related to filesystem mounts. Default: /media

extensions_repo_protocol

Default: https

extensions_repo_domain

Default: downloads.eclecticiq.com

ic_extensions

Do not change. List of values.

Default:

- eclecticiq-extension-all~={{ eiq_ic_version }}.0
- "eclecticiq-extension-api=={{ eiq_ic_api_version }}"

eiq_os_user

Default: eclecticiq

eiq_os_group

Default: eclecticiq

es_protocol

Default: http

es_port

Default: 9200

es_url_list

Do not change.

LC_ALL

Default: en_US.utf8

LANG

Default: en_US.utf8

Prepare managed nodes#

Recommended

Managed nodes should be separate from the control node.

Each managed node must have:

  • One of the operating systems supported by EclecticIQ Intelligence Center:

    • CentOS 7.9

    • RHEL 7.9

  • System Python 3.6.8 or newer.

    Must be the same Python interpreter used by libselinux-python3.

  • libselinux-python3

  • Allow incoming requests for services

  • Filesystem mounts where required.

Install libselinux-python3#

On RedHat-based systems (RHEL, CentOS, etc.), you must install libselinux-python3 on all managed nodes to allow Ansible to execute playbook tasks on them. Run as root:

yum install -y libselinux-python3

Note

Installing libselinux-python3 on CentOS 7 and RHEL 7 should also automatically ensure that a compatible version of Python 3 is installed.

Allow incoming requests for services#

When performing a multi-node install (2 nodes and above), you may need to configure managed nodes to allow incoming connections from EclecticIQ Intelligence Center application node for certain services to work.

The table below lists EclecticIQ Intelligence Center services and the ports they listen on:

Service

Port(s)

Ingestion, Celery

  • 8000

PostgreSQL

  • 5432

Redis

  • 6379

Elasticsearch

  • 9200

  • 9300 (between Elasticsearch nodes only)

Neo4j

  • 7474

  • 7687

  • 4008 (for Neo4jbatcher service)

Filesystem mounts#

(Optional) We recommend provisioning your managed nodes with volume mounts at the following locations:

On all managed nodes

On all managed nodes, we recommend mounting /var/log on its own volume:

Component

Mount point

Min. size (GB)

Recommended size (GB)

Logs

/var/log

10

20

Mount volumes for data stores

The following describes volume mount and sizing recommendations for individual services/node hosting those services:

Component

Mount point

Min. size (GB)

Recommended size (GB)

Elasticsearch

/media/elasticsearch

100

150

Neo4j

/media/neo4j

25

50

PostgreSQL

/media/pgsql

200

300

Redis

/media/redis

10

20