Back up and restore with playbooks#

Danger

These playbooks are provided mainly to aid in the upgrade process. This is not a long-term backup solution.

Backup & restore is a destructive operation. The provided playbooks are destructive. Backup and restore operations will remove the existing contents of EIQ_IC_BACKUP_ROOT before beginning.

Ensure that the EIQ_IC_BACKUP_ROOT environment variable is set to:

a path on a volume with sufficient free space
a unique path on corresponding this backup operation

Limitations:

File based backups are not suitable for long-term archival use-cases, due to incompatible files formats between major software updates.
The playbooks must be run on control node with a large amount of storage.
A stable and fast network connection is required.

You can back up and restore your EclecticIQ Intelligence Center installation with instructions provided here.

The playbooks here provide a basic file-based backup and restore by copying data and configuration files to ${EIQ_IC_BACKUP_ROOT}/eiq.backup on your control node filesystem.

Requirements#

Manage backup destination:

You must manually manage your backup destination. By default, this is set to ${EIQ_IC_BACKUP_ROOT:-.}/eiq.backup, or ./eiq.backup when you run the playbooks.
Running the pb-backup*.yml playbooks more than once will overwrite the contents of ${EIQ_IC_BACKUP_ROOT:-.}/eiq.backup.
Use version control to manage your backup destination, or change EIQ_IC_BACKUP_ROOT each time you make a backup of your system(s).

Tip

To change your backup destination, see Prepare nodes.

Playbook requirements:

Caution

You may have used an .envrc file in earlier versions of the installation playbooks. Rename it to .env, or create an .env file using instructions in Set up .env.

You must have an existing .env file, or reconstruct an .env file from the /etc/eclecticiq/platform_settings.py of an existing EclecticIQ Intelligence Center installation. See Prepare control node.
You must have an existing inventory file. If you do not have an existing inventory file, create a new file from examples available in inventories/ that corresponds to your deployment configuration. See Prepare control node.

Control node requirements:

Control node must have sufficient disk space at EIQ_IC_BACKUP_ROOT. If the EIQ_IC_BACKUP_ROOT environment variable is not set when you run the playbooks, it defaults to your working directory when running the playbooks (./).

Managed node(s) requirements:

EclecticIQ Intelligence Center services and workers must not be running.
You must stop all database services (e.g. PostgreSQL, Elasticsearch, and Redis processes).
When performing a backup, your resulting backup must form a contemporaneous snapshot of the state of your system(s). This means that you must run the intended pb-backup*.yml playbooks and allow them to complete before attempting to restart any of these services.
Conversely, when restoring from a given backup, you must restore all the backups created for a given snapshot before attempting to restart any of these services.

List of playbooks#

Playbook file	Description
`pb-backup_config.yml`	Back up EclecticIQ Intelligence Center configuration files. Files backed up with this playbook must be restored manually.
`pb-backup-es.yml`	Back up Elasticsearch configuration and data directory.
`pb-backup-pg.yml`	Back up PostgreSQL configuration and data directory.
`pb-backup-redis.yml`	Back up Redis configuration and data directory.
`pb-backup_kibana.yml`	Back up Kibana configuration and custom dashboards. Kibana has a separate set of backup instructions. See Run Kibana backup playbook.
`pb-restore-es.yml`	Restore Elasticsearch configuration and data directory.
`pb-restore-pg.yml`	Restore PostgreSQL configuration and data directory.
`pb-restore-redis.yml`	Restore Redis configuration and data directory.
`pb-restore-kibana.yml`	Restore Kibana configuration and custom dashboards.

To have a working backup, you must at least back up your:

EclecticIQ Intelligence Center configuration (pb-backup-config.yml)
PostgreSQL data and configuration (pb-backup-pg.yml)

Run Kibana backup playbook#

You must run the Kibana backup playbook (pb-backup_kibana.yml) before attempting an Upgrade with playbooks to preserve custom configuration and dashboards.

Before following instructions here, you must make sure that:

Kibana and Elasticsearch are running
Control node must be able to connect to Kibana and Elasticsearch

To back up Kibana configuration and dashboards,

Make sure your .env file is loaded:
```
source .env
```

Run the pb-backup_kibana.yml playbook. Kibana and Elasticsearch must be running.

ansible-playbook -i inventories/<inventory_file.yml> pb-backup_kibana.yml

When the backup completes, make sure that ic-kibana-objects.ndjson:

Is successfully created in your backup directory on the control node.
Contains expected contents (e.g. named custom dashboards).

Run backup playbooks#

Tip

Run commands with --tags=all,logstash if your deployment includes Logstash. E.g. ansible-playbook --tags=all,logstash -i inventories/ic-prod-large.yml ...

Make sure your .env file is loaded:
```
source .env
```

Stop all services and workers. Run pb-stop_services.yml:

ansible-playbook -i inventories/<inventory_file.yml> pb-stop_services.yml

Run the backup playbooks:

ansible-playbook -i inventories/<inventory_file.yml> pb-backup_config.yml
ansible-playbook -i inventories/<inventory_file.yml> pb-backup-pg.yml
ansible-playbook -i inventories/<inventory_file.yml> pb-backup-es.yml
ansible-playbook -i inventories/<inventory_file.yml> pb-backup-redis.yml

Wait for the playbooks to complete.

Once playbooks have completed, start all services:

ansible-playbook -i inventories/<inventory_file.yml> pb-start_services.yml

Run restore playbooks#

Caution

EclecticIQ Intelligence Center configuration backups (pb-backup_config.yml) must be manually restored.

EIQ_IC_BACKUP_ROOT must point to an existing backup location.

Tip

Run commands with --tags=all,logstash if your deployment includes Logstash. E.g. ansible-playbook --tags=all,logstash -i inventories/ic-prod-large.yml ...

Make sure your .env file is loaded:
```
source .env
```

Stop all services and workers. Run pb-stop_services.yml:

ansible-playbook -i inventories/<inventory_file.yml> pb-stop_services.yml

Run the restore playbooks:

ansible-playbook -i inventories/<inventory_file.yml> pb-restore-pg.yml
ansible-playbook -i inventories/<inventory_file.yml> pb-restore-es.yml
ansible-playbook -i inventories/<inventory_file.yml> pb-restore-redis.yml

Wait for the playbooks to complete.

Once playbooks have completed, start all services:

ansible-playbook -i inventories/<inventory_file.yml> pb-start_services.yml

Run Kibana restore playbook#

Run the Kibana restore playbook to restore Kibana configuration and dashboards. If you have any custom Kibana configuration or dashboards, you must run this to make them available.

Before following instructions here, you must make sure that:

Kibana and Elasticsearch are running
Control node must be able to connect to Kibana and Elasticsearch
ic-kibana-objects.ndjson must be present in the backup destination directory on your control node.

To restore up Kibana configuration and dashboards,

Make sure your .env file is loaded:
```
source .env
```
Run Kibana restore playbook: pb-restore-kibana.yml

The playbook expects ic-kibana-objects.ndjson, to be in the backup destination directory on the control node.
```
ansible-playbook -i inventories/ic-test-small.yml pb-restore-kibana.yml
```