Before you upgrade: Elasticsearch 7 VM#

This page contains precautions and instructions for managing Elasticsearch before performing an Intelligence Center (IC) upgrade.

You must follow the instructions in Elasticsearch shard limit and Peform pre-upgrade when upgrading from Elasticsearch 6 to 7 when:

  • Upgrading from Elasticsearch 6.x to 7.x.

  • Upgrading from IC 2.8.x and older to 2.9.x and newer.

Have at least 50% free disk space#

During complex index upgrades and reindexing operations, Elasticsearch may require additional disk space to store temporary working files and temporary copies of the existing indices.

Monitor your Elasticsearch partition usage.

Before it reaches 50% of the available space in the partition extend it, so that the new partition size is at least twice as large as the sum of the existing Elasticsearch indices.

Example

If Elasticsearch currently uses 43 GB of disk space, extend the partition where Elasticsearch lives to at least 86 GB.

Elasticsearch shard limit#

Note

This section only applies if you are upgrading EclecticIQ Intelligence Center from 2.8.x and earlier to 2.9.x and later (Elasticsearch 6.x to 7.x).

Elasticsearch Version 7 has a default shard limit per node of 1,000. However, in order to ensure a successful upgrade, we have increased the shard limit to 32,000.

Problem description#

If you upgrade to Elasticsearch version 7 and have more than 1,000 shards, the upgrade will fail and you will receive a warning that you are over-sharded. If you subsequently increase the shard limit and restart the upgrade process, it will fail again and you will get a “duplicated MetaIndex” error.

Cause#

Prior to its version 7, Elastic Search did not impose a shard limit. However, the relatively high number of shards created by EIQ Intelligence Center for its logs and metrics may exceed this limit, which can cause an upgrade to fail. The two processes concerned are logstash (for logs) and statsite (for metrics). Resolution

To resolve this issue, EclecticIQ Intelligence Center takes the following measures:

  • Automatic cleanup of logs and metric data.

  • Higher shard limit

Warning

Before upgrading, it is important that you determine how many shards you have. You can make a call to the GET /_cat/shards endpoint (curl http://<ELASTICSEARCH_URL>/_cat/shards) or use the EclecticIQ Elasticsearch diagnostic tool.

Automatic cleanup#

As of release 2.9, EclecticIQ Intelligence Center automatically cleans up historical statsite* and logstash* indices.

The cleanup tool uses the LOG_RETENTION and METRIC_RETENTION parameters in your Intelligence Center settings file (typically /etc/eclecticiq/platform_settings.py) to determine the cut-off dates for the cleanup. By default, the values of the two parameters are set to 365 days and 730 days respectively, but you can modify these to suit your data retention policies.

Higher shard limit#

Although Elasticsearch recommends a limit of 1,000 for the number of shards. We have changed this default to 32,000 so that upgrade will run smoothly. Once you have successfully upgraded, the cleanup tool referred to above will run automatically.

The high number of shards caused by logstash and statsite shouldn’t affect core search Intelligence Center functionality because the logs and metrics are stored on different indices.

If you need to set the shard limit to a value greater than 32,000, please consult the upgrade recovery guide.

“Duplicated MetaIndex” error#

If you increase your shard limit only after your first attempt to upgrade has failed, and try again to upgrade, the upgrade will fail once more due to a “duplicated MetaIndex” error. Follow the steps below recover from this error.

  1. Set the max_shards_per_node parameter in elasticsearch.yml to a value which is equal to your current number of shards + 10,000.

    To obtain your current number of shards, execute the following command:

    $ curl https://<ELASTICSEARCH_URL>/_cat/shards

  2. Restart Elasticsearch

  3. Open the .meta Elasticsearch index with the following command:

    $ curl -XPOST https://<ELASTICSEARCH_URL>/.meta/_open

  4. Retry the Elasticsearch search upgrade steps in the EclecticIQ Intelligence Center upgrade guide, and then complete the rest of EclecticIQ Intelligence Center upgrade.

Perform pre-upgrade when upgrading from Elasticsearch 6 to 7#

When upgrading from Intelligence Center versions 2.8 and earlier to 2.9 and later, you must do these tasks before upgrading EclecticIQ Intelligence Center:

  • Run the pre-upgrade scripts to migrate Elasticsearch 6 indexes to Elasticsearch 7

  • Upgrade Elasticsearch from 6.x to 7.9.1

This applies to Intelligence Center versions 2.4 to 2.8, which depend on Elasticsearch 6.

Caution

Elasticsearch 7 shard limit

Elasticsearch 7 adds a default shard limit of 1000, which may cause an upgrade from Elasticsearch 6 to 7 to fail.

For more information and remediation, see Elasticsearch shard limit above.

Requirements#

  • Root access to your Intelligence Center host.

    Tip

    Run sudo -i to switch to root.

  • EIQ Python 3.8 packages.

    To make sure that you have these packages installed:

    1. Make sure you’ve set up the repositories per Set up the repositories CentOS.

    2. Run as root:

      yum install python38

Update your YUM repositories#

Set up your package manager by following the instructions here: Set up the repositories CentOS

Install the Elasticsearch pre-upgrade script package#

Install the eclecticiq-platform-backend-preupgrade package. This package contains the pre-upgrade script that migrates your Elasticsearch 6 indexes to Elasticsearch 7.

This installs the upgrade_elasticsearch_indices.py script to the /opt/eclecticiq-platform-backend/share/eclecticiq directory.

To install the package:

  1. Run as root:

    yum install eclecticiq-platform-backend-preupgrade

  2. Follow the on-screen instructions.

Stop EclecticIQ Intelligence Center services#

Before running the newly installed script, stop the Intelligence Center backend services:

# Stop Intelligence Center backend services, Kibana, and Statsite
systemctl stop eclecticiq-platform-backend-services
systemctl stop kibana
systemctl stop statsite

Run the Elasticsearch pre-upgrade script#

Run the pre-upgrade script. Run as root:

# Specify the URL and the port of the Elasticsearch cluster your Intelligence Center instance uses
python3 /opt/eclecticiq-platform-backend/share/eclecticiq/upgrade_elasticsearch_indices.py --es-url <Elasticsearch URL>

# Example
python3 /opt/eclecticiq-platform-backend/share/eclecticiq/upgrade_elasticsearch_indices.py --es-url "http://localhost:9200/"

Tip

Run with the --dry-run flag to simulate the index migration:

python3 /opt/eclecticiq-platform-backend/share/eclecticiq/upgrade_elasticsearch_indices.py --es-url <Elasticsearch URL> --dry-run

Tip

If there are no operations to perform, the script displays:

[INFO  ] All indices are okay. No operation is required.

Upgrade EclecticIQ Intelligence Center#

Once you’ve run the pre-upgrade script successfully, you can continue to:

  • Upgrade Elasticsearch 6 to Elasticsearch 7.9.1

  • Upgrade EclecticIQ Intelligence Center

For more information, see Upgrade EclecticIQ Intelligence Center VM.

Verify the Elasticsearch upgrade#

To verify that Elasticsearch was correctly upgraded, send a cURL request to the Elasticsearch instance:

curl -XGET 'http://localhost:9200'

The reponse includes the current Elasticsearch version:

{
    "name" : "host.server.address.org",
    "cluster_name" : "intel",
    "cluster_uuid" : "bBABELkfTnGHct1D3c5a8f",
    "version" : {
    "number" : "7.9.1", // current Elasticsearch version
    "build_flavor" : "oss",
    "build_type" : "rpm",
    "build_hash" : "059746d648kh64skio6855e4b42b66854972fu91",
    "build_date" : "2020-09-01T21:22:21.964974Z",
    "build_snapshot" : false,
    "lucene_version" : "8.6.2",
    "minimum_wire_compatibility_version" : "6.8.0", // The oldest node version it is possible to perform a rolling upgrade from
    "minimum_index_compatibility_version" : "6.0.0-beta1" // The oldest index version it is possible to index and read from
    },
    "tagline" : "You Know, for Search"
}