Problem description

If you upgrade to Elasticsearch version 7 and have more than 1,000 shards, the upgrade will fail and you will receive a warning that you are over-sharded. If you subsequently increase the shard limit and restart the upgrade process, it will fail again and you will get a “duplicated MetaIndex” error.

Cause

Prior to its version 7, Elastic Search did not impose a shard limit. However, the relatively high number of shards created by EIQ Intelligence Center for its logs and metrics may exceed this limit, which can cause an upgrade to fail. The two processes concerned are logstash (for logs) and statsite (for metrics). Resolution

To resolve this issue, EclecticIQ Intelligence Center takes the following measures:

• Automatic cleanup of logs and metric data.

• Higher shard limit

Automatic cleanup

As of release 2.9, the Intelligence Center automatically cleans up historical statsite* and logstash* indices.

The cleanup tool uses the LOG_RETENTION and METRIC_RETENTION parameters in your Intelligence Center settings file (typically /etc/eclecticiq/platform_settings.py) to determine the cut-off dates for the cleanup. By default, the values of the two parameters are set to 365 days and 730 days respectively, but you can modify these to suit your data retention policies.

Higher shard limit

Although Elasticsearch recommends a limit of 1,000 for the number of shards. We have changed this default to 32,000 so that upgrade will run smoothly. Once you have successfully upgraded, the cleanup tool referred to above will run automatically.

The high number of shards caused by logstash and statsite shouldn’t affect core search Intelligence Center functionality because the logs and metrics are stored on different indices.

If you need to set the shard limit to a value greater than 32,000, please consult the upgrade recovery guide.

“Duplicated MetaIndex” error

If you increase your shard limit only after your first attempt to upgrade has failed, and try again to upgrade, the upgrade will fail once more due to a “duplicated MetaIndex” error. Follow the steps below recover from this error.

1. Set the max_shards_per_node parameter in elasticsearch.yml to a value which is equal to your current number of shards + 10,000.

   To obtain your current number of shards, execute the following command:

   $ curl https://<ELASTICSEARCH_URL>/_cat/shards

2. Restart Elasticsearch

3. Open the .meta Elasticsearch index with the following command:

   $ curl -XPOST https://<ELASTICSEARCH_URL>/.meta/_open

4. Retry the Elasticsearch search upgrade steps in the EclecticIQ Intelligence Center upgrade guide, and then complete the rest of the Intelligence Center upgrade.

Contents

• Requirements

• Update your YUM repositories

• Install the Elasticsearch pre-upgrade script package

• Stop the Intelligence Center services

• Run the Elasticsearch pre-upgrade script

• Upgrade the Intelligence Center

• Verify the Elasticsearch upgrade

Requirements

• Root access to your Intelligence Center host.

  Run sudo -i to switch to root.

• EIQ Python 3.8 packages.

  To make sure that you have these packages installed:

  1. Make sure you’ve set up the repositories per Set up the repositories CentOS.

  2. Run as root:

     yum install python38

Update your YUM repositories

Set up your package manager by following the instructions here: Set up the repositories CentOS

Install the Elasticsearch pre-upgrade script package

Install the eclecticiq-platform-backend-preupgrade package. This package contains the pre-upgrade script that migrates your Elasticsearch 6 indexes to Elasticsearch 7.

This installs the upgrade_elasticsearch_indices.py script to the /opt/eclecticiq-platform-backend/share/eclecticiq directory.

To install the package:

1. Run as root:

   yum install eclecticiq-platform-backend-preupgrade

2. Follow the on-screen instructions.

Stop the Intelligence Center services

Before running the newly installed script, stop the Intelligence Center backend services:

Run the Elasticsearch pre-upgrade script

Run the pre-upgrade script. Run as root:

[INFO  ] All indices are okay. No operation is required.

Upgrade the Intelligence Center

Once you’ve run the pre-upgrade script successfully, you can continue to:

• Upgrade Elasticsearch 6 to Elasticsearch 7.9.1

• Upgrade the Intelligence Center

For more information, see Upgrade the Intelligence Center CentOS.

Verify the Elasticsearch upgrade

To verify that Elasticsearch was correctly upgraded, send a cURL request to the Elasticsearch instance:

The reponse includes the current Elasticsearch version:

{
    "name" : "host.server.address.org",
    "cluster_name" : "intel",
    "cluster_uuid" : "bBABELkfTnGHct1D3c5a8f",
    "version" : {
    "number" : "7.9.1", // current Elasticsearch version
    "build_flavor" : "oss",
    "build_type" : "rpm",
    "build_hash" : "059746d648kh64skio6855e4b42b66854972fu91",
    "build_date" : "2020-09-01T21:22:21.964974Z",
    "build_snapshot" : false,
    "lucene_version" : "8.6.2",
    "minimum_wire_compatibility_version" : "6.8.0", // The oldest node version it is possible to perform a rolling upgrade from
    "minimum_index_compatibility_version" : "6.0.0-beta1" // The oldest index version it is possible to index and read from
    },
    "tagline" : "You Know, for Search"
}