Before you upgrade: Elasticsearch 7 RHEL#
This page contains precautions and instructions for managing Elasticsearch before performing an EclecticIQ Intelligence Center upgrade.
You must follow the instructions in Elasticsearch shard limit and Peform pre-upgrade when upgrading from Elasticsearch 6 to 7 when:
Upgrading from Elasticsearch 6.x to 7.x.
Upgrading from EclecticIQ Intelligence Center 2.8.x and older to 2.9.x and newer.
Have at least 50% free disk space#
During complex index upgrades and reindexing operations, Elasticsearch may require additional disk space to store temporary working files and temporary copies of the existing indices.
Monitor your Elasticsearch partition usage.
Before it reaches 50% of the available space in the partition extend it, so that the new partition size is at least twice as large as the sum of the existing Elasticsearch indices.
Example
If Elasticsearch currently uses 43 GB of disk space, extend the partition where Elasticsearch lives to at least 86 GB.
Elasticsearch shard limit#
Note
This section only applies if you are upgrading EclecticIQ Intelligence Center from 2.8.x and earlier to 2.9.x and later (Elasticsearch 6.x to 7.x).
Elasticsearch Version 7 has a default shard limit per node of 1,000. However, in order to ensure a successful upgrade, we have increased the shard limit to 32,000.
Problem description#
If you upgrade to Elasticsearch version 7 and have more than 1,000 shards, the upgrade will fail and you will receive a warning that you are over-sharded. If you subsequently increase the shard limit and restart the upgrade process, it will fail again and you will get a “duplicated MetaIndex” error.
Cause#
Prior to its version 7, Elastic Search did not impose a shard limit. However, the relatively high number of shards created by EIQ Intelligence Center for its logs and metrics may exceed this limit, which can cause an upgrade to fail. The two processes concerned are logstash (for logs) and statsite (for metrics). Resolution
To resolve this issue, EclecticIQ Intelligence Center takes the following measures:
Automatic cleanup of logs and metric data.
Higher shard limit
Warning
Before upgrading, it is important that you determine how
many shards you have. You can make a call to the
GET /_cat/shards
endpoint (curl http://<ELASTICSEARCH_URL>/_cat/shards
)
or use the EclecticIQ
Elasticsearch diagnostic tool.
Automatic cleanup#
As of release 2.9, EclecticIQ Intelligence Center automatically cleans up
historical statsite*
and logstash*
indices.
The cleanup tool uses the LOG_RETENTION
and
METRIC_RETENTION
parameters in your Intelligence Center settings
file (typically /etc/eclecticiq/platform_settings.py
) to
determine the cut-off dates for the cleanup. By default, the
values of the two parameters are set to 365 days and 730
days respectively, but you can modify these to suit your
data retention policies.
Higher shard limit#
Although Elasticsearch recommends a limit of 1,000 for the number of shards. We have changed this default to 32,000 so that upgrade will run smoothly. Once you have successfully upgraded, the cleanup tool referred to above will run automatically.
The high number of shards caused by logstash
and
statsite
shouldn’t affect core search Intelligence Center
functionality because the logs and metrics are stored on
different indices.
If you need to set the shard limit to a value greater than 32,000, please consult the upgrade recovery guide.
“Duplicated MetaIndex” error#
If you increase your shard limit only after your first attempt to upgrade has failed, and try again to upgrade, the upgrade will fail once more due to a “duplicated MetaIndex” error. Follow the steps below recover from this error.
Set the max_shards_per_node parameter in elasticsearch.yml to a value which is equal to your current number of shards + 10,000.
To obtain your current number of shards, execute the following command:
$ curl https://<ELASTICSEARCH_URL>/_cat/shards
Restart Elasticsearch
Open the
.meta
Elasticsearch index with the following command:$ curl -XPOST https://<ELASTICSEARCH_URL>/.meta/_open
Retry the Elasticsearch search upgrade steps in the EclecticIQ Intelligence Center upgrade guide, and then complete the rest of EclecticIQ Intelligence Center upgrade.
Perform pre-upgrade when upgrading from Elasticsearch 6 to 7#
When upgrading from Intelligence Center versions 2.8 and earlier to 2.9 and later, you must do these tasks before upgrading EclecticIQ Intelligence Center:
Run the pre-upgrade scripts to migrate Elasticsearch 6 indexes to Elasticsearch 7
Upgrade Elasticsearch from 6.x to 7.9.1
This applies to Intelligence Center versions 2.4 to 2.8, which depend on Elasticsearch 6.
Caution
Elasticsearch 7 shard limit
Elasticsearch 7 adds a default shard limit of 1000, which may cause an upgrade from Elasticsearch 6 to 7 to fail.
For more information and remediation, see Elasticsearch shard limit above.
Requirements#
Root access to your Intelligence Center host.
Tip
Run
sudo -i
to switch to root.EIQ Python 3.8 packages.
To make sure that you have these packages installed:
Make sure you’ve set up the repositories per Set up the repositories RHEL.
Run as root:
yum install python38
Update your YUM repositories#
Set up your package manager by following the instructions here: Set up the repositories RHEL
Install the Elasticsearch pre-upgrade script package#
Install the eclecticiq-platform-backend-preupgrade
package. This package contains the pre-upgrade script that
migrates your Elasticsearch 6 indexes to Elasticsearch 7.
This installs the upgrade_elasticsearch_indices.py
script to the
/opt/eclecticiq-platform-backend/share/eclecticiq
directory.
To install the package:
Run as root:
yum install eclecticiq-platform-backend-preupgrade
Follow the on-screen instructions.
Stop EclecticIQ Intelligence Center services#
Before running the newly installed script, stop the Intelligence Center backend services:
# Stop Intelligence Center backend services, Kibana, and Statsite
systemctl stop eclecticiq-platform-backend-services
systemctl stop kibana
systemctl stop statsite
Run the Elasticsearch pre-upgrade script#
Run the pre-upgrade script. Run as root:
# Specify the URL and the port of the Elasticsearch cluster your Intelligence Center instance uses
python3 /opt/eclecticiq-platform-backend/share/eclecticiq/upgrade_elasticsearch_indices.py --es-url <Elasticsearch URL>
# Example
python3 /opt/eclecticiq-platform-backend/share/eclecticiq/upgrade_elasticsearch_indices.py --es-url "http://localhost:9200/"
Tip
Run with the --dry-run
flag to simulate the index
migration:
python3 /opt/eclecticiq-platform-backend/share/eclecticiq/upgrade_elasticsearch_indices.py --es-url <Elasticsearch URL> --dry-run
Tip
If there are no operations to perform, the script displays:
[INFO ] All indices are okay. No operation is required.
Upgrade EclecticIQ Intelligence Center#
Once you’ve run the pre-upgrade script successfully, you can continue to:
Upgrade Elasticsearch 6 to Elasticsearch 7.9.1
Upgrade EclecticIQ Intelligence Center
For more information, see Upgrade EclecticIQ Intelligence Center RHEL.
Verify the Elasticsearch upgrade#
To verify that Elasticsearch was correctly upgraded, send a cURL request to the Elasticsearch instance:
curl -XGET 'http://localhost:9200'
The reponse includes the current Elasticsearch version:
{
"name" : "host.server.address.org",
"cluster_name" : "intel",
"cluster_uuid" : "bBABELkfTnGHct1D3c5a8f",
"version" : {
"number" : "7.9.1", // current Elasticsearch version
"build_flavor" : "oss",
"build_type" : "rpm",
"build_hash" : "059746d648kh64skio6855e4b42b66854972fu91",
"build_date" : "2020-09-01T21:22:21.964974Z",
"build_snapshot" : false,
"lucene_version" : "8.6.2",
"minimum_wire_compatibility_version" : "6.8.0", // The oldest node version it is possible to perform a rolling upgrade from
"minimum_index_compatibility_version" : "6.0.0-beta1" // The oldest index version it is possible to index and read from
},
"tagline" : "You Know, for Search"
}