Elasticsearch 7: “Data too large”

Since release 2.9.0, EclecticIQ Intelligence Center comes bundled with Elasticsearch (ES) 7.9.1. ES 7 adds a new real memory circuit breaker that causes ES nodes to respond with a circuit_breaking_exception error when it detects that memory use has reached 95% of the totally available JVM heap.

Because of this change, you may encounter issues related to available memory where previously at the same workloads, ES would appear to run smoothly.

If your plaform is encountering issues related to Elasticsearch responding with a circuit_breaking_exception error, you can do the following to mitigate:

Increase available memory for ES

The circuit_breaking_exception error occurs only when ES detects that you are about to go over a memory use threshold that would cause it to fail.

Increase the amount of memory available to ES, or move it to its own host where it does not compete with EclecticIQ Intelligence Center for resources to keep your ES nodes running.

(Not recommended) Disable the “real memory circuit breaker”

Warning

This may allow ES to reach an out of memory state and fail.

(Not recommended) To disable the “real memory circuit breaker”, set the indices.breaker.total.use_real_memory parameter in your ES configuration to false.

This allows ES to use the ES 6 parent circuit breaker instead, but disables the safety guarantees that the real memory circuit breaker provides.