Statsite reference CentOS

From EclecticIQ Platform 2.3.0 Statsite replaces StatsD as a metrics aggregator.

Statsite is compatible with StatsD: existing StatsD mappings and indices work with Statsite.

Load the Statsite index mapping template

For Elasticsearch to correctly load and index Statsite-specific fields, and for Kibana to display Statsite data, locally save the Statsite mapping template, and then upload it to Elasticsearch.

Statsite ships with a mapping template that should not require any customization, so that you can use it as is: /opt/statsite/etc/elasticsearch_template.json

1. Upload elasticsearch_template.json to the Elasticsearch instance, to the /_template/statsite-template directory:

   # Push Statsite mapping template to Elasticsearch (Repeat the command if it does not succeed the first time)
   curl -X POST http://127.0.0.1:9200/_template/statsite-template -H 'Content-Type: application/json' -d@/opt/statsite/etc/elasticsearch_template.json
   

2. Restart Elasticsearch:

   # Restart Elasticsearch
   systemctl restart elasticsearch
   

3. When Elasticsearch is up and running again, delete any existing indices, so that Statsite creates new indices with the correct mapping:

   # Delete existing Statsite indices
   curl -X DELETE 'http://localhost:9200/statsite-*'
   

   This creates a new index in Elasticsearch: statsite*.

   The index contains the following Statsite-specific index fields:

   Field	Meaning	Description
   ns	namespace	It refers to a platform component or to a background process, such as API, ingestion, outgoing, and so on. It scopes a set of metrics within specific platform functionality or a specific platform process.
   grp	group	It refers to a group aggregating one or more components and their related metrics, within the context of the platform area specified in the ns namespace. For example, if the namespace is ingestion, components that fall within a group can be incoming feeds and their priority level.
   tgt	target	It refers to a specific platform artifact, such as an index, a task, a worker, a service, and so on.
   act	action	It refers to the the type of metric, such as success, failure, waiting, pending, deduplication, runtime, and so on.

Files and locations

Statsite configuration files	Description
/opt/statsite/etc/statsite.conf	Statsite configuration.
/opt/statsite/etc/statsite.service	Statsite service configuration.
/opt/statsite/etc/elasticsearch_template.json	Elasticsearch index mapping for Statsite.

[statsite]
port=8125
udp_port=8125
bind_address=127.0.0.1
log_level=INFO
flush_interval=10
timer_eps=0.01
set_eps=0.02
stream_cmd=python3 /opt/statsite/bin/elasticsearch_sink.py --base-url=http://localhost:9200 --split-keys=ns.grp.tgt.act --index-by-date
binary_stream=0

[Unit]
Description=network daemon to collect metrics

[Service]
User=nobody
Type=simple
Restart=on-failure
ExecStart=/opt/statsite/bin/statsite -f /opt/statsite/etc/statsite.conf

[Install]
WantedBy=multi-user.target

{
    "template" : "statsite*",
    "settings" : {
        "number_of_shards" : 1
    },
    "mappings" : {
        "counter" : {
            "_source" : { "enabled" : true },
            "properties": {
                "@timestamp": {
                    "type": "date"
                },
                "val": {
                    "type": "double",
                    "index": "not_analyzed"
                },
                "ns": {
                    "type": "string",
                    "index": "not_analyzed"
                },
                "grp": {
                    "type": "string",
                    "index": "not_analyzed"
                },
                "tgt": {
                    "type": "string",
                    "index": "not_analyzed"
                },
                "act": {
                    "type": "string",
                    "index": "not_analyzed"
                }
            }
        },
        "gauge" : {
            "_source" : { "enabled" : true },
            "properties": {
                "@timestamp": {
                    "type": "date"
                },
                "val": {
                    "type": "double",
                    "index": "not_analyzed"
                },
                "ns": {
                    "type": "string",
                    "index": "not_analyzed"
                },
                "grp": {
                    "type": "string",
                    "index": "not_analyzed"
                },
                "tgt": {
                    "type": "string",
                    "index": "not_analyzed"
                },
                "act": {
                    "type": "string",
                    "index": "not_analyzed"
                }
            }
        },
        "timer" : {
            "_source" : { "enabled" : true },
            "properties": {
                "@timestamp": {
                    "type": "date"
                },
                "val": {
                    "type": "double",
                    "index": "not_analyzed"
                },
                "ns": {
                    "type": "string",
                    "index": "not_analyzed"
                },
                "grp": {
                    "type": "string",
                    "index": "not_analyzed"
                },
                "tgt": {
                    "type": "string",
                    "index": "not_analyzed"
                },
                "act": {
                    "type": "string",
                    "index": "not_analyzed"
                }
            }
        },
        "timer_data" : {
            "_source" : { "enabled" : true },
            "properties": {
                "@timestamp": {
                    "type": "date"
                },
                "ns": {
                    "type": "string",
                    "index": "not_analyzed"
                },
                "grp": {
                    "type": "string",
                    "index": "not_analyzed"
                },
                "tgt": {
                    "type": "string",
                    "index": "not_analyzed"
                },
                "act": {
                    "type": "string",
                    "index": "not_analyzed"
                },
                "count": {
                    "type": "integer",
                    "index": "not_analyzed"
                },
                "lower": {
                    "type": "float",
                    "index": "not_analyzed"
                },
                "mean": {
                    "type": "float",
                    "index": "not_analyzed"
                },
                "median": {
                    "type": "float",
                    "index": "not_analyzed"
                },
                "p50": {
                    "type": "float",
                    "index": "not_analyzed"
                },
                "p95": {
                    "type": "float",
                    "index": "not_analyzed"
                },
                "p99": {
                    "type": "float",
                    "index": "not_analyzed"
                },
                "rate": {
                    "type": "float",
                    "index": "not_analyzed"
                },
                "sample_rate": {
                    "type": "float",
                    "index": "not_analyzed"
                },
                "stdev": {
                    "type": "float",
                    "index": "not_analyzed"
                },
                "sum": {
                    "type": "float",
                    "index": "not_analyzed"
                },
                "sum_sq": {
                    "type": "float",
                    "index": "not_analyzed"
                },
                "upper": {
                    "type": "float",
                    "index": "not_analyzed"
                }
            }
        }
    }
}