Statsite reference CentOS

From EclecticIQ Platform 2.3.0 Statsite replaces StatsD as a metrics aggregator.

Statsite is compatible with StatsD: existing StatsD mappings and indices work with Statsite.

Load the Statsite index mapping template

For Elasticsearch to correctly load and index Statsite-specific fields, and for Kibana to display Statsite data, locally save the Statsite mapping template, and then upload it to Elasticsearch.

Statsite ships with a mapping template that should not require any customization, so that you can use it as is:
/opt/statsite/etc/ elasticsearch_template.json

  1. Upload elasticsearch_template.json to the Elasticsearch instance, to the /_template/statsite-template directory:

    # Push Statsite mapping template to Elasticsearch (Repeat the command if it does not succeed the first time)
    curl -X POST http://127.0.0.1:9200/_template/statsite-template -H 'Content-Type: application/json' -d@/opt/statsite/etc/elasticsearch_template.json

  2. Restart Elasticsearch:

    # Restart Elasticsearch
    systemctl restart elasticsearch

  3. When Elasticsearch is up and running again, delete any existing indices, so that Statsite creates new indices with the correct mapping:

    # Delete existing Statsite indices
    curl -X DELETE 'http://localhost:9200/statsite-*'

    This creates a new index in Elasticsearch: statsite*.
    The index contains the following Statsite-specific index fields:

    Field

    Meaning

    Description

    ns

    namespace

    It refers to a platform component or to a background process, such as API, ingestion, outgoing, and so on.
    It scopes a set of metrics within specific platform functionality or a specific platform process.

    grp

    group

    It refers to a group aggregating one or more components and their related metrics, within the context of the platform area specified in the ns namespace.
    For example, if the namespace is ingestion, components that fall within a group can be incoming feeds and their priority level.

    tgt

    target

    It refers to a specific platform artifact, such as an index, a task, a worker, a service, and so on.

    act

    action

    It refers to the the type of metric, such as success, failure, waiting, pending, deduplication, runtime, and so on.

Files and locations


Statsite configuration files

Description

/opt/statsite/etc/statsite.conf

Statsite configuration.

/opt/statsite/etc/statsite.service

Statsite service configuration.

/opt/statsite/etc/elasticsearch_template.json

Elasticsearch index mapping for Statsite.


statsite.conf example
[statsite]
port=8125
udp_port=8125
bind_address=127.0.0.1
log_level=INFO
flush_interval=10
timer_eps=0.01
set_eps=0.02
stream_cmd=python3 /opt/statsite/bin/elasticsearch_sink.py --base-url=http://localhost:9200 --split-keys=ns.grp.tgt.act --index-by-date
binary_stream=0


statsite.service example
[Unit]
Description=network daemon to collect metrics
 
[Service]
User=nobody
Type=simple
Restart=on-failure
ExecStart=/opt/statsite/bin/statsite -f /opt/statsite/etc/statsite.conf
 
[Install]
WantedBy=multi-user.target


elasticsearch_template.json example
{
"template" : "statsite*",
"settings" : {
"number_of_shards" : 1
},
"mappings" : {
"counter" : {
"_source" : { "enabled" : true },
"properties": {
"@timestamp": {
"type": "date"
},
"val": {
"type": "double",
"index": "not_analyzed"
},
"ns": {
"type": "string",
"index": "not_analyzed"
},
"grp": {
"type": "string",
"index": "not_analyzed"
},
"tgt": {
"type": "string",
"index": "not_analyzed"
},
"act": {
"type": "string",
"index": "not_analyzed"
}
}
},
"gauge" : {
"_source" : { "enabled" : true },
"properties": {
"@timestamp": {
"type": "date"
},
"val": {
"type": "double",
"index": "not_analyzed"
},
"ns": {
"type": "string",
"index": "not_analyzed"
},
"grp": {
"type": "string",
"index": "not_analyzed"
},
"tgt": {
"type": "string",
"index": "not_analyzed"
},
"act": {
"type": "string",
"index": "not_analyzed"
}
}
},
"timer" : {
"_source" : { "enabled" : true },
"properties": {
"@timestamp": {
"type": "date"
},
"val": {
"type": "double",
"index": "not_analyzed"
},
"ns": {
"type": "string",
"index": "not_analyzed"
},
"grp": {
"type": "string",
"index": "not_analyzed"
},
"tgt": {
"type": "string",
"index": "not_analyzed"
},
"act": {
"type": "string",
"index": "not_analyzed"
}
}
},
"timer_data" : {
"_source" : { "enabled" : true },
"properties": {
"@timestamp": {
"type": "date"
},
"ns": {
"type": "string",
"index": "not_analyzed"
},
"grp": {
"type": "string",
"index": "not_analyzed"
},
"tgt": {
"type": "string",
"index": "not_analyzed"
},
"act": {
"type": "string",
"index": "not_analyzed"
},
"count": {
"type": "integer",
"index": "not_analyzed"
},
"lower": {
"type": "float",
"index": "not_analyzed"
},
"mean": {
"type": "float",
"index": "not_analyzed"
},
"median": {
"type": "float",
"index": "not_analyzed"
},
"p50": {
"type": "float",
"index": "not_analyzed"
},
"p95": {
"type": "float",
"index": "not_analyzed"
},
"p99": {
"type": "float",
"index": "not_analyzed"
},
"rate": {
"type": "float",
"index": "not_analyzed"
},
"sample_rate": {
"type": "float",
"index": "not_analyzed"
},
"stdev": {
"type": "float",
"index": "not_analyzed"
},
"sum": {
"type": "float",
"index": "not_analyzed"
},
"sum_sq": {
"type": "float",
"index": "not_analyzed"
},
"upper": {
"type": "float",
"index": "not_analyzed"
}
}
}
}
}

See also