Config and log files CentOS#

An overview of all platform configuration, log, and manifest files for system administrators.

Configuration, log and manifest files#

EclecticIQ Platform relies on a number of configuration files to store platform settings you can edit and fine-tune to adapt the behavior of the platform to your system.

Log files record platform events; they hold a history of the platform activities that can provide meaningful context, for example when investigating the possible root causes of a problem.

Manifest files contain metadata that help identify the product like the source/origin of the package containing the platform and its components, release reference number, and version information.

This section describes where the platform configuration, log, and manifest files are stored, and what kind of information each file holds.

Configuration files#

To get a list with the platform configuration files, run the following command(s):

# Returns only platform core config files
find /etc/eclecticiq/ -type f

# Returns platform and third-party components config files
find /etc/eclecticiq* -type f

# Returns backend worker config files that manage
# processes such as discovery, rules, reindexing,
# retention policies, and so on
find /etc/default/eclecticiq* -type f

The response returns a list with the following files:

# Platform core settings
/etc/eclecticiq/platform_settings.py
/etc/eclecticiq/proxy_url
/etc/eclecticiq/opentaxii.yml

# Worker config files that manage
# discovery, rules, reindexing, retention policies, and so on
/etc/default/eclecticiq-platform
/etc/default/eclecticiq-platform-backend-worker-common
/etc/default/eclecticiq-platform-backend-worker-discovery
/etc/default/eclecticiq-platform-backend-worker-discovery-priority
/etc/default/eclecticiq-platform-backend-worker-entity-rules-priority
/etc/default/eclecticiq-platform-backend-worker-extract-rules-priority
/etc/default/eclecticiq-platform-backend-worker-incoming-transports
/etc/default/eclecticiq-platform-backend-worker-incoming-transports-priority
/etc/default/eclecticiq-platform-backend-worker-reindexing
/etc/default/eclecticiq-platform-backend-worker-retention-policies
/etc/default/eclecticiq-platform-backend-worker-retention-policies-priority
/etc/default/eclecticiq-platform-backend-worker-utilities
/etc/default/eclecticiq-platform-backend-worker-utilities-priority

# Systemd unit files for ingestion, search, graph, taxii, tasks
/lib/systemd/system/eclecticiq-neo4jbatcher.service
/lib/systemd/system/eclecticiq-platform-backend-graphindex.service
/lib/systemd/system/eclecticiq-platform-backend-ingestion.service
/lib/systemd/system/[email protected]
/lib/systemd/system/eclecticiq-platform-backend-opentaxii.service
/lib/systemd/system/eclecticiq-platform-backend-scheduler.service
/lib/systemd/system/eclecticiq-platform-backend-searchindex.service
/lib/systemd/system/eclecticiq-platform-backend-services.service
/lib/systemd/system/eclecticiq-platform-backend-web.service
/lib/systemd/system/[email protected]
/lib/systemd/system/eclecticiq-platform-backend-workers.service
/lib/systemd/system/eclecticiq-secrets-setter.service

# Elasticsearch config files
/etc/eclecticiq-elasticsearch/elasticsearch.yml
/etc/eclecticiq-elasticsearch/jvm.options
/etc/eclecticiq-elasticsearch/log4j2.properties
/etc/eclecticiq-elasticsearch/elasticsearch.keystore

# Kibana config file
/etc/eclecticiq-kibana/kibana.yml

# Logstash log aggregation config files
/etc/logstash/conf.d/eclecticiq.conf
/etc/logstash/jvm.options
/etc/logstash/log4j2.properties
/etc/logstash/logstash-sample.conf
/etc/logstash/logstash.yml
/etc/logstash/pipelines.yml
/etc/logstash/startup.options

# Neo4j and neo4jbatcher config files
/etc/eclecticiq-neo4j/template-neo4j.conf
/etc/eclecticiq-neo4j/neo4j.conf
/etc/eclecticiq-neo4j/configured-by-eclecticiq
/etc/eclecticiq-neo4jbatcher/neo4jbatcher.conf

# Nginx web server config files
/etc/eclecticiq-nginx/eclecticiq_servername.local.conf
/etc/eclecticiq-nginx/nginx.conf
/etc/eclecticiq-nginx/nginx.centos.conf
/etc/eclecticiq-nginx/nginx.common.conf
/etc/eclecticiq-nginx/nginx.rhel.conf
/etc/eclecticiq-nginx/proxy_params.conf
/etc/eclecticiq-nginx/locations.conf.d/neo4jbatcher.conf
/etc/eclecticiq-nginx/locations.conf.d/platform-frontend.conf
/etc/eclecticiq-nginx/locations.conf.d/tip-backend.conf
/etc/eclecticiq-nginx/sites.conf.d/eclecticiq-default.conf
/etc/eclecticiq-nginx/ssl/eclecticiq-default.fullchain.pem
/etc/eclecticiq-nginx/ssl/eclecticiq-default.privkey.pem

# Postfix email server config file
/etc/postfix/main.cf

# PostgreSQL config files
/etc/eclecticiq-postgres/configured-by-eclecticiq
/etc/eclecticiq-postgres/eclecticiq-postgres.conf
/etc/eclecticiq-postgres/listen-addresses.conf
/etc/eclecticiq-postgres/pg_hba.conf

# Redis message broker config files
/etc/eclecticiq-redis/configured-by-eclecticiq
/etc/eclecticiq-redis/local.conf
/etc/eclecticiq-redis/redis.conf

# Statsite config files
/opt/statsite/etc/elasticsearch_template.json
/opt/statsite/etc/statsite.conf
/opt/statsite/etc/statsite.service

This overview includes further details about the specific content and purpose of the platform configuration files.

File name and location

Owner

Description

/etc/eclecticiq/platform_settings.py

root:eclecticiq

Contains core platform settings like security key value, authentication bearer token expiration time, URLs pointing to external components Celery-managed tasks, and LDAP configuration.

/etc/eclecticiq/opentaxii.yml

root:eclecticiq

Contains OpenTAXII configuration parameters like URL and port for the service, as well as the designated inbound queue and message broker to use.

/etc/eclecticiq/proxy_url

root:eclecticiq

Contains the IP addresses and host names that should bypass the proxy.

Multiple values are comma-separated.

Note

The no-proxy list must always include the following entries: 127.0.0.1,localhost

/etc/eclecticiq-elasticsearch/elasticsearch.yml

root:elasticsearch

Elasticsearch configuration file.

/etc/eclecticiq-elasticsearch/jvm.options

root:elasticsearch

Defines the JVM configuration for Elasticsearch.

Make sure you allocate enough memory to Elasticsearch’s JVM heap.

Default minimum recommended values:

-Xms4g (minimum heap size) -Xmx4g (maximum heap size)

/etc/eclecticiq-elasticsearch/log4j2.properties

root:elasticsearch

Defines log statement output options.

Based on Log4j.

/etc/eclecticiq-elasticsearch/elasticsearch.keystore

root:elasticsearch

Secure store for sensitive settings.

/etc/eclecticiq-kibana/kibana.yml

root:root

Kibana configuration file.

/etc/logstash/conf.d/eclecticiq.conf

root:root

Defines the data source of the input data stream, and the output destination of the processed data.

By default, the input source is the platform Syslog server, and the output is Elasticsearch.

Default port values:

  • 9514: Syslog port

  • 9200: Elasticsearch port

/etc/logstash/jvm.options

root:root

Defines the JVM configuration for Logstash.

/etc/logstash/log4j2.properties

root:root

Defines log statement output options.

Based on Log4j.

/etc/logstash/logstash.yml

root:root

Main configuration file for Logstash.

/etc/logstash/pipelines.yml

root:root

Defines pipeline configurations for Logstash.

/etc/logstash/startup.options

root:root

Helper file used to create a custom startup script for Logstash.

In general, this file is used when installing Logstash, and is kept for reference.

/etc/eclecticiq-neo4j/neo4j.conf

root:root

Neo4j set up and configuration.

Default min. and max. values for the memory heap: 4096 MB.

Refer to Neo4j official documentation for details on Neo4j memory tuning.

/etc/eclecticiq-neo4j/configured-by-eclecticiq

root:root

Do not edit or delete this file.

If it’s there, Neo4j was installed successfully.

/etc/eclecticiq-neo4jbatcher/neo4jbatcher.conf

root:root

Defines the configuration for Neoj4batcher, the ingestion preprocessor for Neo4j.

/etc/eclecticiq-nginx/sites.conf.d/eclecticiq-default.conf

root:root

Defines a default Nginx configuration for the platform.

/etc/eclecticiq-nginx/nginx.centos.conf

root:root

Defines the designated user with access to Nginx in CentOS: nginx.

/etc/eclecticiq-nginx/nginx.common.conf

root:root

Defines further Nginx configuration parameters for the platform.

/etc/eclecticiq-nginx/proxy_params.conf

root:root

Defines the HTTP headers the Nginx sets as a proxy.

/etc/eclecticiq-nginx/eclecticiq_servername.local.conf

root:root

Defines the name of the server hosting the platform. It takes the same value as the Nginx server_name directive.

/etc/eclecticiq-nginx/locations.conf.d/platform-frontend.conf

root:root

Defines the frontend configuration for the web server.

/etc/eclecticiq-nginx/locations.conf.d/neo4jbatcher.conf

root:root

Defines specific web server configuration for Neo4jbatcher.

/etc/eclecticiq-nginx/locations.conf.d/tip-backend.conf

root:root

Defines the backend configuration for the web server, including the root endpoint exposing the public API, and the endpoint exposing the TAXII server service.

/etc/postfix/main.cf

root:root

Postfix configuration file.

Besides configuring email addresses through the GUI, set SMTP email server options in the main.cf file.

/etc/eclecticiq-postgres/eclecticiq-postgres.conf

root:root

PostgreSQL configuration file.

/etc/eclecticiq-postgres/pg_hba.conf

root:root

PostgreSQL client authentication configuration file.

/etc/eclecticiq-postgres/configured-by-eclecticiq

root:root

Do not edit or delete this file.

If it’s there, PostgreSQL was installed successfully.

/etc/eclecticiq-redis/redis.conf

root:root

Redis configuration file.

/etc/eclecticiq-redis/local.conf

root:root

It defines the default directory of the Redis database, and it stores access credentials to it.

Default directory: /media/redis

/etc/eclecticiq-redis/configured-by-eclecticiq

root:root

Do not edit or delete this file.

If it’s there, Redis was installed successfully.

  • /opt/statsite/etc/elasticsearch_template.json

    /opt/statsite/etc/statsite.conf

    /opt/statsite/etc/statsite.service

root:root

Statsite files for, respectively:

Log files#

To get a list with the log files created by the platform and its components, run the following command:

# Returns all log files in the '/log' subdir
find /var/log -type f

# Returns EclecticIQ Platform-specific log files
# in the '/log' subdir
find /var/log -type f | grep 'eiq*\|eclecticiq*'

The response returns a list with the following files:

# Platform core services and components logs:
# ingestion, graph, OpenTAXII, scheduler, Statsite
/var/log/eclecticiq/eiq-backend-web.log
/var/log/eclecticiq/eiq-graphindex.log
/var/log/eclecticiq/eiq-ingestion.log
/var/log/eclecticiq/eiq-neo4jbatcher.log
/var/log/eclecticiq/eiq-opentaxii.log
/var/log/eclecticiq/eiq-searchindex.log
/var/log/eclecticiq/eiq-scheduler.log
/var/log/eclecticiq/eiq-statsite.log

# Celery task workers
/var/log/eclecticiq/eiq-worker-discovery.log
/var/log/eclecticiq/eiq-worker-discovery-priority.log
/var/log/eclecticiq/eiq-worker-enrichers.log
/var/log/eclecticiq/eiq-worker-enrichers-priority.log
/var/log/eclecticiq/eiq-worker-entity-rules-priority.log
/var/log/eclecticiq/eiq-worker-extract-rules-priority.log
/var/log/eclecticiq/eiq-worker-incoming-transports.log
/var/log/eclecticiq/eiq-worker-incoming-transports-priority.log
/var/log/eclecticiq/eiq-worker-outgoing-feeds.log
/var/log/eclecticiq/eiq-worker-outgoing-feeds-priority.log
/var/log/eclecticiq/eiq-worker-outgoing-transports.log
/var/log/eclecticiq/eiq-worker-outgoing-transports-priority.log
/var/log/eclecticiq/eiq-worker-reindexing.log
/var/log/eclecticiq/eiq-worker-retention-policies.log
/var/log/eclecticiq/eiq-worker-retention-policies-priority.log
/var/log/eclecticiq/eiq-worker-utilities.log
/var/log/eclecticiq/eiq-worker-utilities-priority.log

# Elasticsearch search indexing logs
/var/log/elasticsearch/intel.log
/var/log/elasticsearch/intel-2020-08-06.log
/var/log/elasticsearch/intel_deprecation.log
/var/log/elasticsearch/intel_index_indexing_slowlog.log
/var/log/elasticsearch/intel_index_search_slowlog.log

# Logstash log data aggregation logs
/var/log/logstash/logstash-plain-2020-08-05-1.log.gz
/var/log/logstash/logstash-plain.log
/var/log/logstash/logstash-slowlog-plain.log

# Neo4j graph database log
/var/log/neo4j/debug.log

# Nginx web server logs
/var/log/nginx/access.log
/var/log/nginx/error.log

# PostgreSQL intel database log
/var/log/postgresql/postgresql-2020-08-06.log

# Redis message broker log
/var/log/redis/eclecticiq-redis.log
This overview includes further details about the content of each log file.

File name and location

Owner

Description

/var/log/eclecticiq/eiq-backend-web.log

root:eclecticiq

Platform log file.

It logs core platform information.

/var/log/eclecticiq/eiq-graphindex.log

root:eclecticiq

Neo4j graph database log file.

It logs information about graph database migrations, indices, and graph ingestion queue.

/var/log/eclecticiq/eiq-ingestion.log

root:eclecticiq

Intel ingestion log file.

It logs information about ingestion events, as well as ingested batches and packages.

/var/log/eclecticiq/eiq-neo4jbatcher.log

root:eclecticiq

Neo4jbatcher graph database preprocesser log file.

It logs information about preprocessing data for further funneling and queuing it to be loaded to the graph.

/var/log/eclecticiq/eiq-opentaxii.log

root:eclecticiq

It logs OpenTAXII server log information.

/var/log/eclecticiq/eiq-scheduler.log

root:eclecticiq

It logs information about scheduled tasks; for example, feed runs.

/var/log/eclecticiq/eiq-searchindex.log

root:eclecticiq

Search indexing log file.

It logs information about Elasticsearch data indexing.

/var/log/eclecticiq/eiq-worker-discovery.log

/var/log/eclecticiq/eiq-worker-discovery-priority.log

root:eclecticiq

It logs information about discovery tasks scanning incoming sources, such as enrichers and incoming feeds, to retrieve newly ingested intelligence.

It logs information on the execution order of scheduled tasks, based on priority criteria.

/var/log/eclecticiq/eiq-worker-enrichers.log

/var/log/eclecticiq/eiq-worker-enrichers-priority.log

root:eclecticiq

It logs information about enricher task activity, their intelligence providers, enricher priorities, and enricher-related utility tasks running in the background.

It logs information about the execution order of enrichment tasks, based on priority criteria.

/var/log/eclecticiq/eiq-worker-extract-rules-priority.log

root:eclecticiq

It logs information about the execution order of entity rules, based on priority criteria.

/var/log/eclecticiq/eiq-worker-entity-rules-priority.log

root:eclecticiq

It logs information about the execution order of entity rules, and about modifications to enricher rate limits.

/var/log/eclecticiq/eiq-worker-incoming-transports.log

/var/log/eclecticiq/eiq-worker-incoming-transports-priority.log

root:eclecticiq

It logs information about integrations such as incoming feeds, their intelligence providers, incoming feed priorities, and incoming feed-related tasks running in the background.

It logs information about the execution order of incoming feed task runs, based on priority criteria.

/var/log/eclecticiq/eiq-worker-outgoing-feeds.log

/var/log/eclecticiq/eiq-worker-outgoing-feeds-priority.log

root:eclecticiq

It logs information about integrations such as outgoing feeds, their intelligence providers, outgoing feed priorities, and outgoing feed-related tasks running in the background.

It logs information about the execution order of outgoing feed task runs, based on priority criteria.

/var/log/eclecticiq/eiq-worker-outgoing-transports.log

/var/log/eclecticiq/eiq-worker-outgoing-transports-priority.log

root:eclecticiq

It logs information about data connections to notify about establishing connections, Celery workers syncing with other workers as they start up, and initiating outbound data transmissions; for example, by running an outgoing feed.

It logs information about the execution order of outgoing transport tasks, based on priority criteria.

/var/log/eclecticiq/eiq-worker-reindexing.log

root:eclecticiq

It logs information about synced enricher tasks, intelligence providers, feed transport types, and platform utility tasks.

Redis and Celery take care of task worker indexing, queuing, and syncing.

/var/log/eclecticiq/eiq-worker-retention-policies.log

/var/log/eclecticiq/eiq-worker-retention-policies-priority.log

root:eclecticiq

It logs information about data retention policy events.

It logs information about the execution order of data retention policies, based on priority criteria.

/var/log/eclecticiq/eiq-worker-utilities.log

/var/log/eclecticiq/eiq-worker-utilities-priority.log

root:eclecticiq

It logs information about integrations such as enricher tasks, intelligence providers, and platform utility tasks running in the background.

It logs information about the execution order of task workers, based on priority criteria.

/var/log/elasticsearch/intel_deprecation.log

elasticsearch:elasticsearch

It logs information about deprecated Elasticsearch index mapping types.

/var/log/elasticsearch/intel.log

elasticsearch:elasticsearch

Elasticsearch log file.

It logs Elasticsearch events such as initialization, startup, designated Elasticsearch cluster, and so on.

/var/log/elasticsearch/intel.log.YYYY-MM-DD.log

elasticsearch:elasticsearch

Elasticsearch log file for a specific date.

YYYY-MM-DD (year, month, day) in the file name is replaced by the date the log information refers to.

It logs Elasticsearch events such as initialization, startup, designated Elasticsearch cluster, and so on.

/var/log/elasticsearch/intel_index_indexing_slowlog.log

elasticsearch:elasticsearch

Elasticsearch log file.

It logs Elasticsearch indexing information.

/var/log/elasticsearch/intel_index_search_slowlog.log

elasticsearch:elasticsearch

Elasticsearch log file.

It logs Elasticsearch search index information.

/var/log/logstash/logstash.err

logstash:logstash

It logs Logstash errors and error messages.

/var/log/logstash/logstash-plain.log

logstash:logstash

Most recent Logstash events log file.

/var/log/logstash/logstash-plain-YYYY-MM-DD.log

logstash:logstash

Historical Logstash events log files.

YYYY-MM-DD (year, month, day) in the file name is replaced by the date the log information refers to.

/var/log/neo4j/debug.log

neo4j:neo4j

Neo4j graph database debug log containing detailed information about database events. Useful when troubleshooting.

/var/log/nginx/access.log

root:root

Nginx log file.

It logs web server access information.

/var/log/nginx/error.log

root:root

Nginx log file.

It logs web server error information.

/var/log/postgresql/postgresql-YYYY-MM-DD.log

postgres:root

PostgreSQL log file.

YYYY-MM-DD (year, month, day) in the file name is replaced by the date the log information refers to.

It logs PostgreSQL database ingestion information.

/var/log/redis/eclecticiq-redis.log

redis:redis

Redis log file.

It logs message broker event information about memory usage during copy-write operations and data saving to the database.