Config and log files CentOS#
An overview of all platform configuration, log, and manifest files for system administrators.
Configuration, log and manifest files#
EclecticIQ Platform relies on a number of configuration files to store platform settings you can edit and fine-tune to adapt the behavior of the platform to your system.
Log files record platform events; they hold a history of the platform activities that can provide meaningful context, for example when investigating the possible root causes of a problem.
Manifest files contain metadata that help identify the product like the source/origin of the package containing the platform and its components, release reference number, and version information.
This section describes where the platform configuration, log, and manifest files are stored, and what kind of information each file holds.
Configuration files#
To get a list with the platform configuration files, run the following command(s):
# Returns only platform core config files
find /etc/eclecticiq/ -type f
# Returns platform and third-party components config files
find /etc/eclecticiq* -type f
# Returns backend worker config files that manage
# processes such as discovery, rules, reindexing,
# retention policies, and so on
find /etc/default/eclecticiq* -type f
The response returns a list with the following files:
# Platform core settings
/etc/eclecticiq/platform_settings.py
/etc/eclecticiq/proxy_url
/etc/eclecticiq/opentaxii.yml
# Worker config files that manage
# discovery, rules, reindexing, retention policies, and so on
/etc/default/eclecticiq-platform
/etc/default/eclecticiq-platform-backend-worker-common
/etc/default/eclecticiq-platform-backend-worker-discovery
/etc/default/eclecticiq-platform-backend-worker-discovery-priority
/etc/default/eclecticiq-platform-backend-worker-entity-rules-priority
/etc/default/eclecticiq-platform-backend-worker-extract-rules-priority
/etc/default/eclecticiq-platform-backend-worker-incoming-transports
/etc/default/eclecticiq-platform-backend-worker-incoming-transports-priority
/etc/default/eclecticiq-platform-backend-worker-reindexing
/etc/default/eclecticiq-platform-backend-worker-retention-policies
/etc/default/eclecticiq-platform-backend-worker-retention-policies-priority
/etc/default/eclecticiq-platform-backend-worker-utilities
/etc/default/eclecticiq-platform-backend-worker-utilities-priority
# Systemd unit files for ingestion, search, graph, taxii, tasks
/lib/systemd/system/eclecticiq-neo4jbatcher.service
/lib/systemd/system/eclecticiq-platform-backend-graphindex.service
/lib/systemd/system/eclecticiq-platform-backend-ingestion.service
/lib/systemd/system/[email protected]
/lib/systemd/system/eclecticiq-platform-backend-opentaxii.service
/lib/systemd/system/eclecticiq-platform-backend-scheduler.service
/lib/systemd/system/eclecticiq-platform-backend-searchindex.service
/lib/systemd/system/eclecticiq-platform-backend-services.service
/lib/systemd/system/eclecticiq-platform-backend-web.service
/lib/systemd/system/[email protected]
/lib/systemd/system/eclecticiq-platform-backend-workers.service
/lib/systemd/system/eclecticiq-secrets-setter.service
# Elasticsearch config files
/etc/eclecticiq-elasticsearch/elasticsearch.yml
/etc/eclecticiq-elasticsearch/jvm.options
/etc/eclecticiq-elasticsearch/log4j2.properties
/etc/eclecticiq-elasticsearch/elasticsearch.keystore
# Kibana config file
/etc/eclecticiq-kibana/kibana.yml
# Logstash log aggregation config files
/etc/logstash/conf.d/eclecticiq.conf
/etc/logstash/jvm.options
/etc/logstash/log4j2.properties
/etc/logstash/logstash-sample.conf
/etc/logstash/logstash.yml
/etc/logstash/pipelines.yml
/etc/logstash/startup.options
# Neo4j and neo4jbatcher config files
/etc/eclecticiq-neo4j/template-neo4j.conf
/etc/eclecticiq-neo4j/neo4j.conf
/etc/eclecticiq-neo4j/configured-by-eclecticiq
/etc/eclecticiq-neo4jbatcher/neo4jbatcher.conf
# Nginx web server config files
/etc/eclecticiq-nginx/eclecticiq_servername.local.conf
/etc/eclecticiq-nginx/nginx.conf
/etc/eclecticiq-nginx/nginx.centos.conf
/etc/eclecticiq-nginx/nginx.common.conf
/etc/eclecticiq-nginx/nginx.rhel.conf
/etc/eclecticiq-nginx/proxy_params.conf
/etc/eclecticiq-nginx/locations.conf.d/neo4jbatcher.conf
/etc/eclecticiq-nginx/locations.conf.d/platform-frontend.conf
/etc/eclecticiq-nginx/locations.conf.d/tip-backend.conf
/etc/eclecticiq-nginx/sites.conf.d/eclecticiq-default.conf
/etc/eclecticiq-nginx/ssl/eclecticiq-default.fullchain.pem
/etc/eclecticiq-nginx/ssl/eclecticiq-default.privkey.pem
# Postfix email server config file
/etc/postfix/main.cf
# PostgreSQL config files
/etc/eclecticiq-postgres/configured-by-eclecticiq
/etc/eclecticiq-postgres/eclecticiq-postgres.conf
/etc/eclecticiq-postgres/listen-addresses.conf
/etc/eclecticiq-postgres/pg_hba.conf
# Redis message broker config files
/etc/eclecticiq-redis/configured-by-eclecticiq
/etc/eclecticiq-redis/local.conf
/etc/eclecticiq-redis/redis.conf
# Statsite config files
/opt/statsite/etc/elasticsearch_template.json
/opt/statsite/etc/statsite.conf
/opt/statsite/etc/statsite.service
This overview includes further details about the specific content and purpose of the platform configuration files.
File name and location |
Owner |
Description |
|---|---|---|
/etc/eclecticiq/platform_settings.py |
|
Contains core platform settings like security key value, authentication bearer token expiration time, URLs pointing to external components Celery-managed tasks, and LDAP configuration. |
/etc/eclecticiq/opentaxii.yml |
|
Contains OpenTAXII configuration parameters like URL and port for the service, as well as the designated inbound queue and message broker to use. |
/etc/eclecticiq/proxy_url |
|
Contains the IP addresses and host names that should bypass the proxy. Multiple values are comma-separated. Note The no-proxy list must always include the following entries:
|
/etc/eclecticiq-elasticsearch/elasticsearch.yml |
|
Elasticsearch configuration file. |
/etc/eclecticiq-elasticsearch/jvm.options |
|
Defines the JVM configuration for Elasticsearch. Make sure you allocate enough memory to Elasticsearch’s JVM heap. Default minimum recommended values: -Xms4g (minimum heap size) -Xmx4g (maximum heap size) |
/etc/eclecticiq-elasticsearch/log4j2.properties |
|
Defines log statement output options. Based on Log4j. |
/etc/eclecticiq-elasticsearch/elasticsearch.keystore |
|
Secure store for sensitive settings. |
/etc/eclecticiq-kibana/kibana.yml |
|
Kibana configuration file. |
/etc/logstash/conf.d/eclecticiq.conf |
|
Defines the data source of the input data stream, and the output destination of the processed data. By default, the input source is the platform Syslog server, and the output is Elasticsearch. Default port values:
|
/etc/logstash/jvm.options |
|
Defines the JVM configuration for Logstash. |
/etc/logstash/log4j2.properties |
|
Defines log statement output options. Based on Log4j. |
/etc/logstash/logstash.yml |
|
Main configuration file for Logstash. |
/etc/logstash/pipelines.yml |
|
Defines pipeline configurations for Logstash. |
/etc/logstash/startup.options |
|
Helper file used to create a custom startup script for Logstash. In general, this file is used when installing Logstash, and is kept for reference. |
/etc/eclecticiq-neo4j/neo4j.conf |
|
Neo4j set up and configuration. Default min. and max. values for the memory heap: 4096 MB. Refer to Neo4j official documentation for details on Neo4j memory tuning. |
/etc/eclecticiq-neo4j/configured-by-eclecticiq |
|
Do not edit or delete this file. If it’s there, Neo4j was installed successfully. |
/etc/eclecticiq-neo4jbatcher/neo4jbatcher.conf |
|
Defines the configuration for Neoj4batcher, the ingestion preprocessor for Neo4j. |
/etc/eclecticiq-nginx/sites.conf.d/eclecticiq-default.conf |
|
Defines a default Nginx configuration for the platform. |
/etc/eclecticiq-nginx/nginx.centos.conf |
|
Defines the designated user with access to Nginx in CentOS: nginx. |
/etc/eclecticiq-nginx/nginx.common.conf |
|
Defines further Nginx configuration parameters for the platform. |
/etc/eclecticiq-nginx/proxy_params.conf |
|
Defines the HTTP headers the Nginx sets as a proxy. |
/etc/eclecticiq-nginx/eclecticiq_servername.local.conf |
|
Defines the name of the server hosting the platform. It takes the same value as the Nginx server_name directive. |
/etc/eclecticiq-nginx/locations.conf.d/platform-frontend.conf |
|
Defines the frontend configuration for the web server. |
/etc/eclecticiq-nginx/locations.conf.d/neo4jbatcher.conf |
|
Defines specific web server configuration for Neo4jbatcher. |
/etc/eclecticiq-nginx/locations.conf.d/tip-backend.conf |
|
Defines the backend configuration for the web server, including the root endpoint exposing the public API, and the endpoint exposing the TAXII server service. |
/etc/postfix/main.cf |
|
Besides configuring email addresses through the GUI, set SMTP email server options in the main.cf file. |
/etc/eclecticiq-postgres/eclecticiq-postgres.conf |
|
|
/etc/eclecticiq-postgres/pg_hba.conf |
|
PostgreSQL client authentication configuration file. |
/etc/eclecticiq-postgres/configured-by-eclecticiq |
|
Do not edit or delete this file. If it’s there, PostgreSQL was installed successfully. |
/etc/eclecticiq-redis/redis.conf |
|
|
/etc/eclecticiq-redis/local.conf |
|
It defines the default directory of the Redis database, and it stores access credentials to it. Default directory: /media/redis |
/etc/eclecticiq-redis/configured-by-eclecticiq |
|
Do not edit or delete this file. If it’s there, Redis was installed successfully. |
|
|
Statsite files for, respectively:
|
Log files#
To get a list with the log files created by the platform and its components, run the following command:
# Returns all log files in the '/log' subdir
find /var/log -type f
# Returns EclecticIQ Platform-specific log files
# in the '/log' subdir
find /var/log -type f | grep 'eiq*\|eclecticiq*'
The response returns a list with the following files:
# Platform core services and components logs:
# ingestion, graph, OpenTAXII, scheduler, Statsite
/var/log/eclecticiq/eiq-backend-web.log
/var/log/eclecticiq/eiq-graphindex.log
/var/log/eclecticiq/eiq-ingestion.log
/var/log/eclecticiq/eiq-neo4jbatcher.log
/var/log/eclecticiq/eiq-opentaxii.log
/var/log/eclecticiq/eiq-searchindex.log
/var/log/eclecticiq/eiq-scheduler.log
/var/log/eclecticiq/eiq-statsite.log
# Celery task workers
/var/log/eclecticiq/eiq-worker-discovery.log
/var/log/eclecticiq/eiq-worker-discovery-priority.log
/var/log/eclecticiq/eiq-worker-enrichers.log
/var/log/eclecticiq/eiq-worker-enrichers-priority.log
/var/log/eclecticiq/eiq-worker-entity-rules-priority.log
/var/log/eclecticiq/eiq-worker-extract-rules-priority.log
/var/log/eclecticiq/eiq-worker-incoming-transports.log
/var/log/eclecticiq/eiq-worker-incoming-transports-priority.log
/var/log/eclecticiq/eiq-worker-outgoing-feeds.log
/var/log/eclecticiq/eiq-worker-outgoing-feeds-priority.log
/var/log/eclecticiq/eiq-worker-outgoing-transports.log
/var/log/eclecticiq/eiq-worker-outgoing-transports-priority.log
/var/log/eclecticiq/eiq-worker-reindexing.log
/var/log/eclecticiq/eiq-worker-retention-policies.log
/var/log/eclecticiq/eiq-worker-retention-policies-priority.log
/var/log/eclecticiq/eiq-worker-utilities.log
/var/log/eclecticiq/eiq-worker-utilities-priority.log
# Elasticsearch search indexing logs
/var/log/elasticsearch/intel.log
/var/log/elasticsearch/intel-2020-08-06.log
/var/log/elasticsearch/intel_deprecation.log
/var/log/elasticsearch/intel_index_indexing_slowlog.log
/var/log/elasticsearch/intel_index_search_slowlog.log
# Logstash log data aggregation logs
/var/log/logstash/logstash-plain-2020-08-05-1.log.gz
/var/log/logstash/logstash-plain.log
/var/log/logstash/logstash-slowlog-plain.log
# Neo4j graph database log
/var/log/neo4j/debug.log
# Nginx web server logs
/var/log/nginx/access.log
/var/log/nginx/error.log
# PostgreSQL intel database log
/var/log/postgresql/postgresql-2020-08-06.log
# Redis message broker log
/var/log/redis/eclecticiq-redis.log
This overview includes further details about the content of each log file.
File name and location |
Owner |
Description |
|---|---|---|
/var/log/eclecticiq/eiq-backend-web.log |
|
Platform log file. It logs core platform information. |
/var/log/eclecticiq/eiq-graphindex.log |
|
Neo4j graph database log file. It logs information about graph database migrations, indices, and graph ingestion queue. |
/var/log/eclecticiq/eiq-ingestion.log |
|
Intel ingestion log file. It logs information about ingestion events, as well as ingested batches and packages. |
/var/log/eclecticiq/eiq-neo4jbatcher.log |
|
Neo4jbatcher graph database preprocesser log file. It logs information about preprocessing data for further funneling and queuing it to be loaded to the graph. |
/var/log/eclecticiq/eiq-opentaxii.log |
|
It logs OpenTAXII server log information. |
/var/log/eclecticiq/eiq-scheduler.log |
|
It logs information about scheduled tasks; for example, feed runs. |
/var/log/eclecticiq/eiq-searchindex.log |
|
Search indexing log file. It logs information about Elasticsearch data indexing. |
/var/log/eclecticiq/eiq-worker-discovery.log /var/log/eclecticiq/eiq-worker-discovery-priority.log |
|
It logs information about discovery tasks scanning incoming sources, such as enrichers and incoming feeds, to retrieve newly ingested intelligence. It logs information on the execution order of scheduled tasks, based on priority criteria. |
/var/log/eclecticiq/eiq-worker-enrichers.log /var/log/eclecticiq/eiq-worker-enrichers-priority.log |
|
It logs information about enricher task activity, their intelligence providers, enricher priorities, and enricher-related utility tasks running in the background. It logs information about the execution order of enrichment tasks, based on priority criteria. |
/var/log/eclecticiq/eiq-worker-extract-rules-priority.log |
|
It logs information about the execution order of entity rules, based on priority criteria. |
/var/log/eclecticiq/eiq-worker-entity-rules-priority.log |
|
It logs information about the execution order of entity rules, and about modifications to enricher rate limits. |
/var/log/eclecticiq/eiq-worker-incoming-transports.log /var/log/eclecticiq/eiq-worker-incoming-transports-priority.log |
|
It logs information about integrations such as incoming feeds, their intelligence providers, incoming feed priorities, and incoming feed-related tasks running in the background. It logs information about the execution order of incoming feed task runs, based on priority criteria. |
/var/log/eclecticiq/eiq-worker-outgoing-feeds.log /var/log/eclecticiq/eiq-worker-outgoing-feeds-priority.log |
|
It logs information about integrations such as outgoing feeds, their intelligence providers, outgoing feed priorities, and outgoing feed-related tasks running in the background. It logs information about the execution order of outgoing feed task runs, based on priority criteria. |
/var/log/eclecticiq/eiq-worker-outgoing-transports.log /var/log/eclecticiq/eiq-worker-outgoing-transports-priority.log |
|
It logs information about data connections to notify about establishing connections, Celery workers syncing with other workers as they start up, and initiating outbound data transmissions; for example, by running an outgoing feed. It logs information about the execution order of outgoing transport tasks, based on priority criteria. |
/var/log/eclecticiq/eiq-worker-reindexing.log |
|
It logs information about synced enricher tasks, intelligence providers, feed transport types, and platform utility tasks. Redis and Celery take care of task worker indexing, queuing, and syncing. |
/var/log/eclecticiq/eiq-worker-retention-policies.log /var/log/eclecticiq/eiq-worker-retention-policies-priority.log |
|
It logs information about data retention policy events. It logs information about the execution order of data retention policies, based on priority criteria. |
/var/log/eclecticiq/eiq-worker-utilities.log /var/log/eclecticiq/eiq-worker-utilities-priority.log |
|
It logs information about integrations such as enricher tasks, intelligence providers, and platform utility tasks running in the background. It logs information about the execution order of task workers, based on priority criteria. |
/var/log/elasticsearch/intel_deprecation.log |
|
It logs information about deprecated Elasticsearch index mapping types. |
/var/log/elasticsearch/intel.log |
|
Elasticsearch log file. It logs Elasticsearch events such as initialization, startup, designated Elasticsearch cluster, and so on. |
/var/log/elasticsearch/intel.log.YYYY-MM-DD.log |
|
Elasticsearch log file for a specific date. YYYY-MM-DD (year, month, day) in the file name is replaced by the date the log information refers to. It logs Elasticsearch events such as initialization, startup, designated Elasticsearch cluster, and so on. |
/var/log/elasticsearch/intel_index_indexing_slowlog.log |
|
Elasticsearch log file. It logs Elasticsearch indexing information. |
/var/log/elasticsearch/intel_index_search_slowlog.log |
|
Elasticsearch log file. It logs Elasticsearch search index information. |
/var/log/logstash/logstash.err |
|
It logs Logstash errors and error messages. |
/var/log/logstash/logstash-plain.log |
|
Most recent Logstash events log file. |
/var/log/logstash/logstash-plain-YYYY-MM-DD.log |
|
Historical Logstash events log files. YYYY-MM-DD (year, month, day) in the file name is replaced by the date the log information refers to. |
/var/log/neo4j/debug.log |
|
Neo4j graph database debug log containing detailed information about database events. Useful when troubleshooting. |
/var/log/nginx/access.log |
|
Nginx log file. It logs web server access information. |
/var/log/nginx/error.log |
|
Nginx log file. It logs web server error information. |
/var/log/postgresql/postgresql-YYYY-MM-DD.log |
|
PostgreSQL log file. YYYY-MM-DD (year, month, day) in the file name is replaced by the date the log information refers to. It logs PostgreSQL database ingestion information. |
/var/log/redis/eclecticiq-redis.log |
|
Redis log file. It logs message broker event information about memory usage during copy-write operations and data saving to the database. |