Search | Query Syntax#

Basic operators and syntax#

Search fields in EclecticIQ Intelligence Center use the Elasticsearch DSL query syntax unless otherwise specified.

Query syntax element

Description

AND

Boolean operator between filters.

NOT

Boolean operator between filters.

OR

Boolean operator between filters.

"we want to look up all these keywords"

Use double quotes to group multiple search entries into one single search string.

The search query looks for matches for the string as one token.

keywo*

Searches for words containing criteria.

Use the * character for multiple character wildcards.

key?ord

Searches for words containing criteria.

Use the ? character for single character wildcards.

keyword~

Searches for similar words. Fuzzy search.

"keyword1"^2 AND keyword2

Use the ^${integer} suffix to give more weight to a search term over another in a search query.

+keyword

Prefix + to include a keyword in a search (must be in the search).

-keyword

Prefix - to exclude a keyword from a search (must not be in the search).

/keyw?rd)/

Searches using Elasticsearch DSL query syntax regular expressions.

Wildcards#

  • Currently supported wildcards: * and ?

  • Escape character: \

  • Avoid starting a wildcard query term with * or ?, as it may result in an extremely slow-running query.

    Search and rules use literal values and regex patterns to look for matching values assigned to the corresponding JSON path fields.

Example

Use Boolean operators and wildcards to create more refined searches.

enrichment_extracts.kind:domain AND enrichment_extracts.meta.classification:high

Search with regular expressions#

The regex syntax this field accepts is the Elasticsearch regular expression syntax.

The main peculiarities of the Elasticsearch DSL query syntax are:

  • Anchors – ^ and $ – are implied at the beginning and at the end of the regex.

    Do not include them in the regex pattern input.

  • If you insert explicit anchor characters in the value input field, they are interpreted as literal values.

  • Escape special characters:

    ( . ? + | { } [ ] ( ) " \ / )
    

    To escape a special character, prepend a backslash \ to it.

    Example: \{ \}

    Note

    At the moment, Elasticsearch regular expression syntax optional operators are not supported.

    Note

    Although the regex syntax used in the UI of the Intelligence Center is that of ElasticSearch, the regex syntax used by EclecticIQ Intelligence Center’s ingestion engine is Python regex. In the vast majority of cases this does not lead to conflicts, but if your discovery rule is not working the way you want it to, this may be the reason.

    Here are two differences that commonly cause problems:

    • ElasticSearch searches for whole strings, while Python searches everywhere in a string. For example,

      • Input: Cert-BUND

      • Pattern: ert

      • ES result:

      • Python: will match

    • ElasticSearch regex syntax requires you to leave out anchors, whereas Python regex syntax requires you to include them.

    If you are experiencing problems in this area, contact Customer Support for help in writing the discovery rule you want.

Search in specific JSON fields#

You can also specify JSON paths to search for the corresponding values in specific JSON fields of an entity data structure.

To do so, in the search input field enter the JSON path to the field whose value you want to look up, a semicolon, and the literal or data pattern to look up.

Format:

json.path:${search_string}

Example:

data.types.value:"IP Watchlist"

Search for observable properties#

For example, the following JSON paths point to JSON fields holding observable values:

Field

Description

Example

extracts.kind

Search for a specific observable type.

extracts.kind:ipv4

extracts.meta.classification

Search for observables based on classification :

  • good

  • bad

  • unknown

extracts.meta.classification:bad

extracts.meta.confidence

Search for observables based on confidence :

  • high

  • medium

  • low

extracts.meta.confidence:high

extracts.meta.blacklisted

Search for observables based on blacklist status:

  • true

  • false

extracts.meta.blacklisted:false

enrichment_extracts.kind

Search for a specific enrichment observable type.

enrichment_extracts.kind:ipv4

enrichment_extracts.meta.classification

Search for enrichment observables based on classification :

  • good

  • bad

  • unknown

enrichment_extracts.meta.classification:unknown

enrichment_extracts.meta.confidence

Search for enrichment observables based on confidence :

  • high

  • medium

  • low

enrichment_extracts.meta.confidence:medium

enrichment_extracts.meta.blacklisted

Search for enrichment observables based on blacklist status:

  • true

  • false

enrichment_extracts.meta.blacklisted:true

Search for entities#

Search for entiites by going to Search icon Search > Go to search and browse > Observables.

Tip

You can search for entities by querying the observables they are connected to. See the extracts or meta.bundled_extracts keywords in the Entity attributes table below.

Entity attributes#

The table below lists common attributes you can use to search for entities.

Tip

There are more granular attributes you can search with. Use the search bar’s autocomplete to help you find attributes to use, see individual entity documentation for more information.

Query key

Looks for

Query value examples

created_at

Entities whose creation date matches the specified pattern or literal.

[now-1w TO *], [now-24h TO *], [now-1w TO *], [now-1y TO *], [* TO *]

created_by

Entities whose user ID integer value matches the specified pattern or literal.

*, 1

data.confidence.value

Entities whose observable maliciousness confidence value matches the specified pattern or literal.

high, medium, low, none, unknown

data.kill_chain_phases.kill_chain_name

Entities whose kill chain phase name matches the specified pattern or literal.

reconnaissance, weaponization, delivery, exploitation, installation, command and control, actions on objectives

data.kill_chain_phases.name

Entities whose official or standard name matches the specified pattern or literal.

LMCO Kill Chain

data.kill_chain_phases.ordinality

Entities whose integer order value matches the specified pattern or literal.

This value defines the order of a kill chain phase within a kill chain.

1, 2, 3, 4, 5, 6, 7

data.observable.title

Entities with at least an observable whose title/header matches the specified pattern or literal.

Mirai botnet-related observable

data.producer.identity.name

Entities whose data producer name matches the specified pattern or literal.

The Entity per producer gauge on the dashboard used this field to display the total amount of ingested entities, based on the corresponding producer.

phishtank, hailataxii

data.producer.time_produced

Entities whose creation time at the data producer matches the specified pattern or literal.

Fractions of seconds are optional and may not always be included in the time value.

Input format: yyyy-MM-ddTHH:mm:ssZ

Description: year, month, day, hour, T (the date and time separator), minutes, seconds, time zone

2016-11-08T05\:04\:12Z, 2016-11-08T05\:04\:12\+00\:00, [now-24M/M TO 2016-01-01]

data.producer.time_received

Entities whose reception time at the data producer matches the specified pattern or literal.

Fractions of seconds are optional and may not always be included in the time value.

Input format: yyyy-MM-ddTHH:mm:ssZ

Description: year, month, day, hour, T (the date and time separator), minutes, seconds, time zone

2015-03-26T14\:28\:24Z, 2015-03-26T14\:28\:24\+00\:00, [now-24M/M TO 2016-01-01]

data.sightings_count

Entities with at least an observable that has actually been sighted.

This is a counter whose integer value reports the number of sightings recorded for the corresponding observable.

*, 1, 2, 3

data.timestamp

Entities whose data creation time matches the specified pattern or literal.

Fractions of seconds are optional and may not always be included in the time value.

Input format: yyyy-MM-ddTHH:mm:ssZ

Description: year, month, day, hour, T (the date and time separator), minutes, seconds, time zone

2015-03-26T14\:28\:24Z, 2015-03-26T14\:28\:24\+00\:00, [now-24M/M TO 2016-01-01]

data.type

Entity type. See Table of possible entity types.

indicator

exposure.sighted

Entities with at least an observable that has actually been sighted.

When entities are associated with a sighting, they are exposed.

Entities with exposure.sighted:true have a data.sightings_count value of at least 1.

true, false

meta.source_reliability

Entities whose data source reliability matches the specified pattern or literal.

A, (A B C)

meta.tlp_color

Entities whose TLP color matches the specified pattern or literal.

RED, AMBER, GREEN, WHITE, NONE

meta.tags

Entities whose custom tag values match the specified pattern or literal.

malware, ransomware

tags

Entities whose custom tag and standard taxonomy values match the specified pattern or literal.

malware, ransomware

_exists_:"attached_files"

When added, returns only Report entities with files attached. When excluded, returns Report entities without files attached and all other entities.

_exists_:"attached_files", !(_exists_:"attached_files")

Table of possible entity types:

Entity type

JSON value

Attack Pattern

attack-pattern

Campaign

campaign

Course of Action

course-of-action

Exploit target

exploit-target

Sighting

eclecticiq-sighting

Identity

Identity

Indicator

indicator

Incident

incident

Infrastructure

Infrastructure

Intrusion set

intrusion-set

Location

location

Malware

malware

Malware analysis

malware-analysis

Report

report

Threat actor

threat-actor

Tactics, Techniques, Procedures

ttp

Tool

tool

Search entities by observables#

Entities can have embedded observables. These are usually entities and observables ingested through incoming feeds.

You can search for entities that are directly related to or embed certain observables by using the extracts or enrichment_extracts keyword.

See Search for observables for a list of attributes to use with these keywords.

E.g. To search for entities containing an extract with an ID of 4:

extracts.id: 4

Search entities by outgoing feeds#

Outgoing feeds are indexed as destinations. Search for entities and observables published through an outgoing feed based on the feed name, type, or ID.

Query key

Description

destinations.name

Outgoing feed name.

Example: destinations.name: "Poll TAXII Stand"

Note

destinations.name is only set when an entity is packed for an outgoing feed. It is not updated when the name of that outgoing feed is changed, only when it is re-packed. This can result in entities not carrying the updated name of the outgoing feed it belongs to.

destinations.type

Outgoing feed transport type.

Example: `destinations.type: “TAXII Poll”

destinations.id

(Deprecated. Search using destinations.name instead.)

UUID of outgoing feed. To find the UUID of an outgoing feed, make a request to the /outgoing-feeds Public API endpoint.

Tip

Outgoing feeds have two IDs:

  • a numeric ID.

  • a destination ID, which is a UUID and is only available through a GET /outgoing-feeds request.

To find the UUID of an outgoing feed:

  1. Find the numeric ID of the feed.

    1. Go to Data configuration icon Data configuration > Outgoing feeds and select the feed to open it.

    2. Inspect the URL in your browser’s address bar. The numeric ID of the feed is the value of the detail parameter.

      E.g. In https://ic-playground.eclecticiq.com/main/configuration/outgoing-feeds?tab=detail&detail=5 the feed numeric ID is 5

  2. Make a REST API request to retrieve the feed IDL

    1. Make sure you have an API token.

    2. Make a request to /outgoing-feeds/{id}, where {id} is the numeric ID of the outgoing feed. E.g.:

      curl -H "Authorization: Bearer $APITOKEN" \
        https://ic-playground.eclecticiq.com/api/v2/outgoing-feeds/5
      
    3. Inspect the response. The UUID of the feed is the value of data.destination

Search for observables#

Search for observables by going to Search icon Search > Go to search and browse > Observables.

Observables are indexed and searchable using the extracts and enrichment_extracts keywords.

Tip

enrichment_extracts are extracts created as a result of running an enricher.

When searching for observables in Search icon Search > Go to search and browse > Observables, you can omit the extracts keyword and search directly using attributes.

E.g. To search for an observable with an ID of 4:

id: 4

Observable attributes#

The table below lists common attributes you can use to search observables with.

Entity JSON path

Description

Example values

id

Unique ID of observable.

13

value

Entities with at least an observable whose value matches the specified pattern or literal.

The value attribute has tokens and keyword attributes you can use. See Search | Using Tokenizers.

malware.win32.sample

kind

Entities with at least an observable whose data type matches the specified pattern or literal.

ipv4, name

meta.confidence

Confidence assigned to an observable. Used together with classification to determine maliciousness. See Observable maliciousness.

high, medium, low

meta.classification

Classification assigned to an observable. Used together with confidence to determine maliciousness. See Observable maliciousness.

good, bad, unknown

meta.blacklisted

Whether or not an observable has been ignored by an Observable rules.

true, false

Relationships#

From release 3.3.0 onward, use the relational query UI to search entity relationships.

Search with date and time#

Use only ISO 8601 date-time formats:

  • YYYY-MM-DD

  • YYYY-MM-DDTHHZ

  • YYYY-MM-DDTHH:mmZ

  • YYYY-MM-DDTHH:mm:ssZ

As a complete example, 12:35 p.m. on 25 January 2020 in Bucharest, Romania would be expressed as 2020-01-25T12:35:00+0200

Entity date-time attributes#

There are several entity attributes that you can use to search by date and time. The most commonly used are listed below along with their explanations.

Entity attribute

Description

created_at

The date and time that an entity was created in EclecticIQ Platform.

last_updated_at

The date and time that an entity’s details were last edited in EclecticIQ Intelligence Center.

data.producer.time_produced

The date and time that an incident was created.

This field is specific to incidents.

You have to type the date value as a full string (for example: 2020-01-25T12:35:00+0200).

data.reporter.time_produced

The date and time that an incident was reported.

This field is specific to incidents.

You have to type the date value as a full string (for example: 2020-01-25T12:35:00+0200).

data.time_first_malicious_action

The date and time of the first malicious of an incident.

This field is specific to incidents.

You have to type the date value as a full string (for example: 2020-01-25T12:35:00+0200).

data.timestamp

A specific version of an entity.

data.valid_time_positions.start

The start of the time window during which an indicator is valid.

This field is specific to indicators.

You have to type the date value as a full string (for example: 2020-01-25T12:35:00+0200).

data.valid_time_positions.end

The end of the time window during which an indicator is valid.

This field is specific to indicators.

You have to type the date value as a full string (for example: 2020-01-25T12:35:00+0200).

meta.first_ingest_time

The date and time of an entity’s most recent ingestion into an organization’s cluster of platforms.

You can only search using a single value.

meta.ingest_time

The date and time of an entity’s ingestion into a particular platform.

You can search using a single value or a range of values.

meta.estimated_observed_time

The date and time when an entity is estimated to have been first observed.

You can search using a single value or a range of values.

meta.estimated_threat_start_time

The date and time when the threat posed by an entity is estimated to have started.

You can search using a single value or a range of values.

meta.estimated_threat_end_time

The date and time when the threat posed by an entity is estimated to have ended

You can search using a single value or a range of values.

Date ranges#

Elasticsearch allows you to specify date ranges in your queries. Please be aware however that Elasticsearch treats queries using single dates differently to queries using date ranges. In the case of single dates, Elasticsearch only compares date values; it ignores time values and timezones. That’s why if you use ranges in your search queries you are more likely to get the results you expect.

Tip

If you use single date values in your search queries, make sure that the timezone in your user profile is the same as your system timezone, which is also the timezone of all the entities and observables in your system.

To specify ranges, enclose them either square or curly brackets:

  • [<min_datetime> TO <max_datetime>]

  • {<min_datetime> TO <max_datetime>}

Note

The TO operator must be written in uppercase.

Examples:

Query string

Description

meta.ingest_time:["2020-01-01T00:00:00" TO *]

Returns all entities that were ingested on and after 1 January 2020

meta.ingest_time:{"2020-01-01T00:00:00" TO  "2020-01-04T00:00:00"]

Returns all entities that were ingested on 2 and 3 January

meta.ingest_time:["2020-01-01T13" TO "2020-01-01T14"}

Returns all entities that were ingested on 1 January 2020 from 13:00 up to but excluding 14:00

You can also use the following shortened syntax:

Query string

Description

meta.ingest_time:[now/d TO *]

Returns all entities that were ingested today.

meta.ingest_time:[now/M TO *]

Returns all entities that were ingested this month.

meta.ingest_time:[now-24h TO *]

Returns all entities that were ingested during last 24 hours

User timezone#

The timezone you specified in your platform profile affects the value displayed for the timestamp of, for example, an entity, observable or relationship. It also serves as the default timezone for Elasticsearch queries.

If you specify a timezone in your query, Elasticsearch ignores your user timezone for the actual search. However, the search results are displayed according to your user timezone.

For example:

  • User timezone: CET (+0100)

  • Entity ingestion date-time: 21 Feb 2020 at 7:00 a.m. GMT (+0000)

  • Search string: meta.ingest_time:["2020-02-21T06:30:00+0000" TO "2020-02-21T07:30:00+0000"]

The entity appears in the search results, but because the user timezone is an hour later, the entity is displayed with a timestamp of 8:00 a.m.

This may be confusing; unless you need to share results with users in different timezones, it is preferable to avoid specifying a timezone in a search query.