About observables


Observables are discrete pieces of information that represent properties, attributes, actions, and events.
They record a distinct piece of information, such as: an IP address, a hash, name of a country, name of a city, name of an organization, or the name of an individual; or an event such as the creation of a registry value, or a file deletion or modification.

They are atomic: the information they hold is complete and meaningful, but it cannot be split into smaller components without losing meaning and intelligence value.
They are factual: they record facts with no additional context or background.

You can relate observables with entities to provide context.
If observables are detected in a specific context, or if they are sighted within the organization, they become indicators and sightings, respectively.


About observable access control

Observable access control is based on inheritance: observables inherit the same set of access control permissions as the entities they are related (connected) to.


Observables inside structured text

When you select Skip extraction of observables from unstructured text in the configuration of incoming feeds or a manual file upload, the Intelligence Center filters and ingests only higher-value observables.
Typically, these are observables inside structured text.

  • Observables inside structured content are typically intelligence-richer, and therefore more valuable, than observables inside unstructured content.

  • Observables inside structured content are usually bundled with an entity when they are ingested; they are extracted from structured fields, or parsed from structured CybOX fields.

  • Observables whose values are retrieved from CybOX fields usually have higher intelligence value, and they are more relevant because they are directly related to the parent STIX entity they refer to.

  • In the same way, observables whose relationship to an entity has a link name value are likely to carry more meaning than observables without a link name: entity-observable relationships with a link name provide additional context, and they help understand, for example, how a specific resource is used, or the purpose it serves to an attacker.

You can leverage the intelligence value of these observables.
For example, you may decide to (re)act by instrumenting your prevention/detection components to automatically block or blacklist them.

Example:

  • A URI in a CybOX URIObj field: http://www.evil.com/big/info.php

Use cases:

  • If you select Skip extraction of observables from unstructured text in the configuration of an incoming feed or of a manual file upload, the Intelligence Center filters and ingests only higher-value observables.

  • If you leave Include observables without a link name deselected in the configuration of an outgoing feed, the Intelligence Center includes in the outgoing feed content only observables with the specified link name value(s) describing specific types of relationship between observables and their parent entities.

  • If you select Link name filter > Link types in the configuration of an observable rule, the Intelligence Center applies the rule only to observables with the specified link name value(s) describing specific types of relationship between observables and their parent entities.


Observables inside unstructured text

When you select Skip extraction of observables from unstructured text in the configuration of incoming feeds or a manual file upload, the Intelligence Center excludes these lower-value observables from ingestion.
Deselect this option to include them and to ingest them along with any observables inside structured content.

  • These observables may be of low quality, and they may clutter, rather than enhance, the overall intelligence value of your Intelligence Center data.

  • Observables inside unstructured text are typically not included in CybOX objects, and they usually do not have link names defining their relationships, if any.

  • They can be mentioned inside STIX fields such as headers, titles, descriptions, where they are included for reference.

  • These observables usually have lower intelligence value, and they are less relevant because they are indirectly related to the parent STIX entities they belong to.

Example:

  • A URI in a STIX Reference field: http://www.evil.com/big/info.php

Use cases:

  • If you leave Skip extraction of observables from unstructured text deselected in the configuration of an incoming feed or of a manual file upload, the Intelligence Center filters and ingests also observable data detected in unstructured content and without link names defining their relationships.

  • If you select Include observables without a link name in the configuration of an outgoing feed, the Intelligence Center includes in the outgoing feed content also observables with an undefined link name, that is, with undefined relationships.

  • If you select Link name filter > Include observables without a link name in the configuration of an observable rule, the Intelligence Center applies the rule also to observables with an undefined link name.


images/download/attachments/27460169/structured-unstructured-observables.png
Difference between structured and unstructured observables


Observable vs Observable XML

In the Observables tab of entity detail panes you can filter entity observables by images/download/attachments/3604538/filter.PNG > Relation to review only entity observables that are associated with the entity through a specific type of relation.

Observable and Observable XML are among the available relation options:

  • Observable: the observable content is parsed and analyzed from the parent entity as an observed piece of factual information.
    It is usually derived from structured text, and its intelligence value is usually more relevant.

  • Observable XML: the observable content is extracted from plain text from the delivered observable XML field in the source data.
    It is usually derived from unstructured text, and its intelligence value is usually less relevant.


Observable categories

It is possible to group the observable data the Intelligence Center ingests in the following categories:


Observable

Either a manually added observable, or an ingested observable without any further context than the observable value.

Enrichment observable

An ingested observable as a result of an enrichment operation.

Nested observable

An ingested observable that is embedded in an entity.

Bundled observable

An ingested observable that has one or more relationships with other entities and observables, and that is part of the same ingestion package holding also the related entities and observables it refers to.

Related observable

An ingested observable that has one or more relationships with other entities and observables.


Observable types

The available observable types in the Intelligence Center are:

_constants.py (sourced from EIQ platform-backend)

Author

wouter bolsterlee

Commit

17a58f9f930d83ee862b731813ff472ea3994a37

Timestamp

February, 14, 2022 11:59 AM

Full path

eiq/platform/extracts/_constants.py

Title

[SNYK] Upgrade packages and ignore issues with no upgrade path

Description

**Upgrade packages:**<br> `ipython==7.16.0` => `ipython==7.16.3` == no risk <br> `cairosvg==2.4.2`=> `cairosvg==2.5.2` == no risk <br> `jinja2==2.10.1` => `jinja2==2.11.3` == no risk<br> `pillow==7.2.0` => `pillow==8.3.2` == no risk <br> `pygments==2.6.1` => `pygments==2.7.4` == no risk <br> <br> **Snyk Ignore:** <br> _Removed issues that no longer affect our product._<br> Increase ignore time for following issues:<br> snyk:lic:pip:html2text:GPL-3.0 - can't be applied for 2.9<br> SNYK-PYTHON-PIP-609855 - can't upgrade PIP due to incompatibility with credential escaping<br> SNYK-PYTHON-PIP-1278135 - can't upgrade PIP due to incompatibility with credential escaping<br> SNYK-PYTHON-DATEPARSER-1063229 - no fix available<br> SNYK-PYTHON-CELERY-2314953 - fix can't be apply due to incompatibility with python 3.6<br> SNYK-PYTHON-PILLOW-2329135 - fix can't be apply due to incompatibility with python 3.6<br> SNYK-PYTHON-PILLOW-2331905 - fix can't be apply due to incompatibility with python 3.6<br> SNYK-PYTHON-PILLOW-2331907 - fix can't be apply due to incompatibility with python 3.6<br> SNYK-PYTHON-PILLOW-2331901 - fix can't be apply due to incompatibility with python 3.6<br> SNYK-PYTHON-PILLOW-2397241 - fix can't be apply due to incompatibility with python 3.6<br> SNYK-PYTHON-CRYPTOGRAPHY-1070544 - can't apply fix risk accepted SNYK-PYTHON-PYSAML2-1063038 - can't apply fix risk accepted SNYK-PYTHON-PYSAML2-1063039 - can't apply fix risk accepted See merge request engineering/platform-backend!6465

SUPPORTED_EXTRACT_TYPES = {
"actor-id",
"address",
"asn",
"bank-account",
"card",
"card-owner",
"cce",
"city",
"company",
"country",
"country-code",
"cve",
"cwe",
"domain",
"email",
"email-subject",
"eui-64",
"file",
"forum-name",
"forum-room",
"forum-thread",
"fox-it-portal-uri",
"geo",
"geo-lat",
"geo-long",
"handle",
"hash-authentihash",
"hash-imphash",
"hash-md5",
"hash-rich-pe-header",
"hash-sha1",
"hash-sha256",
"hash-sha512",
"hash-ssdeep",
"hash-vhash",
"host",
"industry",
"inetnum",
"ipv4",
"ipv6",
"ja3-full",
"ja3-hash",
"ja3s-full",
"ja3s-hash",
"mac-48",
"malware",
"mutex",
"name",
"nationality",
"netname",
"organization",
"person",
"port",
"postcode",
"process",
"product",
"registrar",
"rule",
"snort",
"street",
"telephone",
"uri",
"uri-hash-sha256",
"winregistry",
"yara",
}


Get observable types through the API

To send API requests to API endpoints you must have a valid API token.
You pass the API token as a Bearer token with any requests you send to the Intelligence Center API.

To obtain an API token to programmatically authenticate through API requests:

  1. Sign in to the Intelligence Center through the web-based GUI.

  2. Create an API token.

  3. Pass it as a Bearer token in an Authorization HTTP header with any requests you send to the Intelligence Center API.

    The Authorization HTTP header has the following format:
    Authorization: Bearer ${token}

To send an API request to retrieve a JSON response with all supported observable types:

  • Obtain an API Bearer token.

  • Send a request to the /private/extracts/kinds/ API endpoint (keep the trailing slash in the URL):


curl -X GET \
-v \
--insecure \
-i \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "Authorization: Bearer ${token}" \
--url https://${platform_host}/private/extracts/kinds/

The response is a JSON array listing all the supported observable types:


{
"data": [
{
"kind": "geo"
},
{
"kind": "fox-it-portal-uri"
},
{
"kind": "telephone"
},
{
"kind": "uri"
},
{
"kind": "hash-md5"
},
{
"kind": "cve"
},
{
"kind": "nationality"
},
{
"kind": "registrar"
},
{
"kind": "organization"
},
{
"kind": "cce"
},
{
"kind": "uri-hash-sha256"
},
{
"kind": "hash-rich-pe-header"
},
{
"kind": "handle"
},
{
"kind": "mac-48"
},
{
"kind": "forum-thread"
},
{
"kind": "forum-room"
},
{
"kind": "country-code"
},
{
"kind": "hash-imphash"
},
{
"kind": "ja3s-hash"
},
{
"kind": "hash-sha256"
},
{
"kind": "actor-id"
},
{
"kind": "mutex"
},
{
"kind": "forum-name"
},
{
"kind": "file"
},
{
"kind": "malware"
},
{
"kind": "email-subject"
},
{
"kind": "hash-sha1"
},
{
"kind": "postcode"
},
{
"kind": "yara"
},
{
"kind": "cwe"
},
{
"kind": "winregistry"
},
{
"kind": "netname"
},
{
"kind": "city"
},
{
"kind": "hash-sha512"
},
{
"kind": "ipv6"
},
{
"kind": "bank-account"
},
{
"kind": "ja3s-full"
},
{
"kind": "snort"
},
{
"kind": "hash-authentihash"
},
{
"kind": "company"
},
{
"kind": "process"
},
{
"kind": "port"
},
{
"kind": "address"
},
{
"kind": "hash-ssdeep"
},
{
"kind": "rule"
},
{
"kind": "ja3-hash"
},
{
"kind": "host"
},
{
"kind": "asn"
},
{
"kind": "industry"
},
{
"kind": "card"
},
{
"kind": "street"
},
{
"kind": "ipv4"
},
{
"kind": "name"
},
{
"kind": "country"
},
{
"kind": "email"
},
{
"kind": "eui-64"
},
{
"kind": "ja3-full"
},
{
"kind": "person"
},
{
"kind": "card-owner"
},
{
"kind": "inetnum"
},
{
"kind": "hash-vhash"
},
{
"kind": "geo-lat"
},
{
"kind": "geo-long"
},
{
"kind": "domain"
},
{
"kind": "product"
}
]
}