Saving data URIs and raw artifacts

The extracted entity data that is stored inside observables ranges from short, simple data such as email addresses, domain names, IP addresses, and so on, to binary data.
When an entity contains binary data — for example, a file, a memory region, or packet capture (PCAP) data — the data can be represented as either a data URI or a CybOX raw artifact element.

During ingestion, extraction logic handles binary data URI and raw artifact objects embedded in CybOX objects in the following way:

Data URIs are extracted and stored as entity attachments and new hash values:

• The data URI value is recalculated to a new hash: uri-hash-sha256.
  The SHA-256 hash value for uri-hash-sha256 is calculated over the UTF-8 encoding of the data URI string.
  The uri-hash-sha256 hash substitute enables entity correlation among entities containing the same data URI.

• The binary data/raw content embedded in the data URI is decoded and processed:

  • The extracted binary data content is stored as an entity attachment similar to the name="Raw_Artifact" type="ArtifactObj:RawArtifactType" CybOX object.

  • The extracted content is hashed using SHA-512, SHA-256, SHA-1, and MD5.
    Each resulting hash is added to the relevant entities as an observable.