About ingestion discrepancies#

Inconsistent count due to different formats (STIX vs JSON)#

Scenario#

  • A data exchange setup between two platform instances.

  • Data transmission goes through an outgoing and a corresponding incoming feed.

Issue#

  • At the end of a data exchange operation between a data source and a data destination platform instances, an entity count discrepancy may arise if the exchanged data is in a format other than EclecticIQ JSON.

Mitigation#

This is expected behavior.

What looks like a discrepancy is the result of a data optimization and normalization process whose purpose is to separate the wheat from the chaff, and to retain only data with actual intelligence value.

For example, let’s assume the source data format is STIX:

  • STIX and JSON are different formats.

  • STIX is much more nested than JSON, which has a flatter structure.

  • STIX entities are bundled in packages. After ingestion, the package is not needed any longer.

  • The ingestion process needs to parse the source data before ingesting it:

    • During parsing, STIX relationships between the package and the entities it contains are discarded, as they hold no intelligence value.

    • If the source package contains unresolved entities or observables whose idref is resolved during the parsing and the ingestion steps, the original unresolved objects are discarded, as they hold no intelligence value because they are now resolved inside the platform.

    • Relationships between entities or between entities and observables are retained, and they are correctly ingested into the platform.

Observable source not included in the recipient platform data#

Scenario#

Assets#

  • A data exchange setup between two platform instances.

  • Data transmission goes through an outgoing and a corresponding incoming feed.

  • The exchanged data content type is EclecticIQ JSON.

  • Exchanged data include enrichment observables.

Events#

  • The source enricher the observables were originally ingested from is removed from the publisher/source platform instance.

  • Data exchange between the two platform instances occurs, and it includes enrichment observables whose data source is the enricher that was removed from the the publisher/source platform instance.

Issue#

  • The recipient platform instance ingests the enrichment observables, but it does not receive any source information.

    This occurs because the source enricher the publisher/source platform instance originally ingested the observables from is not installed on the publisher/source platform instance when the data exchange occurs.

  • Exchanged enrichment observables are ingested in the recipient platform instance without any source information.

Mitigation#

To retain all the enrichment source information when you export data from a platform instance, do not uninstall an enrichment source before exporting or publishing data that may include content from that source.

Instead, disable an enricher source on the publisher/source platform instance, and then filter the exchanged data on the recipient platform instance using incoming feed configuration options.