Merge entities#
Merge almost identical entities into a master entity and rewire relationships to reduce data noise.
Occasionally, some entities — typically, TTPs and indicators — can exist in EclecticIQ Intelligence Center as multiple, distinct entities even if they share identical titles, descriptions, and types.
They are processed and ingested as separate entities because they have different STIX IDs and timestamps.
This can occur, for example, when the source data is not well-formed.
To reduce data noise, you can merge similar entities into a master entity.
In this context, similar entities have the following characteristics:
Identical content as for title, description, and other STIX data fields.
Different STIX ID.
Different timestamp.
Before you start#
Caution
Use entity merging with caution: it is not possible to undo a merge action.
All merged entities disappear: they are not indexed, and therefore they are not searchable through the GUI.
They persist in the main data storage (PostgreSQL): to search these entities, run a SQL query in PostgreSQL.
Merged entities can generate errors when you export them from a source Intelligence Center instance to a destination one when an edge case occurs.
For example, let’s consider a merge rule running on the source data Intelligence Center that works like this:
Merge all incoming TTPs to a master TTP entity, AND
If 2 incoming TTPs are related — as in: TTP1 -> related TTP -> TTP2 — merge them into a master TTP.
When both conditions apply, this merge rule produces merged entities that have the same node as both source and target of the relationship: Master TTP -> related TTP -> Master TTP.
When the destination Intelligence Center instance ingests data from the source instance, it runs a number of checks to validate incoming content.
Among other things, it checks that there are no circular relations: a relation cannot have the same node as both source and target.
Therefore, the destination Intelligence Center rejects
Master TTP -> related TTP -> Master TTP-type incoming merged entities,
and it returns an error:
Relation source and target cannot be the same.
Similar entities#
In this context, similar entities have the following characteristics:
Identical content as for title, description, and other STIX data fields.
Different STIX ID.
Different timestamp.
From a point of view of information relevance and intelligence value, you can handle these entities like duplicates, and you can decide to merge similar entities into a master entity. You can manually create a new entity, as well as use an existing one as the master entity to merge similar entities into.
To control the merging process, define a merge entity rule with a set of criteria and a merge action. Rules apply to new and to historical, pre-existing entities. Therefore, a merge rule merges new and historical entities into the selected master entity, based on the specified criteria.
When merging similar entities into a master entity, the merge rule handles similar/duplicate entities as follows:
New similar entities that have been already processed, but not yet saved to the database, are ignored because they are duplicates.
Any incoming or outgoing relationships they may have are automatically rewired, so that they refer to the master entity.
Historical, pre-existing similar entities are removed because they are duplicates.
Any incoming or outgoing relationships they may have are automatically rewired, so that they refer to the master entity.
Any existing workflow items merged historical entities may have — for example, workspaces or tasks — are also automatically rewired in the same way.
Merged entities are not deleted from the database, since the Intelligence Center uses them for idref resolution. However, they are not indexed, and therefore not searchable in the Intelligence Center.
You can still search for these entities by running SQL queries in PostgreSQL.
A successful merge action produces also an audit entry recording the main details of the operation.