About purging incoming feeds

Delete an incoming feed to remove from the platform any packages, as well as any entities and observables, ingested through the feed. Purge an incoming feed to completely remove any ingested data through the feed, as well as the feed configuration.

Purging incoming feeds

Purge an incoming feed to completely delete it.
The operation removes any data ingested through the feed since the first execution of the feed, as well as the incoming feed configuration.

You may want to completely delete an incoming feed to stop ingesting data from the feed source, and to remove the option to resume ingestion from the feed source at a later time.
For example, the content an incoming feed delivers may be no longer relevant; the quality of the ingested data may be unsatisfactory; a different, open source feed may provide similar content; it may be necessary to free up budget resources to fund other tools, services and activities that benefit the team.

Consider purging and completely deleting an incoming feed to achieve the following results:

  • Remove from the platform any data ingested from the data source the incoming feed retrieves its content from:

    • Downloaded packages

    • Entities

    • Observables

  • Remove the incoming feed from the platform data sources.

  • Remove the incoming feed configuration and setup.
    Besides the basic feed settings, the feed configuration that is completely removed from the platform includes also other options related to:

    • Entity rules

    • Extract rules

    • Enrichment rules

    • Data retention policies

    • User group allowed sources

    • Outgoing feeds listing the incoming feed in Include source metadata in the outgoing feed configuration.

Purging feeds and data source changes

When you delete or purge an incoming feed, you change the platform data sources map.
Deleting or purging an incoming feed affects other functionality areas of the platform, namely:

Changes to data sources can trigger rules

When the platform ingests duplicate data, it updates existing entities to add the origin of the duplicate data as a new data source.
When changes to the data sources for an entity occur – for example, adding a new data source, renaming an enricher, or deleting an incoming feed – the process triggers entity and observable rules that use the changed data sources as a criterion:

  • The current data sources of the affected entities are reassessed.
    For example, any unavailable sources, such as a deleted incoming feed, are dropped; new sources, such as the origin of a discarded duplicate package, are added.

  • The timestamp recording the most recent entity update is set to now; that is, the time the data source change occurs.

  • Entity and observable rules are triggered to update any entity information that may have changed because of the data source modification.

Changes to data sources can trigger database reindexing

Entity deletion and incoming feed purge actions trigger a database reindex process, because they involve dropping any data sources referring to the deleted incoming feed content and entities:

  • When you delete an entity, you also detach from the deleted entity any data sources referring to it.

  • When you delete or purge an incoming feed, you also remove it as a data source from the entities that refer to it.

Changes to data sources can affect user access

Changes to entity data sources affect user access to entity data.
For example, if the data source an entity refers to changes from group A to group B, group A members who are not also members of group B can no longer access the entity.

Purging feeds and entities with multiple data sources

Deleting or purging an incoming feed that is one of many data sources of an entity does not remove the entity:

  • Entities with multiple sources are kept in the database.

  • They drop the data source corresponding to the incoming feed being deleted or purged.

  • To reflect removing one of the entity's data sources, the platform triggers entity rules and a database reindexing.

To delete entities with multiple data sources, you need to delete or purge all the incoming feeds that are data sources of the entities you want to remove from the platform.

Purging entity data sources and user access

To access platform data, users must belong to at least one user group.
Each user group can access one or more data sources – incoming feeds, enrichers, and other user groups – as defined under the Allowed sources options for the group.

Dropping an entity data source can affect user access to the entity.
For example:

  • A user belongs to a user group with incoming feed C as an allowed source for the group.
    The user can can access an entity with multiple data sources A, B, and C.

  • The incoming feed corresponding to data source C is either deleted or purged.
    The entity is still available in the database with with multiple data sources A and B.

  • The user can no longer access the entity because data source C, the allowed source that would make the entity available to the user, is no longer there.
    To access the entity again, the group the user belongs to needs to have at least data source A or B as an allowed source.