Outgoing feed - SFTP upload#

Note

This article describes how to configure outgoing feeds for a particular feed source. To see how to configure outgoing feeds in general, see Create and configure outgoing feeds.

Specifications

Transport type

SFTP upload

Content type

  • Advanced Entities CSV (IC 3.3.0 and newer)

  • Advanced Observables CSV (IC 3.3.0 and newer)

  • ArcSight CEF

  • EclecticIQ Entities CSV

  • EclecticIQ HTML Report

  • EclecticIQ HTML Report Digest

  • EclecticIQ JSON

  • EclecticIQ Observables CSV

  • EclecticIQ PDF

  • PAN-OS External Dynamic List

  • Plain text value

  • STIX 1.2

  • STIX 2.1

For more information, see Configure content types.

Published data

Entities and observables.

Description

The feed publishes entities and observables in the selected content type to the specified destination location on an FTP server. Each time the outgoing feed task runs, it generates a data package containing zero or more entities, depending on the outgoing feed update strategy, and on the feed data source containing data that match the feed configuration.

Configure the outgoing feed#

  1. Create or edit an outgoing feed.

  2. From the Transport type drop-down menu, select SFTP upload.

  3. From the Content type drop-down menu, select the appropriate content type for the data you want to publish through the outgoing feed.
    The selected content type for the feed should match the actual format of the source data you are publishing through the feed.
    This can vary, depending on the dataset sources you retrieve the data for publication from.

  4. From the Datasets drop-down menu, select one or more existing datasets to use as sources to populate the outgoing feed content.
    For the feed not to be empty, at least one selected dataset should contain entities and observables in the same format as the configured content type for the feed.

  5. From the Update strategy drop-down menu, select the preferred method to populate the outgoing feed with data before publishing it.
    Update strategies help define how content is aggregated and packaged for publication when an outgoing feed task runs:

    • Append: every time the outgoing feed task runs, it fetches only new, unpublished data to generate the content for publication.
      The published packages contain only new entities and observables ingested in the platform after the previous execution of the outgoing feed.

    • Replace: every time the outgoing feed task runs, it fetches new and existing data to generate the content for publication.
      The published packages contain new and existing entities and observables included also in the previous execution of the outgoing feed.

    • Diff : this option is available only for the EclecticIQ Entities CSV and EclecticIQ Observables CSV content types. Every time the outgoing feed task runs, new data is compared against existing data to identify any differences between the two datasets:

      • At entity level: any entities added to or removed from the set, if EclecticIQ Entities CSV is the designated content type for the feed.

      • At observable level: any observable added to or removed from the entities in the set, if EclecticIQ Observables CSV is the designated content type for the feed.

      Depending on the selected CSV content option, each row in the CSV output contains information about one entity being added or removed, or one observable being added or removed.
      An extra diff column is added to the output CSV to indicate if a row, and therefore either an entity or an observable, has been added to or removed from the set.
      This option enables identifying changes in a feed between two executions without downloading the whole feed every time.

      Note

      Update strategies help define how content is aggregated and packaged for publication when an outgoing feed task runs:

      Update strategies rely on the last_updated_at database field to identify entities whose timestamp value was updated since the previous execution of the outgoing feed.
      Entities with a more recent timestamp value compared to the previous execution of the outgoing feed are packaged and included in the published content of the outgoing feed.

      • Changes to the data section of an entity create a new version of the entity.
        They also add a new log entry to the entity history to record the changes.

      • Changes to the meta section of an entity do not create a new version of the entity.
        However, they do update the timestamp value of the last_update_at database field.

  6. In the SFTP server URL field, enter the target sftp:// location on the SFTP server to upload the outgoing feed content to, so as to make it available for retrieval.
    Format: sftp://${sftp_server}:${port}/${path_to_target_directory}.
    Example: sftp://sftp.server.com:22/source-data/for-the-feed.

  7. In the Username field, enter a valid user name to authenticate and be granted the necessary authorization to access the location of the outgoing feed content.

  8. In the Password field, enter a valid password to authenticate and be granted the necessary authorization to access the location of the outgoing feed content.

  9. Select the Include documents attached to entities checkbox to include in the outgoing feed content also any attachments to the published entities such as MS Word documents or PDF files.

  10. Select the Use SSH key checkbox to enable logging in through SSH to apply this security layer to the outgoing feed.

  11. In the SSH private key field, paste the private SSH key you want to use to access the target location where the SFTP upload outgoing feed should publish content to.

    -----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEA3Tz2mr7SZiAMfQyuvBjM9Oi..Z1BjP5CE/Wm/Rr500P
RK+Lh9x5eJPo5CAZ3/ANBE0sTK0ZsDGMak2m1g7..3VHqIxFTz0Ta1d+NAj
wnLe4nOb7/eEJbDPkk05ShhBrJGBKKxb8n104o/..PdzbFMIyNjJzBM2o5y
5A13wiLitEO7nco2WfyYkQzaxCw0AwzlkVHiIyC..71pSzkv6sv+4IDMbT/
XpCo8L6wTarzrywnQsh+etLD6FtTjYbbrvZ8RQM..Hg2qxraAV++HNBYmNW
kbJ+q+rsJxQlaipn2M4lGuQJEfIxELFDyd3XpxP..Un/82NZNXlPmRIopXs
2T91jiLZEUKQw+n73j26adTbteuEaPGSrTZxBLR..yssO0wWomUyILqVeti
+PK+aXKwguI6bxLGZ3of0UH+mGsSl0mkp7kYZCm..OTQtfeRqP8rDSC7DgA
kHc5ajYqh04AzNFaxjRo+M3IGICUaOdKnXd0Fda..QwfoaX4QlRTgLqb7AN
ZTzM9WbmnYoXrx17kZlT3lsCgYEAm757XI3WJVj..WoLj1+v48WyoxZpcai
uv9bT4Cj+lXRS+gdKHK+SH7J3x2CRHVS+WH/SVC..DxuybvebDoT0TkKiCj
BWQaGzCaJqZa+POHK0klvS+9ln0/6k539p95tfX..X4TCzbVG6+gJiX0ysz
Yfehn5MCgYEAkMiKuWHCsVyCab3RUf6XA9gd3qY..fCTIGtS1tR5PgFIV+G
engiVoWc/hkj8SBHZz1n1xLN7KDf8ySU06MDggB..hJ+gXJKy+gf3mF5Kmj
DtkpjGHQzPF6vOe907y5NQLvVFGXUq/FIJZxB8k..fJdHEm2M4=
-----END RSA PRIVATE KEY-----

  12. In the SSH key password field, enter the password, if your SSH key is password-protected.
    If your SSH key is not password-protected, you can leave this field empty.

  13. Select the Host authentication mode checkbox to automatically add and save the new host name and the new host key to the local Paramiko HostKeys dictionary.

  14. To store your changes, click Save; to discard them, click Cancel.

View and retrieve outgoing feed content#

  1. In the top navigation bar click Data configuration > Outgoing feeds.

  2. In the Outgoing feeds view, click anywhere in the row corresponding to the outgoing feed whose content you want to view or retrieve.

  3. In the selected outgoing feed detail pane, click the Created packages tab.

  4. In the Created packages tab, under the Download column header, click the name of a package to download it, and to save it to a target location.