Incoming feed - Amazon S3#

Note

This article describes how to configure incoming feeds for a particular feed source. To see how to configure incoming feeds in general, see Create and configure incoming feeds.

Specifications

Transport type

Amazon S3 poll

Content type

  • Eclectic JSON

  • Email message

  • MISP JSON

  • PDF

  • SpyCloud Breach Data JSON

  • STIX 1.0

  • STIX 1.1

  • STIX 1.1.1

  • STIX 1.2

  • STIX 2.1

  • Text

Ingested data

All data found on the designated source S3 bucket.

Processed data

The platform ingests retrieved data based on the configured content type for the feed. After successfully retrieving and ingesting the data, it is removed from the source bucket

Description

Securely transfer data from selected Amazon S3 buckets.

Requirements#

Users need a Secret access key and an Access key for their own configuration. Sign up and subscribe to the service to obtain the required credentials.

Configure the incoming feed#

  1. Create or edit an incoming feed.

  2. From the Transport type drop-down menu, select Amazon S3 poll.

  3. From the Content type drop-down menu, select the content type to ingest.
    The selected content type must match the source data format you want to retrieve.

  4. In the Secret access key field, enter a secret key. This key is part of your authentication credentials to log in and to access Amazon S3 services.
    Sign up to Amazon Web Services, and create one or more accounts to use their S3 data storage service.

  5. In the Access key field, enter the access key to enable you to authenticate to access Amazon S3 services.

  6. In the Path field, enter the path to the target directory where the source data for the incoming feed is stored, relative to the bucket root.
    Example: path/to/target/dir.

  7. In the Bucket field, enter the name of the Amazon S3 bucket to use as a data source for the feed.
    [Buckets](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html are data containers in the S3 environment. Buckets are region-specific, and their names need to comply with standard DNS naming conventions.
    The default format of the URL to access a bucket is https://${bucket\_name}.s3-${aws-region}.amazonaws.com.

  8. To store your changes, click Save; to discard them, click Cancel.

Test the feed#

  1. In the top navigation bar, click Data Configuration > Incoming feeds.

  2. Click the feed that you just created, using the steps above.

  3. In the Overview view, click Download now.

  4. Click Ingested entities and check that entities have been ingested into the platform.

Or:

  1. In the top navigation bar, click Intelligence > All intelligence > Browse.

  2. Click the Entities tab.

  3. In the top-left corner, click Filter.

  4. From the Source drop-down menu, select the incoming feed you have just created, using the steps.

  5. You can also filter also by entity type: from the Entity drop-down menu, select the entity types you want to include in the filtered results.