Incoming feed - IMAP Email attachment and body fetcher

This article describes how to configure incoming feeds for a particular feed source. To see how to configure incoming feeds in general, see Configure incoming feeds general options.

Specifications

Transport types

IMAP Email attachment and body fetcher

Content type

See Ingestion results.

Ingested data

Ingests emails from a given IMAP server.

Endpoint(s)

N/A

Processed data

See Ingestion results.

This extension downloads emails from a mail folder on an IMAP server and processes them to produce entities.

You can process both the body and attachments of a given email message.

For information on how emails are processed, see Ingestion results.

Contents

Requirements

  • IMAP-enabled email account

  • IMAP user name

  • IMAP password

Known limitations

These email providers are not supported:

  • Microsoft Office 365 Outlook

Configure the incoming feed

  1. Create or edit an incoming feed.

  2. Under Transport and content, fill out these fields:

    Required fields are marked with an asterisk (*).

    Field

    Description

    Transport type*

    Select IMAP Email attachment and body fetcher from the drop-down menu.

    Content type*

    By default, this is set to Email Message.

    See Ingestion results for more information.

    Accept password protected archives

    Select to display the Archive password field.

    Host*

    Enter the address of your IMAP server.

    For example: imap.example.com

    Username

    Enter your IMAP account user name.

    Password

    Enter your IMAP account password.

    Folder*

    By default, this is set to INBOX.

    Set this to the name of the email folder you want to process. For example, My inbox.

    To fetch emails from more than one email folder, set up a new feed per target folder.

    Delete mails

    This permanently deletes all email messages that the feed fetches from your IMAP server, whether or not it manages to successfully ingest data from them.

    Select this to have this feed delete email messages that it fetches from the target IMAP server.

    Body of email

    Select to ingest the body of email messages that this feed processes.

    Attachments of email

    Select to ingest all attachments from email messages that this feed processes.

    The extension automatically detects the file type for these attachments and processes them as per Ingestion results.

    Port*

    Set to 993 by default.

    Set this according to the instructions from your IMAP provider.

    Use SSL

    Selected by default.

    To keyword

    Leave To keyword, From keyword, and Subject keyword empty to fetch all emails from target IMAP folder.

    Enter keyword(s) to only fetch emails whose Subject field contains them.

    See Filtering messages for more information.

    From keyword

    Enter keyword(s) to only fetch emails whose Subject field contains them.

    See Filtering messages for more information.

    Subject keyword

    Enter keyword(s) to only fetch emails whose Subject field contains them.

    See Filtering messages for more information.

    SSL certificate authentication

    Select to enable SSL client certificate authentication.

    When enabled, you can set the SSL certificate and key to use.

    SSL certificate

    Enter the contents of your PEM-formatted certificate chain files.

    It should look like this:

    -----BEGIN CERTIFICATE REQUEST-----
MIICvDCCAaQCAQAwdzELMAkGA1UEBhMCVVMxDTALBgNVBAgMBFV0YWgxDzANBgNV
BAcMBkxpbmRvbjEWMBQGA1UECgwNRGlnaUNlcnQgSW5jLjERMA8GA1UECwwIRGln
[...]
29XI1PpVUNCPQGn9p/eX6Qo7vpDaPybRtA2R7XLKjQaF9oXWeCUqy1hvJac9QFO2
97Ob1alpHPoZ7mWiEuJwjBPii6a9M9G30nUo39lBi1w=
-----END CERTIFICATE REQUEST-----

    SSL key

    Enter the contents of your PEM-formatted SSL certificate key.

    It should look like this:

    -----BEGINRSAPRIVATEKEY-----
MIIEpQIBAAKCAQEA3Tz2mr7SZiAMfQyuvBjM9Oi..Z1BjP5CE/Wm/Rr500P
RK+Lh9x5eJPo5CAZ3/ANBE0sTK0ZsDGMak2m1g7..3VHqIxFTz0Ta1d+NAj
[...]
engiVoWc/hkj8SBHZz1n1xLN7KDf8ySU06MDggB..hJ+gXJKy+gf3mF5Kmj
DtkpjGHQzPF6vOe907y5NQLvVFGXUq/FIJZxB8k..fJdHEm2M4=
-----ENDRSAPRIVATEKEY-----

  3. Store your changes by selecting Save.

Filtering messages

You can set this incoming feed to only fetch messages that contain specific keywords in their To, From, and Subject fields.

To do this, enter one or more keywords in the following fields of your feed configuration:

  • To keyword

  • From keyword

  • Subject keyword

Results of the keyword filters may vary between email providers. We recommend performing a dry run on a small mailbox folder before using the keyword filters on a large target folder.

Ingestion results

This extension downloads email messages from an IMAP server and allows you to ingest:

  • its message body

  • its attachments

or both.

Email message body

To ingest the email message body, select the Body of email option when setting up the feed.

Each email message that the feed fetches from the IMAP server is ingested to produce:

  • a report entity

  • the message body attached to the report as an .eml file

If an email is part of a reply thread, each email is ingested to produce it’s own Report entity.

Resulting Report entities have these fields set:

EclecticIQ Report entity field

Content description

Title

Subject of email.

This is set to the first 60 characters of an email’s subject field.

All emails part of a reply thread will have the same title. E.g., Re: this is an email subject.

Description/Analysis

Email body.

Body of HTML emails (MIME type text/html) is converted to plain text.

Estimated threat start time

Time email was ingested.

Estimated observed time

Time email was ingested.

Attachment

The original email is attached as an .eml file to the resulting Report entity, named <subject_name>.eml.

Attachments for that email are embedded in the resulting .eml file.

Email attachments

To ingest attachments from email messages, select the Attachment of email option when setting up the feed.

For each email message fetched, the feed extracts all attachments for that message. For each extracted attachment:

  1. The extension attempts to detect its content type.

  2. If the attachment is found to be one of the supported content types, the extension processes the attachment as that content type.

    Supported content types are listed in the table below.

  3. If the attachment is not found to be one of the supported content types, it is ingested as a text file.

    The file headers and data preserved in the resulting text file.

The following table describes all supported content types:

Content type

Description

CAPEC XML

Ingest CAPEC XML as TTPs.

EclecticIQ JSON

Ingest EclecticIQ JSON to produce EclecticIQ entitites.

MISP JSON

Ingest MISP JSON to produce Indicators and Incidents.

PDF

Ingest PDF to produce:

  • A Report entity.

  • The text content of the PDF set to the Analysis field of the entity.

  • The original PDF attached to the entity.

SpyCloud Breach Data JSON

Ingest JSON from SpyCloud to produce Incidents.

STIX 1.0

Ingest STIX 1.0 XML.

STIX 1.1

Ingest STIX 1.1 XML.

STIX 1.1.1

Ingest STIX 1.1.1 XML.

STIX 1.2

Ingest STIX 1.2 XML.

STIX 2.1

Ingest STIX 2.1 JSON. See the documentation on how STIX 2.1 data is processed.

Text

If we cannot detect an attachment’s content type, that attachment is ingested as a text file with its file headers and content preserved.

Ingest email attachments as ASCII text files, regardless of actual file type.

This produces:

  • A Report entity.

  • The ASCII text content of the ingested file set to the Assessment field of the entity.

    The UI displays UTF-8 text, but the data is stored as ASCII.

  • The original file attached to the entity.

This content type is compatible with most file types. It treats all attachments as ASCII text files.

File headers are preserved during this conversion.

Deduplicated attachments

When this feed downloads an email attachment, it compares the file hash of the downloaded attachment with files that have already been processed by the Intelligence Center.

If the file hash matches an existing record, the email attachment is not ingested.