Incoming feed - IMAP Email attachment and body fetcher

Note

This article describes how to configure incoming feeds for a particular feed source. To see how to configure incoming feeds in general, see Create and configure incoming feeds.

	Specifications
Transport types	IMAP Email attachment and body fetcher
Content type	See Ingestion results.
Ingested data	Ingests emails from a given IMAP server.
Endpoint(s)	N/A
Processed data	See Ingestion results.

This extension downloads emails from a mail folder on an IMAP server and processes them to produce entities.

You can process both the body and attachments of a given email message.

For information on how emails are processed, see Ingestion results.

Requirements

• IMAP-enabled email account

• IMAP user name

• IMAP password

Known limitations

These email providers are not supported:

• Microsoft Office 365 Outlook

Configure the incoming feed

1. Create or edit an incoming feed.

2. Under Transport and content, fill out these fields:

   Note

   Required fields are marked with an asterisk (*).

   Field	Description
   Transport type*	Select IMAP Email attachment and body fetcher from the drop-down menu.
   Content type*	By default, this is set to Email Message. See Ingestion results for more information.
   Accept password protected archives	Select to display the Archive password field.
   Host*	Enter the address of your IMAP server. For example: imap.example.com
   Username	Enter your IMAP account user name.
   Password	Enter your IMAP account password.
   Folder*	By default, this is set to INBOX. Set this to the name of the email folder you want to process. For example, My inbox. To fetch emails from more than one email folder, set up a new feed per target folder.
   Delete mails	Warning This permanently deletes all email messages that the feed fetches from your IMAP server, whether or not it manages to successfully ingest data from them. Select this to have this feed delete email messages that it fetches from the target IMAP server.
   Body of email	Select to ingest the body of email messages that this feed processes.
   Attachments of email	Select to ingest all attachments from email messages that this feed processes. The extension automatically detects the file type for these attachments and processes them as per Ingestion results.
   Port*	Set to 993 by default. Set this according to the instructions from your IMAP provider.
   Use SSL	Selected by default.
   To keyword	Tip Leave To keyword, From keyword, and Subject keyword empty to fetch all emails from target IMAP folder. Enter keyword(s) to only fetch emails whose Subject field contains them. See Filtering messages for more information.
   From keyword	Enter keyword(s) to only fetch emails whose Subject field contains them. See Filtering messages for more information.
   Subject keyword	Enter keyword(s) to only fetch emails whose Subject field contains them. See Filtering messages for more information.
   SSL certificate authentication	Select to enable SSL client certificate authentication. When enabled, you can set the SSL certificate and key to use.
   SSL certificate	Enter the contents of your PEM-formatted certificate chain files. It should look like this: -----BEGIN CERTIFICATE REQUEST----- MIICvDCCAaQCAQAwdzELMAkGA1UEBhMCVVMxDTALBgNVBAgMBFV0YWgxDzANBgNV BAcMBkxpbmRvbjEWMBQGA1UECgwNRGlnaUNlcnQgSW5jLjERMA8GA1UECwwIRGln [...] 29XI1PpVUNCPQGn9p/eX6Qo7vpDaPybRtA2R7XLKjQaF9oXWeCUqy1hvJac9QFO2 97Ob1alpHPoZ7mWiEuJwjBPii6a9M9G30nUo39lBi1w= -----END CERTIFICATE REQUEST-----
   SSL key	Enter the contents of your PEM-formatted SSL certificate key. It should look like this: -----BEGINRSAPRIVATEKEY----- MIIEpQIBAAKCAQEA3Tz2mr7SZiAMfQyuvBjM9Oi..Z1BjP5CE/Wm/Rr500P RK+Lh9x5eJPo5CAZ3/ANBE0sTK0ZsDGMak2m1g7..3VHqIxFTz0Ta1d+NAj [...] engiVoWc/hkj8SBHZz1n1xLN7KDf8ySU06MDggB..hJ+gXJKy+gf3mF5Kmj DtkpjGHQzPF6vOe907y5NQLvVFGXUq/FIJZxB8k..fJdHEm2M4= -----ENDRSAPRIVATEKEY-----

3. Store your changes by selecting Save.

Filtering messages

You can set this incoming feed to only fetch messages that contain specific keywords in their To, From, and Subject fields.

To do this, enter one or more keywords in the following fields of your feed configuration:

• To keyword

• From keyword

• Subject keyword

Results of the keyword filters may vary between email providers. We recommend performing a dry run on a small mailbox folder before using the keyword filters on a large target folder.

Ingestion results

This extension downloads email messages from an IMAP server and allows you to ingest:

• its message body

• its attachments

or both.

Email message body

To ingest the email message body, select the Body of email option when setting up the feed.

Each email message that the feed fetches from the IMAP server is ingested to produce:

• a report entity

• the message body attached to the report as an .eml file

If an email is part of a reply thread, each email is ingested to produce it’s own Report entity.

Resulting Report entities have these fields set:

EclecticIQ Report entity field	Content description
Title	Subject of email. This is set to the first 60 characters of an email’s subject field. All emails part of a reply thread will have the same title. E.g., Re: this is an email subject.
Description/Analysis	Email body. Body of HTML emails (MIME type text/html) is converted to plain text.
Estimated threat start time	Time email was ingested.
Estimated observed time	Time email was ingested.
Attachment	The original email is attached as an .eml file to the resulting Report entity, named <subject_name>.eml. Attachments for that email are embedded in the resulting .eml file.

Email attachments

To ingest attachments from email messages, select the Attachment of email option when setting up the feed.

For each email message fetched, the feed extracts all attachments for that message. For each extracted attachment:

1. The extension attempts to detect its content type.

2. If the attachment is found to be one of the supported content types, the extension processes the attachment as that content type.

   Supported content types are listed in the table below.

3. If the attachment is not found to be one of the supported content types, it is ingested as a text file.

   The file headers and data preserved in the resulting text file.

The following table describes all supported content types:

Content type	Description
EclecticIQ JSON	Ingest EclecticIQ JSON to produce EclecticIQ entitites.
MISP JSON	Ingest MISP JSON to produce Indicators and Incidents.
PDF	Ingest PDF to produce: A Report entity. The text content of the PDF set to the Analysis field of the entity. The original PDF attached to the entity.
SpyCloud Breach Data JSON	Ingest JSON from SpyCloud to produce Incidents.
STIX 1.0	Ingest STIX 1.0 XML.
STIX 1.1	Ingest STIX 1.1 XML.
STIX 1.1.1	Ingest STIX 1.1.1 XML.
STIX 1.2	Ingest STIX 1.2 XML.
STIX 2.1	Ingest STIX 2.1 JSON. See the documentation on how STIX 2.1 data is processed.
Text	Tip If we cannot detect an attachment’s content type, that attachment is ingested as a text file with its file headers and content preserved. Ingest email attachments as ASCII text files, regardless of actual file type. This produces: A Report entity. The ASCII text content of the ingested file set to the Assessment field of the entity. Note The UI displays UTF-8 text, but the data is stored as ASCII. The original file attached to the entity. This content type is compatible with most file types. It treats all attachments as ASCII text files. File headers are preserved during this conversion.

Deduplicated attachments

When this feed downloads an email attachment, it compares the file hash of the downloaded attachment with files that have already been processed by the Intelligence Center.

If the file hash matches an existing record, the email attachment is not ingested.