Outgoing feed - Palo Alto PAN-OS External Dynamic List#

Note

This article describes how to configure outgoing feeds for a particular feed source. To see how to configure outgoing feeds in general, see Create and configure outgoing feeds.

	Specifications
Transport type	Amazon S3 push FTP upload SFTP upload Mount point upload TAXII inbox TAXII poll
Content type	PAN-OS External Dynamic List
Published data	Sightings.
Description	The feed publishes sightings to instrument a PAN-OS-controlled Palo Alto Networks firewall, so that the device can trigger rule-based follow-up actions. You can use PAN-OS External Dynamic List outgoing feeds to feed back to a PAN-OS-controlled firewall sightings generated from incoming indicators of compromise whose source is the same or another PAN-OS-controlled firewall

You can publish PAN-OS External Dynamic List using any of the supported transport types to make the outgoing feed content available to a Palo Alto Networks PAN-OS-controlled firewall to trigger rule-based follow-up actions.

The data exchange format is Palo Alto Networks External Dynamic List This enables the firewall to dynamically import the lists and to enforce policy without making configuration changes or commits on the firewall.
For this data exchange setup to work, the platform instance publishing the lists needs to be able to store them to a location that the firewall can access to retrieve the lists.

Configure content types#

Create or edit an outgoing feed.
From the Content type drop-down menu, select PAN-OS External Dynamic List.
From the Palo Alto PAN-OS External Dynamic List drop-down menu, select type of list you want to publish through the outgoing feed:
- PAN-OS IP External Dynamic List: the feed outputs an IP address list to feed the Palo Alto Networks firewall with, so that the firewall blocks the included IP addresses.
- PAN-OS Domain External Dynamic List: the feed outputs a domain name list to feed the Palo Alto Networks firewall with, so that the firewall blocks the included domain names.
- PAN-OS URL External Dynamic List: the feed outputs a URL list to feed the Palo Alto Networks firewall with, so that the firewall blocks the included URLs.
Go to the Transport configuration section, and set the appropriate transport type options for the selected transport type for the PAN-OS External Dynamic List content.

Configure transport types#

Amazon S3 push#

The Amazon S3 push transport type for outgoing feeds publishes data in the supported content types to the specified location on the designated Amazon S3 bucket.

From the Transport type drop-down menu, select Amazon S3 push.
In the Secret key field, enter your Amazon S3 push secret key. Sign up to Amazon Web Services, and then create one or more accounts, as necessary, to use their S3 data storage service.
The secret key is part of your authentication credentials to log in to and to access Amazon S3 services.
In the Access key field, enter your Amazon S3 push access key.
Along with your secret key, the access key enables you to authenticate to access Amazon S3 services.
In the Bucket field, enter the name of the Amazon S3 bucket to use as a target location for the outgoing feed published content.
Buckets are data containers in the S3 environment.
Buckets are region-specific, and their names must comply with standard DNS naming conventions.
The default format of the URL to access a bucket is https://${bucket_name}.s3-${aws-region}.amazonaws.com
In the Path field, enter the path to the target directory where the content published through the outgoing feed is stored, relative to the bucket root.
Example: /intel/actors/hacktivis
Under Feed content, you can define the data source and the update strategy for the outgoing feed:
From the Datasets drop-down menu, select one or more existing datasets to use as sources to populate the outgoing feed content.
For the feed not to be empty, at least one selected dataset should contain entities and observables in the same format as the configured content type for the feed.
From the Update strategy drop-down menu, select the preferred method to populate the outgoing feed with data before publishing it:
- Replace: every time the outgoing feed runs, it fetches new and existing data — new and existing entities and observables included in the previous execution of the feed — to generate the content to publish through the feed.

FTP upload#

The FTP upload transport type for outgoing feeds publishes data in the supported content types to the specified location on the designated FTP server.

From the Transport type drop-down menu, select FTP upload.
In the FTP server URL field, enter the target ftp:// location on the FTP server to upload the outgoing feed content to, so as to make it available for retrieval.
Example: ftp://ftp.server.com/feeds/outgoing/folder
In the Username field, enter a valid user name to authenticate and be granted the necessary authorization to access the location of the outgoing feed content.
In the Password field, enter a valid password to authenticate and be granted the necessary authorization to access the location of the outgoing feed content.
Select the Include documents attached to entities checkbox to include in the outgoing feed content also any attachments to the published entities such as MS Word documents or PDF files.
Under Feed content, you can define the data source and the update strategyfor the outgoing feed:
From the Datasets drop-down menu, select one or more existing datasets to use as sources to populate the outgoing feed content.
For the feed not to be empty, at least one selected dataset should contain entities and observables in the same format as the configured content type for the feed.
From the Update strategy drop-down menu, select the preferred method to populate the outgoing feed with data before publishing it:
- Replace: every time the outgoing feed runs, it fetches new and existing data — new and existing entities and observables included in the previous execution of the feed — to generate the content to publish through the feed.

SFTP upload#

The SFTP upload transport type for outgoing feeds publishes data in the supported content types to the specified location on the designated SFTP server.

From the Transport type drop-down menu, select SFTP upload.
In the SFTP server URL field, enter the target sftp:// location on the SFTP server to upload the outgoing feed content to, so as to make it available for retrieval.
Format: sftp://${sftp_server}:${port}/${path_to_target_directory}
Example: sftp://sftp.server.com:22/source-data/for-the-feed
In the Username field, enter a valid user name to authenticate and be granted the necessary authorization to access the location of the outgoing feed content.
In the Password field, enter a valid password to authenticate and be granted the necessary authorization to access the location of the outgoing feed content.
Select the Include documents attached to entities checkbox to include in the outgoing feed content also any attachments to the published entities such as MS Word documents or PDF files.
Select the Use SSH key checkbox to enable logging in through SSH to apply this security layer to the outgoing feed.

In the SSH private key field, paste the private SSH key you want to use to access the target location where the SFTP upload outgoing feed should publish content to.
Example:

-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEA3Tz2mr7SZiAMfQyuvBjM9Oi..Z1BjP5CE/Wm/Rr500P
RK+Lh9x5eJPo5CAZ3/ANBE0sTK0ZsDGMak2m1g7..3VHqIxFTz0Ta1d+NAj
wnLe4nOb7/eEJbDPkk05ShhBrJGBKKxb8n104o/..PdzbFMIyNjJzBM2o5y    
5A13wiLitEO7nco2WfyYkQzaxCw0AwzlkVHiIyC..71pSzkv6sv+4IDMbT/
XpCo8L6wTarzrywnQsh+etLD6FtTjYbbrvZ8RQM..Hg2qxraAV++HNBYmNW
kbJ+q+rsJxQlaipn2M4lGuQJEfIxELFDyd3XpxP..Un/82NZNXlPmRIopXs
2T91jiLZEUKQw+n73j26adTbteuEaPGSrTZxBLR..yssO0wWomUyILqVeti
+PK+aXKwguI6bxLGZ3of0UH+mGsSl0mkp7kYZCm..OTQtfeRqP8rDSC7DgA
kHc5ajYqh04AzNFaxjRo+M3IGICUaOdKnXd0Fda..QwfoaX4QlRTgLqb7AN
ZTzM9WbmnYoXrx17kZlT3lsCgYEAm757XI3WJVj..WoLj1+v48WyoxZpcai
uv9bT4Cj+lXRS+gdKHK+SH7J3x2CRHVS+WH/SVC..DxuybvebDoT0TkKiCj
BWQaGzCaJqZa+POHK0klvS+9ln0/6k539p95tfX..X4TCzbVG6+gJiX0ysz
engiVoWc/hkj8SBHZz1n1xLN7KDf8ySU06MDggB..hJ+gXJKy+gf3mF5Kmj
DtkpjGHQzPF6vOe907y5NQLvVFGXUq/FIJZxB8k..fJdHEm2M4=
-----END RSA PRIVATE KEY-----

In the SSH key password field, enter your SSH key password, if your SSH key is password-protected.
If your SSH key is not password-protected, you can leave this field empty.
Select the Host authentication mode checkbox to automatically add and save the new host name and the new host key to the local Paramiko HostKeys dictionary.
Under Feed content, you can define the data source and the update strategy for the outgoing feed:
From the Datasets drop-down menu, select one or more existing datasets to use as sources to populate the outgoing feed content.
For the feed not to be empty, at least one selected dataset should contain entities and observables in the same format as the configured content type for the feed.
From the Update strategy drop-down menu, select the preferred method to populate the outgoing feed with data before publishing it:
- Replace: every time the outgoing feed runs, it fetches new and existing data — new and existing entities and observables included in the previous execution of the feed — to generate the content to publish through the feed.

Mount point upload#

The Mount point upload transport type for outgoing feeds publishes data in the supported content types to the specified location on a local or network unit.

From the Transport type drop-down menu, select Mount point upload.
In the Mount point path field, enter the path to the local or network unit to save the outgoing feed content to, so as to make it available for retrieval.
Example: /media/feeds/outgoing/folder

Select the Include documents attached to entities checkbox to include in the outgoing feed content also any attachments to the published entities such as MS Word documents or PDF files.

Note

Explicitly whitelist mount point paths to make them accessible to incoming and to outgoing feeds.
If you do not whitelist the mount point path an incoming or an outgoing feed should access to retrieve data for ingestion or for publication, the feed will not be able to fetch or to publish any content.

The /etc/eclecticiq/platform_settings.py configuration file includes dedicated mount point whitelists for ingestion – incoming feeds – and for dissemination – outgoing feeds.

settings.py (sourced from EIQ platform-backend)


Author	Rutger Prins
Commit	ab323b23ebb93fde6c62b124f6823579957bd1d5
Timestamp	August, 27, 2021 08:57 AM
Full path	eiq/platform/settings.py
Title	Merge branch ‘ext-commons-update-2.10.x’ into ‘release-2.10.x’
Description	Extension Commons update 2.10.x See merge request engineering/platform-backend!6075

# Directories that can be accessed from mount point feeds. POLL is for incoming
# feeds, PUSH is for outgoing feeds. Example: ["/mnt/", "/media/"]
MOUNT_POINT_POLL_ALLOWED_DIRECTORIES: Sequence[str] = []
MOUNT_POINT_PUSH_ALLOWED_DIRECTORIES: Sequence[str` = []

MOUNT_POINT_POLL_ALLOWED_DIRECTORIES is a list of allowed mount point paths that incoming feeds can access to fetch data from.
MOUNT_POINT_PUSH_ALLOWED_DIRECTORIES is a list of allowed mount point paths that outgoing feeds can access to publish data to.

To whitelist a mount point path:

Open the configuration file:

sudo vi /etc/eclecticiq/platform_settings.py

Look for MOUNT_POINT_POLL_ALLOWED_DIRECTORIES to make network locations accessible to incoming feeds, or for for MOUNT_POINT_PUSH_ALLOWED_DIRECTORIES to make network locations accessible to outgoing feeds.
Both parameters are lists that take valid directory paths as list elements.
Each path in the list points to a location that incoming feeds can access to fetch the data to be ingested, or that outgoing feeds can access to publish the content of a feed run.
Incoming and outgoing feeds can access files and directories inside the specified locations, based on the configured access rights of the available assets and resources.

Add as many paths to each list as necessary, then save the file and exit.
Example:

# Whitelist specific dirs; specific file types; everything inside subdirs of a dir
MOUNT_POINT_PUSH_ALLOWED_DIRECTORIES = [ "/mnt/", "/media/", "/media/data/" ]

Under Feed content, you can define the data source and the update strategy for the outgoing feed:
From the Datasets drop-down menu, select one or more existing datasets to use as sources to populate the outgoing feed content.
For the feed not to be empty, at least one selected dataset should contain entities and observables in the same format as the configured content type for the feed.
From the Update strategy drop-down menu, select the preferred method to populate the outgoing feed with data before publishing it:
- Replace: every time the outgoing feed runs, it fetches new and existing data — new and existing entities and observables included in the previous execution of the feed — to generate the content to publish through the feed.

TAXII inbox#

Note

Before configuring a TAXII transport type for an incoming or an outgoing feed, make sure that the appropriate TAXII service is correctly configured in the platform system settings.

TAXII inbox and TAXII poll transport types require Cabby. For more information, see official Cabby documentation, the Cabby public repo on GitHub, and the Cabby download page.

The TAXII inbox transport type for outgoing feeds publishes data in the supported content types through the TAXII inbox service:

From the Transport type drop-down menu, select TAXII inbox.
In the Auto Discovery field, enter the URL pointing to a TAXII discovery service.
Feed consumers can send a request to the discovery service to obtain a list of the available TAXII services they can access and poll for content updates.
Example: http://hailataxii.com/taxii-discovery-service
In the Inbox service URL field, enter the URL pointing to the location of the TAXII data collections available through the TAXII inbox service.
Example: https://example.com/taxii-inbox
In the Destination collection name field, enter an existing collection name as the target container for the outgoing feed data.
Example: collection.Default
From the TAXII version drop-down menu, select the TAXII version your system supports:
- 1.0
- 1.1
In the EclecticIQ authentication URL field, enter the URL pointing to the EclecticIQ Platform instance, including the endpoint that takes the user name and password inputs to send them to the authentication mechanism.
Example: https://${platform_host_name}/api/auth
Select the Basic authentication checkbox to fill out the required information, if the data source TAXII server requires basic authentication to access the corresponding TAXII services.
Username: enter a valid user name to authenticate and be granted the necessary authorization to access the location of the outgoing feed content.
Password: enter a valid password to authenticate and be granted the necessary authorization to access the location of the outgoing feed content.
Select the SSL certificate authentication checkbox to fill in the required information, if the TAXII server requires an SSL certificate to authenticate and to authorize access to the corresponding TAXII services.

In the SSL certificate field, copy-paste the content of a valid SSL certificate to authenticate.
SSL certificate file format: .pem
Example:

----BEGIN CERTIFICATE REQUEST-----
MIICvDCCAaQCAQAwdzELMAkGA1UEBhMCVVMxDTALBgNVBAgMBFV0YWgxDzANBgNV
BAcMBkxpbmRvbjEWMBQGA1UECgwNRGlnaUNlcnQgSW5jLjERMA8GA1UECwwIRGln
aUNlcnQxHTAbBgNVBAMMFGV4YW1wbGUuZGlnaWNlcnQuY29tMIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEA8+To7d+2kPWeBv/orU3LVbJwDrSQbeKamCmo
wp5bqDxIwV20zqRb7APUOKYoVEFFOEQs6T6gImnIolhbiH6m4zgZ/CPvWBOkZc+c
1Po2EmvBz+AD5sBdT5kzGQA6NbWyZGldxRthNLOs1efOhdnWFuhI162qmcflgpiI
WDuwq4C9f+YkeJhNn9dF5+owm8cOQmDrV8NNdiTqin8q3qYAHHJRW28glJUCZkTZ
wIaSR6crBQ8TbYNE0dc+Caa3DOIkz1EOsHWzTx+n0zKfqcbgXi4DJx+C1bjptYPR
BPZL8DAeWuA8ebudVT44yEp82G96/Ggcf7F33xMxe0yc+Xa6owIDAQABoAAwDQYJ
KoZIhvcNAQEFBQADggEBAB0kcrFccSmFDmxox0Ne01UIqSsDqHgL+XmHTXJwre6D
hJSZwbvEtOK0G3+dr4Fs11WuUNt5qcLsx5a8uk4G6AKHMzuhLsJ7XZjgmQXGECpY
Q4mC3yT3ZoCGpIXbw+iP3lmEEXgaQL0Tx5LFl/okKbKYwIqNiyKWOMj7ZR/wxWg/
ZDGRs55xuoeLDJ/ZRFf9bI+IaCUd1YrfYcHIl3G87Av+r49YVwqRDT0VDV7uLgqn
29XI1PpVUNCPQGn9p/eX6Qo7vpDaPybRtA2R7XLKjQaF9oXWeCUqy1hvJac9QFO2
97Ob1alpHPoZ7mWiEuJwjBPii6a9M9G30nUo39lBi1w=
-----END CERTIFICATE REQUEST-----

In the SSL key field, copy-paste the content of a valid SSL key to authenticate.
SSL key file format: *.pem
*Example:

-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEA3Tz2mr7SZiAMfQyuvBjM9Oi..Z1BjP5CE/Wm/Rr500P
RK+Lh9x5eJPo5CAZ3/ANBE0sTK0ZsDGMak2m1g7..3VHqIxFTz0Ta1d+NAj
wnLe4nOb7/eEJbDPkk05ShhBrJGBKKxb8n104o/..PdzbFMIyNjJzBM2o5y
5A13wiLitEO7nco2WfyYkQzaxCw0AwzlkVHiIyC..71pSzkv6sv+4IDMbT/
XpCo8L6wTarzrywnQsh+etLD6FtTjYbbrvZ8RQM..Hg2qxraAV++HNBYmNW
kbJ+q+rsJxQlaipn2M4lGuQJEfIxELFDyd3XpxP..Un/82NZNXlPmRIopXs
2T91jiLZEUKQw+n73j26adTbteuEaPGSrTZxBLR..yssO0wWomUyILqVeti
+PK+aXKwguI6bxLGZ3of0UH+mGsSl0mkp7kYZCm..OTQtfeRqP8rDSC7DgA
kHc5ajYqh04AzNFaxjRo+M3IGICUaOdKnXd0Fda..QwfoaX4QlRTgLqb7AN
ZTzM9WbmnYoXrx17kZlT3lsCgYEAm757XI3WJVj..WoLj1+v48WyoxZpcai
uv9bT4Cj+lXRS+gdKHK+SH7J3x2CRHVS+WH/SVC..DxuybvebDoT0TkKiCj
BWQaGzCaJqZa+POHK0klvS+9ln0/6k539p95tfX..X4TCzbVG6+gJiX0ysz
Yfehn5MCgYEAkMiKuWHCsVyCab3RUf6XA9gd3qY..fCTIGtS1tR5PgFIV+G
engiVoWc/hkj8SBHZz1n1xLN7KDf8ySU06MDggB..hJ+gXJKy+gf3mF5Kmj
DtkpjGHQzPF6vOe907y5NQLvVFGXUq/FIJZxB8k..fJdHEm2M4=
-----END RSA PRIVATE KEY-----

In the SSL key password field, enter the SSL password or passphrase for the SSL key.
This field is masked.
Select the SSL verification checkbox, if the TAXII server requires an SSL certificate to authenticate and to access its TAXII services, to verify that it works as expected.
In the SSL CA bundle file path field, enter the path to the CA bundle file containing the root, intermediate, and public certificates for SSL authentication.
The SSL CA bundle specified here is part of the server certificate validation chain.
SSL CA bundle file format: .ca-bundle
To store your changes, click Save; to discard them, click Cancel.
Under Feed content you can define the data source and the update strategy for the outgoing feed:
From the Datasets drop-down menu, select one or more existing datasets to use as sources to populate the outgoing feed content.
For the feed not to be empty, at least one selected dataset should contain entities and observables in the same format as the configured content type for the feed.
From the Update strategy drop-down menu, select the preferred method to populate the outgoing feed with data before publishing it:
- Replace: every time the outgoing feed runs, it fetches new and existing data — new and existing entities and observables included in the previous execution of the feed — to generate the content to publish through the feed.

TAXII poll#

Note

Before configuring a TAXII transport type for an incoming or an outgoing feed, make sure that the appropriate TAXII service is correctly configured in the platform system settings.

TAXII inbox and TAXII poll transport types require Cabby.
For more information, see official Cabby documentation, the Cabby public repo on GitHub, and the Cabby download page.

The TAXII poll transport type for outgoing feeds publishes data in the supported content types through the TAXII poll service:

From the Transport type drop-down menu, select TAXII poll.
Select the Public checkbox to make the outgoing feed available to all platform groups and to all platform users.
Leave it deselected to make the outgoing feed available only to specific groups.
You can select the intended recipient groups in Authorized groups.
Default value: deselected
From the Authorized groups drop-down menu, select one or more groups to grant them access to the feed.
This option restricts access to the outgoing feed only to the selected user groups and to their members.|
Authorized groups is only available when Public is deselected (default setting).
In the Collection name field, enter the name of the TAXII data collection you want to use to consolidate the outgoing feed content.
The data collection name can be max. 1024 characters long, and its XML schema should comply with the xsd:anyURI data type.
Example: MalwareDomainList_Hostlist
Warning

Before deleting a group, check that is not an authorized group in an incoming or an outgoing feed configuration. Deleting a group that is currently selected as an authorized group to access an incoming or an outgoing feed content breaks feed functionality.

If remove such a group:
1. Remove it from the Authorized groups selection in the relevant incoming and/or outgoing feed(s).
2. Proceed to delete the group.
Under Feed content, you can define the data source and the update strategy for the outgoing feed:
From the Datasets drop-down menu, select one or more existing datasets to use as sources to populate the outgoing feed content.
For the feed not to be empty, at least one selected dataset should contain entities and observables in the same format as the configured content type for the feed.
From the Update strategy drop-down menu, select the preferred method to populate the outgoing feed with data before publishing it:
- Replace: every time the outgoing feed runs, it fetches new and existing data — new and existing entities and observables included in the previous execution of the feed — to generate the content to publish through the feed.
To store your changes, click Save; to discard them, click Cancel.

About consuming the feed content#

The PAN-OS External Dynamic List outgoing feed sends the Palo Alto Networks firewall blocklists containing IP domain, and URL sightings.
In a typical scenario, you want the platform to send the firewall only malicious IPs, domains, and URLs that are flagged by reliable sources because the firewall will block them.

It is a good idea to create a dataset in the platform that acts as a data source for the outgoing fee:

If you create a static dataset, make sure you populate it only with entities having appropriate characteristics.
If you create a dynamic dataset, make sure the search query acting as the filtering rule for the dynamic dataset content is set up to populate it with the desired entities.

In the PAN-OS External Dynamic List outgoing feed configuration, make sure you set the appropriate filters for observable maliciousness under Allowed observable states, TLP code, and source reliability.
For example:

Set Allowed observable states to include only malicious ones.
Do not include any observables marked as Safe or Ignore.
Set the source reliability to the desired level.
For example, you may want to exclude data from unreliable sources, as the firewall may automatically apply a blocking policy to it.
Set the TLP code based on the applicable content sharing policies in your organization.
Set the outgoing feed Update strategy to Replace, so that each feed run replaces the existing extended dynamic list content with both existing and new data since the previous feed run.
Extended dynamic lists are not based on incremental changes: each time the firewall reloads them, previous block rules based on the same list are dropped, and new ones are created.
Therefore, you always want each feed output (that is, each published list) to include all the available content up to that moment.

About external dynamic lists#

Palo Alto External Dynamic Lists support IP addresses, domains, and URLs.

The external dynamic lists the platform published through the PAN-OS External Dynamic List outgoing feed support these data types:

IP addresses

The platform exports individual IP addresses to an EDL list.
The exported EDL list holds one record, that is, one IP address, per line.
Exported IP addresses support both ipv4 and ipv6.

Example IP address EDL
`192.168.20.10` `2001:db8:123:1::1` `8.8.8.8` `192.168.20.10`

Domain names

The platform exports domains to an EDL list.
The exported EDL list holds one record, that is, one domain, per line.

Example domain name EDL
`www.instagram.com` `malicious.com` `abc.baddomain.cc`

URLs

The platform exports URLs to an EDL list.
The exported EDL list holds one record, that is, one URL, per line.
Exported URLs should not include the transport protocol prefix. Examples of not allowed protocol prefixes:
- http://
- https://
- ftp://
URLs record format is not case sensitive.
Wildcard characters are supported.

Example URL EDL
`www.wellknownsite.com/bad-page` `www.yahoo.com/search=*` `www.facebook.com/messenger`

Note

You can use PAN-OS External Dynamic List outgoing feeds to feed back to a PAN-OS-controlled firewall sightings generated from incoming indicators of compromise whose source is the same or another PAN-OS-controlled firewall.