Tripwire has been getting more involved in connecting its products to threat intelligence services lately. I described the reasons why we care about threat intelligence, particularly STIX and TAXII, in my article last month: Why We Should Care About STIX & TAXII. My colleagues also talked in more specifics about some of the partners Tripwire is working with in the post Threat Intelligence: Reduce the Gap. If you are thinking about connecting your own products or services to threat intelligence feeds, one of the first questions you are apt to be asking yourself is: how hard is this going to be? So, I thought I would share some code snippets we developed to connect to a TAXII server and show just how easy it is. Before you start to write code, the first step would be to get a TAXII server to connect to. Here are two very easy options for you:
- Hail a TAXII – This is a freely usable TAXII server that is publicly accessible on the internet, put up by our friends at Soltra. It has some open source threat intelligence on it, and that makes it a great place to connect to pull a TAXII feed from. It is not a test-bed though, so it may not be a great place to test non-compliant experimental code.
- Soltra Edge – Soltra, one of Tripwire’s partners, has a freely available TAXII server you can download as a virtual machine and easily stand-up. This would be the option I’d recommend, as you can have it on a virtualization server or on your local system using something like VMWare Workstation, and start to create TAXII feeds and test your code with it.
Now, here is under 70 lines of python code that will help you pull a TAXII feed and extract a particular element you are interested in out of the STIX content it contains. If you want to use some pre-built code, this is definitely not your only option. Soltra includes some TAXII client code with Soltra Edge as well, and you can also find various TAXII bindings and libraries as part of the TAXII project at https://github.com/TAXIIProject. This code is commented extra-well, so you can follow-along with it even if you aren't familiar with Python.
# These two modules handle the HTTP(S) protocol for us
import urllib2
from urlparse import urlparse
# We'll use ElementTree to parse the XML as it's a simple binding which ships with Python
import xml.etree.ElementTree as ET
# These are used to help with HTTP basic auth and generate random message IDs, respectively
import base64
import random
# The namespace of the TAXII document we'll send to the server
taxii_ns = 'http://taxii.mitre.org/messages/taxii_xml_binding-1.1'
# This class parses the CybOX results from the server
class CybOXObject(object):
# xml parameter should be an ElementTree.Element with the CybOX object inside it
# url parameter is the feed location
def __init__(self, xml, baseurl):
# construct a new URL we can use to browse to this object in the Soltra dashboard
p = urlparse(baseurl)
self.url = '%s://%s/object/%s' % (p.scheme, p.netloc, xml.get('id'))
# Find all file hashes using the ElementTree "findall" method. Each
# Simple_Hash_Value XML node can contain multiple hashes separated by a delimiter.
self.hashes = []
for hashval in xml.findall('.//{http://cybox.mitre.org/common-2}Simple_Hash_Value'):
self.hashes.extend(hashval.text.split(hashval.get('delimiter', '##comma##')))
# Internal function constructing an HTTP request to read unread threat data via TAXII
def _taxii_req(url, feedname, username, password):
# Construct an XML document with 'Poll_Request' as the top-level node name.
xmldata = ET.Element('Poll_Request', {'xmlns': taxii_ns,
'message_id': str(random.randint(345271,9999999999)),
'collection_name': feedname})
# Fill in a few TAXII parameters
pp = ET.SubElement(xmldata, 'Poll_Parameters', {'allow_asynch': 'false'})
ET.SubElement(pp, 'Response_Type').text = 'FULL'
ET.SubElement(pp, 'Content_Binding', {'binding_id': 'urn:stix.mitre.org:xml:1.1.1'})
# Create an HTTP request with our XML as the POST data
req = urllib2.Request(url, ET.tostring(xmldata))
# Add some headers to the request and return it
if username is not None and password is not None:
req.add_header('Authorization', 'Basic %s' %
base64.encodestring('%s:%s' %
(username, password)).replace('\n', ''))
req.add_header('Content-Type', 'application/xml')
req.add_header('User-Agent', 'TAXII Client Application')
req.add_header('Accept', 'application/xml')
req.add_header('X-TAXII-Accept', 'urn:taxii.mitre.org:message:xml:1.1')
req.add_header('X-TAXII-Content-Type', 'urn:taxii.mitre.org:message:xml:1.1')
req.add_header('X-TAXII-Protocol', 'urn:taxii.mitre.org:protocol:https:1.0')
return req
# Function to poll a TAXII feed. Returns a list of CybOXObject instances
def poll_feed(url, feedname, username, password):
return [CybOXObject(x, url) for x in
ET.fromstring(urllib2.urlopen(_taxii_req(url, feedname, username, password)).read())
.findall('.//{http://cybox.mitre.org/cybox-2}Object')]
# Example command-line usage
if __name__ == '__main__':
import sys
if len(sys.argv) < 3: print "usage: taxii.py url feedname [username] [password]" sys.exit(-1) url = sys.argv[1] feedname = sys.argv[2] username, password = None, None if len(sys.argv) >= 5:
username = sys.argv[3]
password = sys.argv[4]
for obj in poll_feed(url, feedname, username, password):
if obj.hashes:
print "%s: %s" % (obj.url, obj.hashes)
Source code examples are released under the BSD 2 Clause License found at http://opensource.org/licenses/BSD-2-Clause.
As you can see, this is actually pretty straight-forward. Browsing through the relevant standards, one might think that STIX and TAXII require months of study to successfully consume. But most of the complexity at the XML level lies in CybOX, the dictionary for which needs to be voluminous to handle all the different possible types of threat data. But if your organization is only interested in a few kinds of data, it's easy enough to pick out and handle the elements you want (as the example above does with Simple_Hash_Value) and ignore the rest. In fact, STIX, which acts as a container for CybOX, is never addressed at all here; instead we just use the ".//" syntax in XPath to skip right past it to the actual content. Likewise, while it's possible to create complex feed scenarios with TAXII, starting out as a consuming client is virtually as simple as sending any other HTTP request; in the example above, we constructed a poll request in four lines of code, then added a few TAXII-specific HTTP headers. Users can extend this basic functionality to handle requesting indicators over a range of timestamps, etc., but for the purposes of discovering any threats which have come in since the last poll request (which Soltra supports by default), the code shown here is pretty much all you'll need. So, have you tried consuming external threat data using STIX and TAXII yet? If you were waiting because the setup seemed too complex, I guess you'll need to find another excuse.
Resources:
The Executive’s Guide to the Top 20 Critical Security Controls Tripwire has compiled an e-book, titled The Executive’s Guide to the Top 20 Critical Security Controls: Key Takeaways and Improvement Opportunities, which is available for download [registration form required].Title image courtesy of ShutterStock
