Moses Schwartz, Andy Culler - A Snake in the Bits: Security Automation with Python

Security incident response is an intense, high stress, high skill job that relies heavily on human judgement. Despite that, for reasons that we can't begin to understand, a big part of an incident responder's job seems to be opening numerous browser tabs and copy-pasting bits of text from one system to another. The hard parts of incident response can't be automated, but there are entire classes of busy-work that we can eliminate with a few web hooks and some artisanal Python.

In this talk we're going to discuss how to use Python to automate security incident response team (SIRT) operations. We'll give an overview of what a typical SecOps/SIRT infrastructure looks like, how and where automation fits in, and dive into some code. We'll walk through a simple example, with screenshots and code, of automating a SecOps process. We want to show that getting started with security automation doesn't have to be difficult or expensive (though vendors will happily take your money). Just a little bit of Python can make some great quality of life improvements for incident responders.

https://us.pycon.org/2019/schedule/presentation/153/

Transcript

1. A Snake in the Bits Security Automation with Python Moses Schwartz, Security Automation Engineer moses@box.com | @mosesschwartz | github.com/mosesschwartz Andy Culler, Security Automation Engineer aculler@box.com | github.com/aculler PyCon 2019
2. 2 A Snake in the Bits Incident response / security monitoring infrastructure Before automation Network Devices User Laptops Servers Security Tools SIEM Logs Ticketing System Active Directory Asset Mgmt VirusTotal Playbook Docs ???? ???? Alerts User Email Report Other Sources
3. 3 A Snake in the Bits Build a search query that matches the condition you want to alert on Splunk Alert development
4. 4 A Snake in the Bits Create an alert using your search query Best practice is to create an App to contain all of your custom settings I like to specify a cron schedule for maximum flexibility Ensure your time range matches the schedule Splunk Alert development
5. 5 A Snake in the Bits Trigger when Number of Results is greater than 0 Trigger for each result - Splunk webhooks only include the first row of data Add a Webhook Trigger Action and aim it at your server (we’ll build this in the next step) Splunk Alert development
6. 6 A Snake in the Bits We will use Flask for super simple API development Always include a status/health endpoint splunk_webhook will write the JSON payload with indentation to a file host=“0.0.0.0” exposes this to the world! automation_server.py Receive Splunk webhook payload
7. 7 A Snake in the Bits It’s JSON from our alert! Development tip: modify that alert to run every minute and extend the time range Don’t run it like this in production - there are many tutorials on deploying a Flask app with Nginx or Apache and a WSGI server automation_server.py Run the server and check the output
8. 8 A Snake in the Bits Your code should be under version control, but your passwords shouldn’t! A super lightweight approach is to keep your secrets and settings in a Python file that is NOT checked in with code (don’t forget to add this file to your .gitignore) This file can then be pushed as part of configuration management or manually settings.py Keep secrets out of git!
9. 9 A Snake in the Bits Let’s create a ticket in Jira Create your authenticated JIRA object using the Python library Use the create_issue method to create the ticket and set fields automation_server.py Round two: ticket creation
10. 10 A Snake in the Bits After the next Splunk webhook fires, we’ll have a Jira ticket Right now the description is just a JSON blob of the alert Jira Issue created
11. 11 A Snake in the Bits Create a webhook to do enrichments – start by just extracting user and MD5 and commenting on the ticket Point the URL to your automation server with a new endpoint Filter for Issue created events that match our project and alert name Jira Webhook configuration
12. 12 A Snake in the Bits automation_server.py Add a comment to Jira
13. 13 A Snake in the Bits Returns a dict loaded from JSON: ad_lookup.py Lookup a user in Active Directory {'entries': [{'attributes': { 'cn': 'Moses Schwartz', 'title': 'Staff Security Engineer', 'company': 'Box, Inc', 'department': 'Security Automation', 'employeeID': '1234', 'l': 'Redwood City', 'streetAddress': '900 Jefferson Avenue', # ... tons more fields omitted }}]}
14. 14 A Snake in the Bits automation_server.py Active Directory lookup enrichment
15. 15 A Snake in the Bits virustotal.py Get a file scan report {'scan_id': 'e3b0c44298fc1c149afbf48996f...', 'sha1': 'da39a3ee5e6b4b0d3255bff9560189...', 'resource': 'd41d8cd98f00b204e980098ecf...', 'scan_date': '2019-03-01 23:35:34', 'permalink': 'https://www.virustotal.com/... 'total': 60, 'positives': 0, 'md5': 'd41d8cd98f00b204e9800998ecf8427e' {'scans': {'Bkav': {'detected': False, 'version': '1.3.0.9899', 'result': None, 'update': '20190301'} ....
16. 16 A Snake in the Bits automation_server.py Now with AD and VT enrichments
17. 17 A Snake in the Bits Incident response / security monitoring infrastructure Before automation Network Devices User Laptops Servers Security Tools SIEM Logs Ticketing System Active Directory Asset Mgmt VirusTotal Playbook Docs ???? ???? Alerts User Email Report Other Sources
18. 18 A Snake in the Bits Incident response / security monitoring infrastructure With automation MockScan Splunk Logs Automation server Alerts Jira Create ticket Webhook Active Directory Enrich ticket VirusTotal
19. 19 A Snake in the Bits • Search for and link to previous tickets, populate ticket fields, close duplicate tickets • Run a Splunk search • Lookup DNS and WHOIS records • Run Ansible playbooks • Send a sample to a sandbox • Upload files to Box • Quarantine hosts and grab memory • Pull network packet captures (PCAPs) • Flash a light or connect to other smart devices More things we could automate Anything you can write a script to do
20. 20 A Snake in the Bits Authentication Logging Documentation Input validation Error handling Asynchronous task execution Status/health monitoring Improve Our Tooling How about a framework?
21. 21 A Snake in the Bits Flask extensions such as Flask-RESTPlus can do a lot of work for you Provides consistent API interaction and error messages Automatically generates Swagger docs!! automation_server.py Now using Flask-RESTPlus
22. 22 A Snake in the Bits Swagger (OpenAPI) is a framework for documenting and building APIs Swagger UI allows us to interact with our API through the browser Docstrings of your endpoint classes automatically populate the Swagger documentation Swagger Docs Automatically generated from code
23. 23 A Snake in the Bits Specify a model for the endpoint’s expected input We don’t need to worry about all of the webhook fields, just the ones we use Enable validation to reject bad requests automation_server.py Input validation
24. 24 A Snake in the Bits The model is used to automatically build the example payload Now we can run these endpoints through the web interface Swagger Docs Model specifies payload format
25. 25 A Snake in the Bits We weren’t handling errors in our code before, so we got default web server HTML error messages Flask-RESTPlus will return properly formatted JSON message Adding input validation prevents the error in the first place and provides semi- helpful error messages API interaction Error handling / input validation
26. 26 A Snake in the Bits • Our in-house framework • We’ll get this open sourced • Logging • Authentication • Pre/post-processing plugins • Code-defined API • Swagger spec generation • Input validation • Exception handling Meet Funnel aka. “Not Invented Here Syndrome”
27. 27 A Snake in the Bits Maintaining state is hard: avoid it whenever possible A surprising number of errors can be resolved by waiting and retrying Keep your code modular and small Enrichments and other tasks should be asynchronous: • Individual Jira webhooks for each enrichment • Celery, multiprocessing, asyncio (DIY Python approaches) • StackStorm, Jenkins, Rundeck (DevOps) • AWS Lambda jobs (cloud magic) • Commercial Security Automation Platform (”SOAR”) Done is better than perfect Keep it real and keep it running Everything that can break, will break Source: xkcd.com
28. 28 A Snake in the Bits Management loves colorful numbers: track your metrics! This approach is not specific to security – it can be used for anything Security automation isn’t about replacing people, and it’s not a set-it-and-forget-it solution Existing tools that aren't marketed toward security can work great in this space There is so much low hanging fruit Our job is to make the rest of the team more effective (which is pretty awesome) This niche is a great path into security from development Takeaways Security automation is not magic
29. A Snake in the Bits Security Automation with Python Moses Schwartz, Security Automation Engineer moses@box.com | @mosesschwartz | github.com/mosesschwartz Andy Culler, Security Automation Engineer aculler@box.com | github.com/aculler PyCon 2019 We’re hiring – Bay Area, CA and Austin, TX Come work with us! https://www.box.com/careers