at Scale Making Computers Work for You!

Brad Lhotsky • Systems Security Team Lead
 at Booking.com" " • Recovering" • Perl Programmer" • Linux Systems Admin" • Network Security Specialist" • PostgreSQL Administrator" • ElasticSearch Janitor" • DNS Voyeur" • Starving Author" • OSSEC Team Member!

From 30,000 Feet ‣The Joy in Obsoleting Yourself" ‣OSSEC Instrumentation" ‣Instrumentation with OSSEC" ‣Pitfalls and Caveats" ‣Q & A

No content

DevOps Proliferation

DevOps Goals ‣ Infrastructure as Code" ‣ Continuous Release" ‣ Testing ‣ Easy Deployment and Rollback ‣ Monitoring" ‣ Reduce “Bus Factor”" ‣ Replayability" ‣ Beer

OSSEC Instrumentation ‣ Identify common pain points ‣ Write a script to solve the problem ‣ Commit that script ‣ Deploy that script ‣ Re-use that script ‣ Automatically!

Pain: client.keys

Easy Solution ‣ Server" ‣ ossec-authd ‣ Clients" ‣ agent-auth Or is it?" Agents still need manual run of agent-auth!

So fix it! exec { 'agent-auth': path => [ '/var/ossec/bin' ], timeout => 10, command => "agent-auth -m ${::ossec_server_ip} -A ${::fqdn}", creates => '/var/ossec/etc/client.keys', notify => Service['ossec-hids'], require => Package['ossec-hids-client']; } Example with Puppet

Pain: Monitoring OSSEC

Use Existing Tools Graphite (https://github.com/graphite-project)

Simple Graphite Script #!/bin/bash # Crontab: * * * * * /path/to/this-script.sh LIST_AGENTS=‘/var/ossec/bin/list_agents’ CARBON_HOST=‘graphite.example.com’ CARBON_PORT=2003 " prefix=“security.ossec.$(hostname -s)” ts=$(date +%s) all=$($LIST_AGENTS -a |wc -l) connected=$($LIST_AGENTS -c | wc -l) " echo <

Nagios can monitor values in Graphite and alert on thresholds.

Kibana (http://www.elasticsearch.org/overview/kibana/)

If you missed Vic’s Presentation ..

Pain: Relocating Clients to a New Server

Puppetry $ossec_server_ip = extlookup(‘ossec_server_ip’); " file { '/etc/facter/facts.d/ossec.txt': content => inline_template("prev_ossec_server=<%= @ossec_server_ip %>\n"), require => Service[‘ossec-hids']; } " if ( $ossec_server_ip != $::prev_ossec_server ) { ossec::reset { $ossec_server_ip: } }

Reset the Client " define ossec::reset() { notify { "OSSEC SERVER RESET: $name (prev:$::prev_ossec_server)": } # Remove the Client Keys exec { "ossec-stop": path => [ '/sbin', '/bin', '/usr/bin', '/var/ossec/bin' ], timeout => 10, command => "ossec-control stop"; " "ossec-remove-client-keys": before => Exec['agent-auth'], onlyif => 'test -f /var/ossec/etc/client.keys', command => '/bin/rm -f /var/ossec/etc/client.keys’, require => Exec['ossec-stop']; " "ossec-rids-reset": path => [ '/bin', ‘/usr/bin' ], timeout => 10, command => "rm -f /var/ossec/queue/rids/*", require => Exec['ossec-stop'], notify => Service['ossec-hids']; } } }

Now Auto-Distribute! " $ossec_servers = extlookup(‘ossec_servers’) # Now an array " file { ‘/etc/facter/facts.d/ossec.txt': content => template(‘ossec/ossec_server_fact.erb’); require => Service[‘ossec-hids']; } " # ossec_server_fact.erb <% uuid = scope.lookupvar("::uniqueid"); # Convert HEX to Integer seed = [uuid].pack(‘H*’).unpack('l')[0]; # Use UUID as Random Seed srand(seed); # Get seeded random number in range idx = rand(0 .. @ossec_servers.length-1); -%> prev_ossec_server=<% @ossec_servers[idx] %>

Instrumentation with OSSEC

DevOps ‣ Configuration Management has States ‣ Configuration Files ‣ Application Versions ‣ Resource Status and Definitions ‣ OSSEC has States too! ‣ Log data ‣ System status ‣ Process status ‣ Network status Instrumentation with a Security Tool? SEC

‣ Configuration Management has Actions ‣ Resource CRUD ‣ Also “Run this script, kthxbye.” ‣ OSSEC has Actions too! ‣ ActiveResponse!!! ‣ Mostly, “Run this script, kthxbye.” Instrumentation with a Security Tool? DevOpsSEC

File Integrity Monitoring ‣ Noisy, at best ‣ Things messing with your files are mostly legit: ‣ System Updates ‣ Configuration Management ‣ Software Deployments ‣ Sysadmins Saving the Day

OSSEC v2.8+ ‣ ActiveResponse passes alert->filename ‣ Write a script which does your job ‣ Commit that script ‣ Deploy that script ‣ Re-use that script ‣ Automatically!

Game Plan ‣ Demote FIM Alerts to Level 1, disable email ‣ Fire an ActiveResponse that takes filename ‣ Emit a new log message in our script ‣ Decode the new log message ‣ Alert / Log based on that ‣ … ‣ PROFIT!

FIM Alerting 1 9 " no yes

Splaying Scan Time <% uuid = scope.lookupvar("::uniqueid"); seed = [uuid].pack('H*').unpack('l')[0]; srand(seed); " # Set our runtime for syscheckd hour = rand(4 .. 7); minute = sprintf("%02d", rand(0 .. 59)); -%> no <%= hour %>:<%= minute %>am 82800

FIM Alerting cont’d ossec syscheck no_email_alert Verify file changes.

Problem Solved! No more emails!

Fire ActiveResponse ossec-ar-verify-file ossec-ar-verify-file.py filename no " ossec-ar-verify-file local 106002

Verify Logs Sep 15 00:26:10 ether ossec-ar-verify: file ok (/etc/ cron.d/puppet-job) " " Sep 15 00:26:10 ether ossec-ar-verify: file managed by RPM (/etc/mcollective/facts.yaml) changed outside of RPM " " Sep 15 00:26:10 ether ossec-ar-verify: file unmanaged (/ etc/postfix/aliases.db) changed

FIM Decoder ossec-ar-verify " ossec-ar-verify ^file ^(\S+) action

Parent Rule ossec-ar-verify Verification verify,

Everything is OK 107000 ^file ok File was changed intentionally.

Managed File Change 107000 ^file managed Verified: Unauthorized File Change

Unmanaged File Change 107000 ^file unmanaged Verified: Unknown File Change no_email_alert

OSSEC FIM Results 96% Reduction in Alerting

Pitfalls and Caveats ‣ Who controls inputs? ‣ How resource intensive are your checks? ‣ What if 1,000,000 fire simultaneously? ‣ On the same server? ‣ Think, test, then get some to try to break it.

CVE-2014-5284 ‣ host-deny.sh created files in /tmp ‣ cp /tmp/hosts.$$.deny /etc/hosts.deny ‣ Didn’t properly manage permissions ‣ Would copy, as root, the contents of that file to /etc ‣ Moved from /tmp to /var/ossec and added randomness to file name

Find Me GitHub: https://github.com/reyjrar/ Twitter: @reyjrar Blogging: http://edgeofsanity.net Email: brad.lhotsky@gmail.com