Share
Embed
OSSEC at Scale
Presentation given on applying DevOps mindset to Security with OSSEC as the Focus.
Brad Lhotsky
September 16, 2014
Transcript
41 slides
-
Brad Lhotsky • Systems Security Team Lead at Booking.com" "
• Recovering" • Perl Programmer" • Linux Systems Admin" • Network Security Specialist" • PostgreSQL Administrator" • ElasticSearch Janitor" • DNS Voyeur" • Starving Author" • OSSEC Team Member! -
From 30,000 Feet ‣The Joy in Obsoleting Yourself" ‣OSSEC Instrumentation"
‣Instrumentation with OSSEC" ‣Pitfalls and Caveats" ‣Q & A -
DevOps Goals ‣ Infrastructure as Code" ‣ Continuous Release" ‣
Testing ‣ Easy Deployment and Rollback ‣ Monitoring" ‣ Reduce “Bus Factor”" ‣ Replayability" ‣ Beer -
OSSEC Instrumentation ‣ Identify common pain points ‣ Write a
script to solve the problem ‣ Commit that script ‣ Deploy that script ‣ Re-use that script ‣ Automatically! -
Easy Solution ‣ Server" ‣ ossec-authd ‣ Clients" ‣ agent-auth
Or is it?" Agents still need manual run of agent-auth! -
So fix it! exec { 'agent-auth': path => [ '/var/ossec/bin'
], timeout => 10, command => "agent-auth -m ${::ossec_server_ip} -A ${::fqdn}", creates => '/var/ossec/etc/client.keys', notify => Service['ossec-hids'], require => Package['ossec-hids-client']; } Example with Puppet -
Simple Graphite Script #!/bin/bash # Crontab: * * * *
* /path/to/this-script.sh LIST_AGENTS=‘/var/ossec/bin/list_agents’ CARBON_HOST=‘graphite.example.com’ CARBON_PORT=2003 " prefix=“security.ossec.$(hostname -s)” ts=$(date +%s) all=$($LIST_AGENTS -a |wc -l) connected=$($LIST_AGENTS -c | wc -l) " echo <<EOM $prefix.available $all $ts $prefix.connected $connected $ts EOM | nc $CARBON_HOST $CARBON_PORT -
Puppetry $ossec_server_ip = extlookup(‘ossec_server_ip’); " file { '/etc/facter/facts.d/ossec.txt': content =>
inline_template("prev_ossec_server=<%= @ossec_server_ip %>\n"), require => Service[‘ossec-hids']; } " if ( $ossec_server_ip != $::prev_ossec_server ) { ossec::reset { $ossec_server_ip: } } -
Reset the Client " define ossec::reset() { notify { "OSSEC
SERVER RESET: $name (prev:$::prev_ossec_server)": } # Remove the Client Keys exec { "ossec-stop": path => [ '/sbin', '/bin', '/usr/bin', '/var/ossec/bin' ], timeout => 10, command => "ossec-control stop"; " "ossec-remove-client-keys": before => Exec['agent-auth'], onlyif => 'test -f /var/ossec/etc/client.keys', command => '/bin/rm -f /var/ossec/etc/client.keys’, require => Exec['ossec-stop']; " "ossec-rids-reset": path => [ '/bin', ‘/usr/bin' ], timeout => 10, command => "rm -f /var/ossec/queue/rids/*", require => Exec['ossec-stop'], notify => Service['ossec-hids']; } } } -
Now Auto-Distribute! " $ossec_servers = extlookup(‘ossec_servers’) # Now an array
" file { ‘/etc/facter/facts.d/ossec.txt': content => template(‘ossec/ossec_server_fact.erb’); require => Service[‘ossec-hids']; } " # ossec_server_fact.erb <% uuid = scope.lookupvar("::uniqueid"); # Convert HEX to Integer seed = [uuid].pack(‘H*’).unpack('l')[0]; # Use UUID as Random Seed srand(seed); # Get seeded random number in range idx = rand(0 .. @ossec_servers.length-1); -%> prev_ossec_server=<% @ossec_servers[idx] %> -
DevOps ‣ Configuration Management has States ‣ Configuration Files ‣
Application Versions ‣ Resource Status and Definitions ‣ OSSEC has States too! ‣ Log data ‣ System status ‣ Process status ‣ Network status Instrumentation with a Security Tool? SEC -
‣ Configuration Management has Actions ‣ Resource CRUD ‣ Also
“Run this script, kthxbye.” ‣ OSSEC has Actions too! ‣ ActiveResponse!!! ‣ Mostly, “Run this script, kthxbye.” Instrumentation with a Security Tool? DevOpsSEC -
File Integrity Monitoring ‣ Noisy, at best ‣ Things messing
with your files are mostly legit: ‣ System Updates ‣ Configuration Management ‣ Software Deployments ‣ Sysadmins Saving the Day -
OSSEC v2.8+ ‣ ActiveResponse passes alert->filename ‣ Write a script
which does your job ‣ Commit that script ‣ Deploy that script ‣ Re-use that script ‣ Automatically! -
Game Plan ‣ Demote FIM Alerts to Level 1, disable
email ‣ Fire an ActiveResponse that takes filename ‣ Emit a new log message in our script ‣ Decode the new log message ‣ Alert / Log based on that ‣ … ‣ PROFIT! -
FIM Alerting <global> <alerts> <log_alert_level>1</log_alert_level> <email_alert_level>9</email_alert_level> </alerts> </global> " <syscheck>
<!-- Alert Enhancements --> <auto_ignore>no</auto_ignore> <alert_new_files>yes</alert_new_files> <!-- Normal Stuff —> </syscheck> -
Splaying Scan Time <% uuid = scope.lookupvar("::uniqueid"); seed = [uuid].pack('H*').unpack('l')[0];
srand(seed); " # Set our runtime for syscheckd hour = rand(4 .. 7); minute = sprintf("%02d", rand(0 .. 59)); -%> <syscheck> <!-- Better Scheduling --> <scan_on_start>no</scan_on_start> <scan_time><%= hour %>:<%= minute %>am</scan_time> <frequency>82800</frequency> <!-- Normal Stuff —> </syscheck> -
Fire ActiveResponse <command> <name>ossec-ar-verify-file</name> <executable>ossec-ar-verify-file.py</executable> <expect>filename</expect> <timeout_allowed>no</timeout_allowed> </command> " <active-response>
<command>ossec-ar-verify-file</command> <location>local</location> <rules_id>106002</rules_id> </active-response> -
Verify Logs Sep 15 00:26:10 ether ossec-ar-verify: file ok (/etc/
cron.d/puppet-job) " " Sep 15 00:26:10 ether ossec-ar-verify: file managed by RPM (/etc/mcollective/facts.yaml) changed outside of RPM " " Sep 15 00:26:10 ether ossec-ar-verify: file unmanaged (/ etc/postfix/aliases.db) changed -
FIM Decoder <decoder name="ossec-ar-verify"> <program_name>ossec-ar-verify</program_name> </decoder> " <decoder name="ossec-ar-verify-file"> <parent>ossec-ar-verify</parent>
<prematch>^file </prematch> <regex offset="after_prematch">^(\S+)</regex> <order>action</order> </decoder> -
Everything is OK <rule id="107001" level="3"> <if_sid>107000</if_sid> <match>^file ok </match>
<description>File was changed intentionally. </description> </rule> -
Managed File Change <rule id="107002" level="11"> <if_sid>107000</if_sid> <match>^file managed </match>
<description>Verified: Unauthorized File Change </description> </rule> -
Unmanaged File Change <rule id="107003" level="11"> <if_sid>107000</if_sid> <match>^file unmanaged </match>
<description>Verified: Unknown File Change </description> <options>no_email_alert</options> </rule> -
Pitfalls and Caveats ‣ Who controls inputs? ‣ How resource
intensive are your checks? ‣ What if 1,000,000 fire simultaneously? ‣ On the same server? ‣ Think, test, then get some to try to break it. -
CVE-2014-5284 ‣ host-deny.sh created files in /tmp ‣ cp /tmp/hosts.$$.deny
/etc/hosts.deny ‣ Didn’t properly manage permissions ‣ Would copy, as root, the contents of that file to /etc ‣ Moved from /tmp to /var/ossec and added randomness to file name