Upgrade to Pro — share decks privately, control downloads, hide ads and more …

OSSEC at Scale

Brad Lhotsky
September 16, 2014

OSSEC at Scale

Presentation given on applying DevOps mindset to Security with OSSEC as the Focus.

Brad Lhotsky

September 16, 2014
Tweet

More Decks by Brad Lhotsky

Other Decks in Technology

Transcript

  1. at Scale Making Computers Work for You!

  2. Brad Lhotsky • Systems Security Team Lead
 at Booking.com" "

    • Recovering" • Perl Programmer" • Linux Systems Admin" • Network Security Specialist" • PostgreSQL Administrator" • ElasticSearch Janitor" • DNS Voyeur" • Starving Author" • OSSEC Team Member!
  3. From 30,000 Feet ‣The Joy in Obsoleting Yourself" ‣OSSEC Instrumentation"

    ‣Instrumentation with OSSEC" ‣Pitfalls and Caveats" ‣Q & A
  4. None
  5. DevOps Proliferation

  6. DevOps Goals ‣ Infrastructure as Code" ‣ Continuous Release" ‣

    Testing ‣ Easy Deployment and Rollback ‣ Monitoring" ‣ Reduce “Bus Factor”" ‣ Replayability" ‣ Beer
  7. OSSEC Instrumentation ‣ Identify common pain points ‣ Write a

    script to solve the problem ‣ Commit that script ‣ Deploy that script ‣ Re-use that script ‣ Automatically!
  8. Pain: client.keys

  9. Easy Solution ‣ Server" ‣ ossec-authd ‣ Clients" ‣ agent-auth

    Or is it?" Agents still need manual run of agent-auth!
  10. So fix it! exec { 'agent-auth': path => [ '/var/ossec/bin'

    ], timeout => 10, command => "agent-auth -m ${::ossec_server_ip} -A ${::fqdn}", creates => '/var/ossec/etc/client.keys', notify => Service['ossec-hids'], require => Package['ossec-hids-client']; } Example with Puppet
  11. Pain: Monitoring OSSEC

  12. Use Existing Tools Graphite (https://github.com/graphite-project)

  13. Simple Graphite Script #!/bin/bash # Crontab: * * * *

    * /path/to/this-script.sh LIST_AGENTS=‘/var/ossec/bin/list_agents’ CARBON_HOST=‘graphite.example.com’ CARBON_PORT=2003 " prefix=“security.ossec.$(hostname -s)” ts=$(date +%s) all=$($LIST_AGENTS -a |wc -l) connected=$($LIST_AGENTS -c | wc -l) " echo <<EOM $prefix.available $all $ts $prefix.connected $connected $ts EOM | nc $CARBON_HOST $CARBON_PORT
  14. Nagios can monitor values in Graphite and alert on thresholds.

  15. Kibana (http://www.elasticsearch.org/overview/kibana/)

  16. If you missed Vic’s Presentation ..

  17. Pain: Relocating Clients to a New Server

  18. Puppetry $ossec_server_ip = extlookup(‘ossec_server_ip’); " file { '/etc/facter/facts.d/ossec.txt': content =>

    inline_template("prev_ossec_server=<%= @ossec_server_ip %>\n"), require => Service[‘ossec-hids']; } " if ( $ossec_server_ip != $::prev_ossec_server ) { ossec::reset { $ossec_server_ip: } }
  19. Reset the Client " define ossec::reset() { notify { "OSSEC

    SERVER RESET: $name (prev:$::prev_ossec_server)": } # Remove the Client Keys exec { "ossec-stop": path => [ '/sbin', '/bin', '/usr/bin', '/var/ossec/bin' ], timeout => 10, command => "ossec-control stop"; " "ossec-remove-client-keys": before => Exec['agent-auth'], onlyif => 'test -f /var/ossec/etc/client.keys', command => '/bin/rm -f /var/ossec/etc/client.keys’, require => Exec['ossec-stop']; " "ossec-rids-reset": path => [ '/bin', ‘/usr/bin' ], timeout => 10, command => "rm -f /var/ossec/queue/rids/*", require => Exec['ossec-stop'], notify => Service['ossec-hids']; } } }
  20. Now Auto-Distribute! " $ossec_servers = extlookup(‘ossec_servers’) # Now an array

    " file { ‘/etc/facter/facts.d/ossec.txt': content => template(‘ossec/ossec_server_fact.erb’); require => Service[‘ossec-hids']; } " # ossec_server_fact.erb <% uuid = scope.lookupvar("::uniqueid"); # Convert HEX to Integer seed = [uuid].pack(‘H*’).unpack('l')[0]; # Use UUID as Random Seed srand(seed); # Get seeded random number in range idx = rand(0 .. @ossec_servers.length-1); -%> prev_ossec_server=<% @ossec_servers[idx] %>
  21. Instrumentation with OSSEC

  22. DevOps ‣ Configuration Management has States ‣ Configuration Files ‣

    Application Versions ‣ Resource Status and Definitions ‣ OSSEC has States too! ‣ Log data ‣ System status ‣ Process status ‣ Network status Instrumentation with a Security Tool? SEC
  23. ‣ Configuration Management has Actions ‣ Resource CRUD ‣ Also

    “Run this script, kthxbye.” ‣ OSSEC has Actions too! ‣ ActiveResponse!!! ‣ Mostly, “Run this script, kthxbye.” Instrumentation with a Security Tool? DevOpsSEC
  24. File Integrity Monitoring ‣ Noisy, at best ‣ Things messing

    with your files are mostly legit: ‣ System Updates ‣ Configuration Management ‣ Software Deployments ‣ Sysadmins Saving the Day
  25. OSSEC v2.8+ ‣ ActiveResponse passes alert->filename ‣ Write a script

    which does your job ‣ Commit that script ‣ Deploy that script ‣ Re-use that script ‣ Automatically!
  26. Game Plan ‣ Demote FIM Alerts to Level 1, disable

    email ‣ Fire an ActiveResponse that takes filename ‣ Emit a new log message in our script ‣ Decode the new log message ‣ Alert / Log based on that ‣ … ‣ PROFIT!
  27. FIM Alerting <global> <alerts> <log_alert_level>1</log_alert_level> <email_alert_level>9</email_alert_level> </alerts> </global> " <syscheck>

    <!-- Alert Enhancements --> <auto_ignore>no</auto_ignore> <alert_new_files>yes</alert_new_files> <!-- Normal Stuff —> </syscheck>
  28. Splaying Scan Time <% uuid = scope.lookupvar("::uniqueid"); seed = [uuid].pack('H*').unpack('l')[0];

    srand(seed); " # Set our runtime for syscheckd hour = rand(4 .. 7); minute = sprintf("%02d", rand(0 .. 59)); -%> <syscheck> <!-- Better Scheduling --> <scan_on_start>no</scan_on_start> <scan_time><%= hour %>:<%= minute %>am</scan_time> <frequency>82800</frequency> <!-- Normal Stuff —> </syscheck>
  29. FIM Alerting cont’d <rule id="106002" level="1"> <category>ossec</catgeory> <if_group>syscheck</if_group> <options>no_email_alert</options> <description>Verify

    file changes. </description> </rule>
  30. Problem Solved! No more emails!

  31. Fire ActiveResponse <command> <name>ossec-ar-verify-file</name> <executable>ossec-ar-verify-file.py</executable> <expect>filename</expect> <timeout_allowed>no</timeout_allowed> </command> " <active-response>

    <command>ossec-ar-verify-file</command> <location>local</location> <rules_id>106002</rules_id> </active-response>
  32. Verify Logs Sep 15 00:26:10 ether ossec-ar-verify: file ok (/etc/

    cron.d/puppet-job) " " Sep 15 00:26:10 ether ossec-ar-verify: file managed by RPM (/etc/mcollective/facts.yaml) changed outside of RPM " " Sep 15 00:26:10 ether ossec-ar-verify: file unmanaged (/ etc/postfix/aliases.db) changed
  33. FIM Decoder <decoder name="ossec-ar-verify"> <program_name>ossec-ar-verify</program_name> </decoder> " <decoder name="ossec-ar-verify-file"> <parent>ossec-ar-verify</parent>

    <prematch>^file </prematch> <regex offset="after_prematch">^(\S+)</regex> <order>action</order> </decoder>
  34. Parent Rule <rule id="107000" level="1"> <decoded_as>ossec-ar-verify</decoded_as> <description>Verification</description> <group>verify,</group> </rule>

  35. Everything is OK <rule id="107001" level="3"> <if_sid>107000</if_sid> <match>^file ok </match>

    <description>File was changed intentionally. </description> </rule>
  36. Managed File Change <rule id="107002" level="11"> <if_sid>107000</if_sid> <match>^file managed </match>

    <description>Verified: Unauthorized File Change </description> </rule>
  37. Unmanaged File Change <rule id="107003" level="11"> <if_sid>107000</if_sid> <match>^file unmanaged </match>

    <description>Verified: Unknown File Change </description> <options>no_email_alert</options> </rule>
  38. OSSEC FIM Results 96% Reduction in Alerting

  39. Pitfalls and Caveats ‣ Who controls inputs? ‣ How resource

    intensive are your checks? ‣ What if 1,000,000 fire simultaneously? ‣ On the same server? ‣ Think, test, then get some to try to break it.
  40. CVE-2014-5284 ‣ host-deny.sh created files in /tmp ‣ cp /tmp/hosts.$$.deny

    /etc/hosts.deny ‣ Didn’t properly manage permissions ‣ Would copy, as root, the contents of that file to /etc ‣ Moved from /tmp to /var/ossec and added randomness to file name
  41. Find Me GitHub: https://github.com/reyjrar/ Twitter: @reyjrar Blogging: http://edgeofsanity.net Email: brad.lhotsky@gmail.com