Upgrade to Pro — share decks privately, control downloads, hide ads and more …

OSSEC at Scale

8d96f5c273062cb617255e630fe0705c?s=47 Brad Lhotsky
September 16, 2014

OSSEC at Scale

Presentation given on applying DevOps mindset to Security with OSSEC as the Focus.

8d96f5c273062cb617255e630fe0705c?s=128

Brad Lhotsky

September 16, 2014
Tweet

Transcript

  1. at Scale Making Computers Work for You!

  2. Brad Lhotsky • Systems Security Team Lead
 at Booking.com" "

    • Recovering" • Perl Programmer" • Linux Systems Admin" • Network Security Specialist" • PostgreSQL Administrator" • ElasticSearch Janitor" • DNS Voyeur" • Starving Author" • OSSEC Team Member!
  3. From 30,000 Feet ‣The Joy in Obsoleting Yourself" ‣OSSEC Instrumentation"

    ‣Instrumentation with OSSEC" ‣Pitfalls and Caveats" ‣Q & A
  4. None
  5. DevOps Proliferation

  6. DevOps Goals ‣ Infrastructure as Code" ‣ Continuous Release" ‣

    Testing ‣ Easy Deployment and Rollback ‣ Monitoring" ‣ Reduce “Bus Factor”" ‣ Replayability" ‣ Beer
  7. OSSEC Instrumentation ‣ Identify common pain points ‣ Write a

    script to solve the problem ‣ Commit that script ‣ Deploy that script ‣ Re-use that script ‣ Automatically!
  8. Pain: client.keys

  9. Easy Solution ‣ Server" ‣ ossec-authd ‣ Clients" ‣ agent-auth

    Or is it?" Agents still need manual run of agent-auth!
  10. So fix it! exec { 'agent-auth': path => [ '/var/ossec/bin'

    ], timeout => 10, command => "agent-auth -m ${::ossec_server_ip} -A ${::fqdn}", creates => '/var/ossec/etc/client.keys', notify => Service['ossec-hids'], require => Package['ossec-hids-client']; } Example with Puppet
  11. Pain: Monitoring OSSEC

  12. Use Existing Tools Graphite (https://github.com/graphite-project)

  13. Simple Graphite Script #!/bin/bash # Crontab: * * * *

    * /path/to/this-script.sh LIST_AGENTS=‘/var/ossec/bin/list_agents’ CARBON_HOST=‘graphite.example.com’ CARBON_PORT=2003 " prefix=“security.ossec.$(hostname -s)” ts=$(date +%s) all=$($LIST_AGENTS -a |wc -l) connected=$($LIST_AGENTS -c | wc -l) " echo <<EOM $prefix.available $all $ts $prefix.connected $connected $ts EOM | nc $CARBON_HOST $CARBON_PORT
  14. Nagios can monitor values in Graphite and alert on thresholds.

  15. Kibana (http://www.elasticsearch.org/overview/kibana/)

  16. If you missed Vic’s Presentation ..

  17. Pain: Relocating Clients to a New Server

  18. Puppetry $ossec_server_ip = extlookup(‘ossec_server_ip’); " file { '/etc/facter/facts.d/ossec.txt': content =>

    inline_template("prev_ossec_server=<%= @ossec_server_ip %>\n"), require => Service[‘ossec-hids']; } " if ( $ossec_server_ip != $::prev_ossec_server ) { ossec::reset { $ossec_server_ip: } }
  19. Reset the Client " define ossec::reset() { notify { "OSSEC

    SERVER RESET: $name (prev:$::prev_ossec_server)": } # Remove the Client Keys exec { "ossec-stop": path => [ '/sbin', '/bin', '/usr/bin', '/var/ossec/bin' ], timeout => 10, command => "ossec-control stop"; " "ossec-remove-client-keys": before => Exec['agent-auth'], onlyif => 'test -f /var/ossec/etc/client.keys', command => '/bin/rm -f /var/ossec/etc/client.keys’, require => Exec['ossec-stop']; " "ossec-rids-reset": path => [ '/bin', ‘/usr/bin' ], timeout => 10, command => "rm -f /var/ossec/queue/rids/*", require => Exec['ossec-stop'], notify => Service['ossec-hids']; } } }
  20. Now Auto-Distribute! " $ossec_servers = extlookup(‘ossec_servers’) # Now an array

    " file { ‘/etc/facter/facts.d/ossec.txt': content => template(‘ossec/ossec_server_fact.erb’); require => Service[‘ossec-hids']; } " # ossec_server_fact.erb <% uuid = scope.lookupvar("::uniqueid"); # Convert HEX to Integer seed = [uuid].pack(‘H*’).unpack('l')[0]; # Use UUID as Random Seed srand(seed); # Get seeded random number in range idx = rand(0 .. @ossec_servers.length-1); -%> prev_ossec_server=<% @ossec_servers[idx] %>
  21. Instrumentation with OSSEC

  22. DevOps ‣ Configuration Management has States ‣ Configuration Files ‣

    Application Versions ‣ Resource Status and Definitions ‣ OSSEC has States too! ‣ Log data ‣ System status ‣ Process status ‣ Network status Instrumentation with a Security Tool? SEC
  23. ‣ Configuration Management has Actions ‣ Resource CRUD ‣ Also

    “Run this script, kthxbye.” ‣ OSSEC has Actions too! ‣ ActiveResponse!!! ‣ Mostly, “Run this script, kthxbye.” Instrumentation with a Security Tool? DevOpsSEC
  24. File Integrity Monitoring ‣ Noisy, at best ‣ Things messing

    with your files are mostly legit: ‣ System Updates ‣ Configuration Management ‣ Software Deployments ‣ Sysadmins Saving the Day
  25. OSSEC v2.8+ ‣ ActiveResponse passes alert->filename ‣ Write a script

    which does your job ‣ Commit that script ‣ Deploy that script ‣ Re-use that script ‣ Automatically!
  26. Game Plan ‣ Demote FIM Alerts to Level 1, disable

    email ‣ Fire an ActiveResponse that takes filename ‣ Emit a new log message in our script ‣ Decode the new log message ‣ Alert / Log based on that ‣ … ‣ PROFIT!
  27. FIM Alerting <global> <alerts> <log_alert_level>1</log_alert_level> <email_alert_level>9</email_alert_level> </alerts> </global> " <syscheck>

    <!-- Alert Enhancements --> <auto_ignore>no</auto_ignore> <alert_new_files>yes</alert_new_files> <!-- Normal Stuff —> </syscheck>
  28. Splaying Scan Time <% uuid = scope.lookupvar("::uniqueid"); seed = [uuid].pack('H*').unpack('l')[0];

    srand(seed); " # Set our runtime for syscheckd hour = rand(4 .. 7); minute = sprintf("%02d", rand(0 .. 59)); -%> <syscheck> <!-- Better Scheduling --> <scan_on_start>no</scan_on_start> <scan_time><%= hour %>:<%= minute %>am</scan_time> <frequency>82800</frequency> <!-- Normal Stuff —> </syscheck>
  29. FIM Alerting cont’d <rule id="106002" level="1"> <category>ossec</catgeory> <if_group>syscheck</if_group> <options>no_email_alert</options> <description>Verify

    file changes. </description> </rule>
  30. Problem Solved! No more emails!

  31. Fire ActiveResponse <command> <name>ossec-ar-verify-file</name> <executable>ossec-ar-verify-file.py</executable> <expect>filename</expect> <timeout_allowed>no</timeout_allowed> </command> " <active-response>

    <command>ossec-ar-verify-file</command> <location>local</location> <rules_id>106002</rules_id> </active-response>
  32. Verify Logs Sep 15 00:26:10 ether ossec-ar-verify: file ok (/etc/

    cron.d/puppet-job) " " Sep 15 00:26:10 ether ossec-ar-verify: file managed by RPM (/etc/mcollective/facts.yaml) changed outside of RPM " " Sep 15 00:26:10 ether ossec-ar-verify: file unmanaged (/ etc/postfix/aliases.db) changed
  33. FIM Decoder <decoder name="ossec-ar-verify"> <program_name>ossec-ar-verify</program_name> </decoder> " <decoder name="ossec-ar-verify-file"> <parent>ossec-ar-verify</parent>

    <prematch>^file </prematch> <regex offset="after_prematch">^(\S+)</regex> <order>action</order> </decoder>
  34. Parent Rule <rule id="107000" level="1"> <decoded_as>ossec-ar-verify</decoded_as> <description>Verification</description> <group>verify,</group> </rule>

  35. Everything is OK <rule id="107001" level="3"> <if_sid>107000</if_sid> <match>^file ok </match>

    <description>File was changed intentionally. </description> </rule>
  36. Managed File Change <rule id="107002" level="11"> <if_sid>107000</if_sid> <match>^file managed </match>

    <description>Verified: Unauthorized File Change </description> </rule>
  37. Unmanaged File Change <rule id="107003" level="11"> <if_sid>107000</if_sid> <match>^file unmanaged </match>

    <description>Verified: Unknown File Change </description> <options>no_email_alert</options> </rule>
  38. OSSEC FIM Results 96% Reduction in Alerting

  39. Pitfalls and Caveats ‣ Who controls inputs? ‣ How resource

    intensive are your checks? ‣ What if 1,000,000 fire simultaneously? ‣ On the same server? ‣ Think, test, then get some to try to break it.
  40. CVE-2014-5284 ‣ host-deny.sh created files in /tmp ‣ cp /tmp/hosts.$$.deny

    /etc/hosts.deny ‣ Didn’t properly manage permissions ‣ Would copy, as root, the contents of that file to /etc ‣ Moved from /tmp to /var/ossec and added randomness to file name
  41. Find Me GitHub: https://github.com/reyjrar/ Twitter: @reyjrar Blogging: http://edgeofsanity.net Email: brad.lhotsky@gmail.com