Slide 1

Slide 1 text

at Scale Making Computers Work for You!

Slide 2

Slide 2 text

Brad Lhotsky • Systems Security Team Lead
 at Booking.com" " • Recovering" • Perl Programmer" • Linux Systems Admin" • Network Security Specialist" • PostgreSQL Administrator" • ElasticSearch Janitor" • DNS Voyeur" • Starving Author" • OSSEC Team Member!

Slide 3

Slide 3 text

From 30,000 Feet ‣The Joy in Obsoleting Yourself" ‣OSSEC Instrumentation" ‣Instrumentation with OSSEC" ‣Pitfalls and Caveats" ‣Q & A

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

DevOps Proliferation

Slide 6

Slide 6 text

DevOps Goals ‣ Infrastructure as Code" ‣ Continuous Release" ‣ Testing ‣ Easy Deployment and Rollback ‣ Monitoring" ‣ Reduce “Bus Factor”" ‣ Replayability" ‣ Beer

Slide 7

Slide 7 text

OSSEC Instrumentation ‣ Identify common pain points ‣ Write a script to solve the problem ‣ Commit that script ‣ Deploy that script ‣ Re-use that script ‣ Automatically!

Slide 8

Slide 8 text

Pain: client.keys

Slide 9

Slide 9 text

Easy Solution ‣ Server" ‣ ossec-authd ‣ Clients" ‣ agent-auth Or is it?" Agents still need manual run of agent-auth!

Slide 10

Slide 10 text

So fix it! exec { 'agent-auth': path => [ '/var/ossec/bin' ], timeout => 10, command => "agent-auth -m ${::ossec_server_ip} -A ${::fqdn}", creates => '/var/ossec/etc/client.keys', notify => Service['ossec-hids'], require => Package['ossec-hids-client']; } Example with Puppet

Slide 11

Slide 11 text

Pain: Monitoring OSSEC

Slide 12

Slide 12 text

Use Existing Tools Graphite (https://github.com/graphite-project)

Slide 13

Slide 13 text

Simple Graphite Script #!/bin/bash # Crontab: * * * * * /path/to/this-script.sh LIST_AGENTS=‘/var/ossec/bin/list_agents’ CARBON_HOST=‘graphite.example.com’ CARBON_PORT=2003 " prefix=“security.ossec.$(hostname -s)” ts=$(date +%s) all=$($LIST_AGENTS -a |wc -l) connected=$($LIST_AGENTS -c | wc -l) " echo <

Slide 14

Slide 14 text

Nagios can monitor values in Graphite and alert on thresholds.

Slide 15

Slide 15 text

Kibana (http://www.elasticsearch.org/overview/kibana/)

Slide 16

Slide 16 text

If you missed Vic’s Presentation ..

Slide 17

Slide 17 text

Pain: Relocating Clients to a New Server

Slide 18

Slide 18 text

Puppetry $ossec_server_ip = extlookup(‘ossec_server_ip’); " file { '/etc/facter/facts.d/ossec.txt': content => inline_template("prev_ossec_server=<%= @ossec_server_ip %>\n"), require => Service[‘ossec-hids']; } " if ( $ossec_server_ip != $::prev_ossec_server ) { ossec::reset { $ossec_server_ip: } }

Slide 19

Slide 19 text

Reset the Client " define ossec::reset() { notify { "OSSEC SERVER RESET: $name (prev:$::prev_ossec_server)": } # Remove the Client Keys exec { "ossec-stop": path => [ '/sbin', '/bin', '/usr/bin', '/var/ossec/bin' ], timeout => 10, command => "ossec-control stop"; " "ossec-remove-client-keys": before => Exec['agent-auth'], onlyif => 'test -f /var/ossec/etc/client.keys', command => '/bin/rm -f /var/ossec/etc/client.keys’, require => Exec['ossec-stop']; " "ossec-rids-reset": path => [ '/bin', ‘/usr/bin' ], timeout => 10, command => "rm -f /var/ossec/queue/rids/*", require => Exec['ossec-stop'], notify => Service['ossec-hids']; } } }

Slide 20

Slide 20 text

Now Auto-Distribute! " $ossec_servers = extlookup(‘ossec_servers’) # Now an array " file { ‘/etc/facter/facts.d/ossec.txt': content => template(‘ossec/ossec_server_fact.erb’); require => Service[‘ossec-hids']; } " # ossec_server_fact.erb <% uuid = scope.lookupvar("::uniqueid"); # Convert HEX to Integer seed = [uuid].pack(‘H*’).unpack('l')[0]; # Use UUID as Random Seed srand(seed); # Get seeded random number in range idx = rand(0 .. @ossec_servers.length-1); -%> prev_ossec_server=<% @ossec_servers[idx] %>

Slide 21

Slide 21 text

Instrumentation with OSSEC

Slide 22

Slide 22 text

DevOps ‣ Configuration Management has States ‣ Configuration Files ‣ Application Versions ‣ Resource Status and Definitions ‣ OSSEC has States too! ‣ Log data ‣ System status ‣ Process status ‣ Network status Instrumentation with a Security Tool? SEC

Slide 23

Slide 23 text

‣ Configuration Management has Actions ‣ Resource CRUD ‣ Also “Run this script, kthxbye.” ‣ OSSEC has Actions too! ‣ ActiveResponse!!! ‣ Mostly, “Run this script, kthxbye.” Instrumentation with a Security Tool? DevOpsSEC

Slide 24

Slide 24 text

File Integrity Monitoring ‣ Noisy, at best ‣ Things messing with your files are mostly legit: ‣ System Updates ‣ Configuration Management ‣ Software Deployments ‣ Sysadmins Saving the Day

Slide 25

Slide 25 text

OSSEC v2.8+ ‣ ActiveResponse passes alert->filename ‣ Write a script which does your job ‣ Commit that script ‣ Deploy that script ‣ Re-use that script ‣ Automatically!

Slide 26

Slide 26 text

Game Plan ‣ Demote FIM Alerts to Level 1, disable email ‣ Fire an ActiveResponse that takes filename ‣ Emit a new log message in our script ‣ Decode the new log message ‣ Alert / Log based on that ‣ … ‣ PROFIT!

Slide 27

Slide 27 text

FIM Alerting 1 9 " no yes

Slide 28

Slide 28 text

Splaying Scan Time <% uuid = scope.lookupvar("::uniqueid"); seed = [uuid].pack('H*').unpack('l')[0]; srand(seed); " # Set our runtime for syscheckd hour = rand(4 .. 7); minute = sprintf("%02d", rand(0 .. 59)); -%> no <%= hour %>:<%= minute %>am 82800

Slide 29

Slide 29 text

FIM Alerting cont’d ossec syscheck no_email_alert Verify file changes.

Slide 30

Slide 30 text

Problem Solved! No more emails!

Slide 31

Slide 31 text

Fire ActiveResponse ossec-ar-verify-file ossec-ar-verify-file.py filename no " ossec-ar-verify-file local 106002

Slide 32

Slide 32 text

Verify Logs Sep 15 00:26:10 ether ossec-ar-verify: file ok (/etc/ cron.d/puppet-job) " " Sep 15 00:26:10 ether ossec-ar-verify: file managed by RPM (/etc/mcollective/facts.yaml) changed outside of RPM " " Sep 15 00:26:10 ether ossec-ar-verify: file unmanaged (/ etc/postfix/aliases.db) changed

Slide 33

Slide 33 text

FIM Decoder ossec-ar-verify " ossec-ar-verify ^file ^(\S+) action

Slide 34

Slide 34 text

Parent Rule ossec-ar-verify Verification verify,

Slide 35

Slide 35 text

Everything is OK 107000 ^file ok File was changed intentionally.

Slide 36

Slide 36 text

Managed File Change 107000 ^file managed Verified: Unauthorized File Change

Slide 37

Slide 37 text

Unmanaged File Change 107000 ^file unmanaged Verified: Unknown File Change no_email_alert

Slide 38

Slide 38 text

OSSEC FIM Results 96% Reduction in Alerting

Slide 39

Slide 39 text

Pitfalls and Caveats ‣ Who controls inputs? ‣ How resource intensive are your checks? ‣ What if 1,000,000 fire simultaneously? ‣ On the same server? ‣ Think, test, then get some to try to break it.

Slide 40

Slide 40 text

CVE-2014-5284 ‣ host-deny.sh created files in /tmp ‣ cp /tmp/hosts.$$.deny /etc/hosts.deny ‣ Didn’t properly manage permissions ‣ Would copy, as root, the contents of that file to /etc ‣ Moved from /tmp to /var/ossec and added randomness to file name

Slide 41

Slide 41 text

Find Me GitHub: https://github.com/reyjrar/ Twitter: @reyjrar Blogging: http://edgeofsanity.net Email: [email protected]