monitorama 2014 - audit all the things

96245b31898ba441a053d1a6c0422738?s=47 Jen Andre
May 07, 2014
410

monitorama 2014 - audit all the things

96245b31898ba441a053d1a6c0422738?s=128

Jen Andre

May 07, 2014
Tweet

Transcript

  1. AUDIT ALL THE THINGS Jen Andre

  2. founder, programmer @threatstack github.com/jandre @fun_cuddles

  3. a simple question

  4. None
  5. who is logging into my (machines|applications|SaaS accounts) ! what are

    they are running ! of running apps, what are making network activity, and where ! every kernel module loaded every library every file created everything!!!! but why stop there?
  6. for the love of… why??

  7. None
  8. a reason to be this paranoid?

  9. ! prevention fails

  10. should you care? X ?

  11. “we found no evidence that any customer data was accessed,

    changed or lost”
  12. “we found no evidence that any customer data was accessed,

    changed or lost”
  13. X

  14. “we’re in the cloud!”

  15. “we’re in the cloud!”

  16. None
  17. None
  18. continuous security monitoring auditing + analytics + automation

  19. authentications process activity network activity kernel modules file system apps

    intrusion detection ! “active defense” ! rapid incident response systems services authentications db requests http requests
 AWS api calls SaaS api calls } e.g.
  20. how?

  21. use the host, luke

  22. process auditing linux audit network flow libnetfilter_conntrack login wtmp/audit/pam_loginuid one

    ‘big data’ db to rule them all script codes hopes and dreams? toolbox
  23. linux audit

  24. pros! ! super powerful built into your kernel (>=2.3) “relatively”

    low overhead apt-get install audit
  25. it audits all of the things!

  26. (sort of)

  27. userland audit daemon and tools ! (e.g. redhat auditd, auditctl,

    etc) kernel thread queue kernel threads doing things audit messages the workings netlink socket /var/log/ audit/ audit.log
  28. # files ! -w /etc/shadow -p wa ! # syscalls

    ! -a always,exit -F arch=ARCH -S init_module -S delete_module -k modules ! # follow executable ! -w /sbin/insmod -p x configuration
  29. now for the cons…

  30. type=SYSCALL msg=audit(1383252540.673:8711406): arch=c000003e syscall=59 success=yes exit=0 a0=c27fa8 a1=c24d48 a2=9f8008 a3=7fffc4553ce0

    items=2 ppid=46247 pid=56107 auid=0 uid=0 gid=0 eu type=EXECVE msg=audit(1383252540.673:8711406): argc=2 a0="ping" a1="google.com" type=CWD msg=audit(1383252540.673:8711406): cwd="/opt/" ! type=PATH msg=audit(1383252540.673:8711406): item=0 name="/bin/ping" inode=1048904 dev=08:01 mode=0104755 ouid=0 ogid=0 rdev=00:00 ! type=PATH msg=audit(1383252540.673:8711406): item=1 name=(null) inode=1056827 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 obtuse logging execve(‘ping google.com’)
  31. THIS ONE WEIRD TRICK! ! enable rate limiting or it

    could ‘crash’ your box ! always be listening (or same) ! …relatively stable otherwise ;) auditctl -b 1000 -r 15000 # 1000 buffers, 15000 eps max
  32. redhat auditd, the userland daemon, occasionally wtf-y

  33. yeah…

  34. alternative:! connect directly to netlink socket and write your own

    audit listener
  35. [ { "exe": "/bin/cat", "comm": "cat", "ses": 10, "fsgid": 0,

    "sgid": 0, "egid": 0, "fsuid": 0, "suid": 0, "euid": 0, "gid": 0, "uid": 0, "auid": 4294967295, "pid": 31335, "ppid": 31334, "items": 2, "a3": "7fff3480e600", "a2": "654c88", "a1": "654bc0", "a0": "654dc0", "exit": 0, "success": "yes", "syscall": "execve", "arch": "c000003e", "milli": 99, "epoch": 1399248110, "serial": 855516, "type": "SYSCALL" }, { "a1": "eth0.dhclient", "a0": "cat", "argc": 2, "milli": 99, "epoch": 1399248110, "serial": 855516, "type": "EXECVE" }, { "cwd": "/run/resolvconf/interface", "milli": 99, "epoch": 1399248110, "serial": 855516, "type": "CWD" }, { "ogid": 0, "name": "/bin/cat", "milli": 99, "epoch": 1399248110, "serial": 855516, "type": "PATH" }, json output! grouped sanely!
  36. luajit! for filtering, transformation & alerting

  37. performance improvements, yay! ! libevent + filtering + state machine

    parsing ! = 120% -> 10% CPU usage with AB 10k connections/sec
  38. + authentications

  39. # last jandre pts/1 dev.threatstack. Sun May 4 11:20 -

    01:37 (14:17) jandre pts/0 dev.threatstack. Sun May 4 11:16 still logged in ! # in json format ! { type: ‘USER_PROCESS', pid: 777, line: ‘pts/1', id: 52, user: 'jandre', host: ‘dev.threatstack', exit_status: { termination: 0, code: 0 }, timestamp: Tue May 06 2014 03:50:03 GMT-0700 (PDT), address: ’10.0.0.10’ } } ! ! wtmp
  40. # if pam is built with audit support… ! type=USER_AUTH

    msg=audit(1234877011.791:7731): user pid=26127 uid=0 1 auid=4294967295 ses=4294967295 msg='op=PAM:authentication acct="root" exe="/usr/sbin/ sshd" (hostname=jupiter.example.com, addr=192.168.2.100, terminal=ssh res=success)' plus audit
  41. ! # /etc/pam.d/login, sshd, wherever you care ! /etc/pam.d$ grep

    loginuid * ! login:session required pam_loginuid.so sshd:session required pam_loginuid.so ! # now you get in /var/log/audit/audit.log: ! type=LOGIN msg=audit(1234877011.799:7734): login pid=26125 uid=0 old auid=4294967295 new auid=0 old ses=4294967295 new ses=1172 ! # cat /proc/<login or sshd pid>/sessionid ! 1172 add pam_loginuid
  42. if you do it right!

  43. +network flows

  44. src=192.168.254.130 dst=192.168.254.2 sport=60710 dport=53 packets=1 bytes=56 src=192.168.254.2 dst=192.168.254.130 sport=53 dport=60710

    packets=1 bytes=248 duration=60 src/dst ips src/dst ports or icmp type protocol duration size
  45. netfilter conntrack

  46. apt-get install conntrack ! # if byte tracking is not

    happening sysctl -w net.netfilter.nf_conntrack_acct=1 gettin’ it
  47. # conntrack -E ! [NEW] udp 17 30 src=192.168.254.130 dst=192.168.254.2

    sport=57906 dport=53 [UNREPLIED] src=192.168.254.2 dst=192.168.254.130 sport=53 dport=57906 [UPDATE] udp 17 29 src=192.168.254.130 dst=192.168.254.2 sport=57906 dport=53 src=192.168.254.2 dst=192.168.254.130 sport=53 dport=57906 [NEW] udp 17 30 src=192.168.254.130 dst=192.168.254.2 sport=60057 dport=53 [UNREPLIED] src=192.168.254.2 dst=192.168.254.130 sport=53 dport=60057 [UPDATE] udp 17 30 src=192.168.254.130 dst=192.168.254.2 sport=60057 dport=53 src=192.168.254.2 dst=192.168.254.130 sport=53 dport=60057 [NEW] tcp 6 120 SYN_SENT src=192.168.254.130 dst=74.125.224.229 sport=41293 dport=80 [UNREPLIED] src=74.125.224.229 dst=192.168.254.130 sport=80 dport=41293 [UPDATE] tcp 6 60 SYN_RECV src=192.168.254.130 dst=74.125.224.229 sport=41293 dport=80 src=74.125.224.229 dst=192.168.254.130 sport=80 dport=41293 [UPDATE] tcp 6 432000 ESTABLISHED src=192.168.254.130 dst=74.125.224.229 sport=41293 dport=80 src=74.125.224.229 dst=192.168.254.130 sport=80 dport=41293 [ASSURED] [UPDATE] tcp 6 120 FIN_WAIT src=192.168.254.130 dst=74.125.224.229 sport=41293 dport=80 src=74.125.224.229 dst=192.168.254.130 sport=80 dport=41293 [ASSURED] [UPDATE] tcp 6 60 CLOSE_WAIT src=192.168.254.130 dst=74.125.224.229 sport=41293 dport=80 src=74.125.224.229 dst=192.168.254.130 sport=80 dport=41293 [ASSURED] [UPDATE] tcp 6 30 LAST_ACK src=192.168.254.130 dst=74.125.224.229 sport=41293 dport=80 src=74.125.224.229 dst=192.168.254.130 sport=80 dport=41293 [ASSURED] [UPDATE] tcp 6 120 TIME_WAIT src=192.168.254.130 dst=74.125.224.229 sport=41293 dport=80 src=74.125.224.229 dst=192.168.254.130 sport=80 dport=41293 [ASSURED] [DESTROY] udp 17 src=192.168.254.130 dst=192.168.254.2 sport=60710 dport=53 packets=1 bytes=56 src=192.168.254.2 dst=192.168.254.130 sport=53 dport=60710 packets=1 bytes=248 “realtime”
  48. kind of gross

  49. // `curl google.com` emits this: ! { id: 1018103008, start:

    1399236274, end: 1399236275, duration: 1, protocol: 'tcp', byte_count: 1195, packet_count: 11, src_ip_numeric: 3232300674, dst_ip_numeric: 1127355157, src_ip: '192.168.254.130', dst_ip: '67.50.19.21', src_port: 37814, dst_port: 80 } build a interface with libnetfilter_conntrack :)
  50. { id: 1018103008, start: 1399236274, end: 1399236275, duration: 1, protocol:

    'tcp', byte_count: 1195, packet_count: 11, src_ip_numeric: 3232300674, dst_ip_numeric: 1127355157, src_ip: '192.168.254.130', dst_ip: ’67.50.19.21’, src_domain: ‘localhost’, dst_domain: ‘google.com’, src_port: 37814, dst_port: 80, exe: ‘/usr/bin/curl’, pid: 1004, uid: 10001, user: ‘jandre’, syscall: ‘sys_connect’ session: 1176 } tie it back to your audit system calls
  51. putting it all together

  52. “I know I said I wouldn’t talk about detection, but

    I am” bonus
  53. “is that guy running commands he shouldn’t be?” (e.g. why

    is anyone except chef user MAYBE running gcc on a prod system) ! “are accounts logging in from non-standard locations?” ! “are there anomalies in my traffic?” ! “did some process suddenly start making outbound connections?” go and detect things
  54. now go and audit