monitorama 2014 - audit all the things

Transcript

1. AUDIT ALL THE THINGS Jen Andre
2. founder, programmer @threatstack github.com/jandre @fun_cuddles
3. a simple question
5. who is logging into my (machines|applications|SaaS accounts) ! what are they are running ! of running apps, what are making network activity, and where ! every kernel module loaded every library every file created everything!!!! but why stop there?
6. for the love of… why??
8. a reason to be this paranoid?
9. ! prevention fails
10. should you care? X ?
11. “we found no evidence that any customer data was accessed, changed or lost”
12. “we found no evidence that any customer data was accessed, changed or lost”
13. X
14. “we’re in the cloud!”
15. “we’re in the cloud!”
18. continuous security monitoring auditing + analytics + automation
19. authentications process activity network activity kernel modules file system apps intrusion detection ! “active defense” ! rapid incident response systems services authentications db requests http requests
 AWS api calls SaaS api calls } e.g.
20. how?
21. use the host, luke
22. process auditing linux audit network flow libnetfilter_conntrack login wtmp/audit/pam_loginuid one ‘big data’ db to rule them all script codes hopes and dreams? toolbox
23. linux audit
24. pros! ! super powerful built into your kernel (>=2.3) “relatively” low overhead apt-get install audit
25. it audits all of the things!
26. (sort of)
27. userland audit daemon and tools ! (e.g. redhat auditd, auditctl, etc) kernel thread queue kernel threads doing things audit messages the workings netlink socket /var/log/ audit/ audit.log
28. # files ! -w /etc/shadow -p wa ! # syscalls ! -a always,exit -F arch=ARCH -S init_module -S delete_module -k modules ! # follow executable ! -w /sbin/insmod -p x configuration
29. now for the cons…
30. type=SYSCALL msg=audit(1383252540.673:8711406): arch=c000003e syscall=59 success=yes exit=0 a0=c27fa8 a1=c24d48 a2=9f8008 a3=7fffc4553ce0 items=2 ppid=46247 pid=56107 auid=0 uid=0 gid=0 eu type=EXECVE msg=audit(1383252540.673:8711406): argc=2 a0="ping" a1="google.com" type=CWD msg=audit(1383252540.673:8711406): cwd="/opt/" ! type=PATH msg=audit(1383252540.673:8711406): item=0 name="/bin/ping" inode=1048904 dev=08:01 mode=0104755 ouid=0 ogid=0 rdev=00:00 ! type=PATH msg=audit(1383252540.673:8711406): item=1 name=(null) inode=1056827 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 obtuse logging execve(‘ping google.com’)
31. THIS ONE WEIRD TRICK! ! enable rate limiting or it could ‘crash’ your box ! always be listening (or same) ! …relatively stable otherwise ;) auditctl -b 1000 -r 15000 # 1000 buffers, 15000 eps max
32. redhat auditd, the userland daemon, occasionally wtf-y
33. yeah…
34. alternative:! connect directly to netlink socket and write your own audit listener
35. [ { "exe": "/bin/cat", "comm": "cat", "ses": 10, "fsgid": 0, "sgid": 0, "egid": 0, "fsuid": 0, "suid": 0, "euid": 0, "gid": 0, "uid": 0, "auid": 4294967295, "pid": 31335, "ppid": 31334, "items": 2, "a3": "7fff3480e600", "a2": "654c88", "a1": "654bc0", "a0": "654dc0", "exit": 0, "success": "yes", "syscall": "execve", "arch": "c000003e", "milli": 99, "epoch": 1399248110, "serial": 855516, "type": "SYSCALL" }, { "a1": "eth0.dhclient", "a0": "cat", "argc": 2, "milli": 99, "epoch": 1399248110, "serial": 855516, "type": "EXECVE" }, { "cwd": "/run/resolvconf/interface", "milli": 99, "epoch": 1399248110, "serial": 855516, "type": "CWD" }, { "ogid": 0, "name": "/bin/cat", "milli": 99, "epoch": 1399248110, "serial": 855516, "type": "PATH" }, json output! grouped sanely!
36. luajit! for filtering, transformation & alerting
37. performance improvements, yay! ! libevent + filtering + state machine parsing ! = 120% -> 10% CPU usage with AB 10k connections/sec
38. + authentications
39. # last jandre pts/1 dev.threatstack. Sun May 4 11:20 - 01:37 (14:17) jandre pts/0 dev.threatstack. Sun May 4 11:16 still logged in ! # in json format ! { type: ‘USER_PROCESS', pid: 777, line: ‘pts/1', id: 52, user: 'jandre', host: ‘dev.threatstack', exit_status: { termination: 0, code: 0 }, timestamp: Tue May 06 2014 03:50:03 GMT-0700 (PDT), address: ’10.0.0.10’ } } ! ! wtmp
40. # if pam is built with audit support… ! type=USER_AUTH msg=audit(1234877011.791:7731): user pid=26127 uid=0 1 auid=4294967295 ses=4294967295 msg='op=PAM:authentication acct="root" exe="/usr/sbin/ sshd" (hostname=jupiter.example.com, addr=192.168.2.100, terminal=ssh res=success)' plus audit
41. ! # /etc/pam.d/login, sshd, wherever you care ! /etc/pam.d$ grep loginuid * ! login:session required pam_loginuid.so sshd:session required pam_loginuid.so ! # now you get in /var/log/audit/audit.log: ! type=LOGIN msg=audit(1234877011.799:7734): login pid=26125 uid=0 old auid=4294967295 new auid=0 old ses=4294967295 new ses=1172 ! # cat /proc/<login or sshd pid>/sessionid ! 1172 add pam_loginuid
42. if you do it right!
43. +network flows
44. src=192.168.254.130 dst=192.168.254.2 sport=60710 dport=53 packets=1 bytes=56 src=192.168.254.2 dst=192.168.254.130 sport=53 dport=60710 packets=1 bytes=248 duration=60 src/dst ips src/dst ports or icmp type protocol duration size
45. netfilter conntrack
46. apt-get install conntrack ! # if byte tracking is not happening sysctl -w net.netfilter.nf_conntrack_acct=1 gettin’ it
47. # conntrack -E ! [NEW] udp 17 30 src=192.168.254.130 dst=192.168.254.2 sport=57906 dport=53 [UNREPLIED] src=192.168.254.2 dst=192.168.254.130 sport=53 dport=57906 [UPDATE] udp 17 29 src=192.168.254.130 dst=192.168.254.2 sport=57906 dport=53 src=192.168.254.2 dst=192.168.254.130 sport=53 dport=57906 [NEW] udp 17 30 src=192.168.254.130 dst=192.168.254.2 sport=60057 dport=53 [UNREPLIED] src=192.168.254.2 dst=192.168.254.130 sport=53 dport=60057 [UPDATE] udp 17 30 src=192.168.254.130 dst=192.168.254.2 sport=60057 dport=53 src=192.168.254.2 dst=192.168.254.130 sport=53 dport=60057 [NEW] tcp 6 120 SYN_SENT src=192.168.254.130 dst=74.125.224.229 sport=41293 dport=80 [UNREPLIED] src=74.125.224.229 dst=192.168.254.130 sport=80 dport=41293 [UPDATE] tcp 6 60 SYN_RECV src=192.168.254.130 dst=74.125.224.229 sport=41293 dport=80 src=74.125.224.229 dst=192.168.254.130 sport=80 dport=41293 [UPDATE] tcp 6 432000 ESTABLISHED src=192.168.254.130 dst=74.125.224.229 sport=41293 dport=80 src=74.125.224.229 dst=192.168.254.130 sport=80 dport=41293 [ASSURED] [UPDATE] tcp 6 120 FIN_WAIT src=192.168.254.130 dst=74.125.224.229 sport=41293 dport=80 src=74.125.224.229 dst=192.168.254.130 sport=80 dport=41293 [ASSURED] [UPDATE] tcp 6 60 CLOSE_WAIT src=192.168.254.130 dst=74.125.224.229 sport=41293 dport=80 src=74.125.224.229 dst=192.168.254.130 sport=80 dport=41293 [ASSURED] [UPDATE] tcp 6 30 LAST_ACK src=192.168.254.130 dst=74.125.224.229 sport=41293 dport=80 src=74.125.224.229 dst=192.168.254.130 sport=80 dport=41293 [ASSURED] [UPDATE] tcp 6 120 TIME_WAIT src=192.168.254.130 dst=74.125.224.229 sport=41293 dport=80 src=74.125.224.229 dst=192.168.254.130 sport=80 dport=41293 [ASSURED] [DESTROY] udp 17 src=192.168.254.130 dst=192.168.254.2 sport=60710 dport=53 packets=1 bytes=56 src=192.168.254.2 dst=192.168.254.130 sport=53 dport=60710 packets=1 bytes=248 “realtime”
48. kind of gross
49. // `curl google.com` emits this: ! { id: 1018103008, start: 1399236274, end: 1399236275, duration: 1, protocol: 'tcp', byte_count: 1195, packet_count: 11, src_ip_numeric: 3232300674, dst_ip_numeric: 1127355157, src_ip: '192.168.254.130', dst_ip: '67.50.19.21', src_port: 37814, dst_port: 80 } build a interface with libnetfilter_conntrack :)
50. { id: 1018103008, start: 1399236274, end: 1399236275, duration: 1, protocol: 'tcp', byte_count: 1195, packet_count: 11, src_ip_numeric: 3232300674, dst_ip_numeric: 1127355157, src_ip: '192.168.254.130', dst_ip: ’67.50.19.21’, src_domain: ‘localhost’, dst_domain: ‘google.com’, src_port: 37814, dst_port: 80, exe: ‘/usr/bin/curl’, pid: 1004, uid: 10001, user: ‘jandre’, syscall: ‘sys_connect’ session: 1176 } tie it back to your audit system calls
51. putting it all together
52. “I know I said I wouldn’t talk about detection, but I am” bonus
53. “is that guy running commands he shouldn’t be?” (e.g. why is anyone except chef user MAYBE running gcc on a prod system) ! “are accounts logging in from non-standard locations?” ! “are there anomalies in my traffic?” ! “did some process suddenly start making outbound connections?” go and detect things
54. now go and audit