Upgrade to Pro — share decks privately, control downloads, hide ads and more …

monitorama 2014 - audit all the things

Jen Andre
May 07, 2014
540

monitorama 2014 - audit all the things

Jen Andre

May 07, 2014
Tweet

Transcript

  1. AUDIT ALL THE
    THINGS
    Jen Andre

    View full-size slide



  2. founder, programmer


    @threatstack
    github.com/jandre @fun_cuddles

    View full-size slide

  3. a simple question

    View full-size slide

  4. who is logging into my (machines|applications|SaaS accounts)
    !
    what are they are running
    !
    of running apps, what are making network activity, and where
    !
    every kernel module loaded
    every library
    every file created
    everything!!!!
    but why stop there?

    View full-size slide

  5. for the love of…
    why??

    View full-size slide

  6. a reason to be this
    paranoid?

    View full-size slide

  7. !
    prevention fails

    View full-size slide

  8. should you care?
    X ?

    View full-size slide

  9. “we found no evidence
    that any customer data was
    accessed, changed or lost”

    View full-size slide

  10. “we found no evidence
    that any customer data was
    accessed, changed or lost”

    View full-size slide

  11. “we’re in the cloud!”

    View full-size slide

  12. “we’re in the cloud!”

    View full-size slide

  13. continuous security
    monitoring
    auditing + analytics +
    automation

    View full-size slide

  14. authentications
    process activity
    network activity
    kernel modules
    file system
    apps
    intrusion
    detection
    !
    “active
    defense”
    !
    rapid incident
    response
    systems
    services
    authentications
    db requests
    http requests

    AWS api calls
    SaaS api calls
    }
    e.g.

    View full-size slide

  15. use the host, luke

    View full-size slide

  16. process auditing linux audit
    network flow libnetfilter_conntrack
    login wtmp/audit/pam_loginuid
    one ‘big data’ db to rule them all
    script codes
    hopes and dreams?
    toolbox

    View full-size slide

  17. pros!
    !
    super powerful
    built into your kernel (>=2.3)
    “relatively” low overhead
    apt-get install audit

    View full-size slide

  18. it audits all of the things!

    View full-size slide

  19. userland audit
    daemon and tools
    !
    (e.g. redhat auditd,
    auditctl, etc)
    kernel
    thread
    queue
    kernel threads
    doing things
    audit messages
    the workings
    netlink socket
    /var/log/
    audit/
    audit.log

    View full-size slide

  20. # files
    !
    -w /etc/shadow -p wa
    !
    # syscalls
    !
    -a always,exit -F arch=ARCH -S init_module -S
    delete_module -k modules
    !
    # follow executable
    !
    -w /sbin/insmod -p x
    configuration

    View full-size slide

  21. now for the cons…

    View full-size slide

  22. type=SYSCALL msg=audit(1383252540.673:8711406):
    arch=c000003e syscall=59 success=yes exit=0 a0=c27fa8
    a1=c24d48 a2=9f8008 a3=7fffc4553ce0 items=2 ppid=46247
    pid=56107 auid=0 uid=0 gid=0 eu
    type=EXECVE msg=audit(1383252540.673:8711406): argc=2
    a0="ping" a1="google.com"
    type=CWD msg=audit(1383252540.673:8711406): cwd="/opt/"
    !
    type=PATH msg=audit(1383252540.673:8711406): item=0
    name="/bin/ping" inode=1048904 dev=08:01 mode=0104755
    ouid=0 ogid=0 rdev=00:00
    !
    type=PATH msg=audit(1383252540.673:8711406): item=1
    name=(null) inode=1056827 dev=08:01 mode=0100755 ouid=0
    ogid=0 rdev=00:00
    obtuse logging
    execve(‘ping google.com’)

    View full-size slide

  23. THIS ONE WEIRD TRICK!
    !
    enable rate limiting or it could ‘crash’ your
    box
    !
    always be listening (or same)
    !
    …relatively stable otherwise ;)
    auditctl -b 1000 -r 15000 # 1000 buffers, 15000 eps max

    View full-size slide

  24. redhat auditd, the userland
    daemon, occasionally wtf-y

    View full-size slide

  25. alternative:!
    connect directly to netlink socket
    and write your own audit listener

    View full-size slide

  26. [
    {
    "exe": "/bin/cat",
    "comm": "cat",
    "ses": 10,
    "fsgid": 0,
    "sgid": 0,
    "egid": 0,
    "fsuid": 0,
    "suid": 0,
    "euid": 0,
    "gid": 0,
    "uid": 0,
    "auid": 4294967295,
    "pid": 31335,
    "ppid": 31334,
    "items": 2,
    "a3": "7fff3480e600",
    "a2": "654c88",
    "a1": "654bc0",
    "a0": "654dc0",
    "exit": 0,
    "success": "yes",
    "syscall": "execve",
    "arch": "c000003e",
    "milli": 99,
    "epoch": 1399248110,
    "serial": 855516,
    "type": "SYSCALL"
    },
    {
    "a1": "eth0.dhclient",
    "a0": "cat",
    "argc": 2,
    "milli": 99,
    "epoch": 1399248110,
    "serial": 855516,
    "type": "EXECVE"
    },
    {
    "cwd": "/run/resolvconf/interface",
    "milli": 99,
    "epoch": 1399248110,
    "serial": 855516,
    "type": "CWD"
    },
    {
    "ogid": 0,
    "name": "/bin/cat",
    "milli": 99,
    "epoch": 1399248110,
    "serial": 855516,
    "type": "PATH"
    },
    json output! grouped sanely!

    View full-size slide

  27. luajit! for filtering, transformation &
    alerting

    View full-size slide

  28. performance improvements,
    yay!
    !
    libevent + filtering + state machine parsing
    !
    = 120% -> 10% CPU usage with AB 10k connections/sec

    View full-size slide

  29. + authentications

    View full-size slide

  30. # last
    jandre pts/1 dev.threatstack. Sun May 4 11:20 - 01:37
    (14:17)
    jandre pts/0 dev.threatstack. Sun May 4 11:16 still logged
    in
    !
    # in json format
    !
    { type: ‘USER_PROCESS',
    pid: 777,
    line: ‘pts/1',
    id: 52,
    user: 'jandre',
    host: ‘dev.threatstack',
    exit_status: { termination: 0, code: 0 },
    timestamp: Tue May 06 2014 03:50:03 GMT-0700 (PDT),
    address: ’10.0.0.10’ }
    }
    !
    !
    wtmp

    View full-size slide

  31. # if pam is built with audit support…
    !
    type=USER_AUTH msg=audit(1234877011.791:7731): user
    pid=26127 uid=0 1
    auid=4294967295 ses=4294967295
    msg='op=PAM:authentication acct="root" exe="/usr/sbin/
    sshd"
    (hostname=jupiter.example.com, addr=192.168.2.100,
    terminal=ssh res=success)'
    plus audit

    View full-size slide

  32. !
    # /etc/pam.d/login, sshd, wherever you care
    !
    /etc/pam.d$ grep loginuid *
    !
    login:session required pam_loginuid.so
    sshd:session required pam_loginuid.so
    !
    # now you get in /var/log/audit/audit.log:
    !
    type=LOGIN msg=audit(1234877011.799:7734): login pid=26125
    uid=0
    old auid=4294967295 new auid=0 old ses=4294967295 new ses=1172
    !
    # cat /proc//sessionid
    !
    1172
    add pam_loginuid

    View full-size slide

  33. if you do it right!

    View full-size slide

  34. +network flows

    View full-size slide

  35. src=192.168.254.130 dst=192.168.254.2 sport=60710
    dport=53 packets=1 bytes=56 src=192.168.254.2
    dst=192.168.254.130 sport=53 dport=60710 packets=1
    bytes=248 duration=60
    src/dst ips
    src/dst ports or icmp type
    protocol
    duration
    size

    View full-size slide

  36. netfilter conntrack

    View full-size slide

  37. apt-get install conntrack
    !
    # if byte tracking is not happening
    sysctl -w net.netfilter.nf_conntrack_acct=1
    gettin’ it

    View full-size slide

  38. # conntrack -E
    !
    [NEW] udp 17 30 src=192.168.254.130 dst=192.168.254.2 sport=57906 dport=53
    [UNREPLIED] src=192.168.254.2 dst=192.168.254.130 sport=53 dport=57906
    [UPDATE] udp 17 29 src=192.168.254.130 dst=192.168.254.2 sport=57906 dport=53
    src=192.168.254.2 dst=192.168.254.130 sport=53 dport=57906
    [NEW] udp 17 30 src=192.168.254.130 dst=192.168.254.2 sport=60057 dport=53
    [UNREPLIED] src=192.168.254.2 dst=192.168.254.130 sport=53 dport=60057
    [UPDATE] udp 17 30 src=192.168.254.130 dst=192.168.254.2 sport=60057 dport=53
    src=192.168.254.2 dst=192.168.254.130 sport=53 dport=60057
    [NEW] tcp 6 120 SYN_SENT src=192.168.254.130 dst=74.125.224.229 sport=41293
    dport=80 [UNREPLIED] src=74.125.224.229 dst=192.168.254.130 sport=80 dport=41293
    [UPDATE] tcp 6 60 SYN_RECV src=192.168.254.130 dst=74.125.224.229 sport=41293
    dport=80 src=74.125.224.229 dst=192.168.254.130 sport=80 dport=41293
    [UPDATE] tcp 6 432000 ESTABLISHED src=192.168.254.130 dst=74.125.224.229 sport=41293
    dport=80 src=74.125.224.229 dst=192.168.254.130 sport=80 dport=41293 [ASSURED]
    [UPDATE] tcp 6 120 FIN_WAIT src=192.168.254.130 dst=74.125.224.229 sport=41293
    dport=80 src=74.125.224.229 dst=192.168.254.130 sport=80 dport=41293 [ASSURED]
    [UPDATE] tcp 6 60 CLOSE_WAIT src=192.168.254.130 dst=74.125.224.229 sport=41293
    dport=80 src=74.125.224.229 dst=192.168.254.130 sport=80 dport=41293 [ASSURED]
    [UPDATE] tcp 6 30 LAST_ACK src=192.168.254.130 dst=74.125.224.229 sport=41293
    dport=80 src=74.125.224.229 dst=192.168.254.130 sport=80 dport=41293 [ASSURED]
    [UPDATE] tcp 6 120 TIME_WAIT src=192.168.254.130 dst=74.125.224.229 sport=41293
    dport=80 src=74.125.224.229 dst=192.168.254.130 sport=80 dport=41293 [ASSURED]
    [DESTROY] udp 17 src=192.168.254.130 dst=192.168.254.2 sport=60710 dport=53 packets=1
    bytes=56 src=192.168.254.2 dst=192.168.254.130 sport=53 dport=60710 packets=1 bytes=248
    “realtime”

    View full-size slide

  39. kind of gross

    View full-size slide

  40. // `curl google.com` emits this:
    !
    {
    id: 1018103008,
    start: 1399236274,
    end: 1399236275,
    duration: 1,
    protocol: 'tcp',
    byte_count: 1195,
    packet_count: 11,
    src_ip_numeric: 3232300674,
    dst_ip_numeric: 1127355157,
    src_ip: '192.168.254.130',
    dst_ip: '67.50.19.21',
    src_port: 37814,
    dst_port: 80
    }
    build a interface with
    libnetfilter_conntrack :)

    View full-size slide

  41. {
    id: 1018103008,
    start: 1399236274,
    end: 1399236275,
    duration: 1,
    protocol: 'tcp',
    byte_count: 1195,
    packet_count: 11,
    src_ip_numeric: 3232300674,
    dst_ip_numeric: 1127355157,
    src_ip: '192.168.254.130',
    dst_ip: ’67.50.19.21’,
    src_domain: ‘localhost’,
    dst_domain: ‘google.com’,
    src_port: 37814,
    dst_port: 80,
    exe: ‘/usr/bin/curl’,
    pid: 1004,
    uid: 10001,
    user: ‘jandre’,
    syscall: ‘sys_connect’
    session: 1176
    }
    tie it back to your
    audit system calls

    View full-size slide

  42. putting it all together

    View full-size slide

  43. “I know I said I wouldn’t talk
    about detection, but I am”
    bonus

    View full-size slide

  44. “is that guy running commands he shouldn’t be?”
    (e.g. why is anyone except chef user MAYBE
    running gcc on a prod system)
    !
    “are accounts logging in from non-standard
    locations?”
    !
    “are there anomalies in my traffic?”
    !
    “did some process suddenly start making
    outbound connections?”
    go and detect things

    View full-size slide

  45. now go and audit

    View full-size slide