Bro Deployment Verification and Troubleshooting

A few tips and tricks to help you verify that Bro is working correctly, pinpoint where any issues might be, and how to fix them.

Transcript

1. Verifying and Troubleshooting your Bro Deployment Vlad Grigorescu Bro4Pros 2015 1
2. What Needs to Work? 1. File and protocol analyzers 2. Log generation 3. Notice actions 4. Integrations with other tools 2
3. Analyzers: Requirements 1. All the traffic is being mirrored 2. A single worker is seeing both sides of the same connection 3. The analyzer is being enabled, and is able to parse the traffic 3
4. Analyzers: Requirements 1. All the traffic is being mirrored 2. A single worker is seeing both sides of the same connection 3. The analyzer is being enabled, and is able to parse the traffic 4
5. Mirroring Traffic: Verification • Compare to another source (flow records, firewall logs, etc.) • Generate a PCAP and read it with standalone Bro (bro -r test.pcap) • trace-summary.py 5
6. Mirroring Traffic: Troubleshooting • SPANs, TAPs, mirror ports installed and configured correctly • Devices that block or modify traffic have been accounted for • All traffic paths are mirrored • Duplication is removed 6
7. Analyzers: Requirements 1. All the traffic is being mirrored 2. A single worker is seeing both sides of the same connection 3. The analyzer is being enabled, and is able to parse the traffic 7
8. Distributing Traffic: Verification • Conn log • Protocol logs • Capture loss • Weirds 8
9. Conn Log Verification: I • "Healthy" TCP history: ShA... • S - originator sent SYN • h - responder sent SYN+ACK • A - originator sent ACK
 9
10. Conn Log Verification: I • "Healthy" TCP history: ShA... • S - originator sent SYN • h - responder sent SYN+ACK • A - originator sent ACK • "Unhealthy" TCP history: SAD... • S - originator sent SYN • A - originator sent ACK • D - originator sent data
 10
11. Conn Log Verification: I • "Healthy" TCP history: ShA... • S - originator sent SYN • h - responder sent SYN+ACK • A - originator sent ACK • "Unhealthy" TCP history: SAD... • S - originator sent SYN • A - originator sent ACK • D - originator sent data • Another "unhealthy" TCP option: had... • h - responder sent SYN+ACK • a - responder sent ACK • d - responder sent data
 11
12. Conn Log Verification: II • conn-node-name.bro: 8.8.8.8 999 9.9.9.9 25 SAD worker3-1 9.9.9.9 25 8.8.8.8 999 had worker3-3 12
13. Conn Log Verification: II • conn-node-name.bro: 8.8.8.8 999 9.9.9.9 25 SAD worker2-1 9.9.9.9 25 8.8.8.8 999 had worker3-1 13
14. Protocol Logs: http.log: 1424181556.478007 CfXO9KW7w3sk 3.8.2.2 3821 3.9.7.1 80 0 - - - - - 0 0 302 Moved Temporarily - - - (empty) - - - - - - - - - - - - - 14
15. Capture Loss policy/misc/capture-loss.bro 15
16. Weirds • Missing or poorly balanced traffic: • data_before_established • possible_split_routing • unmatched_HTTP_reply • dns_unmatched_reply • Traffic mangling: • SYN_seq_jump • TCP_seq_underflow_or_misorder • active_connection_reuse 16
17. Distributing Traffic: Verification • Conn log (history field) • Protocol logs • Capture loss • Weirds 17
18. Distributing Traffic: Troubleshooting • Load-balancing hashing. At most IP, port, and protocol. • Don't forget about the NIC hashing! • Are you seeing all the traffic? • Are you getting duplicate traffic? • If loss is too high, you might be overloaded - can you use filters? 18
19. Analyzers: Requirements 1. All the traffic is being mirrored 2. A single worker is seeing both sides of the same connection 3. The analyzer is being enabled, and is able to parse the traffic 19
20. Analyzer Parsing: Verification • Is it being enabled? • Check service in conn.log • trace-summary.py • Is it able to parse the traffic? • policy/frameworks/dpd/packet- segment-logging.bro creates dpd.log: ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto analyzer failure_reason packet_segment 20
21. # Analyzer Failure Reason 93575 HTTP not a http reply line 15043 HTTP not a http request line 14915 SSL Invalid version late in TLS connection. Packet reported version: $version 7569 SSL Invalid headers in SSL connection. $headers 6698 SSL Invalid version in TLS connection. Version: $version 693 DNS DNS_Conn_count_too_large 27 SSL Invalid version in SSL client hello. Version: $version 16 SMTP reply code -1 out of range [The SMTP... 5 SSH malformed ssh identification [QUIT] 4 DHCP no DHCP message type option 3 SMTP reply code -1 out of range 2 SMTP reply code -1 out of range [5.7.2 User...
22. "not a http reply line" ^H\0^?^K\0^P\0Pa0f\xb1\x80^X^N$\xad \x80\0\0^A^A^H^Jp\x9e4\x87\x99\x8d \xde^AExpires: Mon, 02 Aug 1999 00:00:00 GMT^M^JLast-Modified: Tue, 17 Feb 2015 21:59:46 GMT^M^JCache-Control: no-store, no- cache, must-revalidate^M^JCache-Control: post-check=0, pre-check=0^M^JPragma: no- cache^M^J 22
23. dpd_to_pcap.py • Gist: bit.ly/dpd_to_pcap • grep $UID dpd.log | dpd_to_pcap.py | tcpdump -vvvlAnr - 23
24. Log Generation • Look for health monitoring, other periodic activity • Falling behind? Every 5 minutes, log: current_time() - network_time() 24
25. Notice Actions • Custom notice action, scheduled every X minutes - passive service check for your monitoring system 25
26. Integrations with Other Tools 26 0 125 250 375 500 2013-10-01 2013-11-01 2013-12-02 2013-01-02 2013-02-02 2013-03-05 2013-04-05 2013-05-06 2013-06-06 2013-07-07 2013-08-07 2013-09-07 2014-10-08 Alerts