Verifying and Troubleshooting your Bro Deployment Vlad Grigorescu Bro4Pros 2015 1 

What Needs to Work? 1. File and protocol analyzers 2. Log generation 3. Notice actions 4. Integrations with other tools 2 

Analyzers: Requirements 1. All the traffic is being mirrored 2. A single worker is seeing both sides of the same connection 3. The analyzer is being enabled, and is able to parse the traffic 3 

Analyzers: Requirements 1. All the traffic is being mirrored 2. A single worker is seeing both sides of the same connection 3. The analyzer is being enabled, and is able to parse the traffic 4 

Mirroring Traffic: Verification • Compare to another source (flow records, firewall logs, etc.) • Generate a PCAP and read it with standalone Bro (bro -r test.pcap) • trace-summary.py 5 

Mirroring Traffic: Troubleshooting • SPANs, TAPs, mirror ports installed and configured correctly • Devices that block or modify traffic have been accounted for • All traffic paths are mirrored • Duplication is removed 6 

Analyzers: Requirements 1. All the traffic is being mirrored 2. A single worker is seeing both sides of the same connection 3. The analyzer is being enabled, and is able to parse the traffic 7 

Distributing Traffic: Verification • Conn log • Protocol logs • Capture loss • Weirds 8 

Conn Log Verification: I • "Healthy" TCP history: ShA... • S - originator sent SYN • h - responder sent SYN+ACK • A - originator sent ACK
 9 

Conn Log Verification: I • "Healthy" TCP history: ShA... • S - originator sent SYN • h - responder sent SYN+ACK • A - originator sent ACK • "Unhealthy" TCP history: SAD... • S - originator sent SYN • A - originator sent ACK • D - originator sent data
 10 

Conn Log Verification: I • "Healthy" TCP history: ShA... • S - originator sent SYN • h - responder sent SYN+ACK • A - originator sent ACK • "Unhealthy" TCP history: SAD... • S - originator sent SYN • A - originator sent ACK • D - originator sent data • Another "unhealthy" TCP option: had... • h - responder sent SYN+ACK • a - responder sent ACK • d - responder sent data
 11 

Conn Log Verification: II • conn-node-name.bro: 8.8.8.8 999 9.9.9.9 25 SAD worker3-1 9.9.9.9 25 8.8.8.8 999 had worker3-3 12 

Conn Log Verification: II • conn-node-name.bro: 8.8.8.8 999 9.9.9.9 25 SAD worker2-1 9.9.9.9 25 8.8.8.8 999 had worker3-1 13 

Protocol Logs: http.log: 1424181556.478007 CfXO9KW7w3sk 3.8.2.2 3821 3.9.7.1 80 0 - - - - - 0 0 302 Moved Temporarily - - - (empty) - - - - - - - - - - - - - 14 

Capture Loss policy/misc/capture-loss.bro 15 

Weirds • Missing or poorly balanced traffic: • data_before_established • possible_split_routing • unmatched_HTTP_reply • dns_unmatched_reply • Traffic mangling: • SYN_seq_jump • TCP_seq_underflow_or_misorder • active_connection_reuse 16 

Distributing Traffic: Verification • Conn log (history field) • Protocol logs • Capture loss • Weirds 17 

Distributing Traffic: Troubleshooting • Load-balancing hashing. At most IP, port, and protocol. • Don't forget about the NIC hashing! • Are you seeing all the traffic? • Are you getting duplicate traffic? • If loss is too high, you might be overloaded - can you use filters? 18 

Analyzers: Requirements 1. All the traffic is being mirrored 2. A single worker is seeing both sides of the same connection 3. The analyzer is being enabled, and is able to parse the traffic 19 

Analyzer Parsing: Verification • Is it being enabled? • Check service in conn.log • trace-summary.py • Is it able to parse the traffic? • policy/frameworks/dpd/packet- segment-logging.bro creates dpd.log: ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto analyzer failure_reason packet_segment 20 

# Analyzer Failure Reason 93575 HTTP not a http reply line 15043 HTTP not a http request line 14915 SSL Invalid version late in TLS connection. Packet reported version: $version 7569 SSL Invalid headers in SSL connection. $headers 6698 SSL Invalid version in TLS connection. Version: $version 693 DNS DNS_Conn_count_too_large 27 SSL Invalid version in SSL client hello. Version: $version 16 SMTP reply code -1 out of range [The SMTP... 5 SSH malformed ssh identification [QUIT] 4 DHCP no DHCP message type option 3 SMTP reply code -1 out of range 2 SMTP reply code -1 out of range [5.7.2 User... 

"not a http reply line" ^H\0^?^K\0^P\0Pa0f\xb1\x80^X^N$\xad \x80\0\0^A^A^H^Jp\x9e4\x87\x99\x8d \xde^AExpires: Mon, 02 Aug 1999 00:00:00 GMT^M^JLast-Modified: Tue, 17 Feb 2015 21:59:46 GMT^M^JCache-Control: no-store, no- cache, must-revalidate^M^JCache-Control: post-check=0, pre-check=0^M^JPragma: no- cache^M^J 22 

dpd_to_pcap.py • Gist: bit.ly/dpd_to_pcap • grep $UID dpd.log | dpd_to_pcap.py | tcpdump -vvvlAnr - 23 

Log Generation • Look for health monitoring, other periodic activity • Falling behind? Every 5 minutes, log: current_time() - network_time() 24 

Notice Actions • Custom notice action, scheduled every X minutes - passive service check for your monitoring system 25 

Integrations with Other Tools 26 0 125 250 375 500 2013-10-01 2013-11-01 2013-12-02 2013-01-02 2013-02-02 2013-03-05 2013-04-05 2013-05-06 2013-06-06 2013-07-07 2013-08-07 2013-09-07 2014-10-08 Alerts