drbdump - Speaker Deck

Slide 1

Slide 1 text

drbdump Eric Hodel - [email protected] Sunday, June 2, 13

Slide 2

Slide 2 text

Links • Gem: gem install drbdump https://rubygems.org/gems/drbdump • Source: https://github.com/drbrain/drbdump Sunday, June 2, 13

Slide 3

Slide 3 text

drbdump • Like tcpdump • Understands messages • Tracks Marshal allocations • Tracks latency Sunday, June 2, 13

Slide 4

Slide 4 text

Problems • Out-of-order TCP • mDNS • resolv.rb, partial patch • No ﬁltering • IPv6 (bug) Sunday, June 2, 13

Slide 5

Slide 5 text

Running • Basic drbdump • Summary of 10,000 messages drbdump -q -n -c 10000 • Listen only on loopback drbdump -i lo0 Sunday, June 2, 13

Slide 6

Slide 6 text

Demo $ cd $(dirname $(gem which drbdump))/.. $ ruby example/... Sunday, June 2, 13

Slide 7

Slide 7 text

dRuby Protocol • Peer to Peer • Message → Result • 1 to ∞ messages per connection • TCP, UNIX, SSL, etc. • drbdump is TCP-only Sunday, June 2, 13

Slide 8

Slide 8 text

Message Chunk • Chunk size • 32-bit little-endian integer • Marshal stream "\x00\x00\x00\x03\x04\x08T" Sunday, June 2, 13

Slide 9

Slide 9 text

Message Send • Receiver • nil or object id • Method name • Argument count • Arguments • Block or nil Sunday, June 2, 13

Slide 10

Slide 10 text

Message Result • Message status • true → success • false → exception • Result or Exception Sunday, June 2, 13

Slide 11

Slide 11 text

drbdump loop Get packet Add to stream Find ﬁrst chunk Load Error Ignore stream Determine type Update statistics Display packet Sunday, June 2, 13

Slide 12

Slide 12 text

Message-send Output 17:34:49.546074 "druby://10.101.28.77:60919" ⇒ ("druby://10.101.28.77:60913", 70180390657240). respond_to?(:update, false) Sunday, June 2, 13

Slide 13

Slide 13 text

Timestamp 17:34:49.546074 "druby://10.101.28.77:60919" ⇒ ("druby://10.101.28.77:60913", 70180390657240). respond_to?(:update, false) Sunday, June 2, 13

Slide 14

Slide 14 text

Source 17:34:49.546074 "druby://10.101.28.77:60919" ⇒ ("druby://10.101.28.77:60913", 70180390657240). respond_to?(:update, false) Sunday, June 2, 13

Slide 15

Slide 15 text

Message-send 17:34:49.546074 "druby://10.101.28.77:60919" ⇒ ("druby://10.101.28.77:60913", 70180390657240). respond_to?(:update, false) Sunday, June 2, 13

Slide 16

Slide 16 text

Receiver 17:34:49.546074 "druby://10.101.28.77:60919" ⇒ ("druby://10.101.28.77:60913", 70180390657240). respond_to?(:update, false) Sunday, June 2, 13

Slide 17

Slide 17 text

Method Name 17:34:49.546074 "druby://10.101.28.77:60919" ⇒ ("druby://10.101.28.77:60913", 70180390657240). respond_to?(:update, false) Sunday, June 2, 13

Slide 18

Slide 18 text

Arguments 17:34:49.546074 "druby://10.101.28.77:60919" ⇒ ("druby://10.101.28.77:60913", 70180390657240). respond_to?(:update, false) Sunday, June 2, 13

Slide 19

Slide 19 text

Debug with irb …⋯ ⇒ ("druby://10.101.28.77:60913", 70180390657240). respond_to?(:update, false) ^C $ irb -r drb >> DRb.start_service >> DRb::DRbObject.new_with( "druby://10.101.28.77:60913", 70180390657240) Sunday, June 2, 13

Slide 20

Slide 20 text

Message-result Output 17:34:49.546531 "druby://10.101.28.77:60919" ⾨ "druby://10.101.28.77:60913" success: true Sunday, June 2, 13

Slide 21

Slide 21 text

Timestamp 17:34:49.546531 "druby://10.101.28.77:60919" ⾨ "druby://10.101.28.77:60913" success: true Sunday, June 2, 13

Slide 22

Slide 22 text

Destination 17:34:49.546531 "druby://10.101.28.77:60919" ⾨ "druby://10.101.28.77:60913" success: true Sunday, June 2, 13

Slide 23

Slide 23 text

Message-result 17:34:49.546531 "druby://10.101.28.77:60919" ⾨ "druby://10.101.28.77:60913" success: true Sunday, June 2, 13

Slide 24

Slide 24 text

Source 17:34:49.546531 "druby://10.101.28.77:60919" ⾨ "druby://10.101.28.77:60913" success: true Sunday, June 2, 13

Slide 25

Slide 25 text

Status 17:34:49.546531 "druby://10.101.28.77:60919" ⾨ "druby://10.101.28.77:60913" success: true Sunday, June 2, 13

Slide 26

Slide 26 text

Result Object 17:34:49.546531 "druby://10.101.28.77:60919" ⾨ "druby://10.101.28.77:60913" success: true Sunday, June 2, 13

Slide 27

Slide 27 text

Exception Result 17:34:49.563440 "druby://10.101.28.77:60929" ⤂ "druby://10.101.28.77:60920" exception: # Sunday, June 2, 13

Slide 28

Slide 28 text

Exception 17:34:49.563440 "druby://10.101.28.77:60929" ⤂ "druby://10.101.28.77:60920" exception: # Sunday, June 2, 13

Slide 29

Slide 29 text

Exception Object 17:34:49.563440 "druby://10.101.28.77:60929" ⤂ "druby://10.101.28.77:60920" exception: # Sunday, June 2, 13

Slide 30

Slide 30 text

Easy to Match "druby://10.101.28.77:60919" ⇒ ("druby://10.101.28.77:60913" "druby://10.101.28.77:60919" ⾨ "druby://10.101.28.77:60913" "druby://10.101.28.77:60929" ⇒ ("druby://10.101.28.77:60920" "druby://10.101.28.77:60929" ⤂ "druby://10.101.28.77:60920" Sunday, June 2, 13

Slide 31

Slide 31 text

Statistics Output • Basic statistics • Packets, messages, results • Message statistics • Allocations, latency • Peer statistics • Latency Sunday, June 2, 13

Slide 32

Slide 32 text

Basic Statistics 1508 total packets captured 0 Rinda packets captured 56 DRb packets captured 30 messages sent 26 results received 2 exceptions raised Sunday, June 2, 13

Slide 33

Slide 33 text

Message Statistics Messages sent min, avg, max, stddev: call (1 args) 10 sent; 3.0, 6.2, 19.0, 6.7 allocations; 0.285, 1.479, 7.258, 2.171 ms Sunday, June 2, 13

Slide 34

Slide 34 text

Peer Statistics Peers min, avg, max, stddev: 6 messages: "druby://10.101.28.77:60928" to "druby://10.101.28.77:60910"; 0.623, 1.632, 4.877, 1.618 ms Sunday, June 2, 13

Slide 35

Slide 35 text

Peer Statistics Peers min, avg, max, stddev: [...] 4 single-message peers 0.307, 0.791, 1.568, 0.547 ms Sunday, June 2, 13

Slide 36

Slide 36 text

capp • libpcap wrapper • GVL-friendly • MIT license • Simple interface Sunday, June 2, 13

Slide 37

Slide 37 text

capp capp = Capp.live capp.filter = '...' Thread.new do capp.loop do |packet| @queue.enq packet end @queue << nil # no more packets end Sunday, June 2, 13

Slide 38

Slide 38 text

capp Thread.new do while packet = @queue.deq do process packet end end Sunday, June 2, 13

Slide 39

Slide 39 text

capp Thread process packet Queue packet = queue.deq Thread capp_loop() acquire GVL queue.enq packet Sunday, June 2, 13

Slide 40

Slide 40 text

capp Thread Thread process packet Queue Queue acquire GVL capp_loop() queue.enq packet packet = queue.deq Thread capp_loop() acquire GVL queue.enq packet Sunday, June 2, 13

Slide 41

Slide 41 text

capp Capp.open 'lo0' Capp.open 'capture.pcap' Capp.devices # devices and addresses Capp.drop_privileges # for safety capp.stop # end capture Sunday, June 2, 13

Slide 42

Slide 42 text

marshal-structure • Marshal parser • Tracks allocations Sunday, June 2, 13

Slide 43

Slide 43 text

marshal-structure s = Marshal::Structure.new \ Marshal.dump %w[a b] s.count_allocations #=> 4 s.load #=> ["a", "b"] Sunday, June 2, 13

Slide 44

Slide 44 text

marshal-structure s = Marshal::Structure.new \ Marshal.dump %w[a b] s.count_allocations #=> 4 s.load #=> ["a", "b"] Hidden Encoding instance variable Sunday, June 2, 13

Slide 45

Slide 45 text

token_stream s.token_stream.to_a #=> [ :array, 2, :instance_variables, :string, "a", 1, :symbol, "E", :true, :instance_variables, :string, "b", 1, :symbol_link, 0, :true] Sunday, June 2, 13

Slide 46

Slide 46 text

structure s.structure #=> [ :array, 0, 2, [:instance_variables, [:string, 1, "a"], 1, [:symbol, 0, "E"], :true], [:instance_variables, [:string, 2, "b"], 1, [:symbol_link, 0], :true]] Sunday, June 2, 13

Slide 47

Slide 47 text

Slide 48

Slide 48 text

Slide 49

Slide 49 text

Slide 50

Slide 50 text

Slide 51

Slide 51 text

token_stream s.token_stream.to_a #=> [ :array, 2, :instance_variables, :string, "a", 1, :symbol, "E", :true, :instance_variables, :string, "b", 1, :symbol_link, 0, :true] Sunday, June 2, 13