Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Peeking Into Ruby

Jason R Clark
October 01, 2015
Technology
0
100

Peeking Into Ruby

Your Ruby app is in production, but something isn’t quite right. It worked locally, it passed CI… why’s the running app acting weird?

If this sounds familiar, you’re in luck. Multiple tools exist for grappling with a running Ruby app. This talk will introduce a variety of tools and techniques for peeking into what your Ruby app is doing. From Ruby-level method tracing using rbtrace, all the way down to watching kernel syscalls with strace, you can see what your app is doing, and I’ll show you how.

Don’t let your production system go unwatched!

Jason R Clark

October 01, 2015
Tweet

More Decks by Jason R Clark

See All by Jason R Clark
Failing Well
jasonrclark
0
150
Rack 'em, Stack 'em, Web Apps
jasonrclark
2
340
Programming in the Small: Kids, Chickens, and Ruby
jasonrclark
1
130
Real World Docker for the Rubyist
jasonrclark
2
340
GDB: A Gentle Intro
jasonrclark
1
400
Testing The Multiverse - OS Bridge 2015
jasonrclark
0
110
Testing the Multiverse
jasonrclark
0
220
Spelunking in Ruby
jasonrclark
2
500
Programming in the Small - Teaching my 6 Year Old Ruby
jasonrclark
2
230

Other Decks in Technology

See All in Technology
KEDAで始めるイベント駆動システム #k8snovice / keda-tutorial
chmikata
1
130
Semantic KernelでGPTと外部ツールを連携する
takashi1029
1
1.1k
Oracle Database Technology Night #67 Oracle Database High Availability concept
oracle4engineer
PRO
0
580
NestJS をかじってみた / MIERUNE BBQ #01 / #mierune
tacck
0
200
少しずつでも出来るようになりたいもの、なに？ / What do you want to be able to do step by step?
banjun
PRO
1
150
Tech Dojo Introduction Of Monitoring
norimuraz
0
290
ブロックテーマでどう変わる？新しいWordPressのWebサイト制作
tbshiki
0
140
【Oracle Cloud ウェビナー】Oracle Databaseはクラウドに移行するべきか否か 全10ケースをご紹介 (2023年5月24日)
oracle4engineer
PRO
1
100
エンジニアからの情報発信の意義に関して、さまざまな方々が私に教えてくださったことのまとめ
m3hiro3
0
120
IaCのCI/CDを考えよう / JAWS-UG_Okayama_IaC_CICD
taishin
0
120
社内でChatGPTを利用するためのガイドラインを整備した話
yoshikieguchi
0
1k
トラブルシューティングから Linux カーネルに潜り込む
hiboma
7
1.5k

Featured

See All Featured
Clear Off the Table
cherdarchuk
81
300k
Building Applications with DynamoDB
mza
86
5.1k
How GitHub (no longer) Works
holman
299
140k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
110
17k
The Pragmatic Product Professional
lauravandoore
22
3.8k
Teambox: Starting and Learning
jrom
124
8k
GraphQLの誤解/rethinking-graphql
sonatard
41
8.2k
The Illustrated Children's Guide to Kubernetes
chrisshort
26
44k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
351
21k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
270
12k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
46
15k
No one is an island. Learnings from fostering a developers community.
thoeni
13
1.7k

Transcript

  1. https://flic.kr/p/8RePxD
    Peeking Into Ruby
    Tracing Running Code
    @jasonrclark
    http://github.com/jasonrclark/
    peeking_into_ruby

    View Slide

  2. https://flic.kr/p/aVcE68
    rbtrace
    2
    strace
    tcpdump

    View Slide

  3. https://flic.kr/p/8Kj2EC
    Dig Deeper.
    3

    View Slide

  4. The Case of The
    Missing Config File.
    4

    View Slide

  5. 5
    https://flic.kr/p/pGP9hG

    View Slide

  6. 6
    https://flic.kr/p/tRhQNP

    View Slide

  7. 7
    https://flic.kr/p/shT4Lr

    View Slide

  8. 8
    $ ./data_streamer
    ********************************
    Loading data stream processor
    Requiring top_secret gem...
    ********************************
    Hmmm, that's not how I should be
    configured -- Carl

    View Slide

  9. 9
    https://flic.kr/p/eimYHo

    View Slide

  10. https://flic.kr/p/aVcE68
    strace
    10

    View Slide

  11. 11
    $ strace ./data_streamer

    View Slide

  12. View Slide

  13. ETOOMUCHOUTPUT
    13

    View Slide

  14. execve("/bin/echo", ["echo", "boo"], [/* 14 vars */]) =
    brk(0) = 0x20f0000
    access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No
    mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_
    access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No
    open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
    fstat(3, {st_mode=S_IFREG|0644, st_size=30820, ...}) =
    mmap(NULL, 30820, PROT_READ, MAP_PRIVATE, 3, 0) =
    close(3) = 0
    access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No
    open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOE
    read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0
    \34\2\0\0\0\0\0"..., 832) = 832
    fstat(3, {st_mode=S_IFREG|0755, st_size=1729984, ...})
    mmap(NULL, 3836448, PROT_READ|PROT_EXEC, MAP_PRIVATE|MA
    mprotect(0x7f676f4e1000, 2097152, PROT_NONE) = 0

    View Slide

  15. 15
    open(“/etc/ld.so.cache",
    O_RDONLY|O_CLOEXEC) = 3

    View Slide

  16. 16
    open(“/etc/ld.so.cache",
    O_RDONLY|O_CLOEXEC) = 3

    View Slide

  17. 17
    open(“/etc/ld.so.cache",
    O_RDONLY|O_CLOEXEC) = 3

    View Slide

  18. 18
    open(“/etc/ld.so.cache",
    O_RDONLY|O_CLOEXEC) = 3

    View Slide

  19. A Brief Interlude.
    19

    View Slide

  20. 20
    Kernel User

    View Slide

  21. 21
    https://flic.kr/p/5J1zrp

    View Slide

  22. 22
    https://flic.kr/p/4L4zvH

    View Slide

  23. 23
    Kernel User
    syscalls

    View Slide

  24. exec
    fork
    exit
    24
    open
    read
    write
    close

    View Slide

  25. 25
    $ man open 2

    View Slide

  26. OPEN(2) Linux Programmer's Manual
    NAME
    open, openat - open and possibly create a file
    SYNOPSIS
    #include
    #include
    #include
    int open(const char *pathname, int flags);
    int open(const char *pathname, int flags);
    DESCRIPTION
    Given a pathname for a file, open() returns a file
    descriptor, a small, nonnegative integer for use i
    subsequent system calls (read(2), write(2), lseek(
    fcntl(2), etc.). The file descriptor returned by
    successful call will be the lowest-numbered file
    descriptor not currently open for the process.

    View Slide

  27. 27
    Kernel User
    syscalls
    strace

    View Slide

  28. https://flic.kr/p/8Kj2EC
    Look For
    Landmarks.
    28

    View Slide

  29. 29
    $ ./data_streamer
    ********************************
    Loading data stream processor
    Requiring top_secret gem...
    ********************************
    Hmmm, that's not how I should be
    configured -- Carl

    View Slide

  30. 30
    getcwd("/usr/src/app", 4096) = 13
    lstat("/usr", {st_mode=S_IFDIR|0755,
    lstat("/usr/src", {st_mode=S_IFDIR|0755,
    lstat("/usr/src/app", {st_mode=S_IFDIR|0755,
    lstat("/usr/src/app/missing_config", {st_mo
    getcwd("/usr/src/app", 4096) = 13
    write(1,"****************************"...,
    50*****************************) = 50
    write(1,"\n", 1) = 1
    write(1,"Loading data stream processor",
    29Loading data stream processor) = 29
    fstat(7, {st_mode=S_IFREG|0644, st_size=127,
    close(7) = 0
    getuid() = 0
    geteuid() = 0
    getgid() = 0
    getegid() = 0

    View Slide

  31. 31
    fstat(7, {st_mode=S_IFREG|0644, st_size=12
    ioctl(7, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_-1
    ENOTTY (Inappropriate
    read(7, "begin\n File.read(\”/usr/local/sl
    read(7, "", 8192) = 0
    close(7) = 0
    lstat("/usr", {st_mode=S_IFDIR|0755, st_size=
    lstat("/usr/src", {st_mode=S_IFDIR|0755, st
    write(1, "Hmmm, that's not how I"...,
    51Hmmm, that's not how I...) = 51
    write(1, "\n", 1) = 1
    rt_sigaction(SIGINT, {SIG_IGN, [], SA_RESTO
    rt_sigaction(SIGINT, {SIG_DFL, [],SA_RESTO
    write(4, "!", 1) = 1
    munmap(0x7fdaf91a4000, 1052672) = 0
    exit_group(666) = ?
    +++ exited with 154 +++

    View Slide

  32. 32
    open(“/usr/local/slurp.confi”,
    O_RDONLY|O_CLOEXEC)
    = -1 ENOENT (No such file or
    directory)

    View Slide

  33. 33
    open(“/usr/local/slurp.confi”,
    O_RDONLY|O_CLOEXEC)
    = -1 ENOENT (No such file or
    directory)

    View Slide

  34. slurp.confi?
    34

    View Slide

  35. Resources
    35
    Julia Evans: blog and zine
    http://jvns.ca/blog/categories/strace/
    James Golick: How to Debug Anything
    https://www.youtube.com/watch?v=VV7b7fs4VI8

    View Slide

  36. The Case of The
    Unused Method
    36

    View Slide

  37. … Or Was It?
    37

    View Slide

  38. 38
    https://flic.kr/p/dmb7sW

    View Slide

  39. 39
    https://flic.kr/p/a4VW2z

    View Slide

  40. 40
    add_method_tracer

    View Slide

  41. 41
    https://flic.kr/p/3LP9cm

    View Slide

  42. https://flic.kr/p/8Kj2EC
    Proper
    Preparation.
    42

    View Slide

  43. https://flic.kr/p/aVcE68
    rbtrace
    43

    View Slide

  44. 44
    Ruby
    VM
    Our
    Code
    rbtrace

    View Slide

  45. 45
    $ rbtrace --methods puts
    --exec ./script.rb
    puts
    puts
    ...

    View Slide

  46. 46
    # code.rb
    require ‘rbtrace’
    $ ruby code.rb &
    $ rbtrace -p PID --methods puts
    puts
    puts

    View Slide

  47. 47
    $ rbtrace -p PID
    --methods Custom#goo >
    custom_goo.log

    View Slide

  48. 48
    ls -la custom_goo.log
    -rw-r--r-- 1 0 Sep 30 22:12 goo.log

    View Slide

  49. The Case of The
    Occasional Lag
    49

    View Slide

  50. 50
    Heartbeats Received
    background
    job starts

    View Slide

  51. https://flic.kr/p/aVcE68
    rbtrace
    51

    View Slide

  52. 52
    $ rbtrace -p PID --gc -t
    05:54:28.10610 Running background job
    05:54:28.11113 garbage_collect
    05:54:28.56664 garbage_collect
    05:54:33.72790 garbage_collect

    View Slide

  53. 53

    View Slide

  54. Resources
    54
    Aman Gupta: github
    https://github.com/tmm1/rbtrace
    Justin Weiss: How to Debug Ruby
    Performance Problems in Production
    http://www.justinweiss.com/blog/2015/04/06/how-to-
    debug-ruby-performance-problems-in-production/

    View Slide

  55. The Case of The
    Delicate
    Negotiations
    55

    View Slide

  56. 56
    https://flic.kr/p/d8Kvyo

    View Slide

  57. 57
    url = "https://www.google.com/"
    uri = URI.parse(url)
    response = Net::HTTP.get_response(uri)
    ...
    response = Net::HTTP.get_response(uri)

    View Slide

  58. 58
    url = "https://www.google.com/"
    uri = URI.parse(url)
    conn = Net::HTTP.new(uri.host, uri.port)
    conn.use_ssl = true
    conn.keep_alive_timeout = 60
    conn.start
    request = Net::HTTP::Get.new('/')
    response = conn.request(request)
    ...
    response = conn.request(request)

    View Slide

  59. 59
    url = "https://www.google.com/"
    uri = URI.parse(url)
    conn = Net::HTTP.new(uri.host, uri.port)
    conn.use_ssl = true
    conn.keep_alive_timeout = 60
    conn.start
    request = Net::HTTP::Get.new('/')
    response = conn.request(request)
    ...
    response = conn.request(request)

    View Slide

  60. 60
    url = "https://www.google.com/"
    uri = URI.parse(url)
    conn = Net::HTTP.new(uri.host, uri.port)
    conn.use_ssl = true
    conn.keep_alive_timeout = 60
    conn.start
    request = Net::HTTP::Get.new('/')
    response = conn.request(request)
    ...
    response = conn.request(request)

    View Slide

  61. https://flic.kr/p/aVcE68
    tcpdump
    61

    View Slide

  62. 62
    Network
    Stack
    Our
    Code
    tcpdump

    View Slide

  63. https://flic.kr/p/8Kj2EC
    Find
    Boundaries.
    63

    View Slide

  64. 64
    Network
    Stack
    Our
    Code
    Kernel

    View Slide

  65. 65
    $ tcpdump

    View Slide

  66. 66
    17:29:36.569760 IP a6a0111ae486.38923 > nuq05s02-in-
    f16.1e100.net.https: Flags [S], seq 290609768, win
    29200, options [mss 1460,sackOK,TS val 486029 ecr
    0,nop,wscale 7], length 0
    17:29:36.595921 IP nuq05s02-in-f16.1e100.net.https >
    a6a0111ae486.38923: Flags [S.], seq 651712001, ack
    290609769, win 65535, options [mss 1460], length 0
    17:29:36.595996 IP a6a0111ae486.38923 > nuq05s02-in-
    f16.1e100.net.https: Flags [.], ack 1, win 29200, length
    0
    17:29:36.596658 IP a6a0111ae486.38923 > nuq05s02-in-
    f16.1e100.net.https: Flags [P.], seq 1:251, ack 1, win
    29200, length 250
    17:29:36.596937 IP nuq05s02-in-f16.1e100.net.https >
    a6a0111ae486.38923: Flags [.], ack 251, win 65535,
    length 0

    View Slide

  67. ETOOMUCHOUTPUT
    67

    View Slide

  68. 68
    17:29:36.569760 IP a6a0111ae486.38923 > nuq05s02-in-
    f16.1e100.net.https: Flags [S], seq 290609768, win
    29200, options [mss 1460,sackOK,TS val 486029 ecr
    0,nop,wscale 7], length 0
    17:29:36.595921 IP nuq05s02-in-f16.1e100.net.https >
    a6a0111ae486.38923: Flags [S.], seq 651712001, ack
    290609769, win 65535, options [mss 1460], length 0
    17:29:36.595996 IP a6a0111ae486.38923 > nuq05s02-in-
    f16.1e100.net.https: Flags [.], ack 1, win 29200, length
    0
    17:29:36.596658 IP a6a0111ae486.38923 > nuq05s02-in-
    f16.1e100.net.https: Flags [P.], seq 1:251, ack 1, win
    29200, length 250
    17:29:36.596937 IP nuq05s02-in-f16.1e100.net.https >
    a6a0111ae486.38923: Flags [.], ack 251, win 65535,
    length 0

    View Slide

  69. 69
    17:29:36.569760 IP a6a0111ae486.38923 >
    nuq05s02-in-f16.1e100.net.https:
    Flags [S], seq 290609768, win 29200,
    options [mss 1460,sackOK,TS val 486029
    ecr 0,nop,wscale 7], length 0

    View Slide

  70. 70
    17:29:36.569760 IP a6a0111ae486.38923 >
    nuq05s02-in-f16.1e100.net.https:
    Flags [S], seq 290609768, win 29200,
    options [mss 1460,sackOK,TS val 486029
    ecr 0,nop,wscale 7], length 0

    View Slide

  71. 71
    17:29:36.569760 IP a6a0111ae486.38923 >
    nuq05s02-in-f16.1e100.net.https:
    Flags [S], seq 290609768, win 29200,
    options [mss 1460,sackOK,TS val 486029
    ecr 0,nop,wscale 7], length 0
    a6a0111ae486.38923 == me
    nuq05s02-in-f16.1e100.net.https == google

    View Slide

  72. 72
    17:29:36.569760 IP a6a0111ae486.38923 >
    nuq05s02-in-f16.1e100.net.https:
    Flags [S], seq 290609768, win 29200,
    options [mss 1460,sackOK,TS val 486029
    ecr 0,nop,wscale 7], length 0

    View Slide

  73. 73
    Flags
    [S] - SYN (Start Connection)
    [.] - No Flag Set
    [P] - PSH (Push Data)
    [F] - FIN (Finish Connection)

    View Slide

  74. 74
    17:29:36.569760 IP a6a0111ae486.38923 >
    nuq05s02-in-f16.1e100.net.https:
    Flags [S], seq 290609768, win 29200,
    options [mss 1460,sackOK,TS val 486029
    ecr 0,nop,wscale 7], length 0

    View Slide

  75. 75
    url = "https://www.google.com/"
    uri = URI.parse(url)
    response = Net::HTTP.get_response(uri)

    View Slide

  76. 76
    17:29:36 IP me > google: Flags [S], seq 290609768, win
    17:29:36 IP google > me: Flags [S.], seq 651712001, ac
    17:29:36 IP me > google: Flags [.], ack 1, win 29200,
    17:29:36 IP me > google: Flags [P.], seq 1:251, ack 1,
    17:29:36 IP google > me: Flags [.], ack 251, win 65535
    17:29:36 IP google > me: Flags [P.], seq 1:1289, ack 2
    17:29:36 IP me > google: Flags [.], ack 1289, win 3091
    17:29:36 IP google > me: Flags [P.], seq 1289:2577, ac
    … 24 times
    17:29:36 IP google > me: Flags [.], seq 10110:11530, a
    17:29:36 IP me > google: Flags [.], ack 11530, win 568
    17:29:36 IP google > me: Flags [P.], seq 11530:12169,
    17:29:36 IP me > google: Flags [P.], seq 535:566, ack
    17:29:36 IP me > google: Flags [F.], seq 566, ack 1216
    17:29:36 IP google > me: Flags [.], ack 566, win 65535
    17:29:36 IP google > me: Flags [.], ack 567, win 65535
    17:29:36 IP google > me: Flags [F.], seq 12169, ack 56
    17:29:36 IP me > google: Flags [.], ack 12170, win 596

    View Slide

  77. 77
    17:29:36 IP me > google: Flags [S], seq 290609768, win
    17:29:36 IP google > me: Flags [S.], seq 651712001, ac
    17:29:36 IP me > google: Flags [.], ack 1, win 29200,
    17:29:36 IP me > google: Flags [P.], seq 1:251, ack 1,
    17:29:36 IP google > me: Flags [.], ack 251, win 65535
    17:29:36 IP google > me: Flags [P.], seq 1:1289, ack 2
    17:29:36 IP me > google: Flags [.], ack 1289, win 3091
    17:29:36 IP google > me: Flags [P.], seq 1289:2577, ac
    … 24 times
    17:29:36 IP google > me: Flags [.], seq 10110:11530, a
    17:29:36 IP me > google: Flags [.], ack 11530, win 568
    17:29:36 IP google > me: Flags [P.], seq 11530:12169,
    17:29:36 IP me > google: Flags [P.], seq 535:566, ack
    17:29:36 IP me > google: Flags [F.], seq 566, ack 1216
    17:29:36 IP google > me: Flags [.], ack 566, win 65535
    17:29:36 IP google > me: Flags [.], ack 567, win 65535
    17:29:36 IP google > me: Flags [F.], seq 12169, ack 56
    17:29:36 IP me > google: Flags [.], ack 12170, win 596

    View Slide

  78. 78
    17:29:36 IP me > google: Flags [S], seq 290609768, win
    17:29:36 IP google > me: Flags [S.], seq 651712001, ac
    17:29:36 IP me > google: Flags [.], ack 1, win 29200,
    17:29:36 IP me > google: Flags [P.], seq 1:251, ack 1,
    17:29:36 IP google > me: Flags [.], ack 251, win 65535
    17:29:36 IP google > me: Flags [P.], seq 1:1289, ack 2
    17:29:36 IP me > google: Flags [.], ack 1289, win 3091
    17:29:36 IP google > me: Flags [P.], seq 1289:2577, ac
    … 24 times
    17:29:36 IP google > me: Flags [.], seq 10110:11530, a
    17:29:36 IP me > google: Flags [.], ack 11530, win 568
    17:29:36 IP google > me: Flags [P.], seq 11530:12169,
    17:29:36 IP me > google: Flags [P.], seq 535:566, ack
    17:29:36 IP me > google: Flags [F.], seq 566, ack 1216
    17:29:36 IP google > me: Flags [.], ack 566, win 65535
    17:29:36 IP google > me: Flags [.], ack 567, win 65535
    17:29:36 IP google > me: Flags [F.], seq 12169, ack 56
    17:29:36 IP me > google: Flags [.], ack 12170, win 596

    View Slide

  79. 79
    17:29:36 IP me > google: Flags [S], seq 290609768, win
    17:29:36 IP google > me: Flags [S.], seq 651712001, ac
    17:29:36 IP me > google: Flags [.], ack 1, win 29200,
    17:29:36 IP me > google: Flags [P.], seq 1:251, ack 1,
    17:29:36 IP google > me: Flags [.], ack 251, win 65535
    17:29:36 IP google > me: Flags [P.], seq 1:1289, ack 2
    17:29:36 IP me > google: Flags [.], ack 1289, win 3091
    17:29:36 IP google > me: Flags [P.], seq 1289:2577, ac
    … 24 times
    17:29:36 IP google > me: Flags [.], seq 10110:11530, a
    17:29:36 IP me > google: Flags [.], ack 11530, win 568
    17:29:36 IP google > me: Flags [P.], seq 11530:12169,
    17:29:36 IP me > google: Flags [P.], seq 535:566, ack
    17:29:36 IP me > google: Flags [F.], seq 566, ack 1216
    17:29:36 IP google > me: Flags [.], ack 566, win 65535
    17:29:36 IP google > me: Flags [.], ack 567, win 65535
    17:29:36 IP google > me: Flags [F.], seq 12169, ack 56
    17:29:36 IP me > google: Flags [.], ack 12170, win 596

    View Slide

  80. 80
    url = "https://www.google.com/"
    uri = URI.parse(url)
    conn = Net::HTTP.new(uri.host, uri.port)
    conn.use_ssl = true
    conn.keep_alive_timeout = 60
    conn.start
    request = Net::HTTP::Get.new('/')
    response = conn.request(request)
    ...
    response = conn.request(request)

    View Slide

  81. 81
    17:58:12 IP me > google: Flags [S], seq 290609768, win
    17:58:12 IP google > me: Flags [S.], seq 651712001, ac
    17:58:12 IP me > google: Flags [.], ack 1, win 29200,
    17:58:12 IP me > google: Flags [P.], seq 1:251, ack 1,
    17:58:12 IP google > me: Flags [.], ack 251, win 65535
    17:58:12 IP google > me: Flags [P.], seq 1:1289, ack 2
    17:58:12 IP me > google: Flags [.], ack 1289, win 3091
    17:58:12 IP google > me: Flags [P.], seq 1289:2577, ac
    … 24 times
    17:58:12 IP google > me: Flags [.], seq 10110:11530, a
    17:58:12 IP me > google: Flags [.], ack 11530, win 568
    17:58:12 IP google > me: Flags [P.], seq 11530:12169,
    17:58:12 IP me > google: Flags [P.], seq 535:566, ack
    17:58:12 IP me > google: Flags [.], ack 12170, win 596
    No Flags [F]!

    View Slide

  82. 82
    17:48:18 IP me > google: Flags [P.], seq 693:851, ack
    17:48:18 IP google > me: Flags [.], ack 851, win 65535
    17:48:18 IP google > me: Flags [P.], seq 20688:21976,
    17:48:18 IP me > google: Flags [.], ack 21976, win 653
    17:48:18 IP google > me: Flags [.], seq 21976:23396, a
    17:48:18 IP me > google: Flags [.], ack 23396, win 653
    17:48:18 IP google > me: Flags [.], seq 23396:24816, a
    17:48:18 IP me > google: Flags [.], ack 24816, win 653
    17:48:18 IP google > me: Flags [P.], seq 24816:25840,
    17:48:18 IP me > google: Flags [.], ack 25840, win 653
    17:48:18 IP google > me: Flags [.], seq 25840:27260, a
    17:48:18 IP me > google: Flags [.], ack 27260, win 653
    17:48:18 IP google > me: Flags [.], seq 27260:28680, a
    17:48:18 IP me > google: Flags [.], ack 28680, win 653
    17:48:18 IP google > me: Flags [P.], seq 28680:29109,
    17:48:18 IP me > google: Flags [.], ack 29109, win 653

    View Slide

  83. 83
    17:48:18 IP me > google: Flags [P.], seq 693:851, ack
    17:48:18 IP google > me: Flags [.], ack 851, win 65535
    17:48:18 IP google > me: Flags [P.], seq 20688:21976,
    17:48:18 IP me > google: Flags [.], ack 21976, win 653
    17:48:18 IP google > me: Flags [.], seq 21976:23396, a
    17:48:18 IP me > google: Flags [.], ack 23396, win 653
    17:48:18 IP google > me: Flags [.], seq 23396:24816, a
    17:48:18 IP me > google: Flags [.], ack 24816, win 653
    17:48:18 IP google > me: Flags [P.], seq 24816:25840,
    17:48:18 IP me > google: Flags [.], ack 25840, win 653
    17:48:18 IP google > me: Flags [.], seq 25840:27260, a
    17:48:18 IP me > google: Flags [.], ack 27260, win 653
    17:48:18 IP google > me: Flags [.], seq 27260:28680, a
    17:48:18 IP me > google: Flags [.], ack 28680, win 653
    17:48:18 IP google > me: Flags [P.], seq 28680:29109,
    17:48:18 IP me > google: Flags [.], ack 29109, win 653
    ~15 packets
    No New Connection

    View Slide

  84. 84

    View Slide

  85. Resources
    85
    Benjamin Cane: A Quick and Practical
    Reference for tcpdump
    http://bencane.com/2014/10/13/quick-and-practical-
    reference-for-tcpdump/
    Jonan Scheffler: Lifecycle of a Web Request
    https://www.youtube.com/watch?v=U-4md8lYosk

    View Slide

  86. 86
    https://flic.kr/p/ahkHU8

    View Slide

  87. https://flic.kr/p/aVcE68
    rbtrace
    87
    strace
    tcpdump

    View Slide

  88. https://flic.kr/p/8Kj2EC
    Dig Deeper.
    88

    View Slide

  89. https://flic.kr/p/8Kj2EC
    Look For
    Landmarks.
    89

    View Slide

  90. https://flic.kr/p/8Kj2EC
    Proper
    Preparations.
    90

    View Slide

  91. https://flic.kr/p/8Kj2EC
    Find
    Boundaries.
    91

    View Slide

  92. Case Closed!
    92
    https://github.com/jasonrclark/
    peeking_into_ruby
    @jasonrclark

    View Slide