Slide 1

Slide 1 text

https://flic.kr/p/8RePxD GDB A Gentle Intro @jasonrclark http://git.io/v8GbT https://flic.kr/p/o15Y5n

Slide 2

Slide 2 text

2 $ ps -o pid,pcpu,pmem,cmd PID %CPU %MEM CMD 1 0.0 0.1 bash 10 1.8 0.4 ruby service.rb >>

Slide 3

Slide 3 text

3 $ tail production.log [14:30:59 (8985)] INFO : Starting! $ >>

Slide 4

Slide 4 text

4 Threading https://flic.kr/p/4WFtU8

Slide 5

Slide 5 text

Deadlock? https://flic.kr/p/nkwswv

Slide 6

Slide 6 text

GDB 6

Slide 7

Slide 7 text

Gnu Debugger 7

Slide 8

Slide 8 text

MRI 8

Slide 9

Slide 9 text

MRI CRuby 9

Slide 10

Slide 10 text

10 $ gdb -p 10 ... Attaching to process 10 Reading symbols from /usr/local/bin/ ruby...done. [Thread debugging using libthread_db enabled] Loaded symbols for /usr/local/lib/ruby/ 2.2.0/x86_64-linux/thread.so (gdb) >>

Slide 11

Slide 11 text

11 $ gdb -p 10 ... Attaching to process 10 Reading symbols from /usr/local/bin/ ruby...done. [Thread debugging using libthread_db enabled] Loaded symbols for /usr/local/lib/ruby/ 2.2.0/x86_64-linux/thread.so (gdb) >>

Slide 12

Slide 12 text

(gdb) backtrace #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/...:185 #1 0x00007f131f343b03 in native_cond_wait(mutex=0x7f131fcdf5, cond=0x7f131fcdf558) at thread_pthread.c:338 #2 lock_func (timeout_ms=0, mutex=0x7f, th=0x7f131f8c46) at thread.c:4268 #3 rb_mutex_lock (self=139720114712880) at thread.c:4342 #4 0x00007f131f31c9ee in vm_call_cfunc_with_frame (ci=0x0, reg_cfp=0x7f131f1a7f70, th=0x7f131f8c46e0) at vm_insnhelper.c:1380 >>

Slide 13

Slide 13 text

13 RuntimeError: raise ‘Heck’ config.ru:13:in `call' rack-1.6.4/...tempfile_reaper.rb:15:in `call' rack-1.6.4/...lint.rb:49:in `_call' rack-1.6.4/...lint.rb:37:in `call' rack-1.6.4/...showexceptions.rb:24:in `call' rack-1.6.4/...commonlogger.rb:33:in `call' rack-1.6.4/...chunked.rb:54:in `call' rack-1.6.4/...content_length.rb:15:in `call' thin-1.6.3/...connection.rb:86:in `block in thin-1.6.3/...connection.rb:84:in `catch' thin-1.6.3/...connection.rb:84:in `pre_process' thin-1.6.3/...connection.rb:53:in `process' thin-1.6.3/...connection.rb:39:in `receive_data eventmachine-1.0.7/lib/eventmachine.rb:187:in eventmachine-1.0.7/lib/eventmachine.rb:187:in thin-1.6.3/...backends/base.rb:73:in `start' thin-1.6.3/...server.rb:162:in `start'

Slide 14

Slide 14 text

(gdb) backtrace #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/...:185 #1 0x00007f131f343b03 in native_cond_wait(mutex=0x7f131fcdf5, cond=0x7f131fcdf558) at thread_pthread.c:338 #2 lock_func (timeout_ms=0, mutex=0x7f, th=0x7f131f8c46) at thread.c:4268 #3 rb_mutex_lock (self=139720114712880) at thread.c:4342 #4 0x00007f131f31c9ee in vm_call_cfunc_with_frame (ci=0x0, reg_cfp=0x7f131f1a7f70, th=0x7f131f8c46e0) at vm_insnhelper.c:1380

Slide 15

Slide 15 text

(gdb) backtrace #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/...:185 #1 0x00007f131f343b03 in native_cond_wait(mutex=0x7f131fcdf5, cond=0x7f131fcdf558) at thread_pthread.c:338 #2 lock_func (timeout_ms=0, mutex=0x7f, th=0x7f131f8c46) at thread.c:4268 #3 rb_mutex_lock (self=139720114712880) at thread.c:4342 #4 0x00007f131f31c9ee in vm_call_cfunc_with_frame (ci=0x0, reg_cfp=0x7f131f1a7f70, th=0x7f131f8c46e0) at vm_insnhelper.c:1380

Slide 16

Slide 16 text

(gdb) backtrace #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/...:185 #1 0x00007f131f343b03 in native_cond_wait(mutex=0x7f131fcdf5, cond=0x7f131fcdf558) at thread_pthread.c:338 #2 lock_func (timeout_ms=0, mutex=0x7f, th=0x7f131f8c46) at thread.c:4268 #3 rb_mutex_lock (self=139720114712880) at thread.c:4342 #4 0x00007f131f31c9ee in vm_call_cfunc_with_frame (ci=0x0, reg_cfp=0x7f131f1a7f70, th=0x7f131f8c46e0) at vm_insnhelper.c:1380

Slide 17

Slide 17 text

(gdb) backtrace #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/...:185 #1 0x00007f131f343b03 in native_cond_wait(mutex=0x7f131fcdf5, cond=0x7f131fcdf558) at thread_pthread.c:338 #2 lock_func (timeout_ms=0, mutex=0x7f, th=0x7f131f8c46) at thread.c:4268 #3 rb_mutex_lock (self=139720114712880) at thread.c:4342 #4 0x00007f131f31c9ee in vm_call_cfunc_with_frame (ci=0x0, reg_cfp=0x7f131f1a7f70, th=0x7f131f8c46e0) at vm_insnhelper.c:1380

Slide 18

Slide 18 text

18 (gdb) info threads Id Target Id Frame 4 Thread 0x7f131f1b4700 (LWP 11) "ruby-timer-thr" in poll () at ../sysdeps/unix/syscall-temp... 3 Thread 0x7f131db8a700 (LWP 12) “service.rb:6" pthread_cond_wait@@GLIBC_2.3.2 () 2 Thread 0x7f131d988700 (LWP 13) “service.rb:12" vm_exec_core (th=0x7f131d989058... * 1 Thread 0x7f131f1aa700 (LWP 10) "ruby" pthread_cond_wait@@GLIBC... >>

Slide 19

Slide 19 text

19 (gdb) info threads Id Target Id Frame 4 Thread 0x7f131f1b4700 (LWP 11) "ruby-timer-thr" in poll () at ../sysdeps/unix/syscall-temp... 3 Thread 0x7f131db8a700 (LWP 12) “service.rb:6" pthread_cond_wait@@GLIBC_2.3.2 () 2 Thread 0x7f131d988700 (LWP 13) “service.rb:12" vm_exec_core (th=0x7f131d989058... * 1 Thread 0x7f131f1aa700 (LWP 10) "ruby" pthread_cond_wait@@GLIBC...

Slide 20

Slide 20 text

20 (gdb) info threads Id Target Id Frame 4 Thread 0x7f131f1b4700 (LWP 11) "ruby-timer-thr" in poll () at ../sysdeps/unix/syscall-temp... 3 Thread 0x7f131db8a700 (LWP 12) “service.rb:6" pthread_cond_wait@@GLIBC_2.3.2 () 2 Thread 0x7f131d988700 (LWP 13) “service.rb:12" vm_exec_core (th=0x7f131d989058... * 1 Thread 0x7f131f1aa700 (LWP 10) "ruby" pthread_cond_wait@@GLIBC...

Slide 21

Slide 21 text

21 (gdb) thread apply all backtrace Thread 3 (Thread 0x7f131db8a700 (LWP 12)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at .. #1 0x00007f131f343b03 in native_cond_wait #2 lock_func (timeout_ms=0, mutex=0x7f131f #3 rb_mutex_lock (self=139720114712840) at #4 0x00007f131f31c9ee in vm_call_cfunc_wit #5 vm_call_cfunc (th=th@entry=0x7f131fcdda #6 0x00007f131f32c82e in vm_call_method (t #7 0x00007f131f32108b in vm_exec_core (th= #8 0x00007f131f3255d8 in vm_exec (th=0x7f1 #9 0x00007f131f32be0c in invoke_block_from #10 0x00007f131f32bfa0 in vm_invoke_proc (t >>

Slide 22

Slide 22 text

22 (gdb) thread apply all backtrace Thread 3 (Thread 0x7f131db8a700 (LWP 12)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at .. #1 0x00007f131f343b03 in native_cond_wait #2 lock_func (timeout_ms=0, mutex=0x7f131f #3 rb_mutex_lock (self=139720114712840) at #4 0x00007f131f31c9ee in vm_call_cfunc_wit #5 vm_call_cfunc (th=th@entry=0x7f131fcdda #6 0x00007f131f32c82e in vm_call_method (t #7 0x00007f131f32108b in vm_exec_core (th= #8 0x00007f131f3255d8 in vm_exec (th=0x7f1 #9 0x00007f131f32be0c in invoke_block_from >>

Slide 23

Slide 23 text

23 ---Type to continue, or q Thread 1 (Thread 0x7f131f1aa700 (LWP 10)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at .. #1 0x00007f131f343b03 in native_cond_wait #2 lock_func (timeout_ms=0, mutex=0x7f131f #3 rb_mutex_lock (self=139720114712880) at #4 0x00007f131f31c9ee in vm_call_cfunc_wit #5 vm_call_cfunc (th=th@entry=0x7f131f8c46 #6 0x00007f131f32c82e in vm_call_method (t #7 0x00007f131f32108b in vm_exec_core (th= #8 0x00007f131f3255d8 in vm_exec (th=th@en #9 0x00007f131f33319f in rb_iseq_eval_main >>

Slide 24

Slide 24 text

24 (gdb) exit Undefined command: "exit". Try “help". (gdb) >>

Slide 25

Slide 25 text

25 (gdb) exit Undefined command: "exit". Try “help”. (gdb) quit $ >>

Slide 26

Slide 26 text

https://flic.kr/p/o15Y5n 26 Attach gdb -p PID

Slide 27

Slide 27 text

https://flic.kr/p/o15Y5n 27 Inspect backtrace info threads thread apply all backtrace

Slide 28

Slide 28 text

https://flic.kr/p/o15Y5n 28 $$$$ exit quit

Slide 29

Slide 29 text

A Unique Problem 29 https://flic.kr/p/dPiDp3

Slide 30

Slide 30 text

30 [1] pry(main)> [1,1,2,3].uniq! => [1, 2, 3]

Slide 31

Slide 31 text

31 [1] pry(main)> [1,1,2,3].uniq! => [1, 2, 3] [2] pry(main)> [1,2,3].uniq! => nil

Slide 32

Slide 32 text

32 $ cat snowflake.rb p [1,1,2,3].uniq! p [1,2,3].uniq!

Slide 33

Slide 33 text

33 $ cat snowflake.rb p [1,1,2,3].uniq! p [1,2,3].uniq! $ ruby snowflake.rb [1, 2, 3] nil

Slide 34

Slide 34 text

34 $ gdb --args ruby snowflake.rb GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later ... For help, type "help". Type "apropos word" to search for commands related to "word". (gdb) >>

Slide 35

Slide 35 text

35 (gdb) run Starting program: /usr/local/bin/ruby... [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/ x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7ffff7ff5700 (LWP 13)] [1,2,3] nil [Thread 0x7ffff7ff5700 (LWP 13) exited] [Inferior 1 (process 9) exited normally] (gdb) >>

Slide 36

Slide 36 text

36 (gdb) run Starting program: /usr/local/bin/ruby... [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/ x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7ffff7ff5700 (LWP 13)] [1,2,3] nil [Thread 0x7ffff7ff5700 (LWP 13) exited] [Inferior 1 (process 9) exited normally] (gdb) >>

Slide 37

Slide 37 text

37

Slide 38

Slide 38 text

38

Slide 39

Slide 39 text

39

Slide 40

Slide 40 text

Eeeeeeek! https://flic.kr/p/nkwswv

Slide 41

Slide 41 text

41 int the_answer (long value) { return 42; }

Slide 42

Slide 42 text

42 int the_answer (long value) { return 42; }

Slide 43

Slide 43 text

43 int the_answer (long value) { return 42; }

Slide 44

Slide 44 text

44 int the_answer (long value) { return 42; }

Slide 45

Slide 45 text

45 int the_answer (long value) { return 42; }

Slide 46

Slide 46 text

46 static VALUE rb_ary_uniq_bang(VALUE ary) { VALUE hash; long hash_size; rb_ary_modify_check(ary); if (RARRAY_LEN(ary) <= 1) return Qnil; if (rb_block_given_p()) hash = ary_make_hash_by(ary); else hash = ary_make_hash(ary); hash_size = RHASH_SIZE(hash); if (RARRAY_LEN(ary) == hash_size) {

Slide 47

Slide 47 text

47 static VALUE rb_ary_uniq_bang(VALUE ary) { VALUE hash; long hash_size; rb_ary_modify_check(ary); if (RARRAY_LEN(ary) <= 1) return Qnil; if (rb_block_given_p()) hash = ary_make_hash_by(ary); else hash = ary_make_hash(ary); hash_size = RHASH_SIZE(hash); if (RARRAY_LEN(ary) == hash_size) {

Slide 48

Slide 48 text

48 static VALUE rb_ary_uniq_bang(VALUE ary) { VALUE hash; long hash_size; rb_ary_modify_check(ary); if (RARRAY_LEN(ary) <= 1) return Qnil; if (rb_block_given_p()) hash = ary_make_hash_by(ary); else hash = ary_make_hash(ary); hash_size = RHASH_SIZE(hash); if (RARRAY_LEN(ary) == hash_size) {

Slide 49

Slide 49 text

49 $ gdb --args ruby snowflake.rb ... Reading symbols from ruby...done. (gdb) break rb_ary_uniq_bang Breakpoint 1 at 0x19a370: file array.c, line 4162. (gdb) >>

Slide 50

Slide 50 text

50 (gdb) run Starting program: /usr/local/bin/ruby snowflake.rb Breakpoint 1, rb_ary_uniq_bang (ary=93825000955160) at array.c:4162 4162 { (gdb) >>

Slide 51

Slide 51 text

51 (gdb) run Starting program: /usr/local/bin/ruby snowflake.rb Breakpoint 1, rb_ary_uniq_bang (ary=93825000955160) at array.c:4162 4162 { (gdb) >>

Slide 52

Slide 52 text

52 (gdb) run Starting program: /usr/local/bin/ruby snowflake.rb Breakpoint 1, rb_ary_uniq_bang (ary=93825000955160) at array.c:4162 4162 { (gdb) >>

Slide 53

Slide 53 text

https://flic.kr/p/o15Y5n 53 Starting gdb --args ruby snowflake.rb

Slide 54

Slide 54 text

https://flic.kr/p/o15Y5n 54 C functions! int the_answer (long value)

Slide 55

Slide 55 text

https://flic.kr/p/o15Y5n 55 ruby-doc.org rb_ary_uniq_bang

Slide 56

Slide 56 text

https://flic.kr/p/o15Y5n 56 Breakpoints break rb_ary_uniq_bang

Slide 57

Slide 57 text

Looking Around https://flic.kr/p/uq8MA

Slide 58

Slide 58 text

58 (gdb) list 4157 * 4158 */ 4159 4160 static VALUE 4161 rb_ary_uniq_bang(VALUE ary) 4162 { 4163 VALUE hash; 4164 long hash_size; 4165 4166 rb_ary_modify_check(ary); >>

Slide 59

Slide 59 text

59 (gdb) display ary 1: ary = 93825000955160 >>

Slide 60

Slide 60 text

Eeeeeeek! https://flic.kr/p/nkwswv

Slide 61

Slide 61 text

61 struct Data { int value; }

Slide 62

Slide 62 text

62 struct Data { int value; } value 0xC001C0DE

Slide 63

Slide 63 text

63 struct ValuableData { Data data; int priority; } value priority 0xC001C0DE

Slide 64

Slide 64 text

64 struct RBasic { VALUE flags; const VALUE klass; } flags klass 0xBAADC0DE

Slide 65

Slide 65 text

struct RArray { struct RBasic basic; union { struct { long len; union { long capa; VALUE shared; } aux; const VALUE *ptr; } heap; const VALUE ary[RARRAY_EMBED_LEN_MAX]; } as; }; 65 flags klass len capa ...

Slide 66

Slide 66 text

66 RBasic RClass RObject RBignum RRational RFloat RComplex RHash RSymbol RString RArray RRegexp

Slide 67

Slide 67 text

67 (gdb) display ary 1: ary = 93825000955160 flags klass 0x555555da5d18 >>

Slide 68

Slide 68 text

68 (gdb) display *(struct RArray*)ary 1: ary = {basic = {flags = 106503, klass = 9382499777180}, as = {heap = {len = 3, aux = {capa = 5, shared = 5}, ptr = 0x7}, ary = {3, 5, 7}}} >>

Slide 69

Slide 69 text

69 (gdb) display *(struct RArray*)ary 1: ary = {basic = {flags = 106503, klass = 9382499777180}, as = {heap = {len = 3, aux = {capa = 5, shared = 5}, ptr = 0x7}, ary = {3, 5, 7}}} >>

Slide 70

Slide 70 text

70 /* for debug print within C code */ void rb_p(VALUE obj)

Slide 71

Slide 71 text

71 (gdb) call rb_p(ary) [">= 0"] >>

Slide 72

Slide 72 text

https://flic.kr/p/o15Y5n 72 Source list

Slide 73

Slide 73 text

https://flic.kr/p/o15Y5n 73 Look display ary

Slide 74

Slide 74 text

https://flic.kr/p/o15Y5n 74 struct struct C { BOOL scary /* FALSE */ }

Slide 75

Slide 75 text

https://flic.kr/p/o15Y5n 75 Debug Print call rb_p(ary)

Slide 76

Slide 76 text

76 https://flic.kr/p/tLkpZG

Slide 77

Slide 77 text

77 (gdb) call rb_p(ary) [">= 0"] >>

Slide 78

Slide 78 text

78 (gdb) call rb_backtrace() from :1:in `' from :1:in `require' from rubygems.rb:115:in `' from rubygems.rb:1210:in `' from rubygems.rb:1210:in `require' from rubygems/specification.rb:37:in `’ from rubygems/specification.rb:146:in `’ from rubygems/requirement.rb:74:in `default' from rubygems/requirement.rb:74:in `new' from rubygems/requirement.rb:125:in `initialize' from rubygems/requirement.rb:125:in `uniq!' >>

Slide 79

Slide 79 text

79 (gdb) call rb_backtrace() from :1:in `' from :1:in `require' from rubygems.rb:115:in `' from rubygems.rb:1210:in `' from rubygems.rb:1210:in `require' from rubygems/specification.rb:37:in `’ from rubygems/specification.rb:146:in `’ from rubygems/requirement.rb:74:in `default' from rubygems/requirement.rb:74:in `new' from rubygems/requirement.rb:125:in `initialize' from rubygems/requirement.rb:125:in `uniq!' >>

Slide 80

Slide 80 text

80 $ ruby --disable-gems snowflake.rb

Slide 81

Slide 81 text

Move It! https://flic.kr/p/mAst2V

Slide 82

Slide 82 text

82 (gdb) list 4157 * 4158 */ 4159 4160 static VALUE 4161 rb_ary_uniq_bang(VALUE ary) 4162 { 4163 VALUE hash; 4164 long hash_size; 4165 4166 rb_ary_modify_check(ary); >>

Slide 83

Slide 83 text

83 (gdb) next 4166 rb_ary_modify_check(ary); >>

Slide 84

Slide 84 text

84 (gdb) step rb_ary_modify_check (ary=9382499768664) at array.c:312 312 rb_check_frozen(ary); >>

Slide 85

Slide 85 text

85 (gdb) step rb_ary_modify_check (ary=9382499768664) at array.c:312 312 rb_check_frozen(ary); (gdb) list 309 static inline void 310 rb_ary_modify_check(VALUE ary) 311 { 312 rb_check_frozen(ary); 313 } >>

Slide 86

Slide 86 text

86 (gdb) finish Run till exit from #0 rb_ary_modify_check (ary=9382500108300) at array.c:312 4167 if (RARRAY_LEN(ary) <= 1) (gdb) >>

Slide 87

Slide 87 text

87 (gdb) list 4164 long hash_size; 4165 4166 rb_ary_modify_check(ary); 4167 if (RARRAY_LEN(ary) <= 1) 4168 return Qnil; 4169 if (rb_block_given_p()) 4170 hash = ary_make_hash_by(ary); 4171 else 4172 hash = ary_make_hash(ary); 4173 >>

Slide 88

Slide 88 text

88 (gdb) list 4164 long hash_size; 4165 4166 rb_ary_modify_check(ary); 4167 if (RARRAY_LEN(ary) <= 1) 4168 return Qnil; 4169 if (rb_block_given_p()) 4170 hash = ary_make_hash_by(ary); 4171 else 4172 hash = ary_make_hash(ary); 4173 >>

Slide 89

Slide 89 text

89 (gdb) list 4170 hash = ary_make_hash_by(ary); 4171 else 4172 hash = ary_make_hash(ary); 4173 4174 hash_size = RHASH_SIZE(hash); 4175 if (RARRAY_LEN(ary) == hash_size) { 4176 return Qnil; 4177 } 4178 rb_ary_modify_check(ary); 4179 ARY_SET_LEN(ary, 0); >>

Slide 90

Slide 90 text

90 (gdb) continue Continuing. >>

Slide 91

Slide 91 text

91

Slide 92

Slide 92 text

https://flic.kr/p/o15Y5n 92 Movin’ next step finish list continue

Slide 93

Slide 93 text

Silly GDB Tricks https://flic.kr/p/dHofYV

Slide 94

Slide 94 text

94 (gdb) call (void)close(1) (gdb) call (void)close(2) (gdb) call (int)open(“gdb.log”, 2, 0) (gdb) call (int)open(“gdb.log”, 2, 0)

Slide 95

Slide 95 text

95 (gdb) call(rb_p(rb_eval_string_protect( "Thread.list.each do |t| puts t; puts t.backtrace.join(‘\n'); end",(int*)0))) # eval:1:in `backtrace' eval:1:in `block in ' eval:1:in `each' eval:1:in `' drop.rb:2:in `drop' drop.rb:2:in `' >>

Slide 96

Slide 96 text

96 (gdb) call(rb_p(rb_eval_string_protect( "Thread.list.each do |t| puts t; puts t.backtrace.join(‘\n'); end",(int*)0))) # eval:1:in `backtrace' eval:1:in `block in ' eval:1:in `each' eval:1:in `' drop.rb:2:in `drop' drop.rb:2:in `'

Slide 97

Slide 97 text

97 $ cat gdb_script attach 107 thread all apply backtrace call (void)rb_backtrace() quit >>

Slide 98

Slide 98 text

98 $ gdb --batch -x gdb_script ... Thread 2 (Thread 0x7f09a71d070 (LWP 108)): #0 0x00007f09a638e50d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f09a735d in timer_thread_sleep at thread_pthread.c:1442 #2 thread_timer (p=0x7f09a9466018) at thread_pthread.c:1549 #3 0x00007f09a6d9e0a4 in start_thread at pthread_create.c:309 Inferior 1 [process 10] will be detached. $ >>

Slide 99

Slide 99 text

99 $ gem install newrelic_rpm $ nrdebug 1234 Are you sure you want to attach to PID 1234 ('ruby -e loop do end')}? Extracting debug information from this process may cause it to hang, crash, or otherwise malfunction... To continue, type 'continue': Generated 'nrdebug-1234-1447787527.log'

Slide 100

Slide 100 text

100 $ cat ~/.gdbinit define exit quit end >>

Slide 101

Slide 101 text

Recap 101 https://flic.kr/p/rx2WTU

Slide 102

Slide 102 text

Deadlock? https://flic.kr/p/nkwswv

Slide 103

Slide 103 text

A Unique Problem 103 https://flic.kr/p/dPiDp3

Slide 104

Slide 104 text

Looking Around https://flic.kr/p/uq8MA

Slide 105

Slide 105 text

Move It! https://flic.kr/p/mAst2V

Slide 106

Slide 106 text

Silly GDB Tricks https://flic.kr/p/dHofYV

Slide 107

Slide 107 text

https://flic.kr/p/8RePxD ? @jasonrclark http://git.io/v8GbT https://flic.kr/p/ebUyxU