Tweet
Tweet

Slide 1

Slide 1 text

A COMMON TAXONOMY OF BUGS AND HOW TO SQUASH THEM

Slide 2

Slide 2 text

Instinctual Indications…6 Research Methods…9 Practical Taxonomy…13 Bohrbug…17 Schrödinbug…25 Fractalbug…24 Heisenbug…35 Mandelbug…44 Resources…55 Table of Contents

Slide 3

Slide 3 text

Debugging Skills

Slide 4

Slide 4 text

“As you familiarize yourself with the application, you’ll build up some debugging instincts"

Slide 5

Slide 5 text

“Whenever I see something like this happening, the first thing I do is scan the logs to see if this process is completing or is sending a weird message.”

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Debugging Instincts “ ”

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

“Whenever I see something like this happening, the first thing I do is scan the logs to see if this process is completing or is sending a weird message.”

Slide 11

Slide 11 text

“Whenever I see #{x}, I always check #{y}”

Slide 12

Slide 12 text

Instincts are just internalized rulesets

Slide 13

Slide 13 text

Research Methods • containment sometimes takes priority over squashing

Slide 14

Slide 14 text

Research Methods • containment sometimes takes priority over squashing • we can only work with facts

Slide 15

Slide 15 text

Research Methods • containment sometimes takes priority over squashing • we can only work with facts • we can’t squash every bug in this talk

Slide 16

Slide 16 text

Observable Attributes

Slide 17

Slide 17 text

Phenetics { ]

Slide 18

Slide 18 text

Warning: Contrived Scenarios Ahead

Slide 19

Slide 19 text

A COMMON TAXONOMY OF BUGS AND HOW TO SQUASH THEM

Slide 20

Slide 20 text

Taxonomy of Bugs Upsettingly Observable Wildly Chaotic {

Slide 21

Slide 21 text

Upsettingly Observable Bugs

Slide 22

Slide 22 text

Wildly Chaotic Bugs

Slide 23

Slide 23 text

How to Squash Them

Slide 24

Slide 24 text

upsettingly observable bug #1 UPSETTINGLY OBSERVABLE

Slide 25

Slide 25 text

Observable Attributes is the bug observable in production?

Slide 26

Slide 26 text

Observable Attributes is the bug observable in production? can it be reproduced locally?

Slide 27

Slide 27 text

Observable Attributes is the bug observable in production? can it be reproduced locally? does it seem to be restricted to one area?

Slide 28

Slide 28 text

Bohrbug deterministic, highly reproducible UPSETTINGLY OBSERVABLE

Slide 29

Slide 29 text

Bohrbug Commonly found in code,sometimes on server UPSETTINGLY OBSERVABLE

Slide 30

Slide 30 text

Bohrbug likes to hide in complex branching in functions, classes or config UPSETTINGLY OBSERVABLE

Slide 31

Slide 31 text

Bohrbug In the wild: validation UPSETTINGLY OBSERVABLE

Slide 32

Slide 32 text

Bohrbug UPSETTINGLY OBSERVABLE Easy to squash

Slide 33

Slide 33 text

Reproduction & Resolution replicate locally and in test UPSETTINGLY OBSERVABLE

Slide 34

Slide 34 text

Reproduction & Resolution replicate locally and in test write the simple solution UPSETTINGLY OBSERVABLE

Slide 35

Slide 35 text

Reproduction & Resolution replicate locally and in test write the simple solution rewrite to be highly readable and extendable UPSETTINGLY OBSERVABLE

Slide 36

Slide 36 text

Bohrbug UPSETTINGLY OBSERVABLE

Slide 37

Slide 37 text

Bohrbug UPSETTINGLY OBSERVABLE

Slide 38

Slide 38 text

upsettingly observable bug #2 UPSETTINGLY OBSERVABLE

Slide 39

Slide 39 text

Observable Attributes how does this work?

Slide 40

Slide 40 text

Observable Attributes how does this work? does this work?

Slide 41

Slide 41 text

Observable Attributes how does this work? does this work? wait, what is this even testing?

Slide 42

Slide 42 text

Observable Attributes how does this work? does this work? wait, what is this even testing? did this ever work?

Slide 43

Slide 43 text

Schrödinbug stick-like body appendages look like twigs UPSETTINGLY OBSERVABLE

Slide 44

Slide 44 text

Schrödinbug Likes to pretend to be working code. On close inspection, reveals itself to be a bug. UPSETTINGLY OBSERVABLE

Slide 45

Slide 45 text

Schrödinbug UPSETTINGLY OBSERVABLE Type I. Type II.

Slide 46

Slide 46 text

Schrödinbug Type I. Code that never worked UPSETTINGLY OBSERVABLE

Slide 47

Slide 47 text

Schrödinbug Type I. Reveal themselves via side-effects UPSETTINGLY OBSERVABLE

Slide 48

Slide 48 text

Schrödinbug Type I. In the wild: UI shows update but database entry not updated. UPSETTINGLY OBSERVABLE

Slide 49

Slide 49 text

Schrödinbug Type II. Code that doesn’t work how you thought UPSETTINGLY OBSERVABLE

Slide 50

Slide 50 text

Schrödinbug Type II. In the wild: Same function being called multiple times UPSETTINGLY OBSERVABLE

Slide 51

Slide 51 text

Basic Reproduction & Resolution replicate locally and in test write the simple solution rewrite to be highly readable and extendable UPSETTINGLY OBSERVABLE

Slide 52

Slide 52 text

-You “How can I reproduce this without knowing exactly what is happening?”

Slide 53

Slide 53 text

Logging as Verification Tool

Slide 54

Slide 54 text

Git Bisect Tool

Slide 55

Slide 55 text

Reproduction & Resolution reproduce the “broken” state locally and in test

Slide 56

Slide 56 text

Reproduction & Resolution reproduce the “broken” state locally and in test add log statements until you can verify what causes the broken state.

Slide 57

Slide 57 text

Reproduction & Resolution reproduce the “broken” state locally and in test add log statements until you can verify what causes the broken state. if the bug did work at some point, find the point at which it did work.

Slide 58

Slide 58 text

Reproduction & Resolution reproduce the “broken” state locally and in test add log statements until you can verify what causes the broken state. if the bug did work at some point, find the point at which it did work. write tests to represent the configuration and flow of the fixed state

Slide 59

Slide 59 text

Schrödinbug UPSETTINGLY OBSERVABLE

Slide 60

Slide 60 text

Schrödinbug UPSETTINGLY OBSERVABLE

Slide 61

Slide 61 text

wildly chaotic bug #1 WILDLY CHAOTIC

Slide 62

Slide 62 text

Observable Attributes Does it appear non-deterministic?

Slide 63

Slide 63 text

Observable Attributes Does it appear non-deterministic? Does it seem to disappear once you observe or debug it?

Slide 64

Slide 64 text

Heisenbug “now you see it, now you don’t” WILDLY CHAOTIC

Slide 65

Slide 65 text

Heisenbug WILDLY CHAOTIC Type I. Type II.

Slide 66

Slide 66 text

Heisenbug Type I. WILDLY CHAOTIC Lives in code

Slide 67

Slide 67 text

Heisenbug Type II. WILDLY CHAOTIC Lives in data

Slide 68

Slide 68 text

-You “How can I reproduce this without testing on production?”

Slide 69

Slide 69 text

Profiling for Verification https://kcachegrind.github.io/html/CallgrindFormat.html Tool

Slide 70

Slide 70 text

FLAME GRAPHS http://www.brendangregg.com/FlameGraphs/cpu-mysql-updated.svg Tool

Slide 71

Slide 71 text

Heisenbug Type I. WILDLY CHAOTIC Profiling can reveal what is being called and when

Slide 72

Slide 72 text

Heisenbug Type II. WILDLY CHAOTIC Profiling can reveal how much time is being spent.

Slide 73

Slide 73 text

Reproduction & Resolution use profiling to find the trigger state

Slide 74

Slide 74 text

Reproduction & Resolution use profiling to find the trigger state use the app (not fixtures or DB manipulation) to get the data in this state

Slide 75

Slide 75 text

Reproduction & Resolution use profiling to find the trigger state use the app (not fixtures or DB manipulation) to get the data in this state recreate that state in test

Slide 76

Slide 76 text

Reproduction & Resolution use profiling to find the trigger state use the app (not fixtures or DB manipulation) to get the data in this state recreate that state in test follow borhbug instruction

Slide 77

Slide 77 text

Heisenbug WILDLY CHAOTIC

Slide 78

Slide 78 text

Heisenbug WILDLY CHAOTIC

Slide 79

Slide 79 text

wildly chaotic bug #2 WILDLY CHAOTIC

Slide 80

Slide 80 text

Observable Attributes is everything broken?

Slide 81

Slide 81 text

Observable Attributes is everything broken? all of it?

Slide 82

Slide 82 text

Observable Attributes is everything broken? all of it? send help??

Slide 83

Slide 83 text

Mandelbug WILDLY CHAOTIC

Slide 84

Slide 84 text

Mandelbug seems like everything is broken at once WILDLY CHAOTIC

Slide 85

Slide 85 text

Mandelbug people are very upset with you WILDLY CHAOTIC

Slide 86

Slide 86 text

Mandelbug likely an issue with your system, not code WILDLY CHAOTIC

Slide 87

Slide 87 text

“The bug is huge and everywhere at once. SQL: could not connect to server: Connection refused was bubbling up all over the place. Jobs won’t run, emails won’t send, every submit button on the site fatal errored.” on-call log WILDLY CHAOTIC

Slide 88

Slide 88 text

Disk Usage Tool df -h

Slide 89

Slide 89 text

Logging as Verification Tool

Slide 90

Slide 90 text

Reproduction & Resolution use df -h to find if all the storage is being use

Slide 91

Slide 91 text

Reproduction & Resolution use df -h to find if all the storage is being use attempt to connect to server & view logs

Slide 92

Slide 92 text

Reproduction & Resolution use df -h to find if all the storage is being used attempt to connect to server & view logs can that be restarted, rotated or killed at this time?

Slide 93

Slide 93 text

Mandelbug WILDLY CHAOTIC

Slide 94

Slide 94 text

Mandelbug WILDLY CHAOTIC

Slide 95

Slide 95 text

A Practical Taxonomy of Bugs Upsettingly Observable Wildly Chaotic { bohrbug schrödinbug mandelbug heisenbug

Slide 96

Slide 96 text

“Debugging Instincts”

Slide 97

Slide 97 text

“Debugging Instincts”

Slide 98

Slide 98 text

Debugging Skills

Slide 99

Slide 99 text

Observe & Classify

Slide 100

Slide 100 text

Verify with logging and time travel

Slide 101

Slide 101 text

Verify without changing state by profiling

Slide 102

Slide 102 text

Use server tools to observe entire process

Slide 103

Slide 103 text

Observe & Classify Verify with logging and time travel Verify without changing state by profiling Use server tools to observe entire process

Slide 104

Slide 104 text

Build Up Your Own Toolkit and Share it

Slide 105

Slide 105 text

Resources & Further Study • “Linux Debugging Tools I Love”, Julia Evans • Systems Performance, Brendan Gregg • “Why Do Computers Stop and What Can Be Done About It?”, Jim Gray • “Debug Patterns for Efficient High- levelSystemC Debugging”, Frank Rogin, Erhard Fehlauer, Christian Haufe, Sebastian Ohnewald

Slide 106

Slide 106 text

No content