A Practical Taxonomy of Bugs and How to Squash Them-Keep Ruby Weird 2016

Slide 1

Slide 1 text

A PRACTICAL TAXONOMY OF BUGS AND HOW TO SQUASH THEM

Slide 2

Slide 2 text

Instinctual Indications…2 Research Methods…9 Practical Taxonomy…12 Bohrbug…13 Schrödinbug…20 Fractalbug…24 Heisenbug…30 Mandelbug…36 Common Patterns…43 Resources…50 Table of Contents

Slide 3

Slide 3 text

Debugging Skills

Slide 4

Slide 4 text

“As you familiarize yourself with the application, you’ll build up some debugging instincts"

Slide 5

Slide 5 text

“Whenever I see something like this happening, the first thing I do is scan the logs to see if this process is completing or is sending a weird message.”

Slide 6

Slide 6 text

Debugging Instincts “ ”

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

“Whenever I see #{x}, I always check #{y}”

Slide 9

Slide 9 text

Research Methods • containment sometimes takes priority over squashing • we can only work with facts • we can’t squash every bug in this talk

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

Observable Attributes

Slide 12

Slide 12 text

Warning: Contrived Scenarios Ahead

Slide 13

Slide 13 text

A Practical Taxonomy of Bugs Upsettingly Observable Wildly Chaotic {

Slide 14

Slide 14 text

How to Squash Them

Slide 15

Slide 15 text

upsettingly observable bug #1 UPSETTINGLY OBSERVABLE

Slide 16

Slide 16 text

Observable Attributes is the bug observable in production? can it be reproduced locally? does it seem to be restricted to one area?

Slide 17

Slide 17 text

Bohrbug deterministic, highly reproducible UPSETTINGLY OBSERVABLE

Slide 18

Slide 18 text

Bohrbug Commonly found in code,sometimes on server UPSETTINGLY OBSERVABLE

Slide 19

Slide 19 text

Bohrbug likes to hide in complex branching in functions, classes or config UPSETTINGLY OBSERVABLE

Slide 20

Slide 20 text

Bohrbug In the wild: validation UPSETTINGLY OBSERVABLE

Slide 21

Slide 21 text

Reproduction & Resolution replicate locally and in test write the simple solution rewrite to be highly readable and extendable UPSETTINGLY OBSERVABLE

Slide 22

Slide 22 text

Bohrbug UPSETTINGLY OBSERVABLE

Slide 23

Slide 23 text

Bohrbug UPSETTINGLY OBSERVABLE

Slide 24

Slide 24 text

upsettingly observable bug #2 UPSETTINGLY OBSERVABLE

Slide 25

Slide 25 text

Observable Attributes how does this work? does this work? wait, what is this even testing? did this ever work?

Slide 26

Slide 26 text

Schrödinbug stick-like body appendages look like twigs UPSETTINGLY OBSERVABLE

Slide 27

Slide 27 text

Schrödinbug Likes to pretend to be working code. On close inspection, reveals itself to be a bug. UPSETTINGLY OBSERVABLE

Slide 28

Slide 28 text

Schrödinbug In the wild: code that never worked UPSETTINGLY OBSERVABLE

Slide 29

Slide 29 text

Schrödinbug In the wild: it didn’t work how you thought it did UPSETTINGLY OBSERVABLE

Slide 30

Slide 30 text

Logging as Verification Tool

Slide 31

Slide 31 text

Git Bisect Tool

Slide 32

Slide 32 text

Reproduction & Resolution reproduce the “broken” state locally and in test add log statements until you can verify what causes the broken state. if the bug did work at some point, find the point at which it did work. write tests to represent the configuration and flow of the fixed state

Slide 33

Slide 33 text

Schrödinbug UPSETTINGLY OBSERVABLE

Slide 34

Slide 34 text

Schrödinbug UPSETTINGLY OBSERVABLE

Slide 35

Slide 35 text

wildly chaotic bug #1 WILDLY CHAOTIC

Slide 36

Slide 36 text

Observable Attributes Does it appear non-deterministic? Does it seem to disappear once you observe or debug it?

Slide 37

Slide 37 text

Heisenbug “now you see it, now you don’t” WILDLY CHAOTIC

Slide 38

Slide 38 text

Heisenbug WILDLY CHAOTIC In the wild: a heisenbug that lives in code

Slide 39

Slide 39 text

Heisenbug WILDLY CHAOTIC In the wild: a heisenbug that lives in data

Slide 40

Slide 40 text

Profiling for Verification https://kcachegrind.github.io/html/CallgrindFormat.html Tool

Slide 41

Slide 41 text

FLAME GRAPHS http://www.brendangregg.com/FlameGraphs/cpu-mysql-updated.svg Tool

Slide 42

Slide 42 text

Reproduction & Resolution use profiling to find the trigger state use the app (not fixtures or DB manipulation) to get the data in this state recreate that state in test follow borhbug instruction

Slide 43

Slide 43 text

Heisenbug WILDLY CHAOTIC

Slide 44

Slide 44 text

Heisenbug WILDLY CHAOTIC

Slide 45

Slide 45 text

wildly chaotic bug #2 WILDLY CHAOTIC

Slide 46

Slide 46 text

Observable Attributes is everything broken? all of it? send help??

Slide 47

Slide 47 text

Mandelbug WILDLY CHAOTIC

Slide 48

Slide 48 text

Mandelbug seems like everything is broken at once WILDLY CHAOTIC

Slide 49

Slide 49 text

Mandelbug people are very upset with you WILDLY CHAOTIC

Slide 50

Slide 50 text

Mandelbug likely an issue with your system, not code WILDLY CHAOTIC

Slide 51

Slide 51 text

“The bug is huge and everywhere at once. SQL: could not connect to server: Connection refused was bubbling up all over the place. Jobs won’t run, emails won’t send, every submit button on the site fatal errored.” on-call log 24 June 2014 WILDLY CHAOTIC

Slide 52

Slide 52 text

Disk Usage Tool df -h

Slide 53

Slide 53 text

Reproduction & Resolution attempt to connect to server & view logs use df -h to find if all the storage is being used can that be restarted, rotated or killed at this time?

Slide 54

Slide 54 text

Mandelbug WILDLY CHAOTIC

Slide 55

Slide 55 text

Mandelbug WILDLY CHAOTIC

Slide 56

Slide 56 text

A Practical Taxonomy of Bugs Upsettingly Observable Wildly Chaotic { bohrbug schrödinbug mandelbug heisenbug

Slide 57

Slide 57 text

“Debugging Instincts”

Slide 58

Slide 58 text

“Debugging Instincts”

Slide 59

Slide 59 text

No content

Slide 60

Slide 60 text

Resources • “Linux Debugging Tools I Love”, Julia Evans • Systems Performance, Brendan Gregg • “Why Do Computers Stop and What Can Be Done About It?”, Jim Gray • “Debug Patterns for Efficient High- levelSystemC Debugging”, Frank Rogin, Erhard Fehlauer, Christian Haufe, Sebastian Ohnewald