Escaping The Tar Pit at RubyConf 2019

Slide 1

Slide 1 text

Escaping The Tar Pit Ernesto Tagwerker (@etagwerker) RubyConf, November 2019

Slide 2

Slide 2 text

! Hi, I’m from Argentina I live in Philadelphia # I ❤ Open Source

Slide 3

Slide 3 text

Founder & Software Engineer @OmbuLabs

Slide 4

Slide 4 text

Founder & Software Engineer @fastrubyio

Slide 5

Slide 5 text

Open Source Maintainer database_cleaner; bundler-leak; next_rails

Slide 6

Slide 6 text

Inspiration

Slide 7

Slide 7 text

Chapter 1: The Tar Pit

Slide 8

Slide 8 text

“The ﬁercer the struggle, the more entangling the tar, and no beast is so strong or so skillful but that they ultimately sink.” Fred Brooks

Slide 9

Slide 9 text

“Large-system programming has over the past decade been such a tar pit, and many great and powerful beasts have thrashed violently in it.” Fred Brooks

Slide 10

Slide 10 text

We are in the tar pit or trying to avoid it

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

Tar Pit Symptoms > Projects running over budget

Slide 13

Slide 13 text

Tar Pit Symptoms > Projects running over budget > Taking forever to ship small changes

Slide 14

Slide 14 text

Tar Pit Symptoms > Projects running over budget > Taking forever to ship small changes > Sacriﬁcing quality, increasing tech debt

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

Part 1: How to avoid the tar pit

Slide 17

Slide 17 text

“THAT’D BE GREAT” BOSS: “IF YOU COULD COME IN ON THIS GREAT, LEGACY PROJECT AND MAINTAIN IT FROM NOW ON…”

Slide 18

Slide 18 text

“THAT’D BE GREAT” CLIENT: “IF YOU COULD COME IN ON THIS GREAT, LEGACY PROJECT AND MAINTAIN IT FROM NOW ON…”

Slide 19

Slide 19 text

CLIENT: “HOW LONG IS IT GOING TO TAKE?”

Slide 20

Slide 20 text

“BECAUSE I NEED IT BY END OF DAY TODAY” CLIENT: “HOW LONG IS IT GOING TO TAKE?”

Slide 21

Slide 21 text

How can we quickly assess code quality?

Slide 22

Slide 22 text

Part 2: How to get out of the tar pit

Slide 23

Slide 23 text

How can we gradually pay off technical debt?

Slide 24

Slide 24 text

Part 1: Avoiding the tar pit

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

“It can’t be that hard. It’s a Ruby project.”

Slide 27

Slide 27 text

How can we quickly assess code quality?

Slide 28

Slide 28 text

Assessing Quality ✅ Paid Services ✅ Open Source Gems

Slide 29

Slide 29 text

Assessing Quality ✅ Paid Services ➡ Open Source Gems

Slide 30

Slide 30 text

Open Source Gems ✅ Static code analysis

Slide 31

Slide 31 text

Open Source Gems ✅ Static code analysis ✅ Code coverage

Slide 32

Slide 32 text

Open Source Gems ✅ Static code analysis ✅ Code coverage ✅ Code smells

Slide 33

Slide 33 text

What is software quality anyway?

Slide 34

Slide 34 text

There are hundreds of deﬁnitions

Slide 35

Slide 35 text

“The degree to which a system, component, or process meets implicit and explicit requirements.” IEEE

Slide 36

Slide 36 text

“It works as expected.”

Slide 37

Slide 37 text

“It works as expected and it is not a PITA to maintain it.”

Slide 38

Slide 38 text

ISO 9126-1 Software Quality Model ✅ Reliability ✅ Usability ✅ Efﬁciency ✅ Maintainability ✅ Portability

Slide 39

Slide 39 text

ISO 9126-1 Software Quality Model ✅ Reliability ✅ Usability ✅ Efﬁciency ➡ Maintainability ✅ Portability

Slide 40

Slide 40 text

Maintainability

Slide 41

Slide 41 text

Maintainability 1. Code Coverage

Slide 42

Slide 42 text

Maintainability 1. Code Coverage 2. Code Quality

Slide 43

Slide 43 text

Code Coverage SimpleCov (github.com/colszowka/simplecov)

Slide 44

Slide 44 text

Code Coverage # Gemfile group :test do gem "simplecov", require: false end

Slide 45

Slide 45 text

Code Coverage # spec/spec_helper.rb if ENV["COVERAGE"] == "true" require 'simplecov' SimpleCov.start do add_group "Models", "models" add_filter "/spec/" track_files "**/*.rb" end end

Slide 46

Slide 46 text

Code Coverage $ COVERAGE=true rspec

Slide 47

Slide 47 text

Code Coverage

Slide 48

Slide 48 text

Code Quality n options

Slide 49

Slide 49 text

Code Quality n options (ﬂog; ﬂay; reek; churn; RubyCritic; MetricFu; attractor; rubocop; fukuzatsu; turbulence; …)

Slide 50

Slide 50 text

Code Quality RubyCritic (github.com/whitesmith/rubycritic)

Slide 51

Slide 51 text

Churn

Slide 52

Slide 52 text

Churn git

Slide 53

Slide 53 text

Churn $ git log --follow --format=%h default.rb | cat 573c8f5 2ccc5b8 b447bc3

Slide 54

Slide 54 text

Churn $ git log --follow --format=%h default.rb | cat 573c8f5 2ccc5b8 b447bc3 # Churn Count = 3

Slide 55

Slide 55 text

Complexity

Slide 56

Slide 56 text

Complexity flog

Slide 57

Slide 57 text

Complexity $ flog foo.rb => 10.8: flog total

Slide 58

Slide 58 text

Complexity class Foo def yay # 10.8 = a = eval "1+1" # 1.2 + 6.0 + if a == 2 # 1.2 + 1.2 + puts "yay" # 1.2 end end end

Slide 59

Slide 59 text

Complexity $ flog foo.rb => 10.8: flog total

Slide 60

Slide 60 text

Churn vs. Complexity

Slide 61

Slide 61 text

Churn Complexity 50 0 1 100_000

Slide 62

Slide 62 text

Churn Complexity 100 0 1 100_000 user.rb (changed: 2 times; complexity: 100_000 )

Slide 63

Slide 63 text

Churn Complexity 100 0 1 100_000 user.rb (changed: 27 times; complexity: 100)

Slide 64

Slide 64 text

Code Quality $ gem install rubycritic

Slide 65

Slide 65 text

Code Quality $ rubycritic

Slide 66

Slide 66 text

No content

Slide 67

Slide 67 text

GPA Pie Graph Complexity (flog) Code Smells (reek)

Slide 68

Slide 68 text

Churn vs. Complexity Churn (SCM) Complexity (flog)

Slide 69

Slide 69 text

100 0 1 100_000 Complexity Churn

Slide 70

Slide 70 text

100 0 1 100_000 Complexity Churn

Slide 71

Slide 71 text

100 0 1 100_000 “Welcome to the good place.” ❤ Complexity Churn

Slide 72

Slide 72 text

100 0 1 100_000 No one understands these ﬁles but they work. So don’t change them. Complexity Churn

Slide 73

Slide 73 text

100 0 1 100_000 “[…], if the code never changes, it's not costing us money.” Sandi Metz Complexity Churn

Slide 74

Slide 74 text

100 0 1 100_000 Everybody understands these ﬁles but you need to change them often Complexity Churn

Slide 75

Slide 75 text

100 0 1 100_000 Complexity Churn These modules are complex and change a lot… '

Slide 76

Slide 76 text

100 0 1 100_000 Complexity Churn “Sometimes a class becomes so complex that refactoring seems too difﬁcult.” Michael Feathers

Slide 77

Slide 77 text

Are you getting into a tar pit, is it a dumpster ﬁre, or have you found a project which is easy to maintain?

Slide 78

Slide 78 text

100 0 1 100_000 Welcome to the tar pit. Complexity Churn

Slide 79

Slide 79 text

Maintainability 1. Code Coverage (SimpleCov) 2. Code Quality (RubyCritic)

Slide 80

Slide 80 text

Signal #1 Code Coverage

Slide 81

Slide 81 text

Signal #2 Complexity

Slide 82

Slide 82 text

Coverage & Complexity

Slide 83

Slide 83 text

But what if we could calculate a new signal that combines both?

Slide 84

Slide 84 text

Signal #3 StinkScore

Slide 85

Slide 85 text

StinkScore = f(code_quality, code_coverage)

Slide 86

Slide 86 text

StinkScore = f(code_smells, code_complexity, code_coverage)

Slide 87

Slide 87 text

Files which lack tests should be penalized

Slide 88

Slide 88 text

foo.rb complexity: 10 smells: 10 smell points: 100 (10*10)

Slide 89

Slide 89 text

bar.rb complexity: 10 smells: 10 smell points: 100 (10*10)

Slide 90

Slide 90 text

foo.rb complexity: 10 smells: 10 smell points: 100 (10*10) bar.rb complexity: 10 smells: 10 smell points: 100 (10*10)

Slide 91

Slide 91 text

foo.rb and bar.rb are equally stinky

Slide 92

Slide 92 text

foo.rb complexity: 10 smells: 10 code_coverage: 0 stink_score: 10,000 (10*10)*100

Slide 93

Slide 93 text

bar.rb complexity: 10 smells: 10 code_coverage: 100 stink_score: 100 (10*10)*1

Slide 94

Slide 94 text

foo.rb complexity: 10 smells: 10 code_coverage: 0 stink_score: 10,000 (10*10)*100 bar.rb complexity: 10 smells: 10 code_coverage: 100 stink_score: 100 (10*10)*1

Slide 95

Slide 95 text

foo.rb is considerably stinkier than bar.rb

Slide 96

Slide 96 text

How can we automate this analysis?

Slide 97

Slide 97 text

Skunk A Stink Score Calculator (github.com/fastruby/skunk)

Slide 98

Slide 98 text

Skunk $ gem install skunk

Slide 99

Slide 99 text

Skunk $ skunk

Slide 100

Slide 100 text

No content

Slide 101

Slide 101 text

No content

Slide 102

Slide 102 text

Skunk $ skunk ... StinkScore Total: 13231.069999999996 Modules Analysed: 71 StinkScore Average: 186.3530985915493 Worst StinkScore: 2401.75 (lib/rubycritic/source_control_systems/git.rb)

Slide 103

Slide 103 text

Skunk $ skunk ... StinkScore Total: 13231.069999999996 Modules Analysed: 71 StinkScore Average: 186.3530985915493 Worst StinkScore: 2401.75 (lib/rubycritic/source_control_systems/git.rb)

Slide 104

Slide 104 text

RubyCritic module RubyCritic class AnalysedModule def cost smells.map(&:cost).inject(0.0, :+) + (complexity / COMPLEXITY_FACTOR) end end end

Slide 105

Slide 105 text

RubyCritic module RubyCritic class AnalysedModule def cost smells.map(&:cost).inject(0.0, :+) + # From Reek (complexity / COMPLEXITY_FACTOR) end end end

Slide 106

Slide 106 text

RubyCritic module RubyCritic class AnalysedModule def cost smells.map(&:cost).inject(0.0, :+) + # From Reek (complexity / COMPLEXITY_FACTOR) # From Flog end end end

Slide 107

Slide 107 text

Skunk module RubyCritic class AnalysedModule def stink_score return cost if perfect_coverage? cost * (PERFECT_COVERAGE - coverage.to_i) end end end

Slide 108

Slide 108 text

Skunk module RubyCritic class AnalysedModule def stink_score return cost if perfect_coverage? cost * (PERFECT_COVERAGE - coverage.to_i) # Penalty Factor = 100% - 20% => 80 end end end

Slide 109

Slide 109 text

Skunk Warning > skunk-v0.3.1

Slide 110

Slide 110 text

Skunk Warning > skunk-v0.3.1 > COVERAGE=TRUE rake test

Slide 111

Slide 111 text

Skunk Warning > skunk-v0.3.1 > COVERAGE=TRUE rake test > feedback wanted

Slide 112

Slide 112 text

Maintainability 1. Code Coverage (SimpleCov) 2. Code Quality (RubyCritic) 3. StinkScore (Skunk)

Slide 113

Slide 113 text

You are here:

Slide 114

Slide 114 text

Part 2: Getting out of the tar pit

Slide 115

Slide 115 text

So, where do we begin?

Slide 116

Slide 116 text

removing files will decrease our stink_score

Slide 117

Slide 117 text

Find Dead Code Coverband (github.com/danmayer/coverband)

Slide 118

Slide 118 text

No content

Slide 119

Slide 119 text

refactoring complex files will decrease our stink_score

Slide 120

Slide 120 text

100 0 1 100_000 Complexity Churn “Here be dragons. These modules are complex and change a lot…” '

Slide 121

Slide 121 text

100 0% 1 100_000 Complexity Churn “Great candidates for refactoring (paying technical debt!)” #

Slide 122

Slide 122 text

Churn vs. Complexity is not enough to prioritize

Slide 123

Slide 123 text

Stink Score Table

Slide 124

Slide 124 text

Refactor These Files:

Slide 125

Slide 125 text

Refactor These Files:

Slide 126

Slide 126 text

100 0 1 100_000 Complexity Churn git.rb

Slide 127

Slide 127 text

100 0 1 100_000 Complexity Churn git_calculator.rb git_fetcher.rb refactor! git.rb

Slide 128

Slide 128 text

100 0 1 100_000 Complexity Churn git_calculator.rb git_fetcher.rb

Slide 129

Slide 129 text

Skunk $ skunk -b master Base branch (master) average stink score: 79.78 Feature branch (refactor/git) average stink score: 71.79

Slide 130

Slide 130 text

Skunk $ skunk -b master Base branch (master) average stink score: 79.78 Feature branch (refactor/git) average stink score: 71.79 10% better ( ( (

Slide 131

Slide 131 text

writing tests for our app will decrease our stink_score

Slide 132

Slide 132 text

Stink Score Table

Slide 133

Slide 133 text

Write Tests Here:

Slide 134

Slide 134 text

Skunk* $ skunk -b master Base branch (master) average stink score: 79.78 Feature branch (refactor/git) average stink score: 71.79

Slide 135

Slide 135 text

Time StinkScore Average 100 (days) 0 1 100

Slide 136

Slide 136 text

stink_score Code Coverage; Complexity; and Smells

Slide 137

Slide 137 text

stink_score Your compass to get out of the tar pit

Slide 138

Slide 138 text

Thank you! @etagwerker 138 Thank you! ‣ github.com/fastruby/skunk ‣ @OmbuLabs ‣ @fastrubyio ‣ @etagwerker

Slide 139

Slide 139 text

Resources 1. https://github.com/whitesmith/rubycritic 2.https://github.com/colszowka/simplecov 3.https://github.com/metricfu/metric_fu 4.https://github.com/julianrubisch/attractor 5.https://www.fastruby.io/blog/ruby/quality/code-quality-ruby-gems.html 6.https://www.reddit.com/r/ruby/comments/2bq092/rubycritic/ 7.http://jakescruggs.blogspot.com/2008/08/whats-good-ﬂog-score.html 8.http://ruby.sadi.st/Flog.html 9.http://ruby.sadi.st/Flay.html 10.https://github.com/makaroni4/sandi_meter 11.https://www.fastruby.io/blog/code-quality/intruducing-skunk-stink-score- calculator.html

Slide 140

Slide 140 text

Resources 1.https://codeclimate.com/blog/deciphering-ruby-code-metrics/ 2.https://www.stickyminds.com/article/getting-empirical-about-refactoring 3.https://www.sandimetz.com/blog/2017/9/13/breaking-up-the-behemoth 4.https://github.com/troessner/reek 5.https://github.com/seattlerb/ﬂay 6.https://github.com/seattlerb/ﬂog 7.http://www.sqa.net/iso9126.html 8.https://www.slideshare.net/mscottford/important-metrics-for-measuring-code-health 9.https://dilbert.com/strip/2006-12-08 10.https://www.fastruby.io/blog/ruby/quality/code-quality-ruby-gems.html 11.https://www.fastruby.io/blog/code-quality/code-coverage/rubycritic-4-2-0- simplecov-support.html