Escaping The Tar Pit at NYC.rb

Slide 1

Slide 1 text

Escaping The Tar Pit Ernesto Tagwerker (@etagwerker) NYC.rb, May 13th, 2020

Slide 2

Slide 2 text

Hi, I’m from Argentina I live in Philadelphia I ❤ Open Source

Slide 3

Slide 3 text

Open Source Maintainer database_cleaner; bundler-leak; next_rails

Slide 4

Slide 4 text

Founder & Software Engineer OmbuLabs

Slide 5

Slide 5 text

Founder & Software Engineer FastRuby.io

Slide 6

Slide 6 text

Inspiration

Slide 7

Slide 7 text

Chapter 1: The Tar Pit

Slide 8

Slide 8 text

“Large-system programming has over the past decade been such a tar pit, and many great and powerful beasts have thrashed violently in it.” Fred Brooks

Slide 9

Slide 9 text

“The ﬁercer the struggle, the more entangling the tar, and no beast is so strong or so skillful but that they ultimately sink.” Fred Brooks

Slide 10

Slide 10 text

We are in the tar pit or trying to avoid it

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

Tar Pit Symptoms > Projects running over budget

Slide 13

Slide 13 text

Tar Pit Symptoms > Projects running over budget > Taking forever to ship small changes

Slide 14

Slide 14 text

Tar Pit Symptoms > Projects running over budget > Taking forever to ship small changes > Sacriﬁcing quality, increasing tech debt

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

Part 1: How to avoid the tar pit

Slide 17

Slide 17 text

Part 2: How to get out of the tar pit

Slide 18

Slide 18 text

Part 1: Avoiding the tar pit

Slide 19

Slide 19 text

“THAT’D BE GREAT” BOSS: “IF YOU COULD COME IN ON THIS GREAT, LEGACY PROJECT AND MAINTAIN IT FROM NOW ON…”

Slide 20

Slide 20 text

“THAT’D BE GREAT” CLIENT: “IF YOU COULD COME IN ON THIS GREAT, LEGACY PROJECT AND MAINTAIN IT FROM NOW ON…”

Slide 21

Slide 21 text

CLIENT: “HOW LONG IS IT GOING TO TAKE?”

Slide 22

Slide 22 text

“BECAUSE I NEED IT BY END OF DAY TODAY” CLIENT: “HOW LONG IS IT GOING TO TAKE?”

Slide 23

Slide 23 text

How can we quickly assess code quality?

Slide 24

Slide 24 text

Assessing Quality ✅ Paid Services ✅ Open Source Gems

Slide 25

Slide 25 text

Assessing Quality ✅ Paid Services ➡ Open Source Gems

Slide 26

Slide 26 text

Open Source Gems ✅ Static code analysis

Slide 27

Slide 27 text

Open Source Gems ✅ Static code analysis ✅ Code coverage

Slide 28

Slide 28 text

Open Source Gems ✅ Static code analysis ✅ Code coverage ✅ Code smells

Slide 29

Slide 29 text

What is software quality anyway?

Slide 30

Slide 30 text

There are hundreds of deﬁnitions

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

You can deﬁne your own baseline of “software quality”

Slide 34

Slide 34 text

Slide 35

Slide 35 text

“The degree to which a system, component, or process meets implicit and explicit requirements.” IEEE

Slide 36

Slide 36 text

“It works as expected.”

Slide 37

Slide 37 text

“It works as expected and it is not a PITA to maintain it.”

Slide 38

Slide 38 text

ISO 9126-1 Software Quality Model ✅ Reliability ✅ Usability ✅ Efﬁciency ✅ Maintainability ✅ Portability

Slide 39

Slide 39 text

ISO 9126-1 Software Quality Model ✅ Reliability ✅ Usability ✅ Efﬁciency ➡ Maintainability ✅ Portability

Slide 40

Slide 40 text

Maintainability

Slide 41

Slide 41 text

Maintainability 1. Code Coverage

Slide 42

Slide 42 text

Maintainability 1. Code Coverage 2. Code Quality

Slide 43

Slide 43 text

Code Coverage SimpleCov (github.com/colszowka/simplecov)

Slide 44

Slide 44 text

Code Coverage # Gemfile group :test do gem "simplecov", require: false end

Slide 45

Slide 45 text

Code Coverage # spec/spec_helper.rb if ENV["COVERAGE"] == "true" require 'simplecov' SimpleCov.start do add_group "Models", "models" add_filter "/spec/" track_files "**/*.rb" end end

Slide 46

Slide 46 text

Code Coverage $ COVERAGE=true rake test

Slide 47

Slide 47 text

Code Coverage

Slide 48

Slide 48 text

Code Quality n options

Slide 49

Slide 49 text

Code Quality n options (ﬂog; ﬂay; reek; churn; RubyCritic; MetricFu; attractor; rubocop; fukuzatsu; turbulence; …)

Slide 50

Slide 50 text

Code Quality RubyCritic (github.com/whitesmith/rubycritic)

Slide 51

Slide 51 text

Churn

Slide 52

Slide 52 text

Churn git

Slide 53

Slide 53 text

Churn $ git log --follow --format=%h default.rb | cat 573c8f5 2ccc5b8 b447bc3

Slide 54

Slide 54 text

Churn $ git log --follow --format=%h default.rb | cat 573c8f5 2ccc5b8 b447bc3 # Churn Count = 3

Slide 55

Slide 55 text

Complexity

Slide 56

Slide 56 text

Complexity flog

Slide 57

Slide 57 text

Complexity $ flog default.rb => 10.8: flog total

Slide 58

Slide 58 text

Complexity class Default def yay # 10.8 = a = eval "1+1" # 1.2 + 6.0 + if a == 2 # 1.2 + 1.2 + puts "yay" # 1.2 end end end

Slide 59

Slide 59 text

Complexity $ flog user.rb => 10.8: flog total

Slide 60

Slide 60 text

Churn vs. Complexity

Slide 61

Slide 61 text

Churn Complexity 100 0 1 100_000 user.rb (changed: 2 times; complexity: 100_000 )

Slide 62

Slide 62 text

Churn Complexity 100 0 1 100_000 user.rb (changed: 27 times; complexity: 100)

Slide 63

Slide 63 text

Code Quality $ gem install rubycritic

Slide 64

Slide 64 text

Code Quality $ rubycritic

Slide 65

Slide 65 text

No content

Slide 66

Slide 66 text

No content

Slide 67

Slide 67 text

GPA Pie Graph Complexity (flog) Code Smells (reek)

Slide 68

Slide 68 text

Churn vs. Complexity Churn (SCM) Complexity (flog)

Slide 69

Slide 69 text

100 0 1 100_000 Complexity Churn

Slide 70

Slide 70 text

100 0 1 100_000 Complexity Churn

Slide 71

Slide 71 text

100 0 1 100_000 Complexity Churn

Slide 72

Slide 72 text

100 0 1 100_000 “Welcome to the good place.” ❤ Complexity Churn

Slide 73

Slide 73 text

100 0 1 100_000 No one understands these ﬁles but they work. So don’t change them. Complexity Churn

Slide 74

Slide 74 text

100 0 1 100_000 “[…], if the code never changes, it's not costing us money.” Sandi Metz Complexity Churn

Slide 75

Slide 75 text

100 0 1 100_000 Everybody understands these ﬁles but you need to change them often Complexity Churn

Slide 76

Slide 76 text

100 0 1 100_000 Complexity Churn These modules are complex and are constantly changing

Slide 77

Slide 77 text

100 0 1 100_000 Complexity Churn “Sometimes a class becomes so complex that refactoring seems too difﬁcult.” Michael Feathers

Slide 78

Slide 78 text

Are you getting into a tar pit, is it a dumpster ﬁre, or have you found a project which is easy to maintain?

Slide 79

Slide 79 text

100 0 1 100_000 Welcome to the tar pit. Complexity Churn

Slide 80

Slide 80 text

Maintainability 1. Code Coverage (SimpleCov) 2. Code Quality (RubyCritic)

Slide 81

Slide 81 text

Signal #1 Code Coverage

Slide 82

Slide 82 text

Signal #2 Complexity

Slide 83

Slide 83 text

Coverage & Complexity

Slide 84

Slide 84 text

But what if we could calculate a new signal that combines both?

Slide 85

Slide 85 text

Signal #3 SkunkScore

Slide 86

Slide 86 text

SkunkScore = f(code_quality, code_coverage)

Slide 87

Slide 87 text

Complex files which lack tests should be our top priority

Slide 88

Slide 88 text

user.rb complexity: 10 smells: 10 smell points: 100 (10*10)

Slide 89

Slide 89 text

shop.rb complexity: 10 smells: 10 smell points: 100 (10*10)

Slide 90

Slide 90 text

user.rb complexity: 10 smells: 10 smell points: 100 (10*10) shop.rb complexity: 10 smells: 10 smell points: 100 (10*10)

Slide 91

Slide 91 text

user.rb and shop.rb are equally stinky*

Slide 92

Slide 92 text

user.rb complexity: 10 smells: 10 code_coverage: 0 skunk_score: 10,000 (10*10)*100

Slide 93

Slide 93 text

shop.rb complexity: 10 smells: 10 code_coverage: 100 skunk_score: 100 (10*10)*1

Slide 94

Slide 94 text

user.rb complexity: 10 smells: 10 code_coverage: 0 skunk_score: 10,000 (10*10)*100 shop.rb complexity: 10 smells: 10 code_coverage: 100 skunk_score: 100 (10*10)*1

Slide 95

Slide 95 text

shop.rb smells considerably better than user.rb

Slide 96

Slide 96 text

user.rb complexity: 10 smells: 10 code_coverage: 0 skunk_score: 10,000 (10*10)*100 We should write some tests for user.rb shop.rb complexity: 10 smells: 10 code_coverage: 100 skunk_score: 100 (10*10)*1

Slide 97

Slide 97 text

How can we automate this analysis?

Slide 98

Slide 98 text

A Skunk Score Calculator (github.com/fastruby/skunk)

Slide 99

Slide 99 text

Skunk $ gem install skunk

Slide 100

Slide 100 text

Skunk $ skunk

Slide 101

Slide 101 text

No content

Slide 102

Slide 102 text

Skunk $ skunk ... SkunkScore Total: 13231.069999999996 Modules Analysed: 71 SkunkScore Average: 186.3530985915493 Worst SkunkScore: 2401.75 (lib/rubycritic/source_control_systems/git.rb)

Slide 103

Slide 103 text

Skunk $ skunk ... SkunkScore Total: 13231.069999999996 Modules Analysed: 71 SkunkScore Average: 186.3530985915493 Worst SkunkScore: 2401.75 (lib/rubycritic/source_control_systems/git.rb)

Slide 104

Slide 104 text

RubyCritic module RubyCritic class AnalysedModule def cost smells.map(&:cost).inject(0.0, :+) + (complexity / COMPLEXITY_FACTOR) end end end

Slide 105

Slide 105 text

RubyCritic module RubyCritic class AnalysedModule def cost smells.map(&:cost).inject(0.0, :+) + # From Reek (complexity / COMPLEXITY_FACTOR) end end end

Slide 106

Slide 106 text

RubyCritic module RubyCritic class AnalysedModule def cost smells.map(&:cost).inject(0.0, :+) + # From Reek (complexity / COMPLEXITY_FACTOR) # From Flog end end end

Slide 107

Slide 107 text

Skunk module RubyCritic class AnalysedModule def skunk_score return cost if perfect_coverage? cost * (PERFECT_COVERAGE - coverage.to_i) end end end

Slide 108

Slide 108 text

Skunk module RubyCritic class AnalysedModule def skunk_score return cost if perfect_coverage? cost * (PERFECT_COVERAGE - coverage.to_i) # Penalty Factor = 100% - 20% => 80 end end end

Slide 109

Slide 109 text

Maintainability 1. Code Coverage (SimpleCov) 2. Code Quality (RubyCritic) 3. SkunkScore (Skunk)

Slide 110

Slide 110 text

You are here:

Slide 111

Slide 111 text

Part 2: Getting out of the tar pit

Slide 112

Slide 112 text

How can we gradually pay off technical debt?

Slide 113

Slide 113 text

1. Removing files 2. Refactoring files 3. Increasing coverage

Slide 114

Slide 114 text

1. Removing files

Slide 115

Slide 115 text

Find Dead Code Coverband (github.com/danmayer/coverband)

Slide 116

Slide 116 text

No content

Slide 117

Slide 117 text

2.Refactoring files

Slide 118

Slide 118 text

100 0 1 100_000 Complexity Churn

Slide 119

Slide 119 text

100 0 1 100_000 Complexity Churn “Great candidates for refactoring (paying technical debt!)”

Slide 120

Slide 120 text

Churn vs. Complexity is not enough to prioritize

Slide 121

Slide 121 text

Skunk Score Table

Slide 122

Slide 122 text

Refactor These Files:

Slide 123

Slide 123 text

Refactor These Files:

Slide 124

Slide 124 text

100 0 1 100_000 Complexity Churn git.rb

Slide 125

Slide 125 text

100 0 1 100_000 Complexity Churn git_calculator.rb git_fetcher.rb refactor! git.rb

Slide 126

Slide 126 text

Skunk $ skunk -b master Base branch (master) average skunk score: 79.78 Feature branch (refactor/git) average skunk score: 71.79

Slide 127

Slide 127 text

Skunk $ skunk -b master Base branch (master) average skunk score: 79.78 Feature branch (refactor/git) average skunk score: 71.79 10% better

Slide 128

Slide 128 text

2. Asdf 3. Increasing coverage

Slide 129

Slide 129 text

Skunk Score Table

Slide 130

Slide 130 text

Write Tests Here:

Slide 131

Slide 131 text

Maintainability Code Coverage; Complexity; and Smells

Slide 132

Slide 132 text

SkunkScore A new way to communicate about tech debt

Slide 133

Slide 133 text

“I paid off some tech debt with this PR.”

Slide 134

Slide 134 text

“I paid off 5% of our tech debt with this PR.”

Slide 135

Slide 135 text

“This project is in pretty good shape.”

Slide 136

Slide 136 text

“This project has a SkunkScore average of 120.”

Slide 137

Slide 137 text

Skunk Score - Open Source - Configurable

Slide 138

Slide 138 text

Skunk Score Your compass to get out of the tar pit

Slide 139

Slide 139 text

Time SkunkScore Average 100 (days) 0 1 100

Slide 140

Slide 140 text

Thank you! @etagwerker 140 ‣ github.com/fastruby/skunk ‣ @OmbuLabs ‣ @fastrubyio ‣ @etagwerker

Slide 141

Slide 141 text

No content

Slide 142

Slide 142 text

Thank you! @etagwerker 142 Thank you!

Slide 143

Slide 143 text

Resources 1. https://github.com/whitesmith/rubycritic 2.https://github.com/colszowka/simplecov 3.https://github.com/metricfu/metric_fu 4.https://github.com/julianrubisch/attractor 5.https://www.fastruby.io/blog/ruby/quality/code-quality-ruby-gems.html 6.https://www.reddit.com/r/ruby/comments/2bq092/rubycritic/ 7.http://jakescruggs.blogspot.com/2008/08/whats-good-ﬂog-score.html 8.http://ruby.sadi.st/Flog.html 9.http://ruby.sadi.st/Flay.html 10.https://github.com/makaroni4/sandi_meter 11.https://www.fastruby.io/blog/code-quality/intruducing-skunk-stink-score-calculator.html 12.https://github.com/fastruby/skunk

Slide 144

Slide 144 text

Resources 1.https://codeclimate.com/blog/deciphering-ruby-code-metrics/ 2.https://www.stickyminds.com/article/getting-empirical-about-refactoring 3.https://www.sandimetz.com/blog/2017/9/13/breaking-up-the-behemoth 4.https://github.com/troessner/reek 5.https://github.com/seattlerb/ﬂay 6.https://github.com/seattlerb/ﬂog 7.http://www.sqa.net/iso9126.html 8.https://www.slideshare.net/mscottford/important-metrics-for-measuring-code-health 9.https://dilbert.com/strip/2006-12-08 10.https://www.fastruby.io/blog/ruby/quality/code-quality-ruby-gems.html 11.https://www.fastruby.io/blog/code-quality/code-coverage/rubycritic-4-2-0-simplecov- support.html 12.https://www.fastruby.io/blog/code-quality/churn-vs-complexity-vs-coverage.html