Slide 1

Slide 1 text

Memory Leaks 2015 @schneems

Slide 2

Slide 2 text

Leaks are bad

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Obviously memory leaks are bad

Slide 8

Slide 8 text

Why?

Slide 9

Slide 9 text

First…

Slide 10

Slide 10 text

They Call me @Schneems

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

“Favor composition over inheritance

Slide 14

Slide 14 text

Golang rulz Elixr 4 lyfe

Slide 15

Slide 15 text

Rails Commit Bit

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

I like working with wood

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

Plan for seasonal wood movement “

Slide 23

Slide 23 text

Why is memory important?

Slide 24

Slide 24 text

RAM is a limited resource

Slide 25

Slide 25 text

When you run out, the OS starts using the HD

Slide 26

Slide 26 text

Hard Drive

Slide 27

Slide 27 text

RAM

Slide 28

Slide 28 text

swap

Slide 29

Slide 29 text

When you run out of room…

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

Performance Profiling

Slide 33

Slide 33 text

Start with a known

Slide 34

Slide 34 text

Repro the performance problem

Slide 35

Slide 35 text

#include void f(void) { int* x = malloc(10 *sizeof(int)); x[10] = 0; } int main(void) { f(); return 0; }

Slide 36

Slide 36 text

#include void f(void) { int* x = malloc(10 *sizeof(int)); x[10] = 0; } int main(void) { f(); return 0; }

Slide 37

Slide 37 text

Find the right tools

Slide 38

Slide 38 text

$ valgrind --tool=memcheck ./memoryleak ==31747== Memcheck, a memory error detector ==31747== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. ==31747== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info ==31747== Command: ./memoryleak ==31747== ==31747== Invalid write of size 4 ==31747== at 0x100000F4C: f (memoryleak.c:6) ==31747== by 0x100000F73: main (memoryleak.c:10) ==31747== Address 0x100ae34a8 is 0 bytes after a block of size 40 alloc'd ==31747== at 0x100008EBB: malloc (in /usr/local/Cellar/valgrind/3.11.0/lib/ valgrind/vgpreload_memcheck-amd64-darwin.so) ==31747== by 0x100000F43: f (memoryleak.c:5) ==31747== by 0x100000F73: main (memoryleak.c:10) ==31747== ==31747== ==31747== HEAP SUMMARY: ==31747== in use at exit: 35,304 bytes in 431 blocks ==31747== total heap usage: 503 allocs, 72 frees, 41,272 bytes allocated ==31747== ==31747== LEAK SUMMARY: ==31747== definitely lost: 40 bytes in 1 blocks ==31747== indirectly lost: 0 bytes in 0 blocks ==31747== possibly lost: 0 bytes in 0 blocks ==31747== still reachable: 0 bytes in 0 blocks ==31747== suppressed: 35,264 bytes in 430 blocks ==31747== Rerun with --leak-check=full to see details of leaked memory ==31747== ==31747== For counts of detected and suppressed errors, rerun with: -v ==31747== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

Slide 39

Slide 39 text

$ valgrind --tool=memcheck ./memoryleak ==31747== Memcheck, a memory error detector ==31747== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. ==31747== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info ==31747== Command: ./memoryleak ==31747== ==31747== Invalid write of size 4 ==31747== at 0x100000F4C: f (memoryleak.c:6) ==31747== by 0x100000F73: main (memoryleak.c:10) ==31747== Address 0x100ae34a8 is 0 bytes after a block of size 40 alloc'd ==31747== at 0x100008EBB: malloc (in /usr/local/Cellar/valgrind/3.11.0/lib/ valgrind/vgpreload_memcheck-amd64-darwin.so) ==31747== by 0x100000F43: f (memoryleak.c:5) ==31747== by 0x100000F73: main (memoryleak.c:10) ==31747== ==31747== ==31747== HEAP SUMMARY: ==31747== in use at exit: 35,304 bytes in 431 blocks ==31747== total heap usage: 503 allocs, 72 frees, 41,272 bytes allocated ==31747== ==31747== LEAK SUMMARY: ==31747== definitely lost: 40 bytes in 1 blocks ==31747== indirectly lost: 0 bytes in 0 blocks ==31747== possibly lost: 0 bytes in 0 blocks ==31747== still reachable: 0 bytes in 0 blocks ==31747== suppressed: 35,264 bytes in 430 blocks ==31747== Rerun with --leak-check=full to see details of leaked memory ==31747== ==31747== For counts of detected and suppressed errors, rerun with: -v ==31747== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

Slide 40

Slide 40 text

$ valgrind --tool=memcheck ./memoryleak ==31747== Memcheck, a memory error detector ==31747== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. ==31747== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info ==31747== Command: ./memoryleak ==31747== ==31747== Invalid write of size 4 ==31747== at 0x100000F4C: f (memoryleak.c:6) ==31747== by 0x100000F73: main (memoryleak.c:10) ==31747== Address 0x100ae34a8 is 0 bytes after a block of size 40 alloc'd ==31747== at 0x100008EBB: malloc (in /usr/local/Cellar/valgrind/3.11.0/lib/ valgrind/vgpreload_memcheck-amd64-darwin.so) ==31747== by 0x100000F43: f (memoryleak.c:5) ==31747== by 0x100000F73: main (memoryleak.c:10) ==31747== ==31747== ==31747== HEAP SUMMARY: ==31747== in use at exit: 35,304 bytes in 431 blocks ==31747== total heap usage: 503 allocs, 72 frees, 41,272 bytes allocated ==31747== ==31747== LEAK SUMMARY: ==31747== definitely lost: 40 bytes in 1 blocks ==31747== indirectly lost: 0 bytes in 0 blocks ==31747== possibly lost: 0 bytes in 0 blocks ==31747== still reachable: 0 bytes in 0 blocks ==31747== suppressed: 35,264 bytes in 430 blocks ==31747== Rerun with --leak-check=full to see details of leaked memory ==31747== ==31747== For counts of detected and suppressed errors, rerun with: -v ==31747== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

Slide 41

Slide 41 text

Apply the tools to your App

Slide 42

Slide 42 text

How does Ruby Memory?

Slide 43

Slide 43 text

Ruby is a “managed” Language

Slide 44

Slide 44 text

Don’t manually allocate, use Garbage Collector instead

Slide 45

Slide 45 text

2 causes for increased memory use

Slide 46

Slide 46 text

Retained Objects 1)

Slide 47

Slide 47 text

RETAINED = [] 1.upto(100_000_000).each do |i| RETAINED << “#{i}” end

Slide 48

Slide 48 text

7.32 GB

Slide 49

Slide 49 text

WHY?

Slide 50

Slide 50 text

Memory

Slide 51

Slide 51 text

Memory Object slots

Slide 52

Slide 52 text

Memory “1” “2” “3”“4” “5” “6” “7” RETAINED = [] 1.upto(100_000_000).each do |i| RETAINED << “#{i}” end

Slide 53

Slide 53 text

Out of slots! Allocate more memory

Slide 54

Slide 54 text

Memory “1” “2” “3”“4” “5” “6” “7” “10” “8” “9” “11” “12” “13” “14”

Slide 55

Slide 55 text

Allocated Objects 2)

Slide 56

Slide 56 text

1.upto(100_000_000).each do |i| puts "#{i}" end

Slide 57

Slide 57 text

21.8 MB

Slide 58

Slide 58 text

Memory “1” “2” “3”“4” “5” “6” “7”

Slide 59

Slide 59 text

Out of slots! Run GC!

Slide 60

Slide 60 text

Memory “2” “3”“4” “5” “6” “7” 1.upto(100_000_000).each do |i| puts "#{i}" end “1”

Slide 61

Slide 61 text

Memory “2” “3”“4” “5” “6” “7” 1.upto(100_000_000).each do |i| puts "#{i}" end

Slide 62

Slide 62 text

Memory “8” “3”“4” “5” “6” “7” 1.upto(100_000_000).each do |i| puts "#{i}" end “2”

Slide 63

Slide 63 text

GC will save us!

Slide 64

Slide 64 text

def make_objects array = [] 1.upto(100_000_000).each do |i| array << "#{i}" end return nil end make_objects GC.start(full_mark: true, immediate_sweep: true)

Slide 65

Slide 65 text

def make_objects array = [] 1.upto(100_000_000).each do |i| array << "#{i}" end return nil end make_objects GC.start(full_mark: true, immediate_sweep: true)

Slide 66

Slide 66 text

def make_objects array = [] 1.upto(100_000_000).each do |i| array << "#{i}" end return nil end make_objects GC.start(full_mark: true, immediate_sweep: true)

Slide 67

Slide 67 text

Result?

Slide 68

Slide 68 text

7.32 GB

Slide 69

Slide 69 text

WAT

Slide 70

Slide 70 text

GC.start(full_mark: true, immediate_sweep: true) GC.disable # Code goes here before = GC.stat[:total_freed_objects] GC.enable GC.start(full_mark: true, immediate_sweep: true) after = GC.stat[:total_freed_objects] # => “Objects cleared by GC: 100_010_687”

Slide 71

Slide 71 text

Cannot open slots while reference exists

Slide 72

Slide 72 text

After program runs

Slide 73

Slide 73 text

Memory “1” “2” “3”“4” “5” “6” “7” “10” “8” “9” “11” “12” “13” “14”

Slide 74

Slide 74 text

Run GC

Slide 75

Slide 75 text

Memory “1” “2” “3”“4” “5” “6” “7” “10” “8” “9” “11” “12” “13” “14”

Slide 76

Slide 76 text

Ruby releases memory SLOWLY

Slide 77

Slide 77 text

Memory

Slide 78

Slide 78 text

Test Everything (even these slides)

Slide 79

Slide 79 text

How does this affect my Rails app?

Slide 80

Slide 80 text

Retained App

Slide 81

Slide 81 text

Retained Allocated App

Slide 82

Slide 82 text

Mem Usage App

Slide 83

Slide 83 text

Request Starts

Slide 84

Slide 84 text

Request Full .

Slide 85

Slide 85 text

Request Full Full ..

Slide 86

Slide 86 text

Request Full Full Full ...

Slide 87

Slide 87 text

Request Full Full Full Full ....

Slide 88

Slide 88 text

Request Full Full Full Full Full ......

Slide 89

Slide 89 text

Out of memory Full Full Full Full Full

Slide 90

Slide 90 text

Run GC, All objs needed 0_o Full Full Full Full Full

Slide 91

Slide 91 text

Allocate mem Full Full Full Full Full

Slide 92

Slide 92 text

Request Done Full Full Full Full Full ........

Slide 93

Slide 93 text

Eventually

Slide 94

Slide 94 text

GC Objs not used Full Full Full Full Full

Slide 95

Slide 95 text

GC Objs not used

Slide 96

Slide 96 text

Memory increases even without object retention

Slide 97

Slide 97 text

Object creation/ deletion takes RAM

Slide 98

Slide 98 text

Object creation/ deletion takes Time

Slide 99

Slide 99 text

Further Reading:

Slide 100

Slide 100 text

Generational GC

Slide 101

Slide 101 text

What is a memory leak?

Slide 102

Slide 102 text

What is a memory leak?

Slide 103

Slide 103 text

#include void f(void) { int* x = malloc(10 *sizeof(int)); x[10] = 0; } int main(void) { f(); return 0; }

Slide 104

Slide 104 text

We lost the reference

Slide 105

Slide 105 text

We can never free() that memory

Slide 106

Slide 106 text

RETAINED = [] 1.upto(100_000_000).each do |i| RETAINED << “#{i}” end

Slide 107

Slide 107 text

We retained the reference

Slide 108

Slide 108 text

We can never GC that memory

Slide 109

Slide 109 text

Apply the tools to your App

Slide 110

Slide 110 text

Do you have a memory leak?

Slide 111

Slide 111 text

No content

Slide 112

Slide 112 text

No content

Slide 113

Slide 113 text

Ruby memory should plateau

Slide 114

Slide 114 text

Do you have a memory leak?

Slide 115

Slide 115 text

No

Slide 116

Slide 116 text

Memory Bloat

Slide 117

Slide 117 text

Real life is messy

Slide 118

Slide 118 text

Easiest way to stop a memory bloat ???

Slide 119

Slide 119 text

No content

Slide 120

Slide 120 text

Puma Worker Killer

Slide 121

Slide 121 text

Rolling Restarts

Slide 122

Slide 122 text

Memory Triggered Restarts

Slide 123

Slide 123 text

Doesn’t work with containers

Slide 124

Slide 124 text

Puma Worker Killer

Slide 125

Slide 125 text

Concurrency via processes

Slide 126

Slide 126 text

Reduce “worker” count to reduce RAM

Slide 127

Slide 127 text

Hardest way to stop memory bloat ???

Slide 128

Slide 128 text

Debugging and Introspection

Slide 129

Slide 129 text

Who has heard of derailed?

Slide 130

Slide 130 text

derailed_benchmarks

Slide 131

Slide 131 text

Memory at Boot

Slide 132

Slide 132 text

$ bundle exec derailed bundle:mem TOP: 54.1836 MiB mail: 18.9688 MiB mime/types: 17.4453 MiB mail/field: 0.4023 MiB mail/message: 0.3906 MiB action_view/view_paths: 0.4453 MiB action_view/base: 0.4336 MiB

Slide 133

Slide 133 text

Memory at Runtime

Slide 134

Slide 134 text

$ bundle exec derailed exec perf:objects # ... ## allocated memory by location --------------------------------------- 1935548 /Users/richardschneeman/.gem/ruby/2.2.3/gems/actionpack-4.2.3/lib/action_dispa 896100 /Users/richardschneeman/.gem/ruby/2.2.3/gems/pg-0.16.0/lib/pg/result.rb:10 741488 /Users/richardschneeman/.gem/ruby/2.2.3/gems/activerecord-4.2.3/lib/active_rec 689299 /Users/richardschneeman/.gem/ruby/2.2.3/gems/activesupport-4.2.3/lib/active_su 660672 /Users/richardschneeman/.gem/ruby/2.2.3/gems/actionpack-4.2.3/lib/action_dispa 606384 /Users/richardschneeman/Documents/projects/codetriage/app/views/repos/\_repo.h 579384 /Users/richardschneeman/.gem/ruby/2.2.3/gems/activesupport-4.2.3/lib/active\_s 532800 /Users/richardschneeman/.gem/ruby/2.2.3/gems/actionpack-4.2.3/lib/action_dispa 391392 /Users/richardschneeman/.gem/ruby/2.2.3/gems/activerecord-4.2.3/lib/active_rec 385920 /Users/richardschneeman/.gem/ruby/2.2.3/gems/temple-0.7.5/lib/temple/utils.rb:

Slide 135

Slide 135 text

Case Study

Slide 136

Slide 136 text

No content

Slide 137

Slide 137 text

No content

Slide 138

Slide 138 text

No content

Slide 139

Slide 139 text

$ heroku pg:pull

Slide 140

Slide 140 text

$ PATH_TO_HIT=rails/rails bundle exec derailed exec perf:objects Booting: production Running 1 times Total allocated: 21495556 bytes (207735 objects) Total retained: 15041423 bytes (133220 objects) allocated memory by gem ----------------------------------- 19427636 activerecord-4.2.3 1471380 activesupport-4.2.3 214048 actionview-4.2.3 151066 actionpack-4.2.3 121983 codetriage/app 24308 will_paginate-3.0.7 23193 rack-1.6.4 21105 arel-6.0.3 13397 ruby-2.2.3/lib 5560 warden-1.2.3 5536 rack-timeout-0.3.2 3912 activemodel-4.2.3 3000 hashie-3.4.2 2154 devise-3.5.2 1573 omniauth-1.2.2 1536 i18n-0.7.0 1495 railties-4.2.3 787 temple-0.7.6

Slide 141

Slide 141 text

19427636 bytes…

Slide 142

Slide 142 text

19.4 MB

Slide 143

Slide 143 text

Per

Slide 144

Slide 144 text

Request

Slide 145

Slide 145 text

allocated memory by location ----------------------------------- 5131624 /Users/richardschneeman/.gem/ruby/2.2.3/gems/activerec 4616480 /Users/richardschneeman/.gem/ruby/2.2.3/gems/activerec 1336408 /Users/richardschneeman/.gem/ruby/2.2.3/gems/activerec 1336088 /Users/richardschneeman/.gem/ruby/2.2.3/gems/activerec 1330904 /Users/richardschneeman/.gem/ruby/2.2.3/gems/activerec 1325648 /Users/richardschneeman/.gem/ruby/2.2.3/gems/activerec 921807 /Users/richardschneeman/.gem/ruby/2.2.3/gems/activesup 737376 /Users/richardschneeman/.gem/ruby/2.2.3/gems/activerec 596288 /Users/richardschneeman/.gem/ruby/2.2.3/gems/activerec 460720 /Users/richardschneeman/.gem/ruby/2.2.3/gems/activerec

Slide 146

Slide 146 text

???

Slide 147

Slide 147 text

allocated objects by class ----------------------------------- 108939 String 35456 Hash 21605 Array 5890 Proc 5886 RubyVM::Env 5859 ActiveRecord::Attribute::FromDatabase 5759 ActiveRecord::AttributeSet 5759 ActiveRecord::LazyAttributeHash 5731 ActiveRecord::Associations::BelongsToAssociation 5731 Issue 217 ActiveSupport::SafeBuffer 173 MatchData 112 ActionView::OutputBuffer 59 Time 43 ActiveRecord::Relation 42 ActiveRecord::ConnectionAdapters::PostgreSQL::Name 27 ActiveRecord::AssociationRelation 27 User }

Slide 148

Slide 148 text

allocated objects by class ----------------------------------- 108939 String 35456 Hash 21605 Array 5890 Proc 5886 RubyVM::Env 5859 ActiveRecord::Attribute::FromDatabase 5759 ActiveRecord::AttributeSet 5759 ActiveRecord::LazyAttributeHash 5731 ActiveRecord::Associations::BelongsToAssociation 5731 Issue 217 ActiveSupport::SafeBuffer 173 MatchData 112 ActionView::OutputBuffer 59 Time 43 ActiveRecord::Relation 42 ActiveRecord::ConnectionAdapters::PostgreSQL::Name 27 ActiveRecord::AssociationRelation 27 User }

Slide 149

Slide 149 text

allocated objects by class ----------------------------------- 108939 String 35456 Hash 21605 Array 5890 Proc 5886 RubyVM::Env 5859 ActiveRecord::Attribute::FromDatabase 5759 ActiveRecord::AttributeSet 5759 ActiveRecord::LazyAttributeHash 5731 ActiveRecord::Associations::BelongsToAssociation 5731 Issue 217 ActiveSupport::SafeBuffer 173 MatchData 112 ActionView::OutputBuffer 59 Time 43 ActiveRecord::Relation 42 ActiveRecord::ConnectionAdapters::PostgreSQL::Name 27 ActiveRecord::AssociationRelation 27 User

Slide 150

Slide 150 text

27 Users }

Slide 151

Slide 151 text

allocated objects by class ----------------------------------- 108939 String 35456 Hash 21605 Array 5890 Proc 5886 RubyVM::Env 5859 ActiveRecord::Attribute::FromDatabase 5759 ActiveRecord::AttributeSet 5759 ActiveRecord::LazyAttributeHash 5731 ActiveRecord::Associations::BelongsToAssociation 5731 Issue 217 ActiveSupport::SafeBuffer 173 MatchData 112 ActionView::OutputBuffer 59 Time 43 ActiveRecord::Relation 42 ActiveRecord::ConnectionAdapters::PostgreSQL::Name 27 ActiveRecord::AssociationRelation 27 User

Slide 152

Slide 152 text

NOT 5731 Issues }

Slide 153

Slide 153 text

No content

Slide 154

Slide 154 text

Started GET "/rails/rails" for ::1 at 2015-11-04 14:54:55 -0600 ActiveRecord::SchemaMigration Load (19.2ms) SELECT "schema_migrations".* FROM "schema_migration Processing by ReposController#show as HTML Parameters: {"full_name"=>"rails/rails"} Repo Load (21.2ms) SELECT "repos".* FROM "repos" WHERE "repos"."full_name" = $1 LIMIT 1 [["fu source=rack-timeout id=22a6fea79c2f050a9d9999b007298d12 timeout=30000ms service=1006ms state=activ Issue Load (1091.1ms) SELECT "issues".* FROM "issues" WHERE "issues"."repo_id" IN (36) source=rack-timeout id=22a6fea79c2f050a9d9999b007298d12 timeout=30000ms service=2093ms state=activ source=rack-timeout id=22a6fea79c2f050a9d9999b007298d12 timeout=30000ms service=3101ms state=activ source=rack-timeout id=22a6fea79c2f050a9d9999b007298d12 timeout=30000ms service=4027ms state=activ (48.1ms) SELECT COUNT(*) FROM "users" INNER JOIN "repo_subscriptions" ON "users"."id" = "repo_ User Load (53.4ms) SELECT "users".* FROM "users" INNER JOIN "repo_subscriptions" ON "users"."i Rendered subscribers/_avatars.html.slim (138.9ms) CACHE (0.0ms) SELECT COUNT(*) FROM "users" INNER JOIN "repo_subscriptions" ON "users"."id" = "r (0.6ms) SELECT COUNT(count_column) FROM (SELECT 1 AS count_column FROM "users" INNER JOIN "re 36]] Issue Load (41.8ms) SELECT "issues".* FROM "issues" WHERE "issues"."repo_id" = $1 AND "issues" (44.2ms) SELECT COUNT(*) FROM "issues" WHERE "issues"."repo_id" = $1 AND "issues"."state" = $2 Rendered repos/show.html.slim within layouts/application (496.5ms) source=rack-timeout id=22a6fea79c2f050a9d9999b007298d12 timeout=30000ms service=5005ms state=activ Rendered application/_head.html.slim (1360.4ms) Rendered application/_flashes.html.slim (15.9ms) Rendered application/_logo.html.slim (21.5ms) Rendered application/_nav.html.slim (73.0ms) Rendered application/_thoughtbot.html.slim (11.5ms) Rendered application/_footer.html.slim (36.4ms) Completed 200 OK in 5760ms (Views: 1889.1ms | ActiveRecord: 1323.2ms) source=rack-timeout id=22a6fea79c2f050a9d9999b007298d12 timeout=30000ms service=6004ms state=activ

Slide 155

Slide 155 text

Started GET "/rails/rails" for ::1 at 2015-11-04 14:54:55 -0600 ActiveRecord::SchemaMigration Load (19.2ms) SELECT "schema_migrations".* FROM "schema_migration Processing by ReposController#show as HTML Parameters: {"full_name"=>"rails/rails"} Repo Load (21.2ms) SELECT "repos".* FROM "repos" WHERE "repos"."full_name" = $1 LIMIT 1 [["fu source=rack-timeout id=22a6fea79c2f050a9d9999b007298d12 timeout=30000ms service=1006ms state=activ Issue Load (1091.1ms) SELECT "issues".* FROM "issues" WHERE "issues"."repo_id" IN (36) source=rack-timeout id=22a6fea79c2f050a9d9999b007298d12 timeout=30000ms service=2093ms state=activ source=rack-timeout id=22a6fea79c2f050a9d9999b007298d12 timeout=30000ms service=3101ms state=activ source=rack-timeout id=22a6fea79c2f050a9d9999b007298d12 timeout=30000ms service=4027ms state=activ (48.1ms) SELECT COUNT(*) FROM "users" INNER JOIN "repo_subscriptions" ON "users"."id" = "repo_ User Load (53.4ms) SELECT "users".* FROM "users" INNER JOIN "repo_subscriptions" ON "users"."i Rendered subscribers/_avatars.html.slim (138.9ms) CACHE (0.0ms) SELECT COUNT(*) FROM "users" INNER JOIN "repo_subscriptions" ON "users"."id" = "r (0.6ms) SELECT COUNT(count_column) FROM (SELECT 1 AS count_column FROM "users" INNER JOIN "re 36]] Issue Load (41.8ms) SELECT "issues".* FROM "issues" WHERE "issues"."repo_id" = $1 AND "issues" (44.2ms) SELECT COUNT(*) FROM "issues" WHERE "issues"."repo_id" = $1 AND "issues"."state" = $2 Rendered repos/show.html.slim within layouts/application (496.5ms) source=rack-timeout id=22a6fea79c2f050a9d9999b007298d12 timeout=30000ms service=5005ms state=activ Rendered application/_head.html.slim (1360.4ms) Rendered application/_flashes.html.slim (15.9ms) Rendered application/_logo.html.slim (21.5ms) Rendered application/_nav.html.slim (73.0ms) Rendered application/_thoughtbot.html.slim (11.5ms) Rendered application/_footer.html.slim (36.4ms) Completed 200 OK in 5760ms (Views: 1889.1ms | ActiveRecord: 1323.2ms) source=rack-timeout id=22a6fea79c2f050a9d9999b007298d12 timeout=30000ms service=6004ms state=activ

Slide 156

Slide 156 text

Loading EVERY issue for rails/rails repo

Slide 157

Slide 157 text

No content

Slide 158

Slide 158 text

def self.find_by_full_name(full_name) Repo.includes(:issues).find_by!(full_name: full_name) end

Slide 159

Slide 159 text

def self.find_by_full_name(full_name) Repo.includes(:issues).find_by!(full_name: full_name) end

Slide 160

Slide 160 text

Screenshot 2015-10-22 09.39.20

Slide 161

Slide 161 text

Not a memory leak

Slide 162

Slide 162 text

Objects were getting Collected via GC

Slide 163

Slide 163 text

Puma Threads

Slide 164

Slide 164 text

Idle

Slide 165

Slide 165 text

Now Total

Slide 166

Slide 166 text

Requests

Slide 167

Slide 167 text

rails/rail

Slide 168

Slide 168 text

Now Total

Slide 169

Slide 169 text

saltstack/salt

Slide 170

Slide 170 text

Now Total

Slide 171

Slide 171 text

idle

Slide 172

Slide 172 text

Now Total

Slide 173

Slide 173 text

rustlang/rust

Slide 174

Slide 174 text

golang/go

Slide 175

Slide 175 text

isacs/npm

Slide 176

Slide 176 text

Now Total

Slide 177

Slide 177 text

Idle

Slide 178

Slide 178 text

Now Total

Slide 179

Slide 179 text

Was that necessary?

Slide 180

Slide 180 text

No content

Slide 181

Slide 181 text

/rails/rails Unused Eager Loading detected Repo => [:issues] Remove from your finder: :includes => [:issues]

Slide 182

Slide 182 text

No

Slide 183

Slide 183 text

What if there was another cause?

Slide 184

Slide 184 text

Proof that memory tooling provides results

Slide 185

Slide 185 text

My
 Secret

Slide 186

Slide 186 text

https://www.dropbox.com/s/vyd7teot0pg349q/Screenshot%202015-11-05%2011.34.04.png?dl=0

Slide 187

Slide 187 text

$ bundle gem derailed_benchmarks

Slide 188

Slide 188 text

mime-types boot

Slide 189

Slide 189 text

https://www.dropbox.com/s/wdabatbn0crbgw2/Screenshot%202015-11-05%2011.54.12.png?dl=0

Slide 190

Slide 190 text

Thanks @jeremyevans

Slide 191

Slide 191 text

gem “mime-types”, “~> 2.6.1”, require: “mime/types/columnar”

Slide 192

Slide 192 text

https://www.dropbox.com/s/56bw1mewpm7rc65/Screenshot%202015-11-05%2011.38.15.png?dl=0 Omniauth 28% faster

Slide 193

Slide 193 text

https://www.dropbox.com/s/0n72bg81s0rsfdh/Screenshot%202015-11-05%2011.45.58.png?dl=0 36% smaller require memory

Slide 194

Slide 194 text

gem “mail”, “~> 2.6.2”

Slide 195

Slide 195 text

https://www.dropbox.com/s/7j0m6zxl27yl7e7/Screenshot%202015-11-05%2011.47.07.png?dl=0 10% faster requests in Rails

Slide 196

Slide 196 text

1+ years of building and using tools

Slide 197

Slide 197 text

I made results however

Slide 198

Slide 198 text

All these changes

Slide 199

Slide 199 text

No content

Slide 200

Slide 200 text

Now I can reproduce the a basic memory problem at will

Slide 201

Slide 201 text

What did I learn from 1+ years of PRs?

Slide 202

Slide 202 text

Our community has almost zero knowledge of how our programs use memory

Slide 203

Slide 203 text

Most apps are slow due to app code, not framework

Slide 204

Slide 204 text

Your Apps

Slide 205

Slide 205 text

Monitor Production Memory

Slide 206

Slide 206 text

https://www.dropbox.com/s/tzxhepl8o8gizw0/Screenshot%202015-11-05%2014.46.06.png?dl=0

Slide 207

Slide 207 text

Benchmark!

Slide 208

Slide 208 text

$ gem install derailed

Slide 209

Slide 209 text

Puma Worker Killer

Slide 210

Slide 210 text

$ gem install bullet

Slide 211

Slide 211 text

The Future

Slide 212

Slide 212 text

I want a memory tool as easy to use as Ruby

Slide 213

Slide 213 text

Using Heap Dumps to take Production snapshots

Slide 214

Slide 214 text

Better heap analysis

Slide 215

Slide 215 text

Getting this to work with Heroku

Slide 216

Slide 216 text

Can we make this better?

Slide 217

Slide 217 text

allocated memory by location ----------------------------------- 5131624 active_record/result.rb:116 1336408 active_record/attribute_set/builder.rb:30 1336088 active_record/persistence.rb:69 1330904 active_record/core.rb:547 1325648 active_record/attribute_methods.rb:359 921807 active_support/callbacks.rb:81 737376 active_record/core.rb:114 596288 active_record/associations.rb:162 460720 active_record/attribute_set/builder.rb:16

Slide 218

Slide 218 text

Stack information in heap dumps ???

Slide 219

Slide 219 text

What do other languages do ???

Slide 220

Slide 220 text

https://www.dropbox.com/s/rzs0owrqq3rja96/Screenshot%202015-11-05%2014.32.52.png?dl=0

Slide 221

Slide 221 text

https://www.dropbox.com/s/k0j31exhh2ir9oj/Screenshot%202015-11-05%2015.12.49.png?dl=0

Slide 222

Slide 222 text

Can we leverage existing tooling for C?

Slide 223

Slide 223 text

Valgrind ?

Slide 224

Slide 224 text

GDB ?

Slide 225

Slide 225 text

Visual debuggers?

Slide 226

Slide 226 text

Further Reading

Slide 227

Slide 227 text

No content

Slide 228

Slide 228 text

No content

Slide 229

Slide 229 text

No content

Slide 230

Slide 230 text

One last thing

Slide 231

Slide 231 text

No content

Slide 232

Slide 232 text

PSA: Sprockets is now owned by Rails

Slide 233

Slide 233 text

Refactors on the Horizon

Slide 234

Slide 234 text

21 commits makes me #4 contributor

Slide 235

Slide 235 text

Let’s save sprockets

Slide 236

Slide 236 text

No content

Slide 237

Slide 237 text

No content

Slide 238

Slide 238 text

No content