RubyMem: The Leaky Gems Database for Bundler at Ruby Kaigi Takeout 2020

by Ernesto Tagwerker

Slide 1

Slide 1 text

RubyMem: The Leaky Gems Database for Bundler Ernesto Tagwerker (@etagwerker) Ruby Kaigi, 2020

Slide 2

Slide 2 text

Hi, I’m from Argentina

Slide 3

Slide 3 text

Hi, I’m from Argentina I live in Philadelphia

Slide 4

Slide 4 text

Hi, I’m from Argentina I live in Philadelphia I <3 drawing with my daughter

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

Hi, I’m from Argentina I live in Philadelphia I <3 drawing with my daughter I <3 Open Source

Slide 7

Slide 7 text

Open Source Maintainer database_cleaner

Slide 8

Slide 8 text

Open Source Maintainer database_cleaner next_rails

Slide 9

Slide 9 text

Open Source Maintainer database_cleaner next_rails skunk

Slide 10

Slide 10 text

Founder & Software Engineer @OmbuLabs

Slide 11

Slide 11 text

We are hiring! OmbuLabs.com/jobs

Slide 12

Slide 12 text

Founder & Software Engineer @fastrubyio

Slide 13

Slide 13 text

Memory Issues

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

RAM Issues

Slide 17

Slide 17 text

Story Time

Slide 18

Slide 18 text

Hey, can you help me with my project?

Slide 19

Slide 19 text

Sure! How can I help?

Slide 20

Slide 20 text

Can you make my gif search engine faster?

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

Sure! I can do that with a simple cache!

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

I’m going to write my cache in Ruby with a Hash. Simple.

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

Just shipped a bunch of changes which included my cache-ing solution.

Slide 28

Slide 28 text

We bumped the versions of `roda`, `rest-client`, `puma`, and we upgraded the heroku stack.

Slide 29

Slide 29 text

Good job, Ernesto! That was quick and it is much faster now. ❤

Slide 30

Slide 30 text

(A few hours go by and users start seeing an error in production.)

Slide 31

Slide 31 text

I can haz no laser cat gifs!

Slide 32

Slide 32 text

Weird. I’m going to check the logs.

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

Error R14? Memory Quota Exceeded?

Slide 35

Slide 35 text

Hey, I noticed this weird behavior in our production’s memory graph

Slide 36

Slide 36 text

Deployment

Slide 37

Slide 37 text

Deployment Out of memory

Slide 38

Slide 38 text

That looks bad. We ran out of memory overnight.

Slide 39

Slide 39 text

But my code was so small. No way it’s my code’s fault.

Slide 40

Slide 40 text

It’s probably Rails.

Slide 41

Slide 41 text

It’s probably Rails Roda.

Slide 42

Slide 42 text

What’s more likely? That Roda has a memory bug or that your code has a memory bug?

Slide 43

Slide 43 text

Maybe you should check the changelog for our latest deployment?

Slide 44

Slide 44 text

Good idea! I’m gonna start debugging.

Slide 45

Slide 45 text

What tools can I use?

Slide 46

Slide 46 text

I’m going to use `ObjectSpace.each_object` to count the amount of objects by object type.

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

I’m still not sure, I’m going to use `rack-mini-proﬁler` and `memory_proﬁler`

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

&pp=proﬁle-memory

Slide 53

Slide 53 text

&pp=proﬁle-memory

Slide 54

Slide 54 text

(After a few hours I realize that my code is the root cause.)

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

Ooops! That cache causes “memory bloat”

Slide 57

Slide 57 text

I’m going to revert my change and deploy.

Slide 58

Slide 58 text

Deployment

Slide 59

Slide 59 text

Great! What was the root cause of that memory leak?

Slide 60

Slide 60 text

My cache could end up using inﬁnite memory but we only have 512MB in Heroku.

Slide 61

Slide 61 text

Why did you take so long to ﬁx it?

Slide 62

Slide 62 text

I didn’t know whether the leak was in my application code or my dependencies.

Slide 63

Slide 63 text

Lessons Learned

Slide 64

Slide 64 text

Lessons Learned - Anyone can introduce a memory bloat

Slide 65

Slide 65 text

Lessons Learned - Anyone can introduce a memory bloat - It’s easy to introduce it and hard to ﬁnd

Slide 66

Slide 66 text

Lessons Learned - Anyone can introduce a memory bloat - It’s easy to introduce it and hard to ﬁnd - Great proﬁling tools in Ruby ❤

Slide 67

Slide 67 text

If only there was a way to quickly answer this question:

Slide 68

Slide 68 text

Introducing bundler-leak

Slide 69

Slide 69 text

13 group :development do 14 gem "bundler-leak" 15 end

Slide 70

Slide 70 text

$ bundle leak No leaks found

Slide 71

Slide 71 text

Neat! None of my dependencies are known to be leaking memory. I’m going to debug my application code.

Slide 72

Slide 72 text

$ bundle leak Name: net-http-persistent Version: 3.0.0 URL: https://github.com/drbrain/net-http- persistent/pull/98 Title: Memory leak in thread connection pool of net-http-persistent Solution: upgrade to > 3.0.0 Leaks found!

Slide 73

Slide 73 text

$ cat Gemfile.lock | grep net-http-persistent net-http-persistent (3.0.0) net-http-persistent (= 3.0.0) $ cat Gemfile | grep net-http-persistent gem "net-http-persistent", "3.0.0"

Slide 74

Slide 74 text

Do we need to use `net- http-persistent` v3.0.0 or can we upgrade to v3.0.1?

Slide 75

Slide 75 text

$ cat Gemfile | grep net-http-persistent gem "net-http-persistent", "~> 3.0.0" $ bundle update net-http-persistent

Slide 76

Slide 76 text

$ bundle leak No leaks found

Slide 77

Slide 77 text

$ bundle leak No leaks found

Slide 78

Slide 78 text

No more leaky dependencies! Now let’s check again: Are we leaking memory?

Slide 79

Slide 79 text

bundler-leak Implementation Details

Slide 80

Slide 80 text

1. bundler-leak was born thanks to: https://github.com/ASoftCo/leaky-gems (thank you @sergey-alekseev!)

Slide 81

Slide 81 text

No content

Slide 82

Slide 82 text

2. bundler-leak depends on the ruby-mem- advisory-db project

Slide 83

Slide 83 text

No content

Slide 84

Slide 84 text

No content

Slide 85

Slide 85 text

3. bundler-leak is a fork of bundler-audit

Slide 86

Slide 86 text

$ bundle-audit Name: actionpack Version: 3.2.10 URL: http://www.osvdb.org/show/osvdb/91452 Title: XSS vulnerability in sanitize_css in Action Pack Solution: upgrade to ~> 2.3.18, ~> 3.1.12, >= 3.2.13 Advisory: OSVDB-91452 Unpatched versions found!

Slide 87

Slide 87 text

bundler-audit bundler-leak bundle audit bundle leak ruby-sec-advisory-db ruby-mem-advisory-db rubysec rubymem rubysec.com rubymem.com

Slide 88

Slide 88 text

Why? Not all memory leaks are security vulnerabilities

Slide 89

Slide 89 text

Onwards

Slide 90

Slide 90 text

RubyMem needs your help - The more reports we get, the more value we can provide to the community. ❤

Slide 91

Slide 91 text

No content

Slide 92

Slide 92 text

No content

Slide 93

Slide 93 text

RubyMem needs your help - The more reports we get, the more value we can provide to the community. ❤ - We are looking for contributors and co- maintainers!

Slide 94

Slide 94 text

Is it my code or a dependency?

Slide 95

Slide 95 text

Next time you need to go down the rabbit hole… use `bundle leak` ﬁrst.

Slide 96

Slide 96 text

Thank you! @etagwerker 96 ‣ github.com/rubymem/bundler-leak ‣ @etagwerker

Slide 97

Slide 97 text

Thank you! @etagwerker 97 Thanks to: @AlexeevS @nateberkopec @bronzdoc

Slide 98

Slide 98 text

Thank you! @etagwerker 98 Thank you!

Slide 99

Slide 99 text

Resources 1. https://github.com/ohler55/oj/issues/228 2. https://github.com/ohler55/oj/issues/229 3. https://github.com/rubymem/ruby-mem-advisory-db 4. https://github.com/mperham/sidekiq/pull/2598 5. https://samsaffron.com/archive/2015/03/31/debugging-memory-leaks-in-ruby 6. https://github.com/fastruby/gggiiifff.com/pull/9/ﬁles 7. https://github.com/rubyjs/therubyracer/pull/336#issuecomment-88488472 8. https://gist.github.com/wvengen/f1097651c238b2f7f11d 9. http://www.be9.io/2015/09/21/memory-leak/ 10. https://github.com/mperham/sidekiq/issues/4652

Slide 100

Slide 100 text

Resources 1. https://github.com/drbrain/net-http-persistent/pull/98/ﬁles 2. https://github.com/schneems/get_process_mem 3. https://www.toptal.com/ruby/hunting-ruby-memory-issues 4. https://github.com/schneems/derailed_benchmarks 5. https://www.youtube.com/watch?v=UCJsjr8ksDc 6. https://engineering.giphy.com/debugging-memory-leaks-in-swifts-enum-type/ 7. https://techlog360.com/limit-memory-usage-in-google-chrome/ 8. https://www.rubyguides.com/2016/09/object-space/