RubyMem: The Leaky Gems Database for Bundler at Ruby Kaigi Takeout 2020

Out of memory errors are quite tricky. Our first reaction is always the same: "It can't be my code, it must be one of my dependencies!" What if you could quickly check that with bundler?

In this talk you will learn about memory leaks, out of memory errors, and leaky dependencies. You will learn how to use bundler-leak, a community-driven, open source tool that will make your life easier when debugging memory leaks.

Transcript

1. RubyMem: The Leaky Gems Database for Bundler Ernesto Tagwerker (@etagwerker) Ruby Kaigi, 2020
2. Hi, I’m from Argentina
3. Hi, I’m from Argentina I live in Philadelphia
4. Hi, I’m from Argentina I live in Philadelphia I <3 drawing with my daughter
6. Hi, I’m from Argentina I live in Philadelphia I <3 drawing with my daughter I <3 Open Source
7. Open Source Maintainer database_cleaner
8. Open Source Maintainer database_cleaner next_rails
9. Open Source Maintainer database_cleaner next_rails skunk
10. Founder & Software Engineer @OmbuLabs
11. We are hiring! OmbuLabs.com/jobs
12. Founder & Software Engineer @fastrubyio
13. Memory Issues
16. RAM Issues
17. Story Time
18. Hey, can you help me with my project?
19. Sure! How can I help?
20. Can you make my gif search engine faster?
22. Sure! I can do that with a simple cache!
25. I’m going to write my cache in Ruby with a Hash. Simple.
27. Just shipped a bunch of changes which included my cache-ing solution.
28. We bumped the versions of `roda`, `rest-client`, `puma`, and we upgraded the heroku stack.
29. Good job, Ernesto! That was quick and it is much faster now. ❤
30. (A few hours go by and users start seeing an error in production.)
31. I can haz no laser cat gifs!
32. Weird. I’m going to check the logs.
34. Error R14? Memory Quota Exceeded?
35. Hey, I noticed this weird behavior in our production’s memory graph
36. Deployment
37. Deployment Out of memory
38. That looks bad. We ran out of memory overnight.
39. But my code was so small. No way it’s my code’s fault.
40. It’s probably Rails.
41. It’s probably Rails Roda.
42. What’s more likely? That Roda has a memory bug or that your code has a memory bug?
43. Maybe you should check the changelog for our latest deployment?
44. Good idea! I’m gonna start debugging.
45. What tools can I use?
46. I’m going to use `ObjectSpace.each_object` to count the amount of objects by object type.
49. I’m still not sure, I’m going to use `rack-mini-profiler` and `memory_profiler`
52. &pp=profile-memory
53. &pp=profile-memory
54. (After a few hours I realize that my code is the root cause.)
56. Ooops! That cache causes “memory bloat”
57. I’m going to revert my change and deploy.
58. Deployment
59. Great! What was the root cause of that memory leak?
60. My cache could end up using infinite memory but we only have 512MB in Heroku.
61. Why did you take so long to fix it?
62. I didn’t know whether the leak was in my application code or my dependencies.
63. Lessons Learned
64. Lessons Learned - Anyone can introduce a memory bloat
65. Lessons Learned - Anyone can introduce a memory bloat - It’s easy to introduce it and hard to find
66. Lessons Learned - Anyone can introduce a memory bloat - It’s easy to introduce it and hard to find - Great profiling tools in Ruby ❤
67. If only there was a way to quickly answer this question:
68. Introducing bundler-leak
69. 13 group :development do 14 gem "bundler-leak" 15 end
70. $ bundle leak No leaks found
71. Neat! None of my dependencies are known to be leaking memory. I’m going to debug my application code.
72. $ bundle leak Name: net-http-persistent Version: 3.0.0 URL: https://github.com/drbrain/net-http- persistent/pull/98 Title: Memory leak in thread connection pool of net-http-persistent Solution: upgrade to > 3.0.0 Leaks found!
73. $ cat Gemfile.lock | grep net-http-persistent net-http-persistent (3.0.0) net-http-persistent (= 3.0.0) $ cat Gemfile | grep net-http-persistent gem "net-http-persistent", "3.0.0"
74. Do we need to use `net- http-persistent` v3.0.0 or can we upgrade to v3.0.1?
75. $ cat Gemfile | grep net-http-persistent gem "net-http-persistent", "~> 3.0.0" $ bundle update net-http-persistent
76. $ bundle leak No leaks found
77. $ bundle leak No leaks found
78. No more leaky dependencies! Now let’s check again: Are we leaking memory?
79. bundler-leak Implementation Details
80. 1. bundler-leak was born thanks to: https://github.com/ASoftCo/leaky-gems (thank you @sergey-alekseev!)
82. 2. bundler-leak depends on the ruby-mem- advisory-db project
85. 3. bundler-leak is a fork of bundler-audit
86. $ bundle-audit Name: actionpack Version: 3.2.10 URL: http://www.osvdb.org/show/osvdb/91452 Title: XSS vulnerability in sanitize_css in Action Pack Solution: upgrade to ~> 2.3.18, ~> 3.1.12, >= 3.2.13 Advisory: OSVDB-91452 Unpatched versions found!
87. bundler-audit bundler-leak bundle audit bundle leak ruby-sec-advisory-db ruby-mem-advisory-db rubysec rubymem rubysec.com rubymem.com
88. Why? Not all memory leaks are security vulnerabilities
89. Onwards
90. RubyMem needs your help - The more reports we get, the more value we can provide to the community. ❤
93. RubyMem needs your help - The more reports we get, the more value we can provide to the community. ❤ - We are looking for contributors and co- maintainers!
94. Is it my code or a dependency?
95. Next time you need to go down the rabbit hole… use `bundle leak` first.
96. Thank you! @etagwerker 96 ‣ github.com/rubymem/bundler-leak ‣ @etagwerker
97. Thank you! @etagwerker 97 Thanks to: @AlexeevS @nateberkopec @bronzdoc
98. Thank you! @etagwerker 98 Thank you!
99. Resources 1. https://github.com/ohler55/oj/issues/228 2. https://github.com/ohler55/oj/issues/229 3. https://github.com/rubymem/ruby-mem-advisory-db 4. https://github.com/mperham/sidekiq/pull/2598 5. https://samsaffron.com/archive/2015/03/31/debugging-memory-leaks-in-ruby 6. https://github.com/fastruby/gggiiifff.com/pull/9/files 7. https://github.com/rubyjs/therubyracer/pull/336#issuecomment-88488472 8. https://gist.github.com/wvengen/f1097651c238b2f7f11d 9. http://www.be9.io/2015/09/21/memory-leak/ 10. https://github.com/mperham/sidekiq/issues/4652
100. Resources 1. https://github.com/drbrain/net-http-persistent/pull/98/files 2. https://github.com/schneems/get_process_mem 3. https://www.toptal.com/ruby/hunting-ruby-memory-issues 4. https://github.com/schneems/derailed_benchmarks 5. https://www.youtube.com/watch?v=UCJsjr8ksDc 6. https://engineering.giphy.com/debugging-memory-leaks-in-swifts-enum-type/ 7. https://techlog360.com/limit-memory-usage-in-google-chrome/ 8. https://www.rubyguides.com/2016/09/object-space/