Upgrade to Pro — share decks privately, control downloads, hide ads and more …

RubyMem: The Leaky Gems Database for Bundler at Ruby Kaigi Takeout 2020

RubyMem: The Leaky Gems Database for Bundler at Ruby Kaigi Takeout 2020

Out of memory errors are quite tricky. Our first reaction is always the same: "It can't be my code, it must be one of my dependencies!" What if you could quickly check that with bundler?

In this talk you will learn about memory leaks, out of memory errors, and leaky dependencies. You will learn how to use bundler-leak, a community-driven, open source tool that will make your life easier when debugging memory leaks.

F77032adcbe77d2777bb0e0c30873159?s=128

Ernesto Tagwerker

September 04, 2020
Tweet

Transcript

  1. RubyMem: The Leaky Gems Database for Bundler Ernesto Tagwerker (@etagwerker)

    Ruby Kaigi, 2020
  2. Hi, I’m from Argentina

  3. Hi, I’m from Argentina I live in Philadelphia

  4. Hi, I’m from Argentina I live in Philadelphia I <3

    drawing with my daughter
  5. None
  6. Hi, I’m from Argentina I live in Philadelphia I <3

    drawing with my daughter I <3 Open Source
  7. Open Source Maintainer database_cleaner

  8. Open Source Maintainer database_cleaner next_rails

  9. Open Source Maintainer database_cleaner next_rails skunk

  10. Founder & Software Engineer @OmbuLabs

  11. We are hiring! OmbuLabs.com/jobs

  12. Founder & Software Engineer @fastrubyio

  13. Memory Issues

  14. None
  15. None
  16. RAM Issues

  17. Story Time

  18. Hey, can you help me with my project?

  19. Sure! How can I help?

  20. Can you make my gif search engine faster?

  21. None
  22. Sure! I can do that with a simple cache!

  23. None
  24. None
  25. I’m going to write my cache in Ruby with a

    Hash. Simple.
  26. None
  27. Just shipped a bunch of changes which included my cache-ing

    solution.
  28. We bumped the versions of `roda`, `rest-client`, `puma`, and we

    upgraded the heroku stack.
  29. Good job, Ernesto! That was quick and it is much

    faster now. ❤
  30. (A few hours go by and users start seeing an

    error in production.)
  31. I can haz no laser cat gifs!

  32. Weird. I’m going to check the logs.

  33. None
  34. Error R14? Memory Quota Exceeded?

  35. Hey, I noticed this weird behavior in our production’s memory

    graph
  36. Deployment

  37. Deployment Out of memory

  38. That looks bad. We ran out of memory overnight.

  39. But my code was so small. No way it’s my

    code’s fault.
  40. It’s probably Rails.

  41. It’s probably Rails Roda.

  42. What’s more likely? That Roda has a memory bug or

    that your code has a memory bug?
  43. Maybe you should check the changelog for our latest deployment?

  44. Good idea! I’m gonna start debugging.

  45. What tools can I use?

  46. I’m going to use `ObjectSpace.each_object` to count the amount of

    objects by object type.
  47. None
  48. None
  49. I’m still not sure, I’m going to use `rack-mini-profiler` and

    `memory_profiler`
  50. None
  51. None
  52. &pp=profile-memory

  53. &pp=profile-memory

  54. (After a few hours I realize that my code is

    the root cause.)
  55. None
  56. Ooops! That cache causes “memory bloat”

  57. I’m going to revert my change and deploy.

  58. Deployment

  59. Great! What was the root cause of that memory leak?

  60. My cache could end up using infinite memory but we

    only have 512MB in Heroku.
  61. Why did you take so long to fix it?

  62. I didn’t know whether the leak was in my application

    code or my dependencies.
  63. Lessons Learned

  64. Lessons Learned - Anyone can introduce a memory bloat

  65. Lessons Learned - Anyone can introduce a memory bloat -

    It’s easy to introduce it and hard to find
  66. Lessons Learned - Anyone can introduce a memory bloat -

    It’s easy to introduce it and hard to find - Great profiling tools in Ruby ❤
  67. If only there was a way to quickly answer this

    question:
  68. Introducing bundler-leak

  69. 13 group :development do 14 gem "bundler-leak" 15 end

  70. $ bundle leak No leaks found

  71. Neat! None of my dependencies are known to be leaking

    memory. I’m going to debug my application code.
  72. $ bundle leak Name: net-http-persistent Version: 3.0.0 URL: https://github.com/drbrain/net-http- persistent/pull/98

    Title: Memory leak in thread connection pool of net-http-persistent Solution: upgrade to > 3.0.0 Leaks found!
  73. $ cat Gemfile.lock | grep net-http-persistent net-http-persistent (3.0.0) net-http-persistent (=

    3.0.0) $ cat Gemfile | grep net-http-persistent gem "net-http-persistent", "3.0.0"
  74. Do we need to use `net- http-persistent` v3.0.0 or can

    we upgrade to v3.0.1?
  75. $ cat Gemfile | grep net-http-persistent gem "net-http-persistent", "~> 3.0.0"

    $ bundle update net-http-persistent
  76. $ bundle leak No leaks found

  77. $ bundle leak No leaks found

  78. No more leaky dependencies! Now let’s check again: Are we

    leaking memory?
  79. bundler-leak Implementation Details

  80. 1. bundler-leak was born thanks to: https://github.com/ASoftCo/leaky-gems (thank you @sergey-alekseev!)

  81. None
  82. 2. bundler-leak depends on the ruby-mem- advisory-db project

  83. None
  84. None
  85. 3. bundler-leak is a fork of bundler-audit

  86. $ bundle-audit Name: actionpack Version: 3.2.10 URL: http://www.osvdb.org/show/osvdb/91452 Title: XSS

    vulnerability in sanitize_css in Action Pack Solution: upgrade to ~> 2.3.18, ~> 3.1.12, >= 3.2.13 Advisory: OSVDB-91452 Unpatched versions found!
  87. bundler-audit bundler-leak bundle audit bundle leak ruby-sec-advisory-db ruby-mem-advisory-db rubysec rubymem

    rubysec.com rubymem.com
  88. Why? Not all memory leaks are security vulnerabilities

  89. Onwards

  90. RubyMem needs your help - The more reports we get,

    the more value we can provide to the community. ❤
  91. None
  92. None
  93. RubyMem needs your help - The more reports we get,

    the more value we can provide to the community. ❤ - We are looking for contributors and co- maintainers!
  94. Is it my code or a dependency?

  95. Next time you need to go down the rabbit hole…

    use `bundle leak` first.
  96. Thank you! @etagwerker 96 ‣ github.com/rubymem/bundler-leak ‣ @etagwerker

  97. Thank you! @etagwerker 97 Thanks to: @AlexeevS @nateberkopec @bronzdoc

  98. Thank you! @etagwerker 98 Thank you!

  99. Resources 1. https://github.com/ohler55/oj/issues/228 2. https://github.com/ohler55/oj/issues/229 3. https://github.com/rubymem/ruby-mem-advisory-db 4. https://github.com/mperham/sidekiq/pull/2598 5.

    https://samsaffron.com/archive/2015/03/31/debugging-memory-leaks-in-ruby 6. https://github.com/fastruby/gggiiifff.com/pull/9/files 7. https://github.com/rubyjs/therubyracer/pull/336#issuecomment-88488472 8. https://gist.github.com/wvengen/f1097651c238b2f7f11d 9. http://www.be9.io/2015/09/21/memory-leak/ 10. https://github.com/mperham/sidekiq/issues/4652
  100. Resources 1. https://github.com/drbrain/net-http-persistent/pull/98/files 2. https://github.com/schneems/get_process_mem 3. https://www.toptal.com/ruby/hunting-ruby-memory-issues 4. https://github.com/schneems/derailed_benchmarks 5.

    https://www.youtube.com/watch?v=UCJsjr8ksDc 6. https://engineering.giphy.com/debugging-memory-leaks-in-swifts-enum-type/ 7. https://techlog360.com/limit-memory-usage-in-google-chrome/ 8. https://www.rubyguides.com/2016/09/object-space/