Reduce Ruby Memory Usage With These 12 Weird Tricks

Slide 1

Slide 1 text

Halve Your Memory Usage With These 12 Weird Tricks Heroku and AWS hate him! @nateberkopec

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

Ruby is a garbage collected language. But my memory is growing. Therefore, memory leak.

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Solution 1: Dial Back The Instance Counts Discover true steady-state memory usage per instance

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

Myth: Memory usage should look like a long, ﬂat line

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

Aim for 300MB per instance This also applies to Sidekiq

Slide 14

Slide 14 text

Solution 2: Stop allocating so many objects at once

Slide 15

Slide 15 text

Myth: "Shouldn't GC clean the unused objects up after the job completes?"

Slide 16

Slide 16 text

Translated: Memory goes down, right?

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

Thresholds Heap frag free isn't free

Slide 19

Slide 19 text

Thresholds, not timers.

Slide 20

Slide 20 text

Slots run out oldmalloc malloc

Slide 21

Slide 21 text

Heap fragmentation

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

GC::INTERNAL_CONSTANTS { :RVALUE_SIZE=>40, :HEAP_PAGE_OBJ_LIMIT=>408, :HEAP_PAGE_BITMAP_SIZE=>56, :HEAP_PAGE_BITMAP_PLANES=>4 }

Slide 26

Slide 26 text

Malloc and free are suggestions, not commands

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

Heap fragmentation can cause long-term slow "leaks"

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

End result: Ruby Memory usage = Maximum Memory Pressure

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

Allocating less Or, ﬁx your god damn N+1s

Slide 34

Slide 34 text

Solution 2a: Use an APM - Scout, Skylight, New Relic

Slide 35

Slide 35 text

Solution 2b: Use memory_proﬁler or oink

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

oink -- SUMMARY -- Worst Requests: 1. Feb 02 16:26:06, 157524 KB, SportsController#show 2. Feb 02 20:11:54, 134972 KB, DashboardsController#show 3. Feb 02 19:06:13, 131912 KB, DashboardsController#show 4. Feb 02 08:07:46, 115448 KB, GroupsController#show 5. Feb 02 12:19:53, 112924 KB, GroupsController#show 6. Feb 02 13:03:00, 112064 KB, ColorSchemesController#show 7. Feb 02 13:01:59, 109148 KB, SessionsController#create 8. Feb 02 06:11:17, 108456 KB, PublicPagesController#join 9. Feb 02 08:43:06, 94468 KB, CommentsController#create 10. Feb 02 20:49:44, 82340 KB, DashboardsController#show

Slide 38

Slide 38 text

memory_profiler allocated memory by gem ----------------------------------- rubygems x 305879 allocated memory by file ----------------------------------- /home/sam/.rbenv/versions/2.1.0-github/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb x 285433 /home/sam/.rbenv/versions/2.1.0-github/lib/ruby/2.1.0/rubygems/basic_specification.rb x 18597 /home/sam/.rbenv/versions/2.1.0-github/lib/ruby/2.1.0/rubygems.rb x 2218 /home/sam/.rbenv/versions/2.1.0-github/lib/ruby/2.1.0/rubygems/specification.rb x 1169 /home/sam/.rbenv/versions/2.1.0-github/lib/ruby/2.1.0/rubygems/defaults.rb x 520 /home/sam/.rbenv/versions/2.1.0-github/lib/ruby/2.1.0/rubygems/core_ext/kernel_gem.rb x 80 /home/sam/.rbenv/versions/2.1.0-github/lib/ruby/2.1.0/rubygems/version.rb x 80

Slide 39

Slide 39 text

Make your own: objectspace & gc.stat

Slide 40

Slide 40 text

GC.stat - log it! { :count => 9, :heap_allocated_pages => 74, :heap_sorted_length => 75, :heap_allocatable_pages => 0, :heap_available_slots => 30164, :heap_live_slots => 29863, :heap_free_slots => 301, :heap_ﬁnal_slots => 0, # etc etc }

Slide 41

Slide 41 text

ObjectSpace.count_objects { :TOTAL => 30164, :FREE => 235, :T_OBJECT => 297, :T_CLASS => 944, :T_MODULE => 45, :T_FLOAT => 4, # etc etc }

Slide 42

Slide 42 text

Solution 2c: If all else fails, move to Rake tasks Throwaway VMs are better than bloated VMs

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

Solution 3: Gemﬁle audit with derailed

Slide 45

Slide 45 text

$ bundle exec derailed bundle:mem TOP: 54.1836 MiB mail: 18.9688 MiB mime/types: 17.4453 MiB mail/ﬁeld: 0.4023 MiB mail/message: 0.3906 MiB action_view/view_paths: 0.4453 MiB action_view/base: 0.4336 MiB

Slide 46

Slide 46 text

Myth: Dependencies are free!

Slide 47

Slide 47 text

require false for assets! gem 'sass', require: false

Slide 48

Slide 48 text

sprockets/lib/sprockets/ autoload.rb module Sprockets module Autoload autoload :Babel, 'sprockets/autoload/babel' autoload :Closure, 'sprockets/autoload/closure' autoload :CoffeeScript, 'sprockets/autoload/coffee_script' autoload :Eco, 'sprockets/autoload/eco' #etc etc etc

Slide 49

Slide 49 text

Solution 4: jemalloc

Slide 50

Slide 50 text

“emphasizes fragmentation avoidance and scalable concurrency support.”

Slide 51

Slide 51 text

LD_PRELOAD or --with-jemalloc

Slide 52

Slide 52 text

Solution 5: Use copy-on-write Puma, Unicorn or Passenger w/preloading

Slide 53

Slide 53 text

Copy-on-write increases shared memory

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

Myth: Memory usage = sum of RSS Memory is surprisingly difﬁcult to measure

Slide 56

Slide 56 text

memory can be virtual/real shared/private resident/swapped

Slide 57

Slide 57 text

It isn't perfect but it's a start

Slide 58

Slide 58 text

Solution 6: Use a threaded webserver Puma, Passenger Enterprise. Increase concurrency w/lighter methods

Slide 59

Slide 59 text

Mini-Myth: My application isn't thread-safe

Slide 60

Slide 60 text

No content

Slide 61

Slide 61 text

minitest/hell

Slide 62

Slide 62 text

Solution 7: Keep Ruby and gems up-to-date

Slide 63

Slide 63 text

Ruby 2.2+ Rails 4.2+ Watch out for Ruby 2.4

Slide 64

Slide 64 text

Solution 8: Tune malloc

Slide 65

Slide 65 text

MALLOC_ARENA_MAX

Slide 66

Slide 66 text

mallopt

Slide 67

Slide 67 text

Solution 9: Tune GC

Slide 68

Slide 68 text

If you can't read gc.c and understand these variables yourself, don't touch them (yet)

Slide 69

Slide 69 text

GC Tuning can ﬁx: Too many free slots Slow startup Too many or too few GCs

Slide 70

Slide 70 text

Performance Birds of Feather Tomorrow @ 1:15pm

Slide 71

Slide 71 text

350+ pages, 18+ hours of video railsspeed.com

Slide 72

Slide 72 text

No content

Slide 73

Slide 73 text

10kb speedshop.co

Slide 74

Slide 74 text

Thanks! Slides and notes on Twitter @nateberkopec speedshop.co & railsspeed.com Tell me your problems!