RailsConf 2018 Keynote

RailsConf 2018 Keynote

My keynote at RailsConf 2018

F29327647a9cff5c69618bae420792ea?s=128

Aaron Patterson

May 11, 2018
Tweet

Transcript

  1. Just in Time Presentation

  2. Things I’ve been doing besides preparing my talk

  3. None
  4. None
  5. None
  6. None
  7. 4 or 5 slides out of this

  8. Oh, Hello There!

  9. Aaron Patterson

  10. None
  11. None
  12. None
  13. G GitHub

  14. Le Git

  15. git push -f

  16. git checkout -b

  17. git commit

  18. git reflog

  19. git diff I promise there is actual content in this

    presentation
  20. Level 5 Engineer

  21. None
  22. Not Enough XP to ⬆

  23. I LOVE Local Businesses

  24. #LOCAL

  25. #LOCAL

  26. None
  27. #NOTLOCAL

  28. I love local businesses and I hope you….

  29. Got the point

  30. TDBYIL The Day Before Yesterday I Learned

  31. I’ve got 99 problems and I guess they’re all yours?

  32. This Past Year…

  33. I became an Uncle

  34. None
  35. None
  36. I became very famous (in Japan)

  37. 1 Character 81,000 likes

  38. Decided to become a Thought Leader (again)

  39. None
  40. None
  41. None
  42. None
  43. Does thiswork? \u200b

  44. None
  45. None
  46. lol.rb def puts "neat" end

  47. lol.rb def : puts end : : # =>

  48. non-breaking-space.rb def puts "hello" end # => "hello" \u200b \u200b

  49. None
  50. zero-width-non-joiner.rb def puts end "hello" # => "hello" \u200b \u200b

    \u200c \u200c
  51. None
  52. Foo = \u200b

  53. Bar = \u200c

  54. PRODUCTION READY CODE

  55. Performance

  56. We are always getting faster

  57. Team Velocity T T + 1 T + 2 T

    + 3 Velocity
  58. Who is this mythical man?

  59. Why are we slower?

  60. What do we do about it?

  61. Rewrite!

  62. What good is velocity if you’re not getting anywhere?

  63. We can hire someone!

  64. Refactor?

  65. Software Writer PhD in CS

  66. There must be something in between

  67. We have to go deeper

  68. I’m the only one who can fix this

  69. You’re the only one that can fix your app

  70. I ❤ Rails

  71. Profiling / Tuning

  72. Runtime / Memory

  73. Time / Space

  74. Types of Profilers

  75. Exact / Sampling

  76. How Many? Use an Exact profiler

  77. How Much? Use a Sampling profiler

  78. Exact Profiler

  79. Exact Profiler def fast; end def slow; sleep(0.1); end def

    benchmark_me 1000.times { fast } 10.times { slow } end def exact_profiler counter = Hash.new(0) tp = TracePoint.new(:call) do |event| counter[event.method_id] += 1 end tp.enable yield tp.disable counter end result = exact_profiler { benchmark_me } p result # => {:benchmark_me=>1, :fast=>1000, :slow=>10} Listen for an Event Keep Track of Calls
  80. Sampling Profiler

  81. None
  82. Time Sampling Profiler

  83. Sampling Profiler def fast; end def slow; sleep(0.1); end def

    benchmark_me 1000.times { fast } 10.times { slow } end def exact_profiler target = Thread.current samples = Hash.new(0) t = Thread.new do loop do sleep 0.0001 # sample rate samples[target.backtrace_locations.first.label] += 1 end end yield t.kill samples.dup end result = exact_profiler { benchmark_me } p result # => {"sleep"=>6497, "slow"=>1} 6498 samples 99% inside `sleep`
  84. When to use Exact vs Sampling

  85. Absolute Counts: Exact

  86. Reducing Time or Space: Sampling

  87. Not sure: Use Sampling

  88. 99.963%: Sampling Profiler (1 sample)

  89. Runtime Profiling / Tuning

  90. stackprof

  91. Using StackProf require 'stackprof' def fast; end def slow; sleep(0.1);

    end def benchmark_me 1000.times { fast } 10.times { slow } end StackProf.run(mode: :wall, out: 'profile.dump') do benchmark_me end
  92. `stackprof` output $ stackprof profile.dump ================================== Mode: wall(1000) Samples: 999

    (0.20% miss rate) GC: 0 (0.00%) ================================== TOTAL (pct) SAMPLES (pct) FRAME 999 (100.0%) 999 (100.0%) Object#slow 999 (100.0%) 0 (0.0%) Object#benchmark_me 999 (100.0%) 0 (0.0%) block in <main> 999 (100.0%) 0 (0.0%) <main> 999 (100.0%) 0 (0.0%) <main>
  93. Benchmarking Rails Boot

  94. Rails Boot Process Requiring Files "Compiling" Application Compiling Views Time

    Hit Enter Not Ready For Requests First Request Business Logic Second Request Rack Starts Unicorn Starts
  95. Rails Boot Process Requiring Files "Compiling" Application Compiling Views Time

    Restarting the Webserver Warmup Time Business Logic Runtime Running Scripts
  96. Benchmarking Scripts require 'stackprof' StackProf.run(mode: :wall, out: 'profile.dump') do require

    'config/environment' end
  97. Example Output $ stackprof profile.dump ================================== Mode: wall(1000) Samples: 3951

    (6.17% miss rate) GC: 663 (16.78%) ================================== TOTAL (pct) SAMPLES (pct) FRAME 1134 (28.7%) 821 (20.8%) ActiveSupport::Dependencies::Loadable#require 663 (16.8%) 663 (16.8%) (garbage collection) 166 (4.2%) 104 (2.6%) Kernel#require 86 (2.2%) 86 (2.2%) #<Module:0x00007fb24f9f5200>.mechanism 98 (2.5%) 59 (1.5%) Gem::Specification#validate 56 (1.4%) 55 (1.4%) Module#delegate 51 (1.3%) 40 (1.0%) <top (required)> 51 (1.3%) 36 (0.9%) <top (required)> 40 (1.0%) 34 (0.9%) Gem::StubSpecification#data 138 (3.5%) 34 (0.9%) <top (required)> 137 (3.5%) 33 (0.8%) <top (required)> Time in GC
  98. Runtime Tuning

  99. "write faster code"

  100. Is there a faster way to do the same thing?

  101. Do we need to do this thing so much?

  102. Can we defer doing this thing to a later time?

  103. Memory Tuning

  104. What is a GC?

  105. Frees Memory

  106. Allocates Memory

  107. What is MRI’s GC?

  108. Mark and Sweep Generational Incremental Marking Lazy Sweeping Free-list Allocation

  109. Mark and Sweep Generational Incremental Marking Lazy Sweeping Free-list Allocation

    Faster Shorter Pause Shorter Pause
  110. When can the GC collect garbage?

  111. GC.start Allocate Object Allocate Memory (malloc)

  112. What is an Object?

  113. An Object is a 40 byte chunk Ruby Object x

    = "omg this is a very long string!"
  114. An Object is a 40 byte chunk Ruby Object x

    = "omg this is a very long string!" "omg this is a very long string!" `malloc` GC
  115. Ruby Object Empty Slot An Object is a 40 byte

    chunk x = "omg this is a very long string!" "omg this is a very long string!" `malloc` Empty Slot Empty Slot Empty Slot Empty Slot Empty Slot Empty Slot Empty Slot Page
  116. A Slot is a location where a Ruby object may

    or may not be
  117. Tuning Parameters

  118. RUBY_GC_HEAP_INIT_SLOTS $ ruby -e'p GC.stat(:heap_available_slots)' 17935 $ RUBY_GC_HEAP_INIT_SLOTS=600000 ruby -e'p

    GC.stat(:heap_available_slots)' 599166
  119. Rails App Boot (default) $ stackprof profile.dump ================================== Mode: wall(1000)

    Samples: 4434 (8.22% miss rate) GC: 730 (16.46%) ================================== TOTAL (pct) SAMPLES (pct) FRAME 1125 (25.4%) 817 (18.4%) ActiveSupport::Dependencies::Loadable#require 730 (16.5%) 730 (16.5%) (garbage collection) 202 (4.6%) 135 (3.0%) Kernel#require 82 (1.8%) 82 (1.8%) #<Module:0x00007ffd3533ce00>.mechanism 106 (2.4%) 60 (1.4%) Gem::Specification#validate 55 (1.2%) 54 (1.2%) Module#delegate 72 (1.6%) 53 (1.2%) <top (required)> 170 (3.8%) 47 (1.1%) <top (required)> 51 (1.2%) 45 (1.0%) <top (required)> 58 (1.3%) 38 (0.9%) <top (required)> 164 (3.7%) 36 (0.8%) <top (required)> 234 (5.3%) 34 (0.8%) <module:Helpers> 41 (0.9%) 32 (0.7%) Gem::StubSpecification#data 16.5% Spent in GC $ RAILS_ENV=production ruby -I. bench.rb
  120. Rails App Boot (RUBY_GC_HEAP_INIT_SLOTS=600000) $ stackprof profile.dump ================================== Mode: wall(1000)

    Samples: 3269 (7.03% miss rate) GC: 279 (8.53%) ================================== TOTAL (pct) SAMPLES (pct) FRAME 924 (28.3%) 623 (19.1%) ActiveSupport::Dependencies::Loadable#require 279 (8.5%) 279 (8.5%) (garbage collection) 162 (5.0%) 97 (3.0%) Kernel#require 86 (2.6%) 86 (2.6%) #<Module: 0x00007f937b52a7e0>.mechanism 56 (1.7%) 55 (1.7%) Module#delegate 95 (2.9%) 51 (1.6%) Gem::Specification#validate 144 (4.4%) 35 (1.1%) <top (required)> 50 (1.5%) 35 (1.1%) <top (required)> 127 (3.9%) 31 (0.9%) <top (required)> 35 (1.1%) 29 (0.9%) <top (required)> 8.3% Spent in GC $ RUBY_GC_HEAP_INIT_SLOTS=600000 RAILS_ENV=production ruby -I. bench.rb
  121. Boot Time $ time RAILS_ENV=production ruby -I. -rconfig/environment -e' '

    real 0m4.105s user 0m3.464s sys 0m0.607s $ time RUBY_GC_HEAP_INIT_SLOTS=600000 RAILS_ENV=production ruby -I. -rconfig/environment -e' ' real 0m3.662s user 0m3.015s sys 0m0.607s
  122. Control Growth

  123. RUBY_GC_HEAP_FREE_SLOTS_MIN_RATIO RUBY_GC_HEAP_FREE_SLOTS_MAX_RATIO RUBY_GC_HEAP_FREE_SLOTS_GOAL_RATIO

  124. Growth Test Program x = [] a = GC.stat(:count) 600000.times

    do x << Object.new end b = GC.stat(:count) p [a, b]
  125. Heap Growth Patterns Number of Slots 0 150000 300000 450000

    600000 Number of Heap Resizes 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Goal=0.2 Goal=0.7
  126. Try at Home! $ RUBY_GC_HEAP_FREE_SLOTS_MIN_RATIO=0.1 RUBY_GC_HEAP_FREE_SLOTS_MAX_RATIO=0.9 RUBY_GC_HEAP_FREE_SLOTS_GOAL_RATIO=0.7 ./ruby --disable-gems grow_heap.rb

    $ RUBY_GC_HEAP_FREE_SLOTS_MIN_RATIO=0.1 RUBY_GC_HEAP_FREE_SLOTS_MAX_RATIO=0.9 RUBY_GC_HEAP_FREE_SLOTS_GOAL_RATIO=0.2 ./ruby --disable-gems grow_heap.rb
  127. Memory Profiling

  128. stackprof

  129. Benchmarking Scripts require 'stackprof' StackProf.run(mode: :object, out: 'profile.dump') do require

    'config/environment' end
  130. Object Tracing Output $ stackprof profile.dump ================================== Mode: object(1) Samples:

    951217 (0.00% miss rate) GC: 0 (0.00%) ================================== TOTAL (pct) SAMPLES (pct) FRAME 263967 (27.8%) 199899 (21.0%) ActiveSupport::Dependencies::Loadable#require 51931 (5.5%) 43885 (4.6%) Kernel#require 43744 (4.6%) 23606 (2.5%) Kernel#require 19553 (2.1%) 19553 (2.1%) Module#delegate 57342 (6.0%) 17821 (1.9%) <top (required)> 28941 (3.0%) 15809 (1.7%) Gem::StubSpecification#data 60887 (6.4%) 10937 (1.1%) <module:Helpers> 12754 (1.3%) 10183 (1.1%) <top (required)>
  131. Exact Profiling

  132. ObjectSpace

  133. Heap Dumps require 'objspace' ObjectSpace.trace_object_allocations_start require 'config/environment' GC.start File.open("my_dump.json", "w")

    do |f| ObjectSpace.dump_all(output: f) end Enable Allocation Location Recording The Actual Benchmark Clear any garbage Dump the heap
  134. JSON file (1 line = 1 Object) {"address":"0x7fd43461bab0", "type":"ARRAY", "length":6,

    "references": ["0x7fd4342164d8", "0x7fd4338b28d8", "0x7fd4338b28d8"], "file":"/Users/ aaron/git/rails/activesupport/lib/active_support/dependencies.rb", "line":283, "method":"require", "generation":46, "memsize":200, "flags": {"wb_protected":true, "old":true, "uncollectible":true, "marked":true}}
  135. JSON file (1 line = 1 Object) { "address": "0x7fd43461bab0",

    "type": "ARRAY", "length": 6, "references": [ "0x7fd4342164d8", "0x7fd4338b28d8", "0x7fd4338b28d8" ], "file": "/Users/aaron/git/rails/activesupport/lib/active_support/dependencies.rb", "line": 283, "method": "require", "generation": 46, "memsize": 200, "flags": { "wb_protected": true, "old": true, "uncollectible": true, "marked": true } } Object Type References Allocation Location Size in Memory
  136. Very Easy Analysis

  137. How Many Objects? $ wc -l my_dump.json 185253 my_dump.json

  138. What File is Most Popular? $ jq .file my_dump.json |

    sort | uniq -c | sort -n | tail -10 Extract the "file" attribute Count Unique Files Sort Get Top 10
  139. Sample Output 1768 "lib/ruby/2.6.0/x86_64-darwin17/openssl.bundle" 2218 "lib/ruby/gems/2.6.0/gems/sass-3.5.4/lib/sass/engine.rb" 2365 "lib/ruby/2.6.0/rubygems/stub_specification.rb" 2469 "lib/ruby/2.6.0/rubygems/specification.rb"

    2805 "lib/ruby/2.6.0/delegate.rb" 2977 "activesupport/lib/active_support/core_ext/module/delegation.rb" 3187 "lib/ruby/gems/2.6.0/gems/tzinfo-1.2.5/lib/tzinfo/zoneinfo_data_source.rb" 10628 "lib/ruby/1.6.0/rubygems/core_ext/kernel_require.rb" 12448 null 25693 "activesupport/lib/active_support/dependencies.rb"
  140. What Line is Most Popular? $ jq 'select(.file == "activesupport/lib/active_support/

    dependencies.rb") | .line' my_dump.json | sort | uniq -c | sort -n | tail -10 Filter objects allocated in this file
  141. Sample Output $ jq 'select(.file == "/Users/aaron/git/rails/activesupport/lib/ active_support/dependencies.rb") | .line'

    my_dump.json | sort | uniq -c | sort -n | tail -10 2 304 3 102 3 173 3 215 3 289 4 559 4 92 48 277 53 16 25487 283
  142. How Much Memory Does Active Support Allocate?

  143. Active Support Allocations $ jq 'select(.file | contains("activesupport")) | .memsize'

    my_dump.json | awk '{s+=$1} END {print s}' 3584379
  144. Hidden References

  145. Array References a = [:b, :c] Array Symbol Symbol

  146. Lambda References def foo(a, b) lambda { a + b

    } end m = foo(10, 11) Lambda Number Number
  147. Lambda References def foo(a, b) lambda { } end m

    = foo(10, 11) Lambda Number Number
  148. No Escape! def foo(a, b) lambda { } end m

    = foo(10, 11) env = m.binding # lambda environment p env.local_variables # [:a, :b] p env.local_variable_get(:a) # 10 p env.local_variable_get(:b) # 11
  149. Dropped 200GB total

  150. Tools I Want

  151. Rails Boot Process Requiring Files "Compiling" Application Compiling Views Time

    Restarting the Webserver Warmup Time Business Logic Runtime Running Scripts
  152. Benchmarking Scripts require 'stackprof' StackProf.run(mode: :wall, out: 'profile.dump') do require

    'config/environment' end
  153. Rails Boot Process Requiring Files "Compiling" Application Compiling Views Time

    Restarting the Webserver Warmup Time Business Logic Runtime Running Scripts
  154. New `rackup` Options $ ruby -I lib bin/rackup --help Usage:

    rackup [ruby options] [rack options] [rackup config] Profiling options: --heap HEAPFILE Build the application, then dump the heap to HEAPFILE --profile PROFILE Dump CPU or Memory profile to PROFILE (defaults to a tempfile) --profile-mode MODE Profile mode (cpu|wall|object)
  155. Rails Boot Process Requiring Files "Compiling" Application Compiling Views Time

    Restarting the Webserver Warmup Time Business Logic Runtime
  156. Profiling + System Tests

  157. bin/bench-app --url http:// localhost:3000/ --stackprof object

  158. Output GitHub (master [490a985]) - Rails 4.2.10.github (development) 2018-04-19 12:16:11

    -0400 12:16 up 6 days, 19:54, 8 users, load averages: 2.94 2.27 2.39 aaron@TC-4.local:/Users/aaron/github/github pid 35651 - 940 MB RSS - 54.303887s CPU - 2,163,221 objects 9763 loaded features {:TOTAL=>3899911, :FREE=>365601, :T_OBJECT=>370493, :T_CLASS=>138069, :T_MODULE=>32498, :T_FLOAT=>17, :T_STRING=>1001412, :T_REGEXP=>14153, :T_ARRAY=>398461, :T_HASH=>152491, :T_STRUCT=>11063, :T_BIGNUM=>46, :T_FILE=>81, :T_DATA=>166354, :T_MATCH=>37127, :T_COMPLEX=>1, :T_RATIONAL=>907, : T_SYMBOL=>10038, :T_IMEMO=>1083137, :T_NODE=>1729, :T_ICLASS=>116233} {:count=>54, :heap_allocated_pages=>9568, :heap_sorted_length=>9910, :heap_allocatable_pages=>342, :heap_available_slots=>3899911, :heap_live_slots=>3534410, :heap_free_slots=>365501, :heap_final_slots=>0, :heap_marked_slots=>3186821, :heap_eden_pages=>9568, :heap_tomb_pages=>0, :total_al located_pages=>9568, :total_freed_pages=>0, :total_allocated_objects=>37844811, :total_freed_objects=>34310401, :malloc_increase_bytes=>12849616, :malloc_increase_bytes_limit=>128000000, :minor_gc_count=>43, :major_gc_count=>11, :remembered_wb_unprotected_objects=>77457, :remembered_wb_un protected_objects_limit=>154914, :old_objects=>3009873, :old_objects_limit=>6019746, :oldmalloc_increase_bytes=>12850064, :oldmalloc_increase_bytes_limit=>128000000} /Users/aaron/github/github/vendor/ruby/0d9762bcc5e0bb319765c8e40c5524509159003b/bin/ruby ruby 2.4.3p200 (2017-09-15 revision 59914) [x86_64-darwin17] RUBY_GC_HEAP_GROWTH_FACTOR=1.25 RUBY_GC_HEAP_GROWTH_MAX_SLOTS=300000 RUBY_GC_OLDMALLOC_LIMIT=128000000 RUBY_GC_HEAP_INIT_SLOTS=600000 RUBY_GC_MALLOC_LIMIT=128000000 RUBY_GC_HEAP_FREE_SLOTS=600000 10 requests to http://localhost:3000/ as (anonymous) peak memory: 979 MB RSS response size: 504 KB total (50.4 KB/req) response time: 0ms total render time: 0ms total cpu time: 0ms total idle time: 0ms total oob gc: 0 / 0ms in-request gc: 0 / 0ms total allocations: 0 objs total ar objs: 0 objs total smoke: 0 / 0ms total mysql: 0 / 0ms total redis: 0 / 0ms total cache: 0 / 0ms total marshal: 0 / 0ms total zlib: 0 / 0ms total {:TOTAL=>4338897, :FREE=>1023915, :T_OBJECT=>296251, :T_CLASS=>138076, :T_MODULE=>32499, :T_FLOAT=>17, :T_STRING=>911236, :T_REGEXP=>17144, :T_ARRAY=>412142, :T_HASH=>166431, :T_STRUCT=>11006, :T_BIGNUM=>46, :T_FILE=>6, :T_DATA=>166436, :T_MATCH=>401, :T_COMPLEX=>1, :T_RATIONAL=>907, :T_S YMBOL=>10042, :T_IMEMO=>1034234, :T_NODE=>1729, :T_ICLASS=>116378} {Thread::Mutex=>4749, Time=>3263, Proc=>154783, StringIO=>4, UnboundMethod=>1166, Thread=>2, Concurrent::AtomicFixnum=>1, Random=>2, Method=>454, Google::Protobuf::FieldDescriptor=>647, Google::Protobuf::Descriptor=>147, Google::Protobuf::OneofDescriptor=>3, :autoload_i=>663, :autoload=>94, DateTime=>20, Google::Protobuf::EnumDescriptor=>49, Goomba::Sanitizer=>2, Goomba::Selector=>32, PrettyLights::Theme=>1, PrettyLights::Library=>1, Object=>4, FFI::DynamicLibrary=>5, BigDecimal=>3, Keiko=>1, Date=>15, Encoding=>101, ThreadGroup=>1, Binding=>2, RubyVM=>1, ARGF.class=>1, Data=>1, :unknown=>5, JSON::Ext::Generator::State=>3, swig_runtime_data=>1, OpenSSL::Digest::SHA256=>2, ObjectSpace::WeakMap=>1, GRPC::Core::TimeSpec=>3, FFI::FunctionType=>44, FFI::Function=>41, FFI::DynamicLibrary::Symbol=>42, FFI::Type::Mapped=>13, FFI::StructByReference=>4, FFI::VariadicInvoker=>1, FFI::StructLayout=>5, FFI::StructLayout::Number=>8, FFI::StructLayout::Array=>1, FFI::ArrayType=>1, FFI::StructLayout::Pointer=>3, FFI::StructLayout::String=>4, FFI::StructLayout::InnerStruct=>1, FFI::StructByValue=>1, FFI::StructLayout::Mapped=>2, Google::Protobuf::DescriptorPool=>1, MessagePack::Factory=>1, Concurrent::AtomicReference=>1, Rlibmemcached::MemcachedSt=>1, OpenSSL::PKey::EC::Group=>1, OpenSSL::X509::Store=>1, OpenSSL::PKey::DH=>2, RE2::Regexp=>1, FFI::Pointer=>1, FFI::Type::Builtin=>21, OpenSSL::Cipher=>1} {:count=>60, :heap_allocated_pages=>10645, :heap_sorted_length=>10645, :heap_allocatable_pages=>0, :heap_available_slots=>4338897, :heap_live_slots=>3315218, :heap_free_slots=>1023679, :heap_final_slots=>0, :heap_marked_slots=>3314982, :heap_eden_pages=>10645, :heap_tomb_pages=>0, :total_ allocated_pages=>10645, :total_freed_pages=>0, :total_allocated_objects=>42922679, :total_freed_objects=>39607461, :malloc_increase_bytes=>5792, :malloc_increase_bytes_limit=>128000000, :minor_gc_count=>48, :major_gc_count=>12, :remembered_wb_unprotected_objects=>77656, :remembered_wb_unp rotected_objects_limit=>155312, :old_objects=>3144841, :old_objects_limit=>6289682, :oldmalloc_increase_bytes=>6240, :oldmalloc_increase_bytes_limit=>128000000} 3314982 live, 1023915 free slots 979 MB RSS Stackprof results captured in: tmp/stackprof_4.2.10.github_20180419121610.prof
  159. 13mil objects / req

  160. 300k objects / req

  161. None
  162. Make Performance Easy

  163. Make Rails Scalable by Default

  164. Make Your Team Scalable by Default

  165. One More Story

  166. None
  167. Crap Data

  168. Data I acquired (crap data) -100 925 1950 2975 4000

    Time 2017-01-23 15:53:51 2017-01-23 15:53:56 2017-01-23 15:54:01 2017-01-23 15:54:06 2017-01-23 15:54:11 2017-01-23 15:54:16 2017-01-23 15:54:21 2017-01-23 15:54:26 2017-01-23 15:54:31 2017-01-23 15:54:37 2017-01-23 15:54:42 2017-01-23 15:54:47 2017-01-23 15:54:52 2017-01-23 15:54:57 2017-01-23 15:55:02 2017-01-23 15:55:07 2017-01-23 15:55:12 2017-01-23 15:55:17 2017-01-23 15:55:22 2017-01-23 15:55:27 2017-01-23 15:55:32
  169. 40min later…

  170. Raspberry Pi + Motion Sensor

  171. Litter Box

  172. I made this Weight (g) -100 925 1950 2975 4000

    Time 2017-01-23 15:53:51 2017-01-23 15:53:56 2017-01-23 15:54:01 2017-01-23 15:54:06 2017-01-23 15:54:11 2017-01-23 15:54:16 2017-01-23 15:54:21 2017-01-23 15:54:26 2017-01-23 15:54:31 2017-01-23 15:54:37 2017-01-23 15:54:42 2017-01-23 15:54:47 2017-01-23 15:54:52 2017-01-23 15:54:57 2017-01-23 15:55:02 2017-01-23 15:55:07 2017-01-23 15:55:12 2017-01-23 15:55:17 2017-01-23 15:55:22 2017-01-23 15:55:27 2017-01-23 15:55:32 Cat Enters Cat Leaves
  173. ~100g

  174. (Literally Crap Data)

  175. None
  176. Home Automation!

  177. None
  178. None
  179. None
  180. Reed Switch

  181. None
  182. Tilt Sw itch Z-Wave Sensor

  183. On Position Tilt Sw itch Z-Wave Sensor

  184. Off Position Tilt Sw itch Z-Wave Sensor

  185. None
  186. None
  187. None
  188. Next Step: Facebook Integration

  189. None
  190. Thank You!