The fundamentals of scaling rails applications

Slide 1

Slide 1 text

FullStack Developers Israel Hosted by: !1 Fundamentals of scaling rails application ! With real world examples 15.12.2013! Google

Slide 2

Slide 2 text

WHO AM I?

Slide 3

Slide 3 text

Avi Tzurel, Ruby domain leader @KensoDev github - KensoDev

Slide 4

Slide 4 text

Tikal Expertise

Slide 5

Slide 5 text

text • Scaling a monolithic rails application • Caching • Key Value stores (Focusing on Redis) • Search indexing with NoSql enhancements (Focusing on SOLR) Agenda

Slide 6

Slide 6 text

text • Slides -> talk -> questions • Code together (moving at a normal pace) • Difficulties -> join someone else • More slides, more code, same process goes on How is this going to work?

Slide 7

Slide 7 text

text • Computer • Ruby • Vagrant • Virtual Box • Don’t have something? Please join someone else Prerequisites

Slide 8

Slide 8 text

text • Chef • Vagrant • Chef server • Redis • Memcached • SOLR (3.6) • Offloading from user thread • Micro services • Ruby & Rails (Obviously!) Tools & Concepts

Slide 9

Slide 9 text

Configuration management like a boss! Chef • Chef is a part of the “scaler” daily life • Easily (well, that’s a big fucking fat lie) manage and configure server • Chef-server with chef-client are a devops dream • Your code is your server documentation • Ruby!!!!!

Slide 10

Slide 10 text

up, hack, damn it!, destroy Vagrant • Cost effective way to “play” around with your environment configurations • Working with the same exact images as you would on your production env • Having a production env on your own machine is great for bug catching (princess env)

Slide 11

Slide 11 text

workshop-awesome Redis • Super fast key-value store on steroids • Has sets support, sorted sets, arrays and more • Not full ACID • If you don’t turn on journaling, redis may lose data on writes • Love to hate and lean the fsync() process

Slide 12

Slide 12 text

Date=12.12.2012 Memcached / Membase • Super fast memory (with disk backup if it’s membase) store • Storing values in binary • Simple yet effective API for what you need • Still running tons of caching solutions at all sizes of companies from your average 2-3 person startup to Facebook • Absolutely great for the average web app that has 80% reads with 20% writes

Slide 13

Slide 13 text

go look if i’m there Solr • Lucene based search index • Fully blown NoSql solution • The most mature solution out there • Can run on almost any Java server, Jetty, Tomcat (we’ll be running jetty) • The most misunderstood solution out there, people under-use it, we will learn how NOT to do it

Slide 14

Slide 14 text

text The basic rails application

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

text Real world scaled rails application

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

text • Genuine full stack development • Chef for configuration (chef server) • Caching layers (CDN, object cache, page cache) • Key Value storage • Messaging layers • Micro Services written in other languages • Multiple monitoring systems, multiple logs The modern scaled rails application

Slide 19

Slide 19 text

text Step back ! What is scaling?

Slide 20

Slide 20 text

“ “ Scaling - Replacing all components of a car while driving it at 100mph ! Mike Krieger - Instagram

Slide 21

Slide 21 text

Story Time Scaling a rails app, real life story

Slide 22

Slide 22 text

text Setting out to scale an app What do you want to achieve? • Highly available application • Great performance both for api users and app users • Easily manageable and configureable • Cost effective

Slide 23

Slide 23 text

text Mixing up concepts • Scaling is NOT optimization / hardware / storate • Not about less lines of code or specific piece of software • We are going to talk about DESIGNING for scale, not actually scaling • If you don’t design to scale, you won’t scale • If you design to scale, scaling will be painful • There is NO pain free scaling

Slide 24

Slide 24 text

Story Time Pains of scaling 5X in traffic ! App DESIGNED to scale

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

Questions?

Slide 27

Slide 27 text

text Lets get started

Slide 28

Slide 28 text

http://www.bit.ly/scaling-workshop-google

Slide 29

Slide 29 text

www.bit.ly/scaling-workshop-chef

Slide 30

Slide 30 text

text Having a look into our application

Slide 31

Slide 31 text

text Adding features

Slide 32

Slide 32 text

We are going to add & scale features EXACTLY like we would do in real like ! Well, not “we” but regular developers :)

Slide 33

Slide 33 text

text How do we measure? Primitively!

Slide 34

Slide 34 text

text Adding the like count

Slide 35

Slide 35 text

text Adding the review count

Slide 36

Slide 36 text

text Adding the most recent review snippet

Slide 37

Slide 37 text

Now, we need to scale this, it’s blowing up in production

Slide 38

Slide 38 text

Caching

Slide 39

Slide 39 text

Basics • Base class is ActiveSupport::Cache::Store • Has multiple adapters • Redis • Memcached • File (don’t) • Memory (don’t) Caching

Slide 40

Slide 40 text

Types • Full page caching • Action caching • Object caching • Partial caching Caching

Slide 41

Slide 41 text

API • Read • Write • Fetch (decorator for read/write) • Exist? ! NO Get all keys Search for keys Append, Prepend ! Nothing, learn to love it Caching

Slide 42

Slide 42 text

“ “ There are only two hard things in Computer Science: cache invalidation and naming things. ! ! -- Phil Karlton

Slide 43

Slide 43 text

I DO NOT recommend using anything other than partial caching in your rails application ! Caching solution explained

Slide 44

Slide 44 text

text Adding partial caching

Slide 45

Slide 45 text

text Explaining Cache Keys

Slide 46

Slide 46 text

text product:#{id}:review_snippet Model Name id field/property/item

Slide 47

Slide 47 text

text Invalidating the cache

Slide 48

Slide 48 text

after_commit on the model

Slide 49

Slide 49 text

Problems?

Slide 50

Slide 50 text

• Magic strings • Knowledge of keys are in multiple places • Typos break cache invalidation Problems

Slide 51

Slide 51 text

text Using the model for the key

Slide 52

Slide 52 text

text [product, ‘review_snippet’] Model Instance field / property

Slide 53

Slide 53 text

text Invalidating the cache

Slide 54

Slide 54 text

after_commit on the model

Slide 55

Slide 55 text

Exercise ! Cache (partial) all the products on the page, invalidate the cache when a new product is created or when an existing product is updated ! 30 minutes

Slide 56

Slide 56 text

checkout step2 from remote ! git fetch origin ! git checkout -t origin/step2

Slide 57

Slide 57 text

API • http://redis.io • get/set • sets • hashes • sorted sets • Best practice for keys “object_name:#{id}:#{key}” • product:2:likes • product:2:like_count • product:2:like Redis

Slide 58

Slide 58 text

text Adding counts to redis (using hashes)

Slide 59

Slide 59 text

text Adding search

Slide 60

Slide 60 text

text Searching using SOLR

Slide 61

Slide 61 text

text Why use SOLR only for search?

Slide 62

Slide 62 text

text SOLR ALL THE THINGZ!!!

Slide 63

Slide 63 text

THANK YOU Avi Tzurel Email: [email protected]