Using Ruby & Rails in the wild - Gogobot use case

Slide 1

Slide 1 text

݆ૉ Ruby & Rails In the wild Gogobot use-case Avi Tzurel Gogobot Monday, September 10, 12

Slide 2

Slide 2 text

Who Am I? Avi Tzurel http://avi.io http://twitter.com/kensodev http://kensodev.com http://github.com/kensodev Monday, September 10, 12

Slide 3

Slide 3 text

What is Gogobot? Monday, September 10, 12

Slide 4

Slide 4 text

Social Recommendation Engine For Travel Monday, September 10, 12

Slide 5

Slide 5 text

Personalized Recommendations From Friends, not strangers Monday, September 10, 12

Slide 6

Slide 6 text

The Gogobot Architecture Monday, September 10, 12

Slide 7

Slide 7 text

How we make shit work? Monday, September 10, 12

Slide 8

Slide 8 text

CDN + Reverse Proxy Monday, September 10, 12

Slide 9

Slide 9 text

CDN • Edge servers across the world (hundreds) • Caches static assets • CSS, JS, Images • Caches full pages (with smart expire API) Monday, September 10, 12

Slide 10

Slide 10 text

Reverse Proxy • Super fast connection from edges means users get a local experience, with minimal latency • Cache miss? get from load balancer • Logged out traffic almost never hits or affects logged in traffic • scale differently, control differently Monday, September 10, 12

Slide 11

Slide 11 text

CDN + Reverse Proxy Monday, September 10, 12

Slide 12

Slide 12 text

CDN + Reverse Proxy Front End Back End Monday, September 10, 12

Slide 13

Slide 13 text

Front End • Serves user facing content • Communicates with cache layer and the DB • No heavy lifting, respond to the user as fast as possible Monday, September 10, 12

Slide 14

Slide 14 text

Back End • Serves realtime services (Facebook, Twitter) • Heavy lifting thrown at it from the Front End Machines • Hosts workers for the queue service Monday, September 10, 12

Slide 15

Slide 15 text

CDN + Reverse Proxy Front End Back End Monday, September 10, 12

Slide 16

Slide 16 text

Front End Back End Cache Monday, September 10, 12

Slide 17

Slide 17 text

Cache • Memcached cluster • Memcached 1.4+ • 6+ Machines • 4.5 - 15K operations per second • Hosted on Amazon ElastiCache • Solved tons of problems with memcached dying Monday, September 10, 12

Slide 18

Slide 18 text

Front End Back End Cache Monday, September 10, 12

Slide 19

Slide 19 text

Cache MySql MongoDB Redis SOLR Monday, September 10, 12

Slide 20

Slide 20 text

MySql • Master + 2 Slaves • 64G memory for each machine with 400G storage • Hourly backups • Used as the main persistence layer for the site • Reads are from slaves, writes are from master • Logged out traffic will never have access the master Monday, September 10, 12

Slide 21

Slide 21 text

MySql • EBS snapshots are used as backups • Multi region support for Amazon (1a, 1b, 1c) Monday, September 10, 12

Slide 22

Slide 22 text

MongoDB • 6 Shard • 3 Replica sets in each shards • 16HD (100G) raid on each machine • 64G memory for each machine to provide out of memory index for all queries • Used for the Graph Engine + Scoring system Monday, September 10, 12

Slide 23

Slide 23 text

Monday, September 10, 12

Slide 24

Slide 24 text

Redis • Used for key+value store • Cache tagging solution on top of memcached • Queue services is hosted on Redis • Master / Slave replica • different redis cluster for different things Monday, September 10, 12

Slide 25

Slide 25 text

Redis • Different Redis clusters for each need • Indexer • Cache tagging • Realtime push with Node.js to the client • When one down, others behave normally Monday, September 10, 12

Slide 26

Slide 26 text

SOLR • Search index • NoSQL (Schema Less) • Master/Slave • Slave on each app machine, single master • Eventual consistency Monday, September 10, 12

Slide 27

Slide 27 text

Numbers • 400m+ graph users • 10K triggers per user in the scoring system • Grew ~170X last 18 months • Announced 1m registered users a month ago • hit 2m registered users 2 weeks agter Monday, September 10, 12

Slide 28

Slide 28 text

Numbers • HUGE growth in a relatively short time Monday, September 10, 12

Slide 29

Slide 29 text

99.9% uptime Monday, September 10, 12

Slide 30

Slide 30 text

~200ms avg server response time Monday, September 10, 12

Slide 31

Slide 31 text

Logged out user gets a page in 30ms Monday, September 10, 12

Slide 32

Slide 32 text

How can you manage all of this? Monday, September 10, 12

Slide 33

Slide 33 text

Not too technical Monday, September 10, 12

Slide 34

Slide 34 text

Full Ruby & Rails shop Monday, September 10, 12

Slide 35

Slide 35 text

Ruby will increase developer happiness (That’s what it was designed for) Monday, September 10, 12

Slide 36

Slide 36 text

Community Drives you towards Best Practices Monday, September 10, 12

Slide 37

Slide 37 text

TDD Monday, September 10, 12

Slide 38

Slide 38 text

“Sometimes I want to do Ruby because it means I’m more likely to be able to TDD.” Greg Malcolm Monday, September 10, 12

Slide 39

Slide 39 text

Continuos Integration Monday, September 10, 12

Slide 40

Slide 40 text

Continuous Deployment Monday, September 10, 12

Slide 41

Slide 41 text

Monday, September 10, 12

Slide 42

Slide 42 text

Machine Configuration using Chef Monday, September 10, 12

Slide 43

Slide 43 text

SOA Application with Rails Engines Monday, September 10, 12

Slide 44

Slide 44 text

EVERYTHING IS OPEN SOURCE! Monday, September 10, 12

Slide 45

Slide 45 text

Laptop Monday, September 10, 12

Slide 46

Slide 46 text

Perform Later Monday, September 10, 12

Slide 47

Slide 47 text

Rails fixes & features Monday, September 10, 12

Slide 48

Slide 48 text

Resque Monday, September 10, 12

Slide 49

Slide 49 text

Methodologies and sharing ideas Monday, September 10, 12

Slide 50

Slide 50 text

CI Configuration Monday, September 10, 12

Slide 51

Slide 51 text

Full pages crawler Monday, September 10, 12

Slide 52

Slide 52 text

and more... http://github.com/gogobot http://github.com/kensodev Monday, September 10, 12

Slide 53

Slide 53 text

Super small team Monday, September 10, 12

Slide 54

Slide 54 text

Gets a lot done Monday, September 10, 12

Slide 55

Slide 55 text

No QA! Monday, September 10, 12

Slide 56

Slide 56 text

Rails is awesome Monday, September 10, 12

Slide 57

Slide 57 text

“Ruby on Rails is astounding. Using it is like watching a kung-fu movie, where a dozen bad-ass frameworks prepare to beat up the little newcomer only to be handed their asses in a variety of imaginative ways.” Nathan Tarkington, O'Reilly Publishing Monday, September 10, 12

Slide 58

Slide 58 text

No other language / framework is able to do such things Monday, September 10, 12