DevOps in the wild - Gogobot use case

Talk I did in a DevOps conference in Israel.

Transcript

1. ݆ૉ DevOps in the wild Gogobot use-case Avi Tzurel Kenso 1 Wednesday, July 18, 12
2. Who Am I? http://twitter.com/kensodev http://kensodev.com http://github.com/kensodev http://avi.io 2 Wednesday, July 18, 12
3. What is Gogobot? 3 Wednesday, July 18, 12
4. Social Recommendation Engine For Travel 4 Wednesday, July 18, 12
5. Personalized Recommendations From Friends, not strangers 5 Wednesday, July 18, 12
6. The Gogobot Architecture 6 Wednesday, July 18, 12
7. How we make shit work? 7 Wednesday, July 18, 12
8. CDN + Reverse Proxy 8 Wednesday, July 18, 12
9. CDN • Edge servers across the world (hundreds) • Caches static assets • CSS, JS, Images • Caches full pages (with smart expire API) 9 Wednesday, July 18, 12
10. Reverse Proxy • Super fast connection from edges means users get a local experience, with minimal latency • Cache miss? get from load balancer • Logged out traffic almost never hits or affects logged in traffic • scale differently, control differently 10 Wednesday, July 18, 12
11. CDN + Reverse Proxy 11 Wednesday, July 18, 12
12. CDN + Reverse Proxy Front End Back End 12 Wednesday, July 18, 12
13. Front End • Serves user facing content • Communicates with cache layer and the DB • No heavy lifting, respond to the user as fast as possible 13 Wednesday, July 18, 12
14. Back End • Serves realtime services (Facebook, Twitter) • Heavy lifting thrown at it from the Front End Machines • Hosts workers for the queue service 14 Wednesday, July 18, 12
15. CDN + Reverse Proxy Front End Back End 15 Wednesday, July 18, 12
16. Front End Back End Cache 16 Wednesday, July 18, 12
17. Cache • Memcached cluster • Memcached 1.4+ • 6+ Machines • 4.5 - 15K operations per second • Hosted on Amazon ElastiCache • Solved tons of problems with memcached dying 17 Wednesday, July 18, 12
18. Front End Back End Cache 18 Wednesday, July 18, 12
19. Cache MySql MongoDB Redis SOLR 19 Wednesday, July 18, 12
20. MySql • Master + 2 Slaves • 64G memory for each machine with 400G storage • Hourly backups • Used as the main persistence layer for the site • Reads are from slaves, writes are from master • Logged out traffic will never have access the master 20 Wednesday, July 18, 12
21. MySql • EBS snapshots are used as backups • Multi region support for Amazon (1a, 1b, 1c) 21 Wednesday, July 18, 12
22. MongoDB • 3 Shard • 3 Replica sets in each shards • 16HD (100G) raid on each machine • 64G memory for each machine to provide out of memory index for all queries • Used for the Graph Engine + Scoring system 22 Wednesday, July 18, 12
23. Redis • Used for key+value store • Cache tagging solution on top of memcached • Queue services is hosted on Redis • Master / Slave replica • different redis cluster for different things 23 Wednesday, July 18, 12
24. Redis • Different Redis clusters for each need • Indexer • Cache tagging • Realtime push with Node.js to the client • When one down, others behave normally 24 Wednesday, July 18, 12
25. SOLR • Search index • NoSQL (Schema Less) • Master/Slave • Slave on each app machine, single master • Eventual consistency 25 Wednesday, July 18, 12
26. Numbers • 400m+ graph users • 10K triggers per user in the scoring system • Grew ~170X last 18 months • Announced 1m registered users a month ago • hit 2m registered users 2 weeks agter 26 Wednesday, July 18, 12
27. Numbers • HUGE growth in a relatively short time 27 Wednesday, July 18, 12
28. 99.9% uptime 28 Wednesday, July 18, 12
29. ~200ms avg server response time 29 Wednesday, July 18, 12
30. Logged out user gets a page in 30ms 30 Wednesday, July 18, 12
31. How can you manage all of this? 31 Wednesday, July 18, 12
32. Not too technical 32 Wednesday, July 18, 12
33. Embed it in your culture • Developers should support end users • Get Satisfaction for example • Bugs / Problems must be engaged early and often • Be completely and utterly open 33 Wednesday, July 18, 12
34. Communication • Single and agreed line of communication for everything • Chatroom through the day • Pull requests for code review • Email for updates • SMS notification for urgent stuff 34 Wednesday, July 18, 12
35. No QA! 35 Wednesday, July 18, 12
36. TDD 36 Wednesday, July 18, 12
37. Twitter Driven Deployment 37 Wednesday, July 18, 12
38. Use UTest for manual testing Daily 38 Wednesday, July 18, 12
39. Chatroom 39 Wednesday, July 18, 12
40. 40 Wednesday, July 18, 12
41. Keep others involved 41 Wednesday, July 18, 12
42. Ask Questions 42 Wednesday, July 18, 12
43. AUTOMATE! 43 Wednesday, July 18, 12
44. Build and Deploy 44 Wednesday, July 18, 12
45. 45 Wednesday, July 18, 12
46. the Gbot 46 Wednesday, July 18, 12
47. gbot deploy production 47 Wednesday, July 18, 12
48. What can he do? (if you ask him nicely) Or sudo 48 Wednesday, July 18, 12
49. Gbot • Deploy anything, anywhere, anytime • Tell Jokes • Remind us about bugs • Run custom builds • Alert on server issues • cheer us up 49 Wednesday, July 18, 12
50. 50 Wednesday, July 18, 12
51. <keyword> tweet - Returns a link to a tweet about <keyword> 51 Wednesday, July 18, 12
52. Monitor 52 Wednesday, July 18, 12
53. Monitor • Monit • CPU, Memory, Server health • God • process lifecycle 53 Wednesday, July 18, 12
54. Not just Up / Down 54 Wednesday, July 18, 12
55. Measure 55 Wednesday, July 18, 12
56. Measure • NewRelic • measure performance of everything • Ruby • MySql • Memcached • External Services 56 Wednesday, July 18, 12
57. 57 Wednesday, July 18, 12
58. 58 Wednesday, July 18, 12
59. Share! 59 Wednesday, July 18, 12
60. Share internally and externally 60 Wednesday, July 18, 12
61. 61 Wednesday, July 18, 12
62. github.com/gogobot/laptop 62 Wednesday, July 18, 12
63. Chef Recipes 63 Wednesday, July 18, 12
64. Sharing • Share your ideas • talk about features, future plans • plan together • open source your brain 64 Wednesday, July 18, 12
65. Thank you! 65 Wednesday, July 18, 12
66. Questions? 66 Wednesday, July 18, 12