DON'T PANIC: Large scale web development

In Tuenti, all features we develop have to be ready to scale to millions of users when the code is released to production. By using caching patterns, priming caches, dark-launches and other strategies we try to ensure that the infrastructure will cope with the load. In this talk we want to give real, not-trivial examples of code written to scale... and other cool things about scalability in general.

Transcript

1. Jorge Lería & William Viana DON’T PANIC! Large scale web development {jorge@tuenti.com, william@tuenti.com} - Tuenti
3. in general
4. Jorge Lería @jorgeleria jorge@tuenti.com
5. William Viana @vianasw william@tuenti.com
7. WILL
21. Category: scalability Level: advanced Language: spanish Speakers: 2
22. THIS IS NOT ROCKET SCIENCE } {
23. BUT COULD BE DANGEROUS } {
24. “The effect of a small mistake with that amount of users can be a disaster…”
25. Disaster example Back in 2010 we had games in Tuenti We had a system to share sessions with third parties At first it wasn’ t using our Backend Framework And we launched GreenFarm
26. Disaster example /; ;\ __ \\____// COW /{_\_/ `'\____ \___ (o) (o } _____________________________/ :--' ,-,'`@@@@@@@@ @@@@@@ \_ `__\ ;:( @@@@@@@@@ @@@ \___(o'o) :: ) @@@@ @@@@@@ ,'@@( `====' :: : @@@@@: @@@@ `@@@: :: \ @@@@@: @@@@@@@) ( '@@@' ;; /\ /`, @@@@@@@@@\ :@@@@@) ::/ ) {_----------------: :~`,~~; ;;'`; : ) : / `; ; ;;;; : : ; : ; ; : `'`' / : : : : : : )_ \__; ";" :_ ; \_\ `,',' :__\ \ * `,'* \ \ : \ * 8`;'* * `^' \ :/ `^' `-^-' \v/ : \/ Back in 2010 we had games in Tuenti We had a system to share sessions with third parties At first it wasn’ t using our Backend Framework And we launched GreenFarm
27. Disaster example /; ;\ __ \\____// COW /{_\_/ `'\____ \___ (o) (o } _____________________________/ :--' ,-,'`@@@@@@@@ @@@@@@ \_ `__\ ;:( @@@@@@@@@ @@@ \___(o'o) :: ) @@@@ @@@@@@ ,'@@( `====' :: : @@@@@: @@@@ `@@@: :: \ @@@@@: @@@@@@@) ( '@@@' ;; /\ /`, @@@@@@@@@\ :@@@@@) ::/ ) {_----------------: :~`,~~; ;;'`; : ) : / `; ; ;;;; : : ; : ; ; : `'`' / : : : : : : )_ \__; ";" :_ ; \_\ `,',' :__\ \ * `,'* \ \ : \ * 8`;'* * `^' \ :/ `^' `-^-' \v/ : \/ We had a problem with concurrency We detected duplicated gamer IDs The database was corrupted No backup (it happened during the launch)
28. Disaster example /; ;\ __ \\____// DEAD COW /{_\_/ `'\____ \___ X X } _____________________________/ :--' ,-,'`@@@@@@@@ @@@@@@ \_ `__\ ;:( @@@@@@@@@ @@@ \___(o'o) :: ) @@@@ @@@@@@ ,'@@( `----' :: : @@@@@: @@@@ `@@@: |_| :: \ @@@@@: @@@@@@@) ( '@@@' ;; /\ /`, @@@@@@@@@\ :@@@@@) ::/ ) {_----------------: :~`,~~; ;;'`; : ) : / `; ; ;;;; : : ; : ; ; : `'`' / : : : : : : )_ \__; ";" :_ ; \_\ `,',' :__\ \ * `,'* \ \ : \ * 8`;'* * `^' \ :/ `^' `-^-' \v/ : \/ We had a problem with concurrency We detected duplicated gamer IDs The database was corrupted No backup (it happened during the launch) = RESET THE GAME
30. a.- A hat b.- A boa constrictor digesting an elephant c.- Something much more frightening
31. a.- A hat b.- A boa constrictor digesting an elephant c.- Something much more frightening
33. Tuenti... do you still use that?
34. Three days ago we were #2
35. ~16.000.000 users
36. ~3.200.000.000 images
37. How do we manage all of that?
38. With a ton of servers... +1000 servers 550 frontend (PHP) servers. 300 database (MySQL) servers. 100 memcached servers. 100 image servers. etc. (thumbnails, photo processors, chat, batch, dev, stats, backups, HBase).
39. ...and a great team
40. Technologies PHP MySQL memcache HBase Hadoop Hive HipHop nginx Erlang ...
41. Technologies PHP MySQL memcache HBase Hadoop Hive HipHop nginx Erlang ... }Free software (free as in freedom)
42. In software engineering, scalability is the ability of a system, network, or process, to handle growing amounts of work in a graceful manner or its ability to be enlarged to accommodate that growth. It's all about handling growth
43. What is NOT scalability Scalability ≠ Speed Scalability ≠ Throughput (Sometimes slower code is better for scalability)
45. social network != other sites
46. Photos Notifications Master slave architecture Schema divided into logical blocks One cluster per data type
47. Partitioning not limited to databases Partitioned memcache and frontend farms Databases
48. “All magic comes with a price… ...in complexity” - No longer relational (no joins or counts) - Denormalization
49. Thankfully, we have a very good Backend Framework
50. “Backend Framework team is the team responsible for the data access layer in Tuenti, providing services for persistent storage interaction and their support.”
51. Backend framework All data partitioning is transparent Fully integrated memcache as a storage layer Data structures like “Cached lists”
52. Feature disabling/enabling “Allows you to gradually open a feature or disable one that is broken”
54. Dark launches Usually done when we want to fill a cache, before a feature is actually released
55. Asynchronous jobs Running on a queue system with throthing enable to be used for heavy operations.
56. Staging “Test and review a newer version before it is moved into production by making users go to a concrete server where the new code is deployed”
58. Before we start: what makes a photo popular? how much is it going to be used? is it going to be slow? what are our constraints?
59. So, we run some numbers, and decide how to implement a MVP
60. 1st approach: Let’s build something quick with the existing backend (but that still scales)
61. For every user friend...
62. Photos For every user friend... Retrieve all of their photos
63. Photos Likes Comments For every user friend... Retrieve all of their photos And for every photo, retrieve all the likes and comments
64. Of course, this doesn’t scale very well
65. We can do better
66. B F H Users with activities
67. B F H Status Photo Comment Status Photo Photo Users with activities User Events
68. B F H Status Photo Comment Status Photo Photo Users with activities User Events Likes Comments
69. Store the result in a cache
70. So we released it to the world, crossed our fingers, and opened it up little by little, with an eye in the graphs
71. Well, at least that was the plan...
72. Well, at least that was the plan...
73. After the hotfix, it kind of worked and it was enough to validate the product.
75. “Don't listen to the direct reaction, just look at the numbers” Facebook guy (yesterday in the speech)
76. 2nd approach: The generative approach
77. For every user, store every like and comment in a photo into a memcache storage
78. B F H Like on Photo Comment Comment Like on Photo Comment Comment Users with activities Photo events Much easier Smaller memory footprint
83. Do the dark launch, wait to the caches to fill in, and switch to the new implementation
84. Profit
85. Until the beliebers came...
86. “It’s not enough to make it work for the average user, you’ll have to make it for the power users with power user friends as well.” Socrates -
87. Any questions? @jorgeleria @vianasw @TuentiEng