2017-03 Lisa Guo: Scaling Instagram Infrastructure (QCon London 2017, 87p)

Source:
https://www.infoq.com/presentations/instagram-scale-infrastructure/
https://www.youtube.com/watch?v=hnpzNAPiC0E

Transcript

1. SCALING INSTAGRAM INFRA Lisa Guo— March 7th, 2017 lguo@instagram.com
3. 2017 INSTAGRAM HISTORY 2010 2012/4/9 joined Facebook 2014/1 600M users/month
4. INSTAGRAM EVERYDAY 400 Million Users 4+ Billion likes 100 Million photo/video uploads Top account: 110 Million followers
6. SCALING MEANS Scale out Scale up Scale dev team
7. SCALE OUT
8. SCALE OUT
10. SCALE OUT
13. “Let’s all pray that Amazon gets everything sorted out in short order.”
14. INSTAGRAM STACK memcache RabbitMQ PostgreSQL Cassandra Celery Other Services Django
15. STORAGE VS. COMPUTING • Storage: needs to be consistent across data centers • Computing: driven by user traffic, as needed basis
16. SCALE OUT: STORAGE user, media, friendship etc
17. SCALE OUT: STORAGE user, media, friendship etc Master Replica Replica Django Write Read
18. SCALE OUT: STORAGE user, media, friendship etc Master Replica Replica Django Write Read DC1 DC2 DC3
19. SCALE OUT: STORAGE user feeds, activities etc Replica Replica Replica Write - 2 Read - 1
20. SCALE OUT: STORAGE user feeds, activities etc Replica Replica Replica Write - 2 Read - 1
21. COMPUTING
22. Django RabbitMQ PostgreSQL Cassandra Celery Django RabbitMQ PostgreSQL Cassandra Celery memcache DC1 DC2 memcache
23. MEMCACHE • High performance key-value store in memory • Millions of reads/writes per second • Sensitive to network condition • Cross region operation is prohibitive No global consistency
24. feed get Django User R DC1 Django PostgreSQL memcache User C comment set insert
25. Django memcache PostgreSQL User C comment insert set DC1 Django memcache PostgreSQL User R feed get DC2 replication
26. Django memcache PostgreSQL User C comment insert set DC1 Django memcache PostgreSQL User R feed DC2 replication Cache invalidate Cache invalidate get
27. COUNTERS select count(*) from user_likes_media where media_id=12345; 100s ms
28. COUNTER select count from media_likes where media_id=12345; 10s us
29. Cache invalidated All djangos try to access DB
30. MEMCACHE LEASE d1 d2 memcache db time lease-get fill lease-get wait or use stale read from DB lease-set lease-get hit
31. INSTAGRAM STACK - MULTI REGION Django RabbitMQ PostgreSQL Cassandra Celery memcache Django RabbitMQ PostgreSQL Cassandra Celery memcache DC1 DC2
32. SCALING OUT • Capacity • Reliability • Regional failure ready
33. SCALING OUT - CHALLENGES, OPPORTUNITIES • Beyond North America • More localized social network • Direct messaging • Live streaming
34. 20 40 60 80 100 0 2 4 6 8 10 12 14 16 18 20 22 24 User growth Server growth
35. “Don’t count the servers, make the servers count”
36. SCALE UP
37. SCALE UP Use as few CPU instructions as possible Use as few servers as possible
38. SCALE UP Use as few CPU instructions as possible Use as few servers as possible
39. CPU Monitor Optimize Analyze
40. COLLECT struct perf_event_attr pe; pe.type = PERF_TYPE_HARDWARE; pe.config = PERF_COUNT_HW_INSTRUCTIONS; fd = perf_event_open(&pe, 0, -1, -1, 0); ioctl(fd, PERF_EVENT_IOC_ENABLE, 0); <code you want to measure> ioctl(fd, PERF_EVENT_IOC_DISABLE, 0); read(fd, &count, sizeof(long long));
41. DYNOSTATS 20 40 60 80 100 0 2 4 6 8 10 12 14 16 18 20 22 24 Follow Feed Explore
42. REGRESSION 20 40 60 80 100 0 2 4 6 8 10 12 14 16 18 20 22 24
43. With new feature Without new feature
45. CPU Monitor Optimize Analyze
46. PYTHON CPROFILE import cProfile, pstats, StringIO pr = cProfile.Profile() pr.enable() # ... do something ... pr.disable() s = StringIO.StringIO() sortby = 'cumulative' ps = pstats.Stats(pr, stream=s).sort_stats(sortby) ps.print_stats() print s.getvalue()
48. CPU - ANALYZE continuous profiling generate_profile explore --start <start-time> --duration <minutes>
49. CPU - ANALYZE continuous profiling 20 40 60 80 100 0 2 4 6 8 10 12 14 16 18 20 22 24 Caller Callee Callee
51. CPU Monitor Optimize Analyze
52. igcdn-photos-d-a.akamaihd.net/hphotos-ak-xpl1/t51.2885-19/ s300x300/12345678_1234567890_987654321_a.jpg
53. igcdn-photos-d-a.akamaihd.net/hphotos-ak-xpl1/t51.2885-19/ s150x150/12345678_1234567890_987654321_a.jpg igcdn-photos-d-a.akamaihd.net/hphotos-ak-xpl1/t51.2885-19/ s400x600/12345678_1234567890_987654321_a.jpg igcdn-photos-d-a.akamaihd.net/hphotos-ak-xpl1/t51.2885-19/ s200x200/12345678_1234567890_987654321_a.jpg igcdn-photos-d-a.akamaihd.net/hphotos-ak-xpl1/t51.2885-19/ s300x300/12345678_1234567890_987654321_a.jpg
54. CPU - OPTIMIZE
55. igcdn-photos-d-a.akamaihd.net/hphotos-ak-xpl1/t51.2885-19/ s300x300/12345678_1234567890_987654321_a.jpg 150x150 400x600 200x200
56. CPU - OPTIMIZE C is really faster • Candidate functions: • Used extensively • Stable • Cython or C/C++
57. Use as few CPU instructions as possible Use as few servers as possible SCALE UP
58. ONE WEB SERVER Process 1 Shared Memory Private Memory Process N
59. SCALE UP: MEMORY • Run in optimized mode (-O) • Remove dead code Reduce code
60. SCALE UP: MEMORY • Move configuration into shared memory • Disable garbage collection Share more
61. SCALE UP: MEMORY 20+% capacity increase
62. SCALE UP: NETWORK LATENCY Synchronous processing model with long latency ===> Worker starvation and fewer CPU instr executed
63. Stories Feed Django Feed Stories Suggested Users ASYNC IO
64. Use as few CPU instructions as possible Use as few servers as possible Scale up
65. SCALE UP: CHALLENGES, OPPORTUNITIES • Faster python run-time • Async web framework • Better memory analysis • etc etc
66. SCALE DEV TEAM
67. SCALING TEAM 30% engineers joined in last 6 months Bootcampers - 1 week Hack-A-Month - 4 weeks Intern - 12 weeks
68. Comment Filtering Self-harm Prevention Windows App Multiple media in one post Video View Notification Saved Posts First Story Notification Instagram Live Instagram Stories
69. Which server? NewTable or New Column? What Index? Should I cache it? Will I lock up DB? Will I bring down Instagram?
70. WHAT WE WANT • Automatically handle cache • Define relations, not worry about implementations • Self service by product engineers • Infra focuses on scale
71. TAO USER1 USER2 USER3 media posted posted by likes liked by likes liked by
72. Comment Filtering Self-harm Prevention Windows App Multiple media in one post Video View Notification Saved Posts First Story Notification Instagram Live Instagram Stories
73. SOURCE CONTROL Master Live Direct
74. SOURCE CONTROL • Context switching • Code sync/merge overhead • Surprises • Refactor/major upgrade • Performance tracking harder With branches
75. SOURCE CONTROL Master Live Direct
76. SOURCE CONTROL Master Live Direct
77. SOURCE CONTROL • Continous integration • Collaborate easily • Fast bisect and revert • Continuous performance monitoring No branches
78. FEATURE LAUNCH Engineers Employees Dogfooder Some demographics World
79. FEATURE LOAD TEST
80. Once a 40-60 rollouts per day day diff week? !!
81. CHECKS AND BALANCES Code review unittest Code accepted committed Canary To the Wild
83. SCALING MEANS Scale out Scale up Scale dev team
85. TAKEAWAYS Scaling is everybody’s responsibility Scaling is continuous effort Scaling is multi-dimensional
86. QUESTIONS?