2017-03 Lisa Guo: Scaling Instagram Infrastructure (QCon London 2017, 87p)

Slide 1

Slide 1 text

SCALING INSTAGRAM INFRA Lisa Guo— March 7th, 2017 lguo@instagram.com

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

2017 INSTAGRAM HISTORY 2010 2012/4/9 joined Facebook 2014/1 600M users/month

Slide 4

Slide 4 text

INSTAGRAM EVERYDAY 400 Million Users 4+ Billion likes 100 Million photo/video uploads Top account: 110 Million followers

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

SCALING MEANS Scale out Scale up Scale dev team

Slide 7

Slide 7 text

SCALE OUT

Slide 8

Slide 8 text

SCALE OUT

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

SCALE OUT

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

“Let’s all pray that Amazon gets everything sorted out in short order.”

Slide 14

Slide 14 text

INSTAGRAM STACK memcache RabbitMQ PostgreSQL Cassandra Celery Other Services Django

Slide 15

Slide 15 text

STORAGE VS. COMPUTING • Storage: needs to be consistent across data centers • Computing: driven by user traﬃc, as needed basis

Slide 16

Slide 16 text

SCALE OUT: STORAGE user, media, friendship etc

Slide 17

Slide 17 text

SCALE OUT: STORAGE user, media, friendship etc Master Replica Replica Django Write Read

Slide 18

Slide 18 text

SCALE OUT: STORAGE user, media, friendship etc Master Replica Replica Django Write Read DC1 DC2 DC3

Slide 19

Slide 19 text

SCALE OUT: STORAGE user feeds, activities etc Replica Replica Replica Write - 2 Read - 1

Slide 20

Slide 20 text

SCALE OUT: STORAGE user feeds, activities etc Replica Replica Replica Write - 2 Read - 1

Slide 21

Slide 21 text

COMPUTING

Slide 22

Slide 22 text

Django RabbitMQ PostgreSQL Cassandra Celery Django RabbitMQ PostgreSQL Cassandra Celery memcache DC1 DC2 memcache

Slide 23

Slide 23 text

MEMCACHE • High performance key-value store in memory • Millions of reads/writes per second • Sensitive to network condition • Cross region operation is prohibitive No global consistency

Slide 24

Slide 24 text

feed get Django User R DC1 Django PostgreSQL memcache User C comment set insert

Slide 25

Slide 25 text

Django memcache PostgreSQL User C comment insert set DC1 Django memcache PostgreSQL User R feed get DC2 replication

Slide 26

Slide 26 text

Django memcache PostgreSQL User C comment insert set DC1 Django memcache PostgreSQL User R feed DC2 replication Cache invalidate Cache invalidate get

Slide 27

Slide 27 text

COUNTERS select count(*) from user_likes_media where media_id=12345; 100s ms

Slide 28

Slide 28 text

COUNTER select count from media_likes where media_id=12345; 10s us

Slide 29

Slide 29 text

Cache invalidated All djangos try to access DB

Slide 30

Slide 30 text

MEMCACHE LEASE d1 d2 memcache db time lease-get ﬁll lease-get wait or use stale read from DB lease-set lease-get hit

Slide 31

Slide 31 text

INSTAGRAM STACK - MULTI REGION Django RabbitMQ PostgreSQL Cassandra Celery memcache Django RabbitMQ PostgreSQL Cassandra Celery memcache DC1 DC2

Slide 32

Slide 32 text

SCALING OUT • Capacity • Reliability • Regional failure ready

Slide 33

Slide 33 text

SCALING OUT - CHALLENGES, OPPORTUNITIES • Beyond North America • More localized social network • Direct messaging • Live streaming

Slide 34

Slide 34 text

20 40 60 80 100 0 2 4 6 8 10 12 14 16 18 20 22 24 User growth Server growth

Slide 35

Slide 35 text

“Don’t count the servers, make the servers count”

Slide 36

Slide 36 text

SCALE UP

Slide 37

Slide 37 text

SCALE UP Use as few CPU instructions as possible Use as few servers as possible

Slide 38

Slide 38 text

SCALE UP Use as few CPU instructions as possible Use as few servers as possible

Slide 39

Slide 39 text

CPU Monitor Optimize Analyze

Slide 40

Slide 40 text

COLLECT struct perf_event_attr pe; pe.type = PERF_TYPE_HARDWARE; pe.config = PERF_COUNT_HW_INSTRUCTIONS; fd = perf_event_open(&pe, 0, -1, -1, 0); ioctl(fd, PERF_EVENT_IOC_ENABLE, 0);


ioctl(fd, PERF_EVENT_IOC_DISABLE, 0);
read(fd, &count, sizeof(long long));

Slide 41

Slide 41 text

DYNOSTATS 20 40 60 80 100 0 2 4 6 8 10 12 14 16 18 20 22 24 Follow Feed Explore

Slide 42

Slide 42 text

REGRESSION 20 40 60 80 100 0 2 4 6 8 10 12 14 16 18 20 22 24

Slide 43

Slide 43 text

With new feature Without new feature

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

CPU Monitor Optimize Analyze

Slide 46

Slide 46 text

PYTHON CPROFILE import cProfile, pstats, StringIO pr = cProfile.Profile() pr.enable() # ... do something ... pr.disable() s = StringIO.StringIO() sortby = 'cumulative' ps = pstats.Stats(pr, stream=s).sort_stats(sortby) ps.print_stats() print s.getvalue()

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

CPU - ANALYZE continuous proﬁling generate_proﬁle explore --start --duration

Slide 49

Slide 49 text

CPU - ANALYZE continuous proﬁling 20 40 60 80 100 0 2 4 6 8 10 12 14 16 18 20 22 24 Caller Callee Callee

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

CPU Monitor Optimize Analyze

Slide 52

Slide 52 text

igcdn-photos-d-a.akamaihd.net/hphotos-ak-xpl1/t51.2885-19/ s300x300/12345678_1234567890_987654321_a.jpg

Slide 53

Slide 53 text

igcdn-photos-d-a.akamaihd.net/hphotos-ak-xpl1/t51.2885-19/ s150x150/12345678_1234567890_987654321_a.jpg igcdn-photos-d-a.akamaihd.net/hphotos-ak-xpl1/t51.2885-19/ s400x600/12345678_1234567890_987654321_a.jpg igcdn-photos-d-a.akamaihd.net/hphotos-ak-xpl1/t51.2885-19/ s200x200/12345678_1234567890_987654321_a.jpg igcdn-photos-d-a.akamaihd.net/hphotos-ak-xpl1/t51.2885-19/ s300x300/12345678_1234567890_987654321_a.jpg

Slide 54

Slide 54 text

CPU - OPTIMIZE

Slide 55

Slide 55 text

igcdn-photos-d-a.akamaihd.net/hphotos-ak-xpl1/t51.2885-19/ s300x300/12345678_1234567890_987654321_a.jpg 150x150 400x600 200x200

Slide 56

Slide 56 text

CPU - OPTIMIZE C is really faster • Candidate functions: • Used extensively • Stable • Cython or C/C++

Slide 57

Slide 57 text

Use as few CPU instructions as possible Use as few servers as possible SCALE UP

Slide 58

Slide 58 text

ONE WEB SERVER Process 1 Shared Memory Private Memory Process N

Slide 59

Slide 59 text

SCALE UP: MEMORY • Run in optimized mode (-O) • Remove dead code Reduce code

Slide 60

Slide 60 text

SCALE UP: MEMORY • Move conﬁguration into shared memory • Disable garbage collection Share more

Slide 61

Slide 61 text

SCALE UP: MEMORY 20+% capacity increase

Slide 62

Slide 62 text

SCALE UP: NETWORK LATENCY Synchronous processing model with long latency ===> Worker starvation and fewer CPU instr executed

Slide 63

Slide 63 text

Stories Feed Django Feed Stories Suggested Users ASYNC IO

Slide 64

Slide 64 text

Use as few CPU instructions as possible Use as few servers as possible Scale up

Slide 65

Slide 65 text

SCALE UP: CHALLENGES, OPPORTUNITIES • Faster python run-time • Async web framework • Better memory analysis • etc etc

Slide 66

Slide 66 text

SCALE DEV TEAM

Slide 67

Slide 67 text

SCALING TEAM 30% engineers joined in last 6 months Bootcampers - 1 week Hack-A-Month - 4 weeks Intern - 12 weeks

Slide 68

Slide 68 text

Comment Filtering Self-harm Prevention Windows App Multiple media in one post Video View Notiﬁcation Saved Posts First Story Notiﬁcation Instagram Live Instagram Stories

Slide 69

Slide 69 text

Which server? NewTable or New Column? What Index? Should I cache it? Will I lock up DB? Will I bring down Instagram?

Slide 70

Slide 70 text

WHAT WE WANT • Automatically handle cache • Deﬁne relations, not worry about implementations • Self service by product engineers • Infra focuses on scale

Slide 71

Slide 71 text

TAO USER1 USER2 USER3 media posted posted by likes liked by likes liked by

Slide 72

Slide 72 text

Comment Filtering Self-harm Prevention Windows App Multiple media in one post Video View Notiﬁcation Saved Posts First Story Notiﬁcation Instagram Live Instagram Stories

Slide 73

Slide 73 text

SOURCE CONTROL Master Live Direct

Slide 74

Slide 74 text

SOURCE CONTROL • Context switching • Code sync/merge overhead • Surprises • Refactor/major upgrade • Performance tracking harder With branches

Slide 75

Slide 75 text

SOURCE CONTROL Master Live Direct

Slide 76

Slide 76 text

SOURCE CONTROL Master Live Direct

Slide 77

Slide 77 text

SOURCE CONTROL • Continous integration • Collaborate easily • Fast bisect and revert • Continuous performance monitoring No branches

Slide 78

Slide 78 text

FEATURE LAUNCH Engineers Employees Dogfooder Some demographics World

Slide 79

Slide 79 text

FEATURE LOAD TEST

Slide 80

Slide 80 text

Once a 40-60 rollouts per day day diﬀ week? !!

Slide 81

Slide 81 text

CHECKS AND BALANCES Code review unittest Code accepted committed Canary To the Wild

Slide 82

Slide 82 text

No content

Slide 83

Slide 83 text

SCALING MEANS Scale out Scale up Scale dev team

Slide 84

Slide 84 text

No content

Slide 85

Slide 85 text

TAKEAWAYS Scaling is everybody’s responsibility Scaling is continuous eﬀort Scaling is multi-dimensional