Necessary Performance Measures: npm at Scale

583a0cfd3e0ec851166c5c6fa5e506a5?s=47 Raquel Vélez
April 30, 2014
Technology
1
110

Necessary Performance Measures: npm at Scale

Talk on caching and distribution at MelbJS meetup in Melbourne, Australia

583a0cfd3e0ec851166c5c6fa5e506a5?s=128

Raquel Vélez

April 30, 2014
Tweet

More Decks by Raquel Vélez

See All by Raquel Vélez
583a0cfd3e0ec851166c5c6fa5e506a5?s=48 rockbot
0
85
583a0cfd3e0ec851166c5c6fa5e506a5?s=48 rockbot
0
110
583a0cfd3e0ec851166c5c6fa5e506a5?s=48 rockbot
1
88
583a0cfd3e0ec851166c5c6fa5e506a5?s=48 rockbot
4
310
583a0cfd3e0ec851166c5c6fa5e506a5?s=48 rockbot
1
460
583a0cfd3e0ec851166c5c6fa5e506a5?s=48 rockbot
0
68
583a0cfd3e0ec851166c5c6fa5e506a5?s=48 rockbot
0
410
583a0cfd3e0ec851166c5c6fa5e506a5?s=48 rockbot
0
240
583a0cfd3e0ec851166c5c6fa5e506a5?s=48 rockbot
1
250

Other Decks in Technology

See All in Technology
Cb88f765dd38cd942c39aacb5abbea06?s=48 ufoo68
0
110
847a328633b1df6b11cc2f72430025e6?s=48 ks91
0
170
7a0bc9670ff654b04d4361a9afd1bba8?s=48 hcrane
18
1.7k
Cd931bc05e7cee46b9a950a63e47ba4c?s=48 you
0
130
0af658ea4f11fa7bf803115451890d4c?s=48 ararajp
2
430
C786d7772602b488a7972c38c143f61c?s=48 con_mame
3
1.8k
0b28f5164b540e5a31bc63cca39b2917?s=48 curry9999
0
150
55c9804c185f493816433678393cbe92?s=48 sumihiro3
0
180
8120d39cb575881c830287046f06fec8?s=48 daoka
1
2.1k
921515dc03f89bafe3d8f5d0b3ca0b74?s=48 mmarukaw
0
590
74cec195bfb6cb5165256d88cb7fcf0f?s=48 miu_crescent
5
1.4k
43a37af3a99834cd9c4b15806b5b4081?s=48 yoshiokacb
0
260

Featured

See All Featured
78b475797a14c84799063c7cd073962f?s=48 holman
448
130k
D8fc2580cfaca035f666d9e4ee79a7f7?s=48 mongodb
23
3.7k
761be20b5ebd271008bcee1244fc5b52?s=48 matthewcrist
70
7k
7edec06b5435e0dac07a353b2cc26253?s=48 geeforr
330
29k
Fe6b81005d1553accd6b2a28f6a2bef1?s=48 phodgson
84
3.8k
24fc194843a71f10949be18d5a692682?s=48 gr2m
82
11k
61857dafbd287b3027c4dcea9008ad3c?s=48 carmenhchung
18
910
C0b42577cfc2b321be3474618107e933?s=48 samanthasiow
53
6k
E837d2996f52008ace055a08fbfff992?s=48 frogandcode
121
20k
D47656e20ff5e42f125fc5ea0fd636c6?s=48 tmm1
61
7.4k
20bfe76b3d6105641f879fe45cfc9272?s=48 bkeepers
319
53k
A16eb159db895e3b01d3dc95767ad595?s=48 jonyablonski
6
640

Transcript

  1. Necessary Performance Measures: at Scale Brought to you by @rockbot

  2. Package Management for Node.js • The Open Source Dream !

    • ~70k packages • up to 10 million downloads/ day

  3. 221,845,731) !10,000!! !20,000!! !30,000!! !40,000!! !50,000!! !60,000!! !70,000!! !80,000!! !90,000!!

    !-!!!! !10,000,000!! !20,000,000!! !30,000,000!! !40,000,000!! !50,000,000!! !60,000,000!! !70,000,000!! !80,000,000!! !90,000,000!! !100,000,000!! !110,000,000!! !120,000,000!! !130,000,000!! !140,000,000!! !150,000,000!! !160,000,000!! !170,000,000!! !180,000,000!! !190,000,000!! !200,000,000!! !210,000,000!! !220,000,000!! !230,000,000!! !240,000,000!! 8/29/12! 12/7/12! 3/17/13! 6/25/13! 10/3/13! 1/11/14! 4/21/14! Downloads)of)npm)packages)in)previous)month)) &)Total)number)of)npm)Packages)of)node.js)code) !!!Downloads!of!npm!packages!in!previous!month! !!!Total!Number!of!npm!Packages!

  4. Performance = Perception • Best request vs worst request •

    Focus on the few, super-slow instances: improve performance like whoa ! • Caching vs Distribution

  5. Caching

  6. Data Layer Query Caching Database-level performance enhancements ! innoDB buffer

    pool, CouchDB’s ACID architecture

  7. Query Layer Save CPU Preprocessed/aggregated data; scheduled handling of data

    ! CouchDB views, Download Counts

  8. App Layer Per-User Data that doesn’t change per user !

    Storing user session data in Redis

  9. Edge Server Layer Centralized/Global CDN, static assets ! Fastly

  10. Client Layer Distributed HTTP headers, local storage, etc. ! Browser

    caching

  11. Distribution

  12. Split Load More boxes, more specialized boxes read-only, write-only !

    One process per box Redis, CouchDB, ElasticSearch

  13. Add Resiliency Multi-cloud, multi-zone East/West, multiple datacenters ! Fallback to

    least effort/ most common usage i.e. read-only

  14. Boost with Third-Party Services Global content distribution network CDN

  15. The End?

  16. Thank You! ! Raquel Vélez @rockbot raquel@ js.com