Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Necessary Performance Measures: npm at Scale

583a0cfd3e0ec851166c5c6fa5e506a5?s=47 Raquel Vélez
April 30, 2014
Technology
1
110

Necessary Performance Measures: npm at Scale

Talk on caching and distribution at MelbJS meetup in Melbourne, Australia

583a0cfd3e0ec851166c5c6fa5e506a5?s=128

Raquel Vélez

April 30, 2014
Tweet

More Decks by Raquel Vélez

See All by Raquel Vélez
583a0cfd3e0ec851166c5c6fa5e506a5?s=48 rockbot
0
91
583a0cfd3e0ec851166c5c6fa5e506a5?s=48 rockbot
0
110
583a0cfd3e0ec851166c5c6fa5e506a5?s=48 rockbot
1
89
583a0cfd3e0ec851166c5c6fa5e506a5?s=48 rockbot
4
310
583a0cfd3e0ec851166c5c6fa5e506a5?s=48 rockbot
1
480
583a0cfd3e0ec851166c5c6fa5e506a5?s=48 rockbot
0
69
583a0cfd3e0ec851166c5c6fa5e506a5?s=48 rockbot
0
420
583a0cfd3e0ec851166c5c6fa5e506a5?s=48 rockbot
0
240
583a0cfd3e0ec851166c5c6fa5e506a5?s=48 rockbot
1
270

Other Decks in Technology

See All in Technology
79c9f45e8a3cd4bfa9d7292adb3a6dc0?s=48 kraj
0
5.4k
3a4e9f8c3cb529de8b1ac11fadb45d3d?s=48 yunon_phys
1
480
E7c0cdb6e91562ab5995348cd2b2110e?s=48 gkzz
0
380
Ad449f3f4e595ade283a30c4f66010be?s=48 seto_hi
0
150
Cb88f765dd38cd942c39aacb5abbea06?s=48 ufoo68
0
110
Dca4f8abd1e78940052ee52c85c4d2ed?s=48 shimacos
2
370
928b1395afedebb5ee48d44ab917c5f1?s=48 ryusa
2
440
9cdb1d21375062bc2cd0d1358c2fa1f0?s=48 noir_neo
0
130
79c9f45e8a3cd4bfa9d7292adb3a6dc0?s=48 kraj
0
4.5k
13d936e697fe0f4fa96f926d0a712f6c?s=48 sansanbuildersbox
PRO
0
210
79c9f45e8a3cd4bfa9d7292adb3a6dc0?s=48 kraj
0
5.2k
11d5350db3f389d0f5749b91ee11a916?s=48 indigo13love
2
650

Featured

See All Featured
5b9fe87ec1faa67a4599782930f45ec9?s=48 sstephenson
144
12k
Af6a12710770ad9a7cfd08c97efd40d3?s=48 pedronauck
652
110k
Fde6c3a50ca518a909ef35c22c0ee0e4?s=48 destraynor
223
47k
465724d73fe3a92c0879fdfb43a3a6f3?s=48 philhawksworth
192
8.8k
Ae14cc4491ac334f9cd23f9f93b4305e?s=48 orderedlist
PRO
329
35k
C44e1f7e22c3f23cff7bc130871047ef?s=48 eileencodes
113
25k
C0390e8b11079f8761c0cd3274ed61cb?s=48 productmarketing
5
660
B3d6434763caa0ef5dc4b792662c49f7?s=48 smashingmag
229
18k
C9b18d9dff88a9dd2393364c2b3b21bd?s=48 colly
187
14k
3d6ace9554821d552146413bcdf874f6?s=48 trishagee
20
2.2k
25c7c18223fb42a4c6ae1c8db6f50f9b?s=48 mojombo
358
62k
465724d73fe3a92c0879fdfb43a3a6f3?s=48 philhawksworth
190
17k

Transcript

  1. Necessary Performance Measures: at Scale Brought to you by @rockbot

  2. Package Management for Node.js • The Open Source Dream !

    • ~70k packages • up to 10 million downloads/ day

  3. 221,845,731) !10,000!! !20,000!! !30,000!! !40,000!! !50,000!! !60,000!! !70,000!! !80,000!! !90,000!!

    !-!!!! !10,000,000!! !20,000,000!! !30,000,000!! !40,000,000!! !50,000,000!! !60,000,000!! !70,000,000!! !80,000,000!! !90,000,000!! !100,000,000!! !110,000,000!! !120,000,000!! !130,000,000!! !140,000,000!! !150,000,000!! !160,000,000!! !170,000,000!! !180,000,000!! !190,000,000!! !200,000,000!! !210,000,000!! !220,000,000!! !230,000,000!! !240,000,000!! 8/29/12! 12/7/12! 3/17/13! 6/25/13! 10/3/13! 1/11/14! 4/21/14! Downloads)of)npm)packages)in)previous)month)) &)Total)number)of)npm)Packages)of)node.js)code) !!!Downloads!of!npm!packages!in!previous!month! !!!Total!Number!of!npm!Packages!

  4. Performance = Perception • Best request vs worst request •

    Focus on the few, super-slow instances: improve performance like whoa ! • Caching vs Distribution

  5. Caching

  6. Data Layer Query Caching Database-level performance enhancements ! innoDB buffer

    pool, CouchDB’s ACID architecture

  7. Query Layer Save CPU Preprocessed/aggregated data; scheduled handling of data

    ! CouchDB views, Download Counts

  8. App Layer Per-User Data that doesn’t change per user !

    Storing user session data in Redis

  9. Edge Server Layer Centralized/Global CDN, static assets ! Fastly

  10. Client Layer Distributed HTTP headers, local storage, etc. ! Browser

    caching

  11. Distribution

  12. Split Load More boxes, more specialized boxes read-only, write-only !

    One process per box Redis, CouchDB, ElasticSearch

  13. Add Resiliency Multi-cloud, multi-zone East/West, multiple datacenters ! Fallback to

    least effort/ most common usage i.e. read-only

  14. Boost with Third-Party Services Global content distribution network CDN

  15. The End?

  16. Thank You! ! Raquel Vélez @rockbot raquel@ js.com