High Performance Backend For Mercari

Slide 1

Slide 1 text

High Performance Backend For Mercari Tatsuhiko Kubo@cubicdaiya ࣌Ӎࡇ@2015/03/21

Slide 2

Slide 2 text

ࣗݾ঺հ • ٱอୡ඙(Tatsuhiko Kubo) • bokko@cubicdaiya • Software Engineer in Infrastructure Engineering • Mercari, Inc.

Slide 3

Slide 3 text

OSS࡞ͬͨΓίϯτϦϏϡʔτͨ͠Γ

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

ओཁKPIσʔλ μ΢ϯϩʔυ਺ ߪೖֹۚ ग़඼਺ ਺஋ 1000ສDLಥഁ ݄ؒ਺ेԯԁ 1೔਺ेສ඼Ҏ্

Slide 7

Slide 7 text

ͷେࡶ೺ͳγεςϜߏ੒ ͘͞ΒͱAWSͷϋΠϒϦου

Slide 8

Slide 8 text

ৄ͘͠͸ͪ͜Β http://bit.ly/18Os1Dc

Slide 9

Slide 9 text

Agenda • High performance API server with PHP!!! • High performance networking with nginx • High performance Application with Go

Slide 10

Slide 10 text

Agenda • High performance API server with PHP!!! • High performance networking with nginx • High performance Application with Go

Slide 11

Slide 11 text

ͱ͋ΔAPIαʔόͷฏۉϨεϙϯελΠϜ(1೔ؒ) ϐʔΫ࣌Ͱ΋30ms୆ʂ

Slide 12

Slide 12 text

࣮ࡍʹ͸΋ͬͱ࣮ߦ࣌ؒͷ௕͍API΋͋Γ·͢ e.g. ߪೖτϥϯβΫγϣϯ

Slide 13

Slide 13 text

Why API server very fast? • ܰྔͰߴ଎ͳPHPϑϨʔϜϫʔΫ(DietCake) • Ωϟογϡ • ඇಉظॲཧ • ۚͷ஄ؙMySQL on ioDrive • ͦ΋ͦ΋Ϩεϙϯε͕JSON onlyͳͷͰ͍ܰ

Slide 14

Slide 14 text

http://dietcake.github.io/

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

Ωϟογϡ • memcachedͱRedisΛซ༻ • ௨ৗͷΩϟογϡ͸جຊతʹmemcached • Redis͸Ϧετܕ͕ޮՌతͳՕॴ͕͋ͬͯͦ͜ Ͱར༻͍ͯ͠Δ • ͦͷલஈͰ͞ΒʹmemcachedͰΩϟογϡ

Slide 17

Slide 17 text

Ωϟογϡͷ΢ΥʔϜΞοϓ • ΞΫηε͕ଟ͍ & ߋ৽ස౓͕ߴ͍ & Ωϟογϡͷύλʔϯ͕ଟ͍ & ݁ߏॏ͍API • Ϣʔβ͔ΒͷΞΫηεͷࡍʹΩϟογϡΛੜ੒ͯ͠ ΋ޮՌ͕ബ͍ • ཪͰͻͨ͢ΒΩϟογϡΛੜ੒͢ΔσʔϞϯΛಈ͔͢ • ޙड़

Slide 18

Slide 18 text

ඇಉظॲཧ • APIαʔό͸ͳΔ͸΍ͰϨεϙϯεΛฦ͍ͨ͠ • ͔͠͠ɺதʹ͸Ͳ͏ͯ͠΋ॏ͍ॲཧ͕͋Δˣ • ΩϟογϡɺݕࡧΠϯσοΫεͷߋ৽ • ϝʔϧɺSMSͷૹ৴ • δϣϒΩϡʔ&ϫʔΧʔͰඇಉظʹॲཧ͢Δ

Slide 19

Slide 19 text

δϣϒΩϡʔͱϫʔΧʔ • δϣϒΩϡʔ͸Q4M • ϫʔΧʔ͸php-Parallel-Prefork • PHPͰpreforkσʔϞϯ͕؆୯ʹ࡞ΕΔ

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

Agenda • High performance API server with PHP!!! • High performance networking with nginx • High performance Application with Go

Slide 22

Slide 22 text

OHJOY

Slide 23

Slide 23 text

ϝϧΧϦʹ͓͚Δnginxͷओͳ໾ׂ • APIϦΫΤετͷϩʔυόϥϯε • SSLλʔϛωʔγϣϯ & SPDYήʔτ΢ΣΠ • େྔಉ࣌઀ଓ΁ͷඋ͑(C10K) • HTTPS௨৴ͷߴ଎Խ • TLS Session (Cache|Tickets)

Slide 24

Slide 24 text

TLS Session Cache • TLSϋϯυγΣΠΫͷηογϣϯ৘ใΛαʔόʹ Ωϟογϡ • nginxͰ͸ڞ༗ϝϞϦ্ʹΩϟογϡ͞ΕΔ • ࣍ճͷTLSϋϯυγΣΠΫΛলུ • CPUϦιʔεͷ࡟ݮ΍ϨΠςϯγղফͷޮՌ͕͋Δ

Slide 25

Slide 25 text

TLS Session Cache with nginx

Slide 26

Slide 26 text

TLS Session Tickets • ҉߸Խͨ͠ηογϣϯ৘ใ(νέοτ)ΛΫϥ ΠΞϯτʹ౉͢ • νέοτΛݩʹTLSηογϣϯΛ࠶։ • HTTPSαʔόෳ਺୆ͰηογϣϯΩϟογϡ Λڞ༗Ͱ͖Δ • αϙʔτ͍ͯ͠Δ୺຤͕গͳ͍ͷ͕೉఺

Slide 27

Slide 27 text

TLS Session Tickets with nginx

Slide 28

Slide 28 text

SPDY • ࠷৽ͷiOS΍AndroidͰ͸αʔό͕ରԠͯ͠ ͍Ε͹ࣗಈతʹSPDY/3.1͕ར༻͞ΕΔ • ΫϥΠΞϯτଆͰରԠ͠ͳͯ͘΋Α͍

Slide 29

Slide 29 text

SPDY with nginx

Slide 30

Slide 30 text

TLS Session (Cache|Tickets) ಋೖλΠϛϯά

Slide 31

Slide 31 text

TLS Session (Cache|Tickets) ಋೖλΠϛϯά

Slide 32

Slide 32 text

SPDY 41%:ಋೖͨ͠೔ͷϐʔΫ

Slide 33

Slide 33 text

Agenda • High performance API server with PHP!!! • High performance networking with nginx • High performance Application with Go

Slide 34

Slide 34 text

mercari.go • ϝϧΧϦͷϝΠϯ։ൃݴޠ͸PHP • ΠϯϑϥνʔϜͩͱGoɺNodeɺRubyɺPerl౳͍ Ζ͍Ζ(ΈΜͳಘҙ෼໺͕ҧ͏) • ࠷ۙ͸ͪΐͬͱͨ͠CLI΍σʔϞϯɺύϑΥʔϚϯ ε͕ཁٻ͞ΕΔίϯϙʔωϯτΛGoͰ։ൃ͢Δػ ձ͕૿͑ͯΔ

Slide 35

Slide 35 text

mercari.go • nginx-build • cachectl • slackboard • Ϋϩʔζυ • Gaurun, refreshcache

Slide 36

Slide 36 text

cɺconsul΋࢖ͬͯΔ͔Β(਒͑੠)

Slide 37

Slide 37 text

ͳ͓ɺόάͬͯΔ໛༷ ઌि͜ͷόάΛճආ͢Δίʔυॻ͍ͯ·ͨ͠

Slide 38

Slide 38 text

mercari.go • nginx-build • cachectl • slackboard • Ϋϩʔζυ • Gaurun, refreshcache

Slide 39

Slide 39 text

Gaurun • ൚༻ϓογϡ௨஌αʔό • ୹࣌ؒͰ਺ेɺ਺ඦສ݅ͷϓογϡ௨஌͕Մೳ • HTTPͱJSONΛϕʔεʹͨ͠γϯϓϧͳAPI • ϝϧΧϦ಺ͷϓογϡ௨஌ɺେମGaurunͰஔ ͖׵͑ͨ

Slide 40

Slide 40 text

൚༻ϓογϡ௨஌γεςϜͷΞʔΩςΫνϟ PS CBUDIαʔό

Slide 41

Slide 41 text

Gaurun HTTP API "1* ղઆ 1045QVTI ϓογϡ௨஌ϦΫΤετड෇ (&5TUBUHP (PϥϯλΠϜͷ׆ಈঢ়گΛऔಘ (&5TUBUBQQ ಺෦Ωϡʔ νϟωϧ ͷ࢖༻ྔ΍ ϓογϡͷ੒ޭࣦഊ਺ͷऔಘ (&5DPOpHBQQ αʔόઃఆ 50.- Λऔಘ

Slide 42

Slide 42 text

POST /pushͷϦΫΤετϘσΟ

Slide 43

Slide 43 text

GET /stat/go $ curl -s http://127.0.0.1:1056/stat/go { "time": 1423270089195474419, "go_version": "go1.4", "go_os": "linux", "go_arch": "amd64", "cpu_num": 24, "goroutine_num": 55, "gomaxprocs": 24, "cgo_call_num": 124016, "memory_alloc": 10309424, "memory_total_alloc": 295474683712, "memory_sys": 25557240, "memory_lookups": 214058, "memory_mallocs": 2118264299, "memory_frees": 2118230111, ɾ ɾ

Slide 44

Slide 44 text

GET /stat/app $ curl -s http://127.0.0.1:1056/stat/app { "queue_max": 40960, // ಺෦ΩϡʔͷαΠζ "queue_usage": 0, // ಺෦Ωϡʔͷ࢖༻ྔ(͙͢ࡹ͚ΔͷͰେମ0) "ios": { "push_success": 31804, // iOSϓογϡ௨஌ͷ੒ޭ਺ "push_error": 17 // iOSϓογϡ௨஌ͷࣦഊ਺ }, "android": { "push_success": 18683, // Androidϓογϡ௨஌ͷ੒ޭ਺ "push_error": 233 // Androidϓογϡ௨஌ͷࣦഊ਺ } $

Slide 45

Slide 45 text

GaurunΛ࣮ߦ͍ͯ͠Δ༷ࢠͰ͢

Slide 46

Slide 46 text

mercari.go • nginx-build • cachectl • slackboard • Ϋϩʔζυ • Gaurun, refreshcache

Slide 47

Slide 47 text

refreshcache • ࠷ॳͷํͰݴͬͯͨಛఆAPIͷϨεϙϯεͷ ΩϟογϡΛੜ੒͢ΔσʔϞϯ • ҰఆִؒͰฒྻʹ݁ߏͳྔͷHTTPϦΫΤετ ΛૹΓଓ͚Δ • ݩʑRubyͰॻ͔Ε͍ͯͨ(஗͔ͬͨ)

Slide 48

Slide 48 text

ݩͷRubyίʔυ Parallel.map(params_list, :in_processes => in_processes) do |params| (தུ) res = http_request('GET', url, params) (தུ) end.compact params_listͷαΠζ͕େ͖͍ ↓ େྔʹfork͢ΔͷͰॏ͍

Slide 49

Slide 49 text

GoͰॻ͖௚ͨ͠ done := make(chan error) size := len(paramCombs.Params) var errors []error for i := 0; i < size; i++ { go makeCache(URL, paramCombs.Params[i], done) } for i := 0; i < size; i++ { err := <-done if err != nil { errors = append(errors, err) } }

Slide 50

Slide 50 text

࣮ߦαʔόͷάϥϑ(CPU) ϦϥΠτͨ͠λΠϛϯά

Slide 51

Slide 51 text

࣮ߦαʔόͷάϥϑ(LA) ϦϥΠτͨ͠λΠϛϯά

Slide 52

Slide 52 text

·ͱΊ • ϝϧΧϦͷߴ଎ͳόοΫΤϯυͷ࢓૊Έʹ͍ͭͯ ͍͔ͭ͘ղઆ͠·ͨ͠ • ϝΠϯ͸PHP͚ͩͲ࠷ۙ͸Goͷར༻ࣄྫ૿͑ͯΔ • Go͸ૣ͍ɾ଎͍ɾ͍ܰ • Gaurun͸ͦͷ͏ͪOSSԽ༧ఆ