Upgrade to Pro — share decks privately, control downloads, hide ads and more …

High Performance Backend For Mercari

High Performance Backend For Mercari

Tatsuhiko Kubo

March 21, 2015
Tweet

More Decks by Tatsuhiko Kubo

Other Decks in Technology

Transcript

  1. High Performance Backend
    For Mercari
    Tatsuhiko [email protected]
    ࣌Ӎࡇ@2015/03/21

    View Slide

  2. ࣗݾ঺հ
    • ٱอୡ඙(Tatsuhiko Kubo)
    [email protected]
    • Software Engineer in Infrastructure Engineering
    • Mercari, Inc.

    View Slide

  3. OSS࡞ͬͨΓίϯτϦϏϡʔτͨ͠Γ

    View Slide

  4. View Slide

  5. View Slide

  6. ओཁKPIσʔλ
    μ΢ϯϩʔυ਺
    ߪೖֹۚ
    ग़඼਺
    ਺஋
    1000ສDLಥഁ
    ݄ؒ਺ेԯԁ
    1೔਺ेສ඼Ҏ্

    View Slide

  7. ͷେࡶ೺ͳγεςϜߏ੒
    ͘͞ΒͱAWSͷϋΠϒϦου

    View Slide

  8. ৄ͘͠͸ͪ͜Β
    http://bit.ly/18Os1Dc

    View Slide

  9. Agenda
    • High performance API server with PHP!!!
    • High performance networking with nginx
    • High performance Application with Go

    View Slide

  10. Agenda
    • High performance API server with PHP!!!
    • High performance networking with nginx
    • High performance Application with Go

    View Slide

  11. ͱ͋ΔAPIαʔόͷฏۉϨεϙϯελΠϜ(1೔ؒ)
    ϐʔΫ࣌Ͱ΋30ms୆ʂ

    View Slide

  12. ࣮ࡍʹ͸΋ͬͱ࣮ߦ࣌ؒͷ௕͍API΋͋Γ·͢
    e.g. ߪೖτϥϯβΫγϣϯ

    View Slide

  13. Why API server very fast?
    • ܰྔͰߴ଎ͳPHPϑϨʔϜϫʔΫ(DietCake)
    • Ωϟογϡ
    • ඇಉظॲཧ
    • ۚͷ஄ؙMySQL on ioDrive
    • ͦ΋ͦ΋Ϩεϙϯε͕JSON onlyͳͷͰ͍ܰ

    View Slide

  14. http://dietcake.github.io/

    View Slide

  15. View Slide

  16. Ωϟογϡ
    • memcachedͱRedisΛซ༻
    • ௨ৗͷΩϟογϡ͸جຊతʹmemcached
    • Redis͸Ϧετܕ͕ޮՌతͳՕॴ͕͋ͬͯͦ͜
    Ͱར༻͍ͯ͠Δ
    • ͦͷલஈͰ͞ΒʹmemcachedͰΩϟογϡ

    View Slide

  17. Ωϟογϡͷ΢ΥʔϜΞοϓ
    • ΞΫηε͕ଟ͍ & ߋ৽ස౓͕ߴ͍ &
    Ωϟογϡͷύλʔϯ͕ଟ͍ & ݁ߏॏ͍API
    • Ϣʔβ͔ΒͷΞΫηεͷࡍʹΩϟογϡΛੜ੒ͯ͠
    ΋ޮՌ͕ബ͍
    • ཪͰͻͨ͢ΒΩϟογϡΛੜ੒͢ΔσʔϞϯΛಈ͔͢
    • ޙड़

    View Slide

  18. ඇಉظॲཧ
    • APIαʔό͸ͳΔ͸΍ͰϨεϙϯεΛฦ͍ͨ͠
    • ͔͠͠ɺதʹ͸Ͳ͏ͯ͠΋ॏ͍ॲཧ͕͋Δˣ
    • ΩϟογϡɺݕࡧΠϯσοΫεͷߋ৽
    • ϝʔϧɺSMSͷૹ৴
    • δϣϒΩϡʔ&ϫʔΧʔͰඇಉظʹॲཧ͢Δ

    View Slide

  19. δϣϒΩϡʔͱϫʔΧʔ
    • δϣϒΩϡʔ͸Q4M
    • ϫʔΧʔ͸php-Parallel-Prefork
    • PHPͰpreforkσʔϞϯ͕؆୯ʹ࡞ΕΔ

    View Slide

  20. View Slide

  21. Agenda
    • High performance API server with PHP!!!
    • High performance networking with nginx
    • High performance Application with Go

    View Slide

  22. OHJOY

    View Slide

  23. ϝϧΧϦʹ͓͚Δnginxͷओͳ໾ׂ
    • APIϦΫΤετͷϩʔυόϥϯε
    • SSLλʔϛωʔγϣϯ & SPDYήʔτ΢ΣΠ
    • େྔಉ࣌઀ଓ΁ͷඋ͑(C10K)
    • HTTPS௨৴ͷߴ଎Խ
    • TLS Session (Cache|Tickets)

    View Slide

  24. TLS Session Cache
    • TLSϋϯυγΣΠΫͷηογϣϯ৘ใΛαʔόʹ
    Ωϟογϡ
    • nginxͰ͸ڞ༗ϝϞϦ্ʹΩϟογϡ͞ΕΔ
    • ࣍ճͷTLSϋϯυγΣΠΫΛলུ
    • CPUϦιʔεͷ࡟ݮ΍ϨΠςϯγղফͷޮՌ͕͋Δ

    View Slide

  25. TLS Session Cache with nginx

    View Slide

  26. TLS Session Tickets
    • ҉߸Խͨ͠ηογϣϯ৘ใ(νέοτ)ΛΫϥ
    ΠΞϯτʹ౉͢
    • νέοτΛݩʹTLSηογϣϯΛ࠶։
    • HTTPSαʔόෳ਺୆ͰηογϣϯΩϟογϡ
    Λڞ༗Ͱ͖Δ
    • αϙʔτ͍ͯ͠Δ୺຤͕গͳ͍ͷ͕೉఺

    View Slide

  27. TLS Session Tickets with nginx

    View Slide

  28. SPDY
    • ࠷৽ͷiOS΍AndroidͰ͸αʔό͕ରԠͯ͠
    ͍Ε͹ࣗಈతʹSPDY/3.1͕ར༻͞ΕΔ
    • ΫϥΠΞϯτଆͰରԠ͠ͳͯ͘΋Α͍

    View Slide

  29. SPDY with nginx

    View Slide

  30. TLS Session (Cache|Tickets)
    ಋೖλΠϛϯά

    View Slide

  31. TLS Session (Cache|Tickets)
    ಋೖλΠϛϯά

    View Slide

  32. SPDY
    41%:ಋೖͨ͠೔ͷϐʔΫ

    View Slide

  33. Agenda
    • High performance API server with PHP!!!
    • High performance networking with nginx
    • High performance Application with Go

    View Slide

  34. mercari.go
    • ϝϧΧϦͷϝΠϯ։ൃݴޠ͸PHP
    • ΠϯϑϥνʔϜͩͱGoɺNodeɺRubyɺPerl౳͍
    Ζ͍Ζ(ΈΜͳಘҙ෼໺͕ҧ͏)
    • ࠷ۙ͸ͪΐͬͱͨ͠CLI΍σʔϞϯɺύϑΥʔϚϯ
    ε͕ཁٻ͞ΕΔίϯϙʔωϯτΛGoͰ։ൃ͢Δػ
    ձ͕૿͑ͯΔ

    View Slide

  35. mercari.go
    • nginx-build
    • cachectl
    • slackboard
    • Ϋϩʔζυ
    • Gaurun, refreshcache

    View Slide

  36. cɺconsul΋࢖ͬͯΔ͔Β(਒͑੠)

    View Slide

  37. ͳ͓ɺόάͬͯΔ໛༷
    ઌि͜ͷόάΛճආ͢Δίʔυॻ͍ͯ·ͨ͠

    View Slide

  38. mercari.go
    • nginx-build
    • cachectl
    • slackboard
    • Ϋϩʔζυ
    • Gaurun, refreshcache

    View Slide

  39. Gaurun
    • ൚༻ϓογϡ௨஌αʔό
    • ୹࣌ؒͰ਺ेɺ਺ඦສ݅ͷϓογϡ௨஌͕Մೳ
    • HTTPͱJSONΛϕʔεʹͨ͠γϯϓϧͳAPI
    • ϝϧΧϦ಺ͷϓογϡ௨஌ɺେମGaurunͰஔ
    ͖׵͑ͨ

    View Slide

  40. ൚༻ϓογϡ௨஌γεςϜͷΞʔΩςΫνϟ
    PS
    CBUDIαʔό

    View Slide

  41. Gaurun HTTP API
    "1* ղઆ
    1045QVTI ϓογϡ௨஌ϦΫΤετड෇
    (&5TUBUHP (PϥϯλΠϜͷ׆ಈঢ়گΛऔಘ
    (&5TUBUBQQ
    ಺෦Ωϡʔ νϟωϧ
    ͷ࢖༻ྔ΍
    ϓογϡͷ੒ޭࣦഊ਺ͷऔಘ
    (&5DPOpHBQQ αʔόઃఆ 50.-
    Λऔಘ

    View Slide

  42. POST /pushͷϦΫΤετϘσΟ

    View Slide

  43. GET /stat/go
    $ curl -s http://127.0.0.1:1056/stat/go
    {
    "time": 1423270089195474419,
    "go_version": "go1.4",
    "go_os": "linux",
    "go_arch": "amd64",
    "cpu_num": 24,
    "goroutine_num": 55,
    "gomaxprocs": 24,
    "cgo_call_num": 124016,
    "memory_alloc": 10309424,
    "memory_total_alloc": 295474683712,
    "memory_sys": 25557240,
    "memory_lookups": 214058,
    "memory_mallocs": 2118264299,
    "memory_frees": 2118230111,
    ɾ
    ɾ

    View Slide

  44. GET /stat/app
    $ curl -s http://127.0.0.1:1056/stat/app
    {
    "queue_max": 40960, // ಺෦ΩϡʔͷαΠζ
    "queue_usage": 0, // ಺෦Ωϡʔͷ࢖༻ྔ(͙͢ࡹ͚ΔͷͰେମ0)
    "ios": {
    "push_success": 31804, // iOSϓογϡ௨஌ͷ੒ޭ਺
    "push_error": 17 // iOSϓογϡ௨஌ͷࣦഊ਺
    },
    "android": {
    "push_success": 18683, // Androidϓογϡ௨஌ͷ੒ޭ਺
    "push_error": 233 // Androidϓογϡ௨஌ͷࣦഊ਺
    }
    $

    View Slide

  45. GaurunΛ࣮ߦ͍ͯ͠Δ༷ࢠͰ͢

    View Slide

  46. mercari.go
    • nginx-build
    • cachectl
    • slackboard
    • Ϋϩʔζυ
    • Gaurun, refreshcache

    View Slide

  47. refreshcache
    • ࠷ॳͷํͰݴͬͯͨಛఆAPIͷϨεϙϯεͷ
    ΩϟογϡΛੜ੒͢ΔσʔϞϯ
    • ҰఆִؒͰฒྻʹ݁ߏͳྔͷHTTPϦΫΤετ
    ΛૹΓଓ͚Δ
    • ݩʑRubyͰॻ͔Ε͍ͯͨ(஗͔ͬͨ)

    View Slide

  48. ݩͷRubyίʔυ
    Parallel.map(params_list, :in_processes => in_processes) do |params|
    (தུ)
    res = http_request('GET', url, params)
    (தུ)
    end.compact
    params_listͷαΠζ͕େ͖͍

    େྔʹfork͢ΔͷͰॏ͍

    View Slide

  49. GoͰॻ͖௚ͨ͠
    done := make(chan error)
    size := len(paramCombs.Params)
    var errors []error
    for i := 0; i < size; i++ {
    go makeCache(URL, paramCombs.Params[i], done)
    }
    for i := 0; i < size; i++ {
    err := <-done
    if err != nil {
    errors = append(errors, err)
    }
    }

    View Slide

  50. ࣮ߦαʔόͷάϥϑ(CPU)
    ϦϥΠτͨ͠λΠϛϯά

    View Slide

  51. ࣮ߦαʔόͷάϥϑ(LA)
    ϦϥΠτͨ͠λΠϛϯά

    View Slide

  52. ·ͱΊ
    • ϝϧΧϦͷߴ଎ͳόοΫΤϯυͷ࢓૊Έʹ͍ͭͯ
    ͍͔ͭ͘ղઆ͠·ͨ͠
    • ϝΠϯ͸PHP͚ͩͲ࠷ۙ͸Goͷར༻ࣄྫ૿͑ͯΔ
    • Go͸ૣ͍ɾ଎͍ɾ͍ܰ
    • Gaurun͸ͦͷ͏ͪOSSԽ༧ఆ

    View Slide