High Performance Backend For Mercari

High Performance Backend For Mercari

5d74d743eabd2bf7d4d2f68b9d3c727d?s=128

Tatsuhiko Kubo

March 21, 2015
Tweet

Transcript

  1. High Performance Backend For Mercari Tatsuhiko Kubo@cubicdaiya ࣌Ӎࡇ@2015/03/21

  2. ࣗݾ঺հ • ٱอୡ඙(Tatsuhiko Kubo) • bokko@cubicdaiya • Software Engineer in

    Infrastructure Engineering • Mercari, Inc.
  3. OSS࡞ͬͨΓίϯτϦϏϡʔτͨ͠Γ

  4. None
  5. None
  6. ओཁKPIσʔλ μ΢ϯϩʔυ਺ ߪೖֹۚ ग़඼਺ ਺஋ 1000ສDLಥഁ ݄ؒ਺ेԯԁ 1೔਺ेສ඼Ҏ্

  7. ͷେࡶ೺ͳγεςϜߏ੒ ͘͞ΒͱAWSͷϋΠϒϦου

  8. ৄ͘͠͸ͪ͜Β http://bit.ly/18Os1Dc

  9. Agenda • High performance API server with PHP!!! • High

    performance networking with nginx • High performance Application with Go
  10. Agenda • High performance API server with PHP!!! • High

    performance networking with nginx • High performance Application with Go
  11. ͱ͋ΔAPIαʔόͷฏۉϨεϙϯελΠϜ(1೔ؒ) ϐʔΫ࣌Ͱ΋30ms୆ʂ

  12. ࣮ࡍʹ͸΋ͬͱ࣮ߦ࣌ؒͷ௕͍API΋͋Γ·͢ e.g. ߪೖτϥϯβΫγϣϯ

  13. Why API server very fast? • ܰྔͰߴ଎ͳPHPϑϨʔϜϫʔΫ(DietCake) • Ωϟογϡ •

    ඇಉظॲཧ • ۚͷ஄ؙMySQL on ioDrive • ͦ΋ͦ΋Ϩεϙϯε͕JSON onlyͳͷͰ͍ܰ
  14. http://dietcake.github.io/

  15. None
  16. Ωϟογϡ • memcachedͱRedisΛซ༻ • ௨ৗͷΩϟογϡ͸جຊతʹmemcached • Redis͸Ϧετܕ͕ޮՌతͳՕॴ͕͋ͬͯͦ͜ Ͱར༻͍ͯ͠Δ • ͦͷલஈͰ͞ΒʹmemcachedͰΩϟογϡ

  17. Ωϟογϡͷ΢ΥʔϜΞοϓ • ΞΫηε͕ଟ͍ & ߋ৽ස౓͕ߴ͍ & Ωϟογϡͷύλʔϯ͕ଟ͍ & ݁ߏॏ͍API •

    Ϣʔβ͔ΒͷΞΫηεͷࡍʹΩϟογϡΛੜ੒ͯ͠ ΋ޮՌ͕ബ͍ • ཪͰͻͨ͢ΒΩϟογϡΛੜ੒͢ΔσʔϞϯΛಈ͔͢ • ޙड़
  18. ඇಉظॲཧ • APIαʔό͸ͳΔ͸΍ͰϨεϙϯεΛฦ͍ͨ͠ • ͔͠͠ɺதʹ͸Ͳ͏ͯ͠΋ॏ͍ॲཧ͕͋Δˣ • ΩϟογϡɺݕࡧΠϯσοΫεͷߋ৽ • ϝʔϧɺSMSͷૹ৴ •

    δϣϒΩϡʔ&ϫʔΧʔͰඇಉظʹॲཧ͢Δ
  19. δϣϒΩϡʔͱϫʔΧʔ • δϣϒΩϡʔ͸Q4M • ϫʔΧʔ͸php-Parallel-Prefork • PHPͰpreforkσʔϞϯ͕؆୯ʹ࡞ΕΔ

  20. None
  21. Agenda • High performance API server with PHP!!! • High

    performance networking with nginx • High performance Application with Go
  22. OHJOY

  23. ϝϧΧϦʹ͓͚Δnginxͷओͳ໾ׂ • APIϦΫΤετͷϩʔυόϥϯε • SSLλʔϛωʔγϣϯ & SPDYήʔτ΢ΣΠ • େྔಉ࣌઀ଓ΁ͷඋ͑(C10K) •

    HTTPS௨৴ͷߴ଎Խ • TLS Session (Cache|Tickets)
  24. TLS Session Cache • TLSϋϯυγΣΠΫͷηογϣϯ৘ใΛαʔόʹ Ωϟογϡ • nginxͰ͸ڞ༗ϝϞϦ্ʹΩϟογϡ͞ΕΔ • ࣍ճͷTLSϋϯυγΣΠΫΛলུ

    • CPUϦιʔεͷ࡟ݮ΍ϨΠςϯγղফͷޮՌ͕͋Δ
  25. TLS Session Cache with nginx

  26. TLS Session Tickets • ҉߸Խͨ͠ηογϣϯ৘ใ(νέοτ)ΛΫϥ ΠΞϯτʹ౉͢ • νέοτΛݩʹTLSηογϣϯΛ࠶։ • HTTPSαʔόෳ਺୆ͰηογϣϯΩϟογϡ

    Λڞ༗Ͱ͖Δ • αϙʔτ͍ͯ͠Δ୺຤͕গͳ͍ͷ͕೉఺
  27. TLS Session Tickets with nginx

  28. SPDY • ࠷৽ͷiOS΍AndroidͰ͸αʔό͕ରԠͯ͠ ͍Ε͹ࣗಈతʹSPDY/3.1͕ར༻͞ΕΔ • ΫϥΠΞϯτଆͰରԠ͠ͳͯ͘΋Α͍

  29. SPDY with nginx

  30. TLS Session (Cache|Tickets) ಋೖλΠϛϯά

  31. TLS Session (Cache|Tickets) ಋೖλΠϛϯά

  32. SPDY 41%:ಋೖͨ͠೔ͷϐʔΫ

  33. Agenda • High performance API server with PHP!!! • High

    performance networking with nginx • High performance Application with Go
  34. mercari.go • ϝϧΧϦͷϝΠϯ։ൃݴޠ͸PHP • ΠϯϑϥνʔϜͩͱGoɺNodeɺRubyɺPerl౳͍ Ζ͍Ζ(ΈΜͳಘҙ෼໺͕ҧ͏) • ࠷ۙ͸ͪΐͬͱͨ͠CLI΍σʔϞϯɺύϑΥʔϚϯ ε͕ཁٻ͞ΕΔίϯϙʔωϯτΛGoͰ։ൃ͢Δػ ձ͕૿͑ͯΔ

  35. mercari.go • nginx-build • cachectl • slackboard • Ϋϩʔζυ •

    Gaurun, refreshcache
  36. cɺconsul΋࢖ͬͯΔ͔Β(਒͑੠)

  37. ͳ͓ɺόάͬͯΔ໛༷ ઌि͜ͷόάΛճආ͢Δίʔυॻ͍ͯ·ͨ͠

  38. mercari.go • nginx-build • cachectl • slackboard • Ϋϩʔζυ •

    Gaurun, refreshcache
  39. Gaurun • ൚༻ϓογϡ௨஌αʔό • ୹࣌ؒͰ਺ेɺ਺ඦສ݅ͷϓογϡ௨஌͕Մೳ • HTTPͱJSONΛϕʔεʹͨ͠γϯϓϧͳAPI • ϝϧΧϦ಺ͷϓογϡ௨஌ɺେମGaurunͰஔ ͖׵͑ͨ

  40. ൚༻ϓογϡ௨஌γεςϜͷΞʔΩςΫνϟ PS CBUDIαʔό

  41. Gaurun HTTP API "1* ղઆ 1045QVTI ϓογϡ௨஌ϦΫΤετड෇ (&5TUBUHP (PϥϯλΠϜͷ׆ಈঢ়گΛऔಘ (&5TUBUBQQ

    ಺෦Ωϡʔ νϟωϧ ͷ࢖༻ྔ΍ ϓογϡͷ੒ޭࣦഊ਺ͷऔಘ (&5DPOpHBQQ αʔόઃఆ 50.- Λऔಘ
  42. POST /pushͷϦΫΤετϘσΟ

  43. GET /stat/go $ curl -s http://127.0.0.1:1056/stat/go { "time": 1423270089195474419, "go_version":

    "go1.4", "go_os": "linux", "go_arch": "amd64", "cpu_num": 24, "goroutine_num": 55, "gomaxprocs": 24, "cgo_call_num": 124016, "memory_alloc": 10309424, "memory_total_alloc": 295474683712, "memory_sys": 25557240, "memory_lookups": 214058, "memory_mallocs": 2118264299, "memory_frees": 2118230111, ɾ ɾ
  44. GET /stat/app $ curl -s http://127.0.0.1:1056/stat/app { "queue_max": 40960, //

    ಺෦ΩϡʔͷαΠζ "queue_usage": 0, // ಺෦Ωϡʔͷ࢖༻ྔ(͙͢ࡹ͚ΔͷͰେମ0) "ios": { "push_success": 31804, // iOSϓογϡ௨஌ͷ੒ޭ਺ "push_error": 17 // iOSϓογϡ௨஌ͷࣦഊ਺ }, "android": { "push_success": 18683, // Androidϓογϡ௨஌ͷ੒ޭ਺ "push_error": 233 // Androidϓογϡ௨஌ͷࣦഊ਺ } $
  45. GaurunΛ࣮ߦ͍ͯ͠Δ༷ࢠͰ͢

  46. mercari.go • nginx-build • cachectl • slackboard • Ϋϩʔζυ •

    Gaurun, refreshcache
  47. refreshcache • ࠷ॳͷํͰݴͬͯͨಛఆAPIͷϨεϙϯεͷ ΩϟογϡΛੜ੒͢ΔσʔϞϯ • ҰఆִؒͰฒྻʹ݁ߏͳྔͷHTTPϦΫΤετ ΛૹΓଓ͚Δ • ݩʑRubyͰॻ͔Ε͍ͯͨ(஗͔ͬͨ)

  48. ݩͷRubyίʔυ Parallel.map(params_list, :in_processes => in_processes) do |params| (தུ) res =

    http_request('GET', url, params) (தུ) end.compact params_listͷαΠζ͕େ͖͍ ↓ େྔʹfork͢ΔͷͰॏ͍
  49. GoͰॻ͖௚ͨ͠ done := make(chan error) size := len(paramCombs.Params) var errors

    []error for i := 0; i < size; i++ { go makeCache(URL, paramCombs.Params[i], done) } for i := 0; i < size; i++ { err := <-done if err != nil { errors = append(errors, err) } }
  50. ࣮ߦαʔόͷάϥϑ(CPU) ϦϥΠτͨ͠λΠϛϯά

  51. ࣮ߦαʔόͷάϥϑ(LA) ϦϥΠτͨ͠λΠϛϯά

  52. ·ͱΊ • ϝϧΧϦͷߴ଎ͳόοΫΤϯυͷ࢓૊Έʹ͍ͭͯ ͍͔ͭ͘ղઆ͠·ͨ͠ • ϝΠϯ͸PHP͚ͩͲ࠷ۙ͸Goͷར༻ࣄྫ૿͑ͯΔ • Go͸ૣ͍ɾ଎͍ɾ͍ܰ • Gaurun͸ͦͷ͏ͪOSSԽ༧ఆ