High Performance Backend For Mercari

Transcript

1. High Performance Backend
   For Mercari
   Tatsuhiko Kubo@cubicdaiya
   ࣌Ӎࡇ@2015/03/21
   

   View Slide

2. ࣗݾ঺հ
   • ٱอୡ඙(Tatsuhiko Kubo)
   • bokko@cubicdaiya
   • Software Engineer in Infrastructure Engineering
   • Mercari, Inc.
   

   View Slide

3. OSS࡞ͬͨΓίϯτϦϏϡʔτͨ͠Γ
   

   View Slide

4. View Slide

5. View Slide

6. ओཁKPIσʔλ
   μ΢ϯϩʔυ਺
   ߪೖֹۚ
   ग़඼਺
   ਺஋
   1000ສDLಥഁ
   ݄ؒ਺ेԯԁ
   1೔਺ेສ඼Ҏ্
   

   View Slide

7. ͷେࡶ೺ͳγεςϜߏ੒
   ͘͞ΒͱAWSͷϋΠϒϦου
   

   View Slide

8. ৄ͘͠͸ͪ͜Β
   http://bit.ly/18Os1Dc
   

   View Slide

9. Agenda
   • High performance API server with PHP!!!
   • High performance networking with nginx
   • High performance Application with Go
   

   View Slide

10. Agenda
    • High performance API server with PHP!!!
    • High performance networking with nginx
    • High performance Application with Go
    

    View Slide

11. ͱ͋ΔAPIαʔόͷฏۉϨεϙϯελΠϜ(1೔ؒ)
    ϐʔΫ࣌Ͱ΋30ms୆ʂ
    

    View Slide

12. ࣮ࡍʹ͸΋ͬͱ࣮ߦ࣌ؒͷ௕͍API΋͋Γ·͢
    e.g. ߪೖτϥϯβΫγϣϯ
    

    View Slide

13. Why API server very fast?
    • ܰྔͰߴ଎ͳPHPϑϨʔϜϫʔΫ(DietCake)
    • Ωϟογϡ
    • ඇಉظॲཧ
    • ۚͷ஄ؙMySQL on ioDrive
    • ͦ΋ͦ΋Ϩεϙϯε͕JSON onlyͳͷͰ͍ܰ
    

    View Slide

14. http://dietcake.github.io/
    

    View Slide

15. View Slide

16. Ωϟογϡ
    • memcachedͱRedisΛซ༻
    • ௨ৗͷΩϟογϡ͸جຊతʹmemcached
    • Redis͸Ϧετܕ͕ޮՌతͳՕॴ͕͋ͬͯͦ͜
    Ͱར༻͍ͯ͠Δ
    • ͦͷલஈͰ͞ΒʹmemcachedͰΩϟογϡ
    

    View Slide

17. Ωϟογϡͷ΢ΥʔϜΞοϓ
    • ΞΫηε͕ଟ͍ & ߋ৽ස౓͕ߴ͍ &
    Ωϟογϡͷύλʔϯ͕ଟ͍ & ݁ߏॏ͍API
    • Ϣʔβ͔ΒͷΞΫηεͷࡍʹΩϟογϡΛੜ੒ͯ͠
    ΋ޮՌ͕ബ͍
    • ཪͰͻͨ͢ΒΩϟογϡΛੜ੒͢ΔσʔϞϯΛಈ͔͢
    • ޙड़
    

    View Slide

18. ඇಉظॲཧ
    • APIαʔό͸ͳΔ͸΍ͰϨεϙϯεΛฦ͍ͨ͠
    • ͔͠͠ɺதʹ͸Ͳ͏ͯ͠΋ॏ͍ॲཧ͕͋Δˣ
    • ΩϟογϡɺݕࡧΠϯσοΫεͷߋ৽
    • ϝʔϧɺSMSͷૹ৴
    • δϣϒΩϡʔ&ϫʔΧʔͰඇಉظʹॲཧ͢Δ
    

    View Slide

19. δϣϒΩϡʔͱϫʔΧʔ
    • δϣϒΩϡʔ͸Q4M
    • ϫʔΧʔ͸php-Parallel-Prefork
    • PHPͰpreforkσʔϞϯ͕؆୯ʹ࡞ΕΔ
    

    View Slide

20. View Slide

21. Agenda
    • High performance API server with PHP!!!
    • High performance networking with nginx
    • High performance Application with Go
    

    View Slide

22. OHJOY
    

    View Slide

23. ϝϧΧϦʹ͓͚Δnginxͷओͳ໾ׂ
    • APIϦΫΤετͷϩʔυόϥϯε
    • SSLλʔϛωʔγϣϯ & SPDYήʔτ΢ΣΠ
    • େྔಉ࣌઀ଓ΁ͷඋ͑(C10K)
    • HTTPS௨৴ͷߴ଎Խ
    • TLS Session (Cache|Tickets)
    

    View Slide

24. TLS Session Cache
    • TLSϋϯυγΣΠΫͷηογϣϯ৘ใΛαʔόʹ
    Ωϟογϡ
    • nginxͰ͸ڞ༗ϝϞϦ্ʹΩϟογϡ͞ΕΔ
    • ࣍ճͷTLSϋϯυγΣΠΫΛলུ
    • CPUϦιʔεͷ࡟ݮ΍ϨΠςϯγղফͷޮՌ͕͋Δ
    

    View Slide

25. TLS Session Cache with nginx
    

    View Slide

26. TLS Session Tickets
    • ҉߸Խͨ͠ηογϣϯ৘ใ(νέοτ)ΛΫϥ
    ΠΞϯτʹ౉͢
    • νέοτΛݩʹTLSηογϣϯΛ࠶։
    • HTTPSαʔόෳ਺୆ͰηογϣϯΩϟογϡ
    Λڞ༗Ͱ͖Δ
    • αϙʔτ͍ͯ͠Δ୺຤͕গͳ͍ͷ͕೉఺
    

    View Slide

27. TLS Session Tickets with nginx
    

    View Slide

28. SPDY
    • ࠷৽ͷiOS΍AndroidͰ͸αʔό͕ରԠͯ͠
    ͍Ε͹ࣗಈతʹSPDY/3.1͕ར༻͞ΕΔ
    • ΫϥΠΞϯτଆͰରԠ͠ͳͯ͘΋Α͍
    

    View Slide

29. SPDY with nginx
    

    View Slide

30. TLS Session (Cache|Tickets)
    ಋೖλΠϛϯά
    

    View Slide

31. TLS Session (Cache|Tickets)
    ಋೖλΠϛϯά
    

    View Slide

32. SPDY
    41%:ಋೖͨ͠೔ͷϐʔΫ
    

    View Slide

33. Agenda
    • High performance API server with PHP!!!
    • High performance networking with nginx
    • High performance Application with Go
    

    View Slide

34. mercari.go
    • ϝϧΧϦͷϝΠϯ։ൃݴޠ͸PHP
    • ΠϯϑϥνʔϜͩͱGoɺNodeɺRubyɺPerl౳͍
    Ζ͍Ζ(ΈΜͳಘҙ෼໺͕ҧ͏)
    • ࠷ۙ͸ͪΐͬͱͨ͠CLI΍σʔϞϯɺύϑΥʔϚϯ
    ε͕ཁٻ͞ΕΔίϯϙʔωϯτΛGoͰ։ൃ͢Δػ
    ձ͕૿͑ͯΔ
    

    View Slide

35. mercari.go
    • nginx-build
    • cachectl
    • slackboard
    • Ϋϩʔζυ
    • Gaurun, refreshcache
    

    View Slide

36. cɺconsul΋࢖ͬͯΔ͔Β(਒͑੠)
    

    View Slide

37. ͳ͓ɺόάͬͯΔ໛༷
    ઌि͜ͷόάΛճආ͢Δίʔυॻ͍ͯ·ͨ͠
    

    View Slide

38. mercari.go
    • nginx-build
    • cachectl
    • slackboard
    • Ϋϩʔζυ
    • Gaurun, refreshcache
    

    View Slide

39. Gaurun
    • ൚༻ϓογϡ௨஌αʔό
    • ୹࣌ؒͰ਺ेɺ਺ඦສ݅ͷϓογϡ௨஌͕Մೳ
    • HTTPͱJSONΛϕʔεʹͨ͠γϯϓϧͳAPI
    • ϝϧΧϦ಺ͷϓογϡ௨஌ɺେମGaurunͰஔ
    ͖׵͑ͨ
    

    View Slide

40. ൚༻ϓογϡ௨஌γεςϜͷΞʔΩςΫνϟ
    PS
    CBUDIαʔό
    

    View Slide

41. Gaurun HTTP API
    "1* ղઆ
    1045QVTI ϓογϡ௨஌ϦΫΤετड෇
    (&5TUBUHP (PϥϯλΠϜͷ׆ಈঢ়گΛऔಘ
    (&5TUBUBQQ
    ಺෦Ωϡʔ νϟωϧ
    ͷ࢖༻ྔ΍
    ϓογϡͷ੒ޭࣦഊ਺ͷऔಘ
    (&5DPOpHBQQ αʔόઃఆ 50.-
    Λऔಘ
    

    View Slide

42. POST /pushͷϦΫΤετϘσΟ
    

    View Slide

43. GET /stat/go
    $ curl -s http://127.0.0.1:1056/stat/go
    {
    "time": 1423270089195474419,
    "go_version": "go1.4",
    "go_os": "linux",
    "go_arch": "amd64",
    "cpu_num": 24,
    "goroutine_num": 55,
    "gomaxprocs": 24,
    "cgo_call_num": 124016,
    "memory_alloc": 10309424,
    "memory_total_alloc": 295474683712,
    "memory_sys": 25557240,
    "memory_lookups": 214058,
    "memory_mallocs": 2118264299,
    "memory_frees": 2118230111,
    ɾ
    ɾ
    

    View Slide

44. GET /stat/app
    $ curl -s http://127.0.0.1:1056/stat/app
    {
    "queue_max": 40960, // ಺෦ΩϡʔͷαΠζ
    "queue_usage": 0, // ಺෦Ωϡʔͷ࢖༻ྔ(͙͢ࡹ͚ΔͷͰେମ0)
    "ios": {
    "push_success": 31804, // iOSϓογϡ௨஌ͷ੒ޭ਺
    "push_error": 17 // iOSϓογϡ௨஌ͷࣦഊ਺
    },
    "android": {
    "push_success": 18683, // Androidϓογϡ௨஌ͷ੒ޭ਺
    "push_error": 233 // Androidϓογϡ௨஌ͷࣦഊ਺
    }
    $
    

    View Slide

45. GaurunΛ࣮ߦ͍ͯ͠Δ༷ࢠͰ͢
    

    View Slide

46. mercari.go
    • nginx-build
    • cachectl
    • slackboard
    • Ϋϩʔζυ
    • Gaurun, refreshcache
    

    View Slide

47. refreshcache
    • ࠷ॳͷํͰݴͬͯͨಛఆAPIͷϨεϙϯεͷ
    ΩϟογϡΛੜ੒͢ΔσʔϞϯ
    • ҰఆִؒͰฒྻʹ݁ߏͳྔͷHTTPϦΫΤετ
    ΛૹΓଓ͚Δ
    • ݩʑRubyͰॻ͔Ε͍ͯͨ(஗͔ͬͨ)
    

    View Slide

48. ݩͷRubyίʔυ
    Parallel.map(params_list, :in_processes => in_processes) do |params|
    (தུ)
    res = http_request('GET', url, params)
    (தུ)
    end.compact
    params_listͷαΠζ͕େ͖͍
    ↓
    େྔʹfork͢ΔͷͰॏ͍
    

    View Slide

49. GoͰॻ͖௚ͨ͠
    done := make(chan error)
    size := len(paramCombs.Params)
    var errors []error
    for i := 0; i < size; i++ {
    go makeCache(URL, paramCombs.Params[i], done)
    }
    for i := 0; i < size; i++ {
    err := <-done
    if err != nil {
    errors = append(errors, err)
    }
    }
    

    View Slide

50. ࣮ߦαʔόͷάϥϑ(CPU)
    ϦϥΠτͨ͠λΠϛϯά
    

    View Slide

51. ࣮ߦαʔόͷάϥϑ(LA)
    ϦϥΠτͨ͠λΠϛϯά
    

    View Slide

52. ·ͱΊ
    • ϝϧΧϦͷߴ଎ͳόοΫΤϯυͷ࢓૊Έʹ͍ͭͯ
    ͍͔ͭ͘ղઆ͠·ͨ͠
    • ϝΠϯ͸PHP͚ͩͲ࠷ۙ͸Goͷར༻ࣄྫ૿͑ͯΔ
    • Go͸ૣ͍ɾ଎͍ɾ͍ܰ
    • Gaurun͸ͦͷ͏ͪOSSԽ༧ఆ
    

    View Slide