Upgrade to Pro — share decks privately, control downloads, hide ads and more …

1000万ダウンロードアプリ『メルカリ』を支える技術

 1000万ダウンロードアプリ『メルカリ』を支える技術

dots. Summit 2015

Tatsuhiko Kubo

February 07, 2015
Tweet

More Decks by Tatsuhiko Kubo

Other Decks in Technology

Transcript

  1. 1000ສμ΢ϯϩʔυΞϓϦ
    ʰϝϧΧϦʱ
    Λࢧ͑Δٕज़
    גࣜձࣾϝϧΧϦ
    ΤϯδχΞ
    ٱอୡ඙
    [email protected]
    dots. summit 2015

    View Slide

  2. ࣗݾ঺հ
    • ٱอୡ඙(bokko)
    • @cubicdaiya
    • Software Engineer in Infrastructure Engineering
    • OSS Contributor
    • e.g. nginx, mruby, ngx_mruby, ngx_lua, h2o, etc…

    View Slide

  3. ʰϝϧΧϦʱʹ͍ͭͯ

    View Slide

  4. View Slide

  5. ओཁKPIσʔλ
    μ΢ϯϩʔυ਺
    ߪೖֹۚ
    ग़඼਺
    ਺஋
    1000ສDLಥഁ
    ݄ؒ਺ेԯԁ
    1೔਺ेສ඼Ҏ্
    ΞϓϦΛެ։͔ͯ͠Β໿1೥ͱ൒೥Ͱ
    1000ສμ΢ϯϩʔυୡ੒ʂ(2013೥7݄ެ։)

    View Slide

  6. View Slide

  7. ʰϝϧΧϦʱΛࢧ͑Δ̋̋

    View Slide

  8. τϐοΫ
    • ϝϧΧϦΛࢧ͑Δ։ൃମ੍
    • ϝϧΧϦΛࢧ͑ΔΠϯϑϥ
    • ϝϧΧϦΛࢧ͑ΔαϒγεςϜ
    • ϝϧΧϦΛࢧ͑ΔGo

    View Slide

  9. τϐοΫ
    • ϝϧΧϦΛࢧ͑Δ։ൃମ੍
    • ϝϧΧϦΛࢧ͑ΔΠϯϑϥ
    • ϝϧΧϦΛࢧ͑ΔαϒγεςϜ
    • ϝϧΧϦΛࢧ͑ΔGo

    View Slide

  10. ʰϝϧΧϦʱΛࢧ͑Δ։ൃମ੍
    • ΤϯδχΞ໿20ਓ(QAɺCS෦໳΋͋Δ)
    • ΠϯϑϥνʔϜ͸ࣗ෼ؚΊͯ5ਓ
    • ৭ΜͳόοΫΤϯυΛ࣋ͭΤϯδχΞ͕ू·ͬͯ·͢
    • Ϛωʔδϟ৬΍্ڃΤϯδχΞ৬΋͋Γ·͕͢ɺϑϥοτͳ
    งғؾͰ͢

    View Slide

  11. ʰϝϧΧϦʱΛࢧ͑Δ։ൃମ੍
    • Github্ͰPull Requestϕʔεͷ։ൃ
    • RedmineͰλεΫ؅ཧ
    • Slackϑϧ׆༻

    View Slide

  12. ʰϝϧΧϦʱΛࢧ͑Δ։ൃମ੍
    • Qiita:TeamͰυΩϡϝϯτڞ༗
    • Travis-CIʹΑΔܧଓతΠϯςάϨʔγϣϯ
    • Ansible PlaybookʹΑΔϓϩϏδϣχϯά
    • DockerʹΑΔݸਓ։ൃ؀ڥ(Vagrantͷਓ΋͍·͢)

    View Slide

  13. τϐοΫ
    • ϝϧΧϦΛࢧ͑Δ։ൃମ੍
    • ϝϧΧϦΛࢧ͑ΔΠϯϑϥ
    • ϝϧΧϦΛࢧ͑ΔαϒγεςϜ
    • ϝϧΧϦΛࢧ͑ΔPHP
    • ϝϧΧϦΛࢧ͑ΔGo

    View Slide

  14. ʰϝϧΧϦʱΛࢧ͑ΔΠϯϑϥ
    • ͘͞Βͷઐ༻αʔό
    • ͘͞ΒΫϥ΢υ
    • Amazon Web Service
    • Google Cloud Platform
    • Treasure Data
    • etc…

    View Slide

  15. ʰϝϧΧϦʱΛࢧ͑ΔΠϯϑϥ
    • αʔό୆਺͸100୆Ҏ্
    • ΞϓϦ͸JP൛ͱUS൛͕͋Δ
    • JP൛͸͘͞ΒɺUS൛͸AWS্Ͱӡ༻
    • ࡢ೥ͷޙ൒͔ΒJP൛͸͘͞Βͷઐ༻αʔό͕ϝΠϯ
    • ioDrive࢖ͬͯ·͢

    View Slide

  16. τϐοΫ
    • ϝϧΧϦΛࢧ͑Δ։ൃମ੍
    • ϝϧΧϦΛࢧ͑ΔΠϯϑϥ
    • ϝϧΧϦΛࢧ͑ΔαϒγεςϜ
    • ϝϧΧϦΛࢧ͑ΔGo

    View Slide

  17. େࡶ೺ͳશମ૾

    View Slide

  18. શ෦͓఻͑͢Δͷ͸ແཧͳͷͰ
    ͍͔ͭ͘঺հ͍͖ͯ͠·͢

    View Slide

  19. ϩʔυόϥϯαʔ

    View Slide

  20. OHJOY

    View Slide

  21. nginx
    • APIϦΫΤετͷϩʔυόϥϯε
    • SSLλʔϛωʔγϣϯ & SPDYήʔτ΢ΣΠ
    • ίϯςϯπΩϟογϡ

    View Slide

  22. HTTPS௨৴ͷߴ଎Խ
    • TLS Session Cache
    • TLS Session Tickets
    • SPDY

    View Slide

  23. TLS Session Cache
    • TLSϋϯυγΣΠΫͷηογϣϯ৘ใΛαʔόʹ
    Ωϟογϡ
    • nginxͰ͸ڞ༗ϝϞϦ্ʹΩϟογϡ͞ΕΔ
    • ࣍ճͷTLSϋϯυγΣΠΫΛলུ
    • CPUϦιʔεͷ࡟ݮ΍ϨΠςϯγղফͷޮՌ͕͋Δ

    View Slide

  24. TLS Session Cache with nginx

    View Slide

  25. TLS Session Tickets
    • ҉߸Խͨ͠ηογϣϯ৘ใ(νέοτ)ΛΫϥ
    ΠΞϯτʹ౉͢
    • νέοτΛݩʹTLSηογϣϯΛ࠶։
    • HTTPSαʔόෳ਺୆ͰηογϣϯΩϟογϡ
    Λڞ༗Ͱ͖Δ
    • αϙʔτ͍ͯ͠Δ୺຤͕গͳ͍ͷ͕೉఺

    View Slide

  26. TLS Session Tickets with nginx

    View Slide

  27. SPDY
    • WebΛߴ଎Խ͢ΔͨΊͷϓϩτίϧ
    • nginx-1.5.10Ҏ߱ͰSPDY3.1͕ར༻Մ
    • ࠷৽ͷiOS΍AndroidͰ͸αʔό͕ରԠͯ͠
    ͍Ε͹ࣗಈతʹSPDY͕ར༻͞ΕΔ

    View Slide

  28. SPDY with nginx

    View Slide

  29. άϥϑͰݟΔ࠷దԽͷޮՌ
    5-44FTTJPO $BDIFc5JDLFUT
    ಋೖλΠϛϯά

    View Slide

  30. άϥϑͰݟΔ࠷దԽͷޮՌ
    5-44FTTJPO $BDIFc5JDLFUT
    ಋೖλΠϛϯά

    View Slide

  31. άϥϑͰݟΔ࠷దԽͷޮՌ
    41%:ಋೖͨ͠೔ͷϐʔΫ

    View Slide

  32. "1*αʔό

    View Slide

  33. View Slide

  34. Apache + mod_php
    • ϝϧΧϦͷதͰҰ൪ٕज़తʹރΕͨίϯϙʔωϯτ
    • DietCakeͱ͍͏ܰྔͰߴ଎ͳϑϨʔϜϫʔΫΛར༻
    • ฏۉϨεϙϯελΠϜ͕ϐʔΫ࣌Ͱ΋100msલޙ
    • APIαʔό͸΄ͱΜͲϘτϧωοΫʹͳ͍ͬͯͳ͍

    View Slide

  35. σʔλετΞ

    View Slide

  36. .Z42- 3FEJT NFNDBDIFE 4PMS -VDFOF

    View Slide

  37. ϝϧΧϦΛࢧ͑ΔσʔλετΞ
    • MySQL
    • αʔϏεͷίΞσʔλ֨ೲ৔ॴ
    • αʔϏεͷن໛֦େʹ൐͍ਫฏ෼ׂ͕੝Μ
    • Redisɺmemcached
    • ΞϓϦέʔγϣϯΩϟογϡϨΠϠʔ
    • Solr(Lucene)
    • ݕࡧ

    View Slide

  38. ඇಉظॲཧͷͨΊͷ
    ΩϡʔͱδϣϒϫʔΧʔ

    View Slide

  39. 2.QIQ1BSBMMFM1SFGPSL

    View Slide

  40. ॏ͍ॲཧ͸2.ʹΤϯΩϡʔͯ͠ඇಉظͰ࣮ߦ

    View Slide

  41. PHP-Paralle-Prefork
    • PHPͷpreforkσʔϞϯϑϨʔϜϫʔΫ
    • ฐࣾͷαʔόαΠυΤϯδχΞ͕։ൃ
    • ϓϩηε୯ҐͰλεΫΛฒྻ࣮ߦ
    • Q4M͔ΒσΩϡʔͨ͠λεΫΛฒྻ࣮ߦ͍ͯ͠Δ
    • PHPͰ΋͜͜·ͰͰ͖Δʂ

    View Slide

  42. ϩάσʔλ෼ੳج൫

    View Slide

  43. View Slide

  44. ΞΫηεϩά΍ΞϓϦέʔγϣϯͷϩάΛ
    'MVFOUE΍όονͰ֤ॴʹసૹ

    View Slide

  45. ϝϧΧϦΛࢧ͑Δ
    ϩά&σʔλ෼ੳج൫
    ίϯϙʔωϯτ ༻్
    'MVFOUE ֤छϩάసૹ
    5SFBTVSFEBUB #JH2VFSZ σʔλ෼ੳ
    4 ϩάͷόοΫΞοϓ
    ,JCBOB&MBTUJDTFBSDI ௚ۙͷϩά΍σʔλͷ౷ܭΛνΣοΫ
    /PSJLSB ϦΞϧλΠϜͰσʔλूܭɾ؂ࢹ

    View Slide

  46. τϐοΫ
    • ϝϧΧϦΛࢧ͑Δ։ൃମ੍
    • ϝϧΧϦΛࢧ͑ΔΠϯϑϥ
    • ϝϧΧϦΛࢧ͑ΔαϒγεςϜ
    • ϝϧΧϦΛࢧ͑ΔGo

    View Slide

  47. ʰϝϧΧϦʱΛࢧ͑ΔGo
    • PHPͷΠϝʔδ͕ڧ͍ฐࣾͰ͕͢ɺRuby΍Nodeɺ
    Goͱ͍ͬͨݴޠ΋ར༻͍ͯ͠·͢
    • ࠷ۙPerlʹΑΔϞχλϦϯάπʔϧ͕ՃΘΓ·ͨ͠
    • ࠓճ͸ϝϧΧϦͰঃʑʹ૿͍͑ͯΔGoͷϓϩμΫτ
    Λ঺հ͠·͢

    View Slide

  48. ʰϝϧΧϦʱΛࢧ͑ΔGo
    • nginx-build
    • cachectl
    • Slackboard
    • ൚༻ϓογϡ௨஌αʔό(·ͩඇެ։)

    View Slide

  49. nginx-build
    • nginxͷϏϧυπʔϧ
    • ҎԼͷ࡞ۀΛࣗಈԽ
    • nginxͷιʔείʔυͷμ΢ϯϩʔυ
    • ґଘϥΠϒϥϦ΍3rd PartyϞδϡʔϧͷ૊ΈࠐΈ
    • nginxͷRPM࡞੒࣌ʹར༻͍ͯ͠Δ
    • https://github.com/cubicdaiya/nginx-build

    View Slide

  50. nginx-buildͰnginxΛϏϧυ

    View Slide

  51. cachectl
    • OSͷϖʔδΩϟογϡ੍ޚπʔϧ
    • ಛఆͷϑΝΠϧͷϖʔδΩϟογϡͷར༻ঢ়گͷ֬ೝ΍
    ϖʔδΩϟογϡͷ࡟আ͕Ͱ͖Δ
    • ϖʔδΩϟογϡΛ࡟আ͢ΔσʔϞϯ(cachectld)͕෇ଐ
    • APIαʔόͷڊେͳϩάϑΝΠϧΛϖʔδΩϟογϡ͔
    Βఆظతʹ௥͍ग़͢ͷʹར༻͍ͯ͠Δ(εϫοϓରࡦ)
    • https://github.com/cubicdaiya/cachectl

    View Slide

  52. cachectldͷઃఆͱىಈ

    View Slide

  53. APIαʔόʹcachectldΛ
    ಋೖͨ݁͠Ռɺ
    "1*αʔό͕εϫοϓ͠ͳ͘ͳͬͨ
    DBDIFDUMEΛىಈͨ͠λΠϛϯά

    View Slide

  54. Slackboard
    • SlackϓϩΩγαʔό
    • SlackͷHook URL΍νϟϯωϧͷ৘ใΛҰݩ؅ཧ&λά෇͚
    • Slackͷઃఆ͕֤αʔόʹ෼ࢄ͢ΔͷΛ๷͙
    • ྡͷ੮ͷਓᐌ͘ʮ͋͋ɺikachanΈ͍ͨͳ΍ͭͶʯ
    • ikachan஌Βͳ͔ͬͨ…
    • https://github.com/cubicdaiya/slackboard

    View Slide

  55. 4MBDLͷઃఆ͸TMBDLCPBSEαʔόͰҰݩ؅ཧ

    View Slide

  56. खൈ͖γϯϓϧͳWeb UI
    ҰମԿετϥοϓͳΜͩɾɾɾ

    View Slide

  57. ൚༻ϓογϡ௨஌αʔό
    • HTTPϕʔεͷγϯϓϧͳAPIͱे෼ͳϞχλϦϯάػೳ
    • े෼ͳύϑΥʔϚϯε(਺ઍʙ਺ສQPS)
    • δϟʔφϧϩάʹΑΔΫϥογϡϦΧόϦʔػೳ
    • ୹࣌ؒͰ਺ेສ݅ͷϓογϡ௨஌Λߦ͏͜ͱ͕Ͱ͖Δ
    • ઌʑि͋ͨΓ͔Βຊ൪Քಇத
    • ͍͍Ͷɺίϝϯτɺ঎඼ߪೖͷϓογϡ௨஌ɺetc…
    • ΋͏ͪΐͬͱ࣮੷ੵΜͩΒOSSԽ͍ͨ͠

    View Slide

  58. ൚༻ϓογϡ௨஌γεςϜͷΞʔΩςΫνϟ
    PS
    CBUDIαʔό

    View Slide

  59. HTTP API
    "1* ղઆ
    1045QVTI ϓογϡ௨஌ϦΫΤετड෇
    (&5TUBUHP (PϥϯλΠϜͷ׆ಈঢ়گΛऔಘ
    (&5TUBUBQQ
    ಺෦Ωϡʔ νϟωϧ
    ͷ࢖༻ྔ΍
    ϓογϡͷ੒ޭࣦഊ਺ͷऔಘ
    (&5DPOpHBQQ αʔόઃఆ 50.-
    Λऔಘ

    View Slide

  60. POST /pushͷϦΫΤετϘσΟ

    View Slide

  61. GET /stat/go
    $ curl -s http://127.0.0.1:1056/stat/go
    {
    "time": 1423270089195474419,
    "go_version": "go1.4",
    "go_os": "linux",
    "go_arch": "amd64",
    "cpu_num": 24,
    "goroutine_num": 55,
    "gomaxprocs": 24,
    "cgo_call_num": 124016,
    "memory_alloc": 10309424,
    "memory_total_alloc": 295474683712,
    "memory_sys": 25557240,
    "memory_lookups": 214058,
    "memory_mallocs": 2118264299,
    "memory_frees": 2118230111,



    View Slide

  62. GET /stat/app
    $ curl -s http://127.0.0.1:1056/stat/app
    {
    "queue_max": 40960, // 内部キューのサイズ
    "queue_usage": 0, // 内部キューの使用量(すぐ捌けるので大体0)
    "ios": {
    "push_success": 31804, // iOSプッシュ通知の成功数
    "push_error": 17 // iOSプッシュ通知の失敗数
    },
    "android": {
    "push_success": 18683, // Androidプッシュ通知の成功数
    "push_error": 233 // Androidプッシュ通知の失敗数
    }
    $

    View Slide

  63. GoΛબ୒͢Δཧ༝
    • ύϑΥʔϚϯε͕ྑͯ͘ੜ࢈ੑ͕ߴ͍
    • ͲͷϓϩμΫτ΋਺࣌ؒͰ࠷ॳͷόʔδϣϯΛॻ͍ͨ
    • ࣮ߦϑΝΠϧ͕୯ҰόΠφϦͳͷͰ؅ཧָ͕
    • ฒྻॲཧ͕؆୯ʹॻ͚Δ
    • PHPͷऑ͍෦෼Λ͏·͘ิ׬Ͱ͖Δ

    View Slide

  64. ·ͱΊ
    • ϝϧΧϦ͸μ΢ϯϩʔυ਺1000ສΛӽ͑ΔਓؾϑϦϚΞϓϦ
    • ͘͞Β΍AWS౳ͷΠϯϑϥ΍Ϋϥ΢υαʔϏεͷ্
    ʹߏங͞Ε͍ͯΔ
    • ֤αϒγεςϜ͸ׂΓͱΦʔιυοΫε
    • ݴޠ͸PHP͚ͩͰͳ͘Go΍Node౳ɺෳ਺ͷݴޠΛঢ়
    گʹԠͯ͡࢖͍෼͚ͯ·͢

    View Slide

  65. ਓࡐืूதʂ

    View Slide