Upgrade to Pro — share decks privately, control downloads, hide ads and more …

1000万ダウンロードアプリ『メルカリ』を支える技術

 1000万ダウンロードアプリ『メルカリ』を支える技術

dots. Summit 2015

Tatsuhiko Kubo

February 07, 2015
Tweet

More Decks by Tatsuhiko Kubo

Other Decks in Technology

Transcript

  1. 1000ສμ΢ϯϩʔυΞϓϦ
    ʰϝϧΧϦʱ
    Λࢧ͑Δٕज़
    גࣜձࣾϝϧΧϦ
    ΤϯδχΞ
    ٱอୡ඙
    [email protected]
    dots. summit 2015

    View full-size slide

  2. ࣗݾ঺հ
    • ٱอୡ඙(bokko)
    • @cubicdaiya
    • Software Engineer in Infrastructure Engineering
    • OSS Contributor
    • e.g. nginx, mruby, ngx_mruby, ngx_lua, h2o, etc…

    View full-size slide

  3. ʰϝϧΧϦʱʹ͍ͭͯ

    View full-size slide

  4. ओཁKPIσʔλ
    μ΢ϯϩʔυ਺
    ߪೖֹۚ
    ग़඼਺
    ਺஋
    1000ສDLಥഁ
    ݄ؒ਺ेԯԁ
    1೔਺ेສ඼Ҏ্
    ΞϓϦΛެ։͔ͯ͠Β໿1೥ͱ൒೥Ͱ
    1000ສμ΢ϯϩʔυୡ੒ʂ(2013೥7݄ެ։)

    View full-size slide

  5. ʰϝϧΧϦʱΛࢧ͑Δ̋̋

    View full-size slide

  6. τϐοΫ
    • ϝϧΧϦΛࢧ͑Δ։ൃମ੍
    • ϝϧΧϦΛࢧ͑ΔΠϯϑϥ
    • ϝϧΧϦΛࢧ͑ΔαϒγεςϜ
    • ϝϧΧϦΛࢧ͑ΔGo

    View full-size slide

  7. τϐοΫ
    • ϝϧΧϦΛࢧ͑Δ։ൃମ੍
    • ϝϧΧϦΛࢧ͑ΔΠϯϑϥ
    • ϝϧΧϦΛࢧ͑ΔαϒγεςϜ
    • ϝϧΧϦΛࢧ͑ΔGo

    View full-size slide

  8. ʰϝϧΧϦʱΛࢧ͑Δ։ൃମ੍
    • ΤϯδχΞ໿20ਓ(QAɺCS෦໳΋͋Δ)
    • ΠϯϑϥνʔϜ͸ࣗ෼ؚΊͯ5ਓ
    • ৭ΜͳόοΫΤϯυΛ࣋ͭΤϯδχΞ͕ू·ͬͯ·͢
    • Ϛωʔδϟ৬΍্ڃΤϯδχΞ৬΋͋Γ·͕͢ɺϑϥοτͳ
    งғؾͰ͢

    View full-size slide

  9. ʰϝϧΧϦʱΛࢧ͑Δ։ൃମ੍
    • Github্ͰPull Requestϕʔεͷ։ൃ
    • RedmineͰλεΫ؅ཧ
    • Slackϑϧ׆༻

    View full-size slide

  10. ʰϝϧΧϦʱΛࢧ͑Δ։ൃମ੍
    • Qiita:TeamͰυΩϡϝϯτڞ༗
    • Travis-CIʹΑΔܧଓతΠϯςάϨʔγϣϯ
    • Ansible PlaybookʹΑΔϓϩϏδϣχϯά
    • DockerʹΑΔݸਓ։ൃ؀ڥ(Vagrantͷਓ΋͍·͢)

    View full-size slide

  11. τϐοΫ
    • ϝϧΧϦΛࢧ͑Δ։ൃମ੍
    • ϝϧΧϦΛࢧ͑ΔΠϯϑϥ
    • ϝϧΧϦΛࢧ͑ΔαϒγεςϜ
    • ϝϧΧϦΛࢧ͑ΔPHP
    • ϝϧΧϦΛࢧ͑ΔGo

    View full-size slide

  12. ʰϝϧΧϦʱΛࢧ͑ΔΠϯϑϥ
    • ͘͞Βͷઐ༻αʔό
    • ͘͞ΒΫϥ΢υ
    • Amazon Web Service
    • Google Cloud Platform
    • Treasure Data
    • etc…

    View full-size slide

  13. ʰϝϧΧϦʱΛࢧ͑ΔΠϯϑϥ
    • αʔό୆਺͸100୆Ҏ্
    • ΞϓϦ͸JP൛ͱUS൛͕͋Δ
    • JP൛͸͘͞ΒɺUS൛͸AWS্Ͱӡ༻
    • ࡢ೥ͷޙ൒͔ΒJP൛͸͘͞Βͷઐ༻αʔό͕ϝΠϯ
    • ioDrive࢖ͬͯ·͢

    View full-size slide

  14. τϐοΫ
    • ϝϧΧϦΛࢧ͑Δ։ൃମ੍
    • ϝϧΧϦΛࢧ͑ΔΠϯϑϥ
    • ϝϧΧϦΛࢧ͑ΔαϒγεςϜ
    • ϝϧΧϦΛࢧ͑ΔGo

    View full-size slide

  15. େࡶ೺ͳશମ૾

    View full-size slide

  16. શ෦͓఻͑͢Δͷ͸ແཧͳͷͰ
    ͍͔ͭ͘঺հ͍͖ͯ͠·͢

    View full-size slide

  17. ϩʔυόϥϯαʔ

    View full-size slide

  18. nginx
    • APIϦΫΤετͷϩʔυόϥϯε
    • SSLλʔϛωʔγϣϯ & SPDYήʔτ΢ΣΠ
    • ίϯςϯπΩϟογϡ

    View full-size slide

  19. HTTPS௨৴ͷߴ଎Խ
    • TLS Session Cache
    • TLS Session Tickets
    • SPDY

    View full-size slide

  20. TLS Session Cache
    • TLSϋϯυγΣΠΫͷηογϣϯ৘ใΛαʔόʹ
    Ωϟογϡ
    • nginxͰ͸ڞ༗ϝϞϦ্ʹΩϟογϡ͞ΕΔ
    • ࣍ճͷTLSϋϯυγΣΠΫΛলུ
    • CPUϦιʔεͷ࡟ݮ΍ϨΠςϯγղফͷޮՌ͕͋Δ

    View full-size slide

  21. TLS Session Cache with nginx

    View full-size slide

  22. TLS Session Tickets
    • ҉߸Խͨ͠ηογϣϯ৘ใ(νέοτ)ΛΫϥ
    ΠΞϯτʹ౉͢
    • νέοτΛݩʹTLSηογϣϯΛ࠶։
    • HTTPSαʔόෳ਺୆ͰηογϣϯΩϟογϡ
    Λڞ༗Ͱ͖Δ
    • αϙʔτ͍ͯ͠Δ୺຤͕গͳ͍ͷ͕೉఺

    View full-size slide

  23. TLS Session Tickets with nginx

    View full-size slide

  24. SPDY
    • WebΛߴ଎Խ͢ΔͨΊͷϓϩτίϧ
    • nginx-1.5.10Ҏ߱ͰSPDY3.1͕ར༻Մ
    • ࠷৽ͷiOS΍AndroidͰ͸αʔό͕ରԠͯ͠
    ͍Ε͹ࣗಈతʹSPDY͕ར༻͞ΕΔ

    View full-size slide

  25. SPDY with nginx

    View full-size slide

  26. άϥϑͰݟΔ࠷దԽͷޮՌ
    5-44FTTJPO $BDIFc5JDLFUT
    ಋೖλΠϛϯά

    View full-size slide

  27. άϥϑͰݟΔ࠷దԽͷޮՌ
    5-44FTTJPO $BDIFc5JDLFUT
    ಋೖλΠϛϯά

    View full-size slide

  28. άϥϑͰݟΔ࠷దԽͷޮՌ
    41%:ಋೖͨ͠೔ͷϐʔΫ

    View full-size slide

  29. "QBDIFNPE@QIQ

    View full-size slide

  30. Apache + mod_php
    • ϝϧΧϦͷதͰҰ൪ٕज़తʹރΕͨίϯϙʔωϯτ
    • DietCakeͱ͍͏ܰྔͰߴ଎ͳϑϨʔϜϫʔΫΛར༻
    • ฏۉϨεϙϯελΠϜ͕ϐʔΫ࣌Ͱ΋100msલޙ
    • APIαʔό͸΄ͱΜͲϘτϧωοΫʹͳ͍ͬͯͳ͍

    View full-size slide

  31. σʔλετΞ

    View full-size slide

  32. .Z42- 3FEJT NFNDBDIFE 4PMS -VDFOF

    View full-size slide

  33. ϝϧΧϦΛࢧ͑ΔσʔλετΞ
    • MySQL
    • αʔϏεͷίΞσʔλ֨ೲ৔ॴ
    • αʔϏεͷن໛֦େʹ൐͍ਫฏ෼ׂ͕੝Μ
    • Redisɺmemcached
    • ΞϓϦέʔγϣϯΩϟογϡϨΠϠʔ
    • Solr(Lucene)
    • ݕࡧ

    View full-size slide

  34. ඇಉظॲཧͷͨΊͷ
    ΩϡʔͱδϣϒϫʔΧʔ

    View full-size slide

  35. 2.QIQ1BSBMMFM1SFGPSL

    View full-size slide

  36. ॏ͍ॲཧ͸2.ʹΤϯΩϡʔͯ͠ඇಉظͰ࣮ߦ

    View full-size slide

  37. PHP-Paralle-Prefork
    • PHPͷpreforkσʔϞϯϑϨʔϜϫʔΫ
    • ฐࣾͷαʔόαΠυΤϯδχΞ͕։ൃ
    • ϓϩηε୯ҐͰλεΫΛฒྻ࣮ߦ
    • Q4M͔ΒσΩϡʔͨ͠λεΫΛฒྻ࣮ߦ͍ͯ͠Δ
    • PHPͰ΋͜͜·ͰͰ͖Δʂ

    View full-size slide

  38. ϩάσʔλ෼ੳج൫

    View full-size slide

  39. ΞΫηεϩά΍ΞϓϦέʔγϣϯͷϩάΛ
    'MVFOUE΍όονͰ֤ॴʹసૹ

    View full-size slide

  40. ϝϧΧϦΛࢧ͑Δ
    ϩά&σʔλ෼ੳج൫
    ίϯϙʔωϯτ ༻్
    'MVFOUE ֤छϩάసૹ
    5SFBTVSFEBUB #JH2VFSZ σʔλ෼ੳ
    4 ϩάͷόοΫΞοϓ
    ,JCBOB&MBTUJDTFBSDI ௚ۙͷϩά΍σʔλͷ౷ܭΛνΣοΫ
    /PSJLSB ϦΞϧλΠϜͰσʔλूܭɾ؂ࢹ

    View full-size slide

  41. τϐοΫ
    • ϝϧΧϦΛࢧ͑Δ։ൃମ੍
    • ϝϧΧϦΛࢧ͑ΔΠϯϑϥ
    • ϝϧΧϦΛࢧ͑ΔαϒγεςϜ
    • ϝϧΧϦΛࢧ͑ΔGo

    View full-size slide

  42. ʰϝϧΧϦʱΛࢧ͑ΔGo
    • PHPͷΠϝʔδ͕ڧ͍ฐࣾͰ͕͢ɺRuby΍Nodeɺ
    Goͱ͍ͬͨݴޠ΋ར༻͍ͯ͠·͢
    • ࠷ۙPerlʹΑΔϞχλϦϯάπʔϧ͕ՃΘΓ·ͨ͠
    • ࠓճ͸ϝϧΧϦͰঃʑʹ૿͍͑ͯΔGoͷϓϩμΫτ
    Λ঺հ͠·͢

    View full-size slide

  43. ʰϝϧΧϦʱΛࢧ͑ΔGo
    • nginx-build
    • cachectl
    • Slackboard
    • ൚༻ϓογϡ௨஌αʔό(·ͩඇެ։)

    View full-size slide

  44. nginx-build
    • nginxͷϏϧυπʔϧ
    • ҎԼͷ࡞ۀΛࣗಈԽ
    • nginxͷιʔείʔυͷμ΢ϯϩʔυ
    • ґଘϥΠϒϥϦ΍3rd PartyϞδϡʔϧͷ૊ΈࠐΈ
    • nginxͷRPM࡞੒࣌ʹར༻͍ͯ͠Δ
    • https://github.com/cubicdaiya/nginx-build

    View full-size slide

  45. nginx-buildͰnginxΛϏϧυ

    View full-size slide

  46. cachectl
    • OSͷϖʔδΩϟογϡ੍ޚπʔϧ
    • ಛఆͷϑΝΠϧͷϖʔδΩϟογϡͷར༻ঢ়گͷ֬ೝ΍
    ϖʔδΩϟογϡͷ࡟আ͕Ͱ͖Δ
    • ϖʔδΩϟογϡΛ࡟আ͢ΔσʔϞϯ(cachectld)͕෇ଐ
    • APIαʔόͷڊେͳϩάϑΝΠϧΛϖʔδΩϟογϡ͔
    Βఆظతʹ௥͍ग़͢ͷʹར༻͍ͯ͠Δ(εϫοϓରࡦ)
    • https://github.com/cubicdaiya/cachectl

    View full-size slide

  47. cachectldͷઃఆͱىಈ

    View full-size slide

  48. APIαʔόʹcachectldΛ
    ಋೖͨ݁͠Ռɺ
    "1*αʔό͕εϫοϓ͠ͳ͘ͳͬͨ
    DBDIFDUMEΛىಈͨ͠λΠϛϯά

    View full-size slide

  49. Slackboard
    • SlackϓϩΩγαʔό
    • SlackͷHook URL΍νϟϯωϧͷ৘ใΛҰݩ؅ཧ&λά෇͚
    • Slackͷઃఆ͕֤αʔόʹ෼ࢄ͢ΔͷΛ๷͙
    • ྡͷ੮ͷਓᐌ͘ʮ͋͋ɺikachanΈ͍ͨͳ΍ͭͶʯ
    • ikachan஌Βͳ͔ͬͨ…
    • https://github.com/cubicdaiya/slackboard

    View full-size slide

  50. 4MBDLͷઃఆ͸TMBDLCPBSEαʔόͰҰݩ؅ཧ

    View full-size slide

  51. खൈ͖γϯϓϧͳWeb UI
    ҰମԿετϥοϓͳΜͩɾɾɾ

    View full-size slide

  52. ൚༻ϓογϡ௨஌αʔό
    • HTTPϕʔεͷγϯϓϧͳAPIͱे෼ͳϞχλϦϯάػೳ
    • े෼ͳύϑΥʔϚϯε(਺ઍʙ਺ສQPS)
    • δϟʔφϧϩάʹΑΔΫϥογϡϦΧόϦʔػೳ
    • ୹࣌ؒͰ਺ेສ݅ͷϓογϡ௨஌Λߦ͏͜ͱ͕Ͱ͖Δ
    • ઌʑि͋ͨΓ͔Βຊ൪Քಇத
    • ͍͍Ͷɺίϝϯτɺ঎඼ߪೖͷϓογϡ௨஌ɺetc…
    • ΋͏ͪΐͬͱ࣮੷ੵΜͩΒOSSԽ͍ͨ͠

    View full-size slide

  53. ൚༻ϓογϡ௨஌γεςϜͷΞʔΩςΫνϟ
    PS
    CBUDIαʔό

    View full-size slide

  54. HTTP API
    "1* ղઆ
    1045QVTI ϓογϡ௨஌ϦΫΤετड෇
    (&5TUBUHP (PϥϯλΠϜͷ׆ಈঢ়گΛऔಘ
    (&5TUBUBQQ
    ಺෦Ωϡʔ νϟωϧ
    ͷ࢖༻ྔ΍
    ϓογϡͷ੒ޭࣦഊ਺ͷऔಘ
    (&5DPOpHBQQ αʔόઃఆ 50.-
    Λऔಘ

    View full-size slide

  55. POST /pushͷϦΫΤετϘσΟ

    View full-size slide

  56. GET /stat/go
    $ curl -s http://127.0.0.1:1056/stat/go
    {
    "time": 1423270089195474419,
    "go_version": "go1.4",
    "go_os": "linux",
    "go_arch": "amd64",
    "cpu_num": 24,
    "goroutine_num": 55,
    "gomaxprocs": 24,
    "cgo_call_num": 124016,
    "memory_alloc": 10309424,
    "memory_total_alloc": 295474683712,
    "memory_sys": 25557240,
    "memory_lookups": 214058,
    "memory_mallocs": 2118264299,
    "memory_frees": 2118230111,



    View full-size slide

  57. GET /stat/app
    $ curl -s http://127.0.0.1:1056/stat/app
    {
    "queue_max": 40960, // 内部キューのサイズ
    "queue_usage": 0, // 内部キューの使用量(すぐ捌けるので大体0)
    "ios": {
    "push_success": 31804, // iOSプッシュ通知の成功数
    "push_error": 17 // iOSプッシュ通知の失敗数
    },
    "android": {
    "push_success": 18683, // Androidプッシュ通知の成功数
    "push_error": 233 // Androidプッシュ通知の失敗数
    }
    $

    View full-size slide

  58. GoΛબ୒͢Δཧ༝
    • ύϑΥʔϚϯε͕ྑͯ͘ੜ࢈ੑ͕ߴ͍
    • ͲͷϓϩμΫτ΋਺࣌ؒͰ࠷ॳͷόʔδϣϯΛॻ͍ͨ
    • ࣮ߦϑΝΠϧ͕୯ҰόΠφϦͳͷͰ؅ཧָ͕
    • ฒྻॲཧ͕؆୯ʹॻ͚Δ
    • PHPͷऑ͍෦෼Λ͏·͘ิ׬Ͱ͖Δ

    View full-size slide

  59. ·ͱΊ
    • ϝϧΧϦ͸μ΢ϯϩʔυ਺1000ສΛӽ͑ΔਓؾϑϦϚΞϓϦ
    • ͘͞Β΍AWS౳ͷΠϯϑϥ΍Ϋϥ΢υαʔϏεͷ্
    ʹߏங͞Ε͍ͯΔ
    • ֤αϒγεςϜ͸ׂΓͱΦʔιυοΫε
    • ݴޠ͸PHP͚ͩͰͳ͘Go΍Node౳ɺෳ਺ͷݴޠΛঢ়
    گʹԠͯ͡࢖͍෼͚ͯ·͢

    View full-size slide

  60. ਓࡐืूதʂ

    View full-size slide