1000万ダウンロードアプリ『メルカリ』を支える技術

dots. Summit 2015

Transcript

1. 1000ສμ΢ϯϩʔυΞϓϦ
   ʰϝϧΧϦʱ
   Λࢧ͑Δٕज़
   גࣜձࣾϝϧΧϦ
   ΤϯδχΞ
   ٱอୡ඙
   bokko@mercariapp.com
   dots. summit 2015
   

   View Slide

2. ࣗݾ঺հ
   • ٱอୡ඙(bokko)
   • @cubicdaiya
   • Software Engineer in Infrastructure Engineering
   • OSS Contributor
   • e.g. nginx, mruby, ngx_mruby, ngx_lua, h2o, etc…
   

   View Slide

3. ʰϝϧΧϦʱʹ͍ͭͯ
   

   View Slide

4. View Slide

5. ओཁKPIσʔλ
   μ΢ϯϩʔυ਺
   ߪೖֹۚ
   ग़඼਺
   ਺஋
   1000ສDLಥഁ
   ݄ؒ਺ेԯԁ
   1೔਺ेສ඼Ҏ্
   ΞϓϦΛެ։͔ͯ͠Β໿1೥ͱ൒೥Ͱ
   1000ສμ΢ϯϩʔυୡ੒ʂ(2013೥7݄ެ։)
   

   View Slide

6. View Slide

7. ʰϝϧΧϦʱΛࢧ͑Δ̋̋
   

   View Slide

8. τϐοΫ
   • ϝϧΧϦΛࢧ͑Δ։ൃମ੍
   • ϝϧΧϦΛࢧ͑ΔΠϯϑϥ
   • ϝϧΧϦΛࢧ͑ΔαϒγεςϜ
   • ϝϧΧϦΛࢧ͑ΔGo
   

   View Slide

9. τϐοΫ
   • ϝϧΧϦΛࢧ͑Δ։ൃମ੍
   • ϝϧΧϦΛࢧ͑ΔΠϯϑϥ
   • ϝϧΧϦΛࢧ͑ΔαϒγεςϜ
   • ϝϧΧϦΛࢧ͑ΔGo
   

   View Slide

10. ʰϝϧΧϦʱΛࢧ͑Δ։ൃମ੍
    • ΤϯδχΞ໿20ਓ(QAɺCS෦໳΋͋Δ)
    • ΠϯϑϥνʔϜ͸ࣗ෼ؚΊͯ5ਓ
    • ৭ΜͳόοΫΤϯυΛ࣋ͭΤϯδχΞ͕ू·ͬͯ·͢
    • Ϛωʔδϟ৬΍্ڃΤϯδχΞ৬΋͋Γ·͕͢ɺϑϥοτͳ
    งғؾͰ͢
    

    View Slide

11. ʰϝϧΧϦʱΛࢧ͑Δ։ൃମ੍
    • Github্ͰPull Requestϕʔεͷ։ൃ
    • RedmineͰλεΫ؅ཧ
    • Slackϑϧ׆༻
    

    View Slide

12. ʰϝϧΧϦʱΛࢧ͑Δ։ൃମ੍
    • Qiita:TeamͰυΩϡϝϯτڞ༗
    • Travis-CIʹΑΔܧଓతΠϯςάϨʔγϣϯ
    • Ansible PlaybookʹΑΔϓϩϏδϣχϯά
    • DockerʹΑΔݸਓ։ൃ؀ڥ(Vagrantͷਓ΋͍·͢)
    

    View Slide

13. τϐοΫ
    • ϝϧΧϦΛࢧ͑Δ։ൃମ੍
    • ϝϧΧϦΛࢧ͑ΔΠϯϑϥ
    • ϝϧΧϦΛࢧ͑ΔαϒγεςϜ
    • ϝϧΧϦΛࢧ͑ΔPHP
    • ϝϧΧϦΛࢧ͑ΔGo
    

    View Slide

14. ʰϝϧΧϦʱΛࢧ͑ΔΠϯϑϥ
    • ͘͞Βͷઐ༻αʔό
    • ͘͞ΒΫϥ΢υ
    • Amazon Web Service
    • Google Cloud Platform
    • Treasure Data
    • etc…
    

    View Slide

15. ʰϝϧΧϦʱΛࢧ͑ΔΠϯϑϥ
    • αʔό୆਺͸100୆Ҏ্
    • ΞϓϦ͸JP൛ͱUS൛͕͋Δ
    • JP൛͸͘͞ΒɺUS൛͸AWS্Ͱӡ༻
    • ࡢ೥ͷޙ൒͔ΒJP൛͸͘͞Βͷઐ༻αʔό͕ϝΠϯ
    • ioDrive࢖ͬͯ·͢
    

    View Slide

16. τϐοΫ
    • ϝϧΧϦΛࢧ͑Δ։ൃମ੍
    • ϝϧΧϦΛࢧ͑ΔΠϯϑϥ
    • ϝϧΧϦΛࢧ͑ΔαϒγεςϜ
    • ϝϧΧϦΛࢧ͑ΔGo
    

    View Slide

17. େࡶ೺ͳશମ૾
    

    View Slide

18. શ෦͓఻͑͢Δͷ͸ແཧͳͷͰ
    ͍͔ͭ͘঺հ͍͖ͯ͠·͢
    

    View Slide

19. ϩʔυόϥϯαʔ
    

    View Slide

20. OHJOY
    

    View Slide

21. nginx
    • APIϦΫΤετͷϩʔυόϥϯε
    • SSLλʔϛωʔγϣϯ & SPDYήʔτ΢ΣΠ
    • ίϯςϯπΩϟογϡ
    

    View Slide

22. HTTPS௨৴ͷߴ଎Խ
    • TLS Session Cache
    • TLS Session Tickets
    • SPDY
    

    View Slide

23. TLS Session Cache
    • TLSϋϯυγΣΠΫͷηογϣϯ৘ใΛαʔόʹ
    Ωϟογϡ
    • nginxͰ͸ڞ༗ϝϞϦ্ʹΩϟογϡ͞ΕΔ
    • ࣍ճͷTLSϋϯυγΣΠΫΛলུ
    • CPUϦιʔεͷ࡟ݮ΍ϨΠςϯγղফͷޮՌ͕͋Δ
    

    View Slide

24. TLS Session Cache with nginx
    

    View Slide

25. TLS Session Tickets
    • ҉߸Խͨ͠ηογϣϯ৘ใ(νέοτ)ΛΫϥ
    ΠΞϯτʹ౉͢
    • νέοτΛݩʹTLSηογϣϯΛ࠶։
    • HTTPSαʔόෳ਺୆ͰηογϣϯΩϟογϡ
    Λڞ༗Ͱ͖Δ
    • αϙʔτ͍ͯ͠Δ୺຤͕গͳ͍ͷ͕೉఺
    

    View Slide

26. TLS Session Tickets with nginx
    

    View Slide

27. SPDY
    • WebΛߴ଎Խ͢ΔͨΊͷϓϩτίϧ
    • nginx-1.5.10Ҏ߱ͰSPDY3.1͕ར༻Մ
    • ࠷৽ͷiOS΍AndroidͰ͸αʔό͕ରԠͯ͠
    ͍Ε͹ࣗಈతʹSPDY͕ར༻͞ΕΔ
    

    View Slide

28. SPDY with nginx
    

    View Slide

29. άϥϑͰݟΔ࠷దԽͷޮՌ
    5-44FTTJPO $BDIFc5JDLFUT
    ಋೖλΠϛϯά
    

    View Slide

30. άϥϑͰݟΔ࠷దԽͷޮՌ
    5-44FTTJPO $BDIFc5JDLFUT
    ಋೖλΠϛϯά
    

    View Slide

31. άϥϑͰݟΔ࠷దԽͷޮՌ
    41%:ಋೖͨ͠೔ͷϐʔΫ
    

    View Slide

32. "1*αʔό
    

    View Slide

33. "QBDIFNPE@QIQ
    

    View Slide

34. Apache + mod_php
    • ϝϧΧϦͷதͰҰ൪ٕज़తʹރΕͨίϯϙʔωϯτ
    • DietCakeͱ͍͏ܰྔͰߴ଎ͳϑϨʔϜϫʔΫΛར༻
    • ฏۉϨεϙϯελΠϜ͕ϐʔΫ࣌Ͱ΋100msલޙ
    • APIαʔό͸΄ͱΜͲϘτϧωοΫʹͳ͍ͬͯͳ͍
    

    View Slide

35. σʔλετΞ
    

    View Slide

36. .Z42- 3FEJT NFNDBDIFE 4PMS -VDFOF
    
    

    View Slide

37. ϝϧΧϦΛࢧ͑ΔσʔλετΞ
    • MySQL
    • αʔϏεͷίΞσʔλ֨ೲ৔ॴ
    • αʔϏεͷن໛֦େʹ൐͍ਫฏ෼ׂ͕੝Μ
    • Redisɺmemcached
    • ΞϓϦέʔγϣϯΩϟογϡϨΠϠʔ
    • Solr(Lucene)
    • ݕࡧ
    

    View Slide

38. ඇಉظॲཧͷͨΊͷ
    ΩϡʔͱδϣϒϫʔΧʔ
    

    View Slide

39. 2.QIQ1BSBMMFM1SFGPSL
    

    View Slide

40. ॏ͍ॲཧ͸2.ʹΤϯΩϡʔͯ͠ඇಉظͰ࣮ߦ
    

    View Slide

41. PHP-Paralle-Prefork
    • PHPͷpreforkσʔϞϯϑϨʔϜϫʔΫ
    • ฐࣾͷαʔόαΠυΤϯδχΞ͕։ൃ
    • ϓϩηε୯ҐͰλεΫΛฒྻ࣮ߦ
    • Q4M͔ΒσΩϡʔͨ͠λεΫΛฒྻ࣮ߦ͍ͯ͠Δ
    • PHPͰ΋͜͜·ͰͰ͖Δʂ
    

    View Slide

42. ϩάσʔλ෼ੳج൫
    

    View Slide

43. View Slide

44. ΞΫηεϩά΍ΞϓϦέʔγϣϯͷϩάΛ
    'MVFOUE΍όονͰ֤ॴʹసૹ
    

    View Slide

45. ϝϧΧϦΛࢧ͑Δ
    ϩά&σʔλ෼ੳج൫
    ίϯϙʔωϯτ ༻్
    'MVFOUE ֤छϩάసૹ
    5SFBTVSFEBUB #JH2VFSZ σʔλ෼ੳ
    4 ϩάͷόοΫΞοϓ
    ,JCBOB&MBTUJDTFBSDI ௚ۙͷϩά΍σʔλͷ౷ܭΛνΣοΫ
    /PSJLSB ϦΞϧλΠϜͰσʔλूܭɾ؂ࢹ
    

    View Slide

46. τϐοΫ
    • ϝϧΧϦΛࢧ͑Δ։ൃମ੍
    • ϝϧΧϦΛࢧ͑ΔΠϯϑϥ
    • ϝϧΧϦΛࢧ͑ΔαϒγεςϜ
    • ϝϧΧϦΛࢧ͑ΔGo
    

    View Slide

47. ʰϝϧΧϦʱΛࢧ͑ΔGo
    • PHPͷΠϝʔδ͕ڧ͍ฐࣾͰ͕͢ɺRuby΍Nodeɺ
    Goͱ͍ͬͨݴޠ΋ར༻͍ͯ͠·͢
    • ࠷ۙPerlʹΑΔϞχλϦϯάπʔϧ͕ՃΘΓ·ͨ͠
    • ࠓճ͸ϝϧΧϦͰঃʑʹ૿͍͑ͯΔGoͷϓϩμΫτ
    Λ঺հ͠·͢
    

    View Slide

48. ʰϝϧΧϦʱΛࢧ͑ΔGo
    • nginx-build
    • cachectl
    • Slackboard
    • ൚༻ϓογϡ௨஌αʔό(·ͩඇެ։)
    

    View Slide

49. nginx-build
    • nginxͷϏϧυπʔϧ
    • ҎԼͷ࡞ۀΛࣗಈԽ
    • nginxͷιʔείʔυͷμ΢ϯϩʔυ
    • ґଘϥΠϒϥϦ΍3rd PartyϞδϡʔϧͷ૊ΈࠐΈ
    • nginxͷRPM࡞੒࣌ʹར༻͍ͯ͠Δ
    • https://github.com/cubicdaiya/nginx-build
    

    View Slide

50. nginx-buildͰnginxΛϏϧυ
    

    View Slide

51. cachectl
    • OSͷϖʔδΩϟογϡ੍ޚπʔϧ
    • ಛఆͷϑΝΠϧͷϖʔδΩϟογϡͷར༻ঢ়گͷ֬ೝ΍
    ϖʔδΩϟογϡͷ࡟আ͕Ͱ͖Δ
    • ϖʔδΩϟογϡΛ࡟আ͢ΔσʔϞϯ(cachectld)͕෇ଐ
    • APIαʔόͷڊେͳϩάϑΝΠϧΛϖʔδΩϟογϡ͔
    Βఆظతʹ௥͍ग़͢ͷʹར༻͍ͯ͠Δ(εϫοϓରࡦ)
    • https://github.com/cubicdaiya/cachectl
    

    View Slide

52. cachectldͷઃఆͱىಈ
    

    View Slide

53. APIαʔόʹcachectldΛ
    ಋೖͨ݁͠Ռɺ
    "1*αʔό͕εϫοϓ͠ͳ͘ͳͬͨ
    DBDIFDUMEΛىಈͨ͠λΠϛϯά
    

    View Slide

54. Slackboard
    • SlackϓϩΩγαʔό
    • SlackͷHook URL΍νϟϯωϧͷ৘ใΛҰݩ؅ཧ&λά෇͚
    • Slackͷઃఆ͕֤αʔόʹ෼ࢄ͢ΔͷΛ๷͙
    • ྡͷ੮ͷਓᐌ͘ʮ͋͋ɺikachanΈ͍ͨͳ΍ͭͶʯ
    • ikachan஌Βͳ͔ͬͨ…
    • https://github.com/cubicdaiya/slackboard
    

    View Slide

55. 4MBDLͷઃఆ͸TMBDLCPBSEαʔόͰҰݩ؅ཧ
    

    View Slide

56. खൈ͖γϯϓϧͳWeb UI
    ҰମԿετϥοϓͳΜͩɾɾɾ
    

    View Slide

57. ൚༻ϓογϡ௨஌αʔό
    • HTTPϕʔεͷγϯϓϧͳAPIͱे෼ͳϞχλϦϯάػೳ
    • े෼ͳύϑΥʔϚϯε(਺ઍʙ਺ສQPS)
    • δϟʔφϧϩάʹΑΔΫϥογϡϦΧόϦʔػೳ
    • ୹࣌ؒͰ਺ेສ݅ͷϓογϡ௨஌Λߦ͏͜ͱ͕Ͱ͖Δ
    • ઌʑि͋ͨΓ͔Βຊ൪Քಇத
    • ͍͍Ͷɺίϝϯτɺ঎඼ߪೖͷϓογϡ௨஌ɺetc…
    • ΋͏ͪΐͬͱ࣮੷ੵΜͩΒOSSԽ͍ͨ͠
    

    View Slide

58. ൚༻ϓογϡ௨஌γεςϜͷΞʔΩςΫνϟ
    PS
    CBUDIαʔό
    

    View Slide

59. HTTP API
    "1* ղઆ
    1045QVTI ϓογϡ௨஌ϦΫΤετड෇
    (&5TUBUHP (PϥϯλΠϜͷ׆ಈঢ়گΛऔಘ
    (&5TUBUBQQ
    ಺෦Ωϡʔ νϟωϧ
    ͷ࢖༻ྔ΍
    ϓογϡͷ੒ޭࣦഊ਺ͷऔಘ
    (&5DPOpHBQQ αʔόઃఆ 50.-
    Λऔಘ
    

    View Slide

60. POST /pushͷϦΫΤετϘσΟ
    

    View Slide

61. GET /stat/go
    $ curl -s http://127.0.0.1:1056/stat/go
    {
    "time": 1423270089195474419,
    "go_version": "go1.4",
    "go_os": "linux",
    "go_arch": "amd64",
    "cpu_num": 24,
    "goroutine_num": 55,
    "gomaxprocs": 24,
    "cgo_call_num": 124016,
    "memory_alloc": 10309424,
    "memory_total_alloc": 295474683712,
    "memory_sys": 25557240,
    "memory_lookups": 214058,
    "memory_mallocs": 2118264299,
    "memory_frees": 2118230111,
    ・
    ・
    ・
    

    View Slide

62. GET /stat/app
    $ curl -s http://127.0.0.1:1056/stat/app
    {
    "queue_max": 40960, // 内部キューのサイズ
    "queue_usage": 0, // 内部キューの使用量(すぐ捌けるので大体0)
    "ios": {
    "push_success": 31804, // iOSプッシュ通知の成功数
    "push_error": 17 // iOSプッシュ通知の失敗数
    },
    "android": {
    "push_success": 18683, // Androidプッシュ通知の成功数
    "push_error": 233 // Androidプッシュ通知の失敗数
    }
    $
    

    View Slide

63. GoΛબ୒͢Δཧ༝
    • ύϑΥʔϚϯε͕ྑͯ͘ੜ࢈ੑ͕ߴ͍
    • ͲͷϓϩμΫτ΋਺࣌ؒͰ࠷ॳͷόʔδϣϯΛॻ͍ͨ
    • ࣮ߦϑΝΠϧ͕୯ҰόΠφϦͳͷͰ؅ཧָ͕
    • ฒྻॲཧ͕؆୯ʹॻ͚Δ
    • PHPͷऑ͍෦෼Λ͏·͘ิ׬Ͱ͖Δ
    

    View Slide

64. ·ͱΊ
    • ϝϧΧϦ͸μ΢ϯϩʔυ਺1000ສΛӽ͑ΔਓؾϑϦϚΞϓϦ
    • ͘͞Β΍AWS౳ͷΠϯϑϥ΍Ϋϥ΢υαʔϏεͷ্
    ʹߏங͞Ε͍ͯΔ
    • ֤αϒγεςϜ͸ׂΓͱΦʔιυοΫε
    • ݴޠ͸PHP͚ͩͰͳ͘Go΍Node౳ɺෳ਺ͷݴޠΛঢ়
    گʹԠͯ͡࢖͍෼͚ͯ·͢
    

    View Slide

65. ਓࡐืूதʂ
    

    View Slide