フリマアプリ「メルカリ」の急成長を支えるエンジニアリング

 フリマアプリ「メルカリ」の急成長を支えるエンジニアリング

5d74d743eabd2bf7d4d2f68b9d3c727d?s=128

Tatsuhiko Kubo

October 03, 2015
Tweet

Transcript

  1. ϑϦϚΞϓϦ ʮϝϧΧϦʯͷ ٸ੒௕Λࢧ͑Δ ΤϯδχΞϦϯά גࣜձࣾϝϧΧϦ ΤϯδχΞ ٱอୡ඙ bokko@mercari.com PHPΧϯϑΝϨϯε 2015

  2. @cubicdaiya / Tatsuhiko Kubo • Software Engineer @ Mercari, Inc.

    • Infrastructure Engineering • Skills • C, Go, Lua, nginx,… • OSS developer • ngx_small_light, ngx_dynamic_upstream, nginx-build, slackboard, cachectl, gaurun, …
  3. ISUCON5༧બ 2ҐͰಥഁ͠·ͨ͠ ݴޠ͸PerlͰ͢ νʔϜGoBold @kazeburo @shmorimo @cubicdaiya

  4. None
  5. None
  6. None
  7. ओཁKPIσʔλ μ΢ϯϩʔυ਺ ߪೖֹۚ ग़඼਺ ਺஋ 2000ສDL(JP+US) ݄ؒ਺ेԯԁ 1೔਺ेສ඼Ҏ্

  8. 4ਓ 30ਓऑ ΠϯϑϥνʔϜ ૯ΤϯδχΞ਺

  9. 100+ 13,000+ αʔό୆਺ ඵؒϦΫΤετ਺ (ϐʔΫ࣌) JP only

  10. Πϯϑϥ͸͘͞ΒͱAWSͷϋΠϒϦουߏ੒ • ͘͞Βͷઐ༻αʔό • ͘͞ΒͷΫϥ΢υ • Amazon Web Services •

    EC2, S3, SES౳
  11. ઐ༻αʔόͱΫϥ΢υͷ࢖͍෼͚ • ઐ༻αʔό͸ίετύϑΥʔϚϯε͕ߴ͍ • 1୆͋ͨΓͷੑೳ͕ߴ͍ͷͰ୆਺ΛগͳΊʹͰ͖Δ • MySQL on ioDrive •

    Ϋϥ΢υ͸ॊೈੑ͕ߴ͍ • ಥൃతͳෛՙ΁ͷରԠ • ࢼݧɾ࢖͍ࣺͯ༻ʹαʔόΛ֬อ͠΍͍͢
  12. PHPͰߴ଎ͳAPIαʔόΛ࣮ݱ

  13. PHPͰߴ଎ͳAPIαʔόΛ࣮ݱ • ֤ॴͰΧϦΧϦʹνϡʔχϯά • ܰྔͳϑϨʔϜϫʔΫΛར༻(dietcake) • Ωϟογϡ(memcached, Redis) • ඇಉظॲཧ(Q4M

    + php-Parallel-Prefork) • ۚͷ஄ؙMySQL on ioDrive
  14. http://dietcake.github.io/

  15. php-Parallel-Prefork • prefork server framework for PHP • https://github.com/travail/php-parallel-prefork •

    ඇಉظॲཧʹར༻ • Q4M͔ΒδϣϒΛσΩϡʔɺϚϧνϓϩηεͰฒྻʹॲཧ
  16. New RelicʹΑΔϞχλϦϯά

  17. PHP5.3 -> 5.6ʢ2015೥7݄ʣ

  18. nginxʹΑΔϋΠύϑΥʔϚϯε ωοτϫʔΩϯά

  19. TLSλʔϛωʔγϣϯɺSPDYήʔτ΢ΣΠ ɾɾɾ ɾɾɾ "1* "QBDIF NPE@QIQ OHJOY OHJOY HTTPS or

    SPDY HTTP "1* "QBDIF NPE@QIQ "1* "QBDIF NPE@QIQ ɾɾɾ ɾɾɾ OHJOY
  20. HTTPSऴ୺αʔόͱͯ͠ͷnginx • େྔͷಉ࣌઀ଓ(਺ສ)Λܰշʹࡹ͚Δ • APIαʔόͷϩʔυόϥϯε΋݉ͶΔ • ߴ଎ͳHTTPS௨৴ͷఏڙ • TLS Session

    (Cache|Tickets), OCSP Stapling • SPDY/3.1ɺHTTP/2ͷήʔτ΢ΣΠ • ϝϧΧϦ͸SPDY/3.1Λར༻
  21. L7(HTTP)ϩʔυόϥϯε "1* "QBDIF 4PMS search "QBDIF 4PMS "QBDIF 4PMS OHJOY

    "1* "1* search
  22. ݕࡧ݁ՌͷΩϟογϡ "1* OHJOY "QBDIF 4PMS search

  23. ݕࡧ݁ՌͷΩϟογϡ(Ωϟογϡ͕ͳ͍৔߹) "1* OHJOY "QBDIF 4PMS 1:search 2:proxy 3:response 5:response 4:cached

    response
  24. ݕࡧ݁ՌͷΩϟογϡ(Ωϟογϡ͕͋Δ৔߹) "1* OHJOY "QBDIF 4PMS 1:search 2:response

  25. https://speakerdeck.com/cubicdaiya/nginxfalsepahuomansutiyuningu ଓ͖͸WebͰ

  26. SlackͰChatOps

  27. SlackͰͰ͖Δ͜ͱʢϝϧΧϦͷ৔߹ʣ • σϓϩΠ • ۈଵ؅ཧ • Ξϥʔτ௨஌ • CIͷखಈ࣮ߦ •

    etc…
  28. σϓϩΠ

  29. ۈଵ؅ཧ

  30. Ξϥʔτ௨஌

  31. CIͷखಈ࣮ߦ

  32. ࣦഊ͢Δͱɺ

  33. SlackϓϩΩγαʔόܦ༝Ͱ Ͳ͔͜ΒͰ΋ϙετ

  34. ͍ΖΜͳαʔό͔ΒSlackʹ௨஌͢Δࡍͷ໰୊఺ • ֤αʔό্ͷΫϥΠΞϯτ͕Incomming WebhooksͷURLΛ஌͍ͬͯΔඞཁ͕͋Δ • ΫϥΠΞϯτͷར༻ݴޠ͕όϥόϥ • ௨஌ॲཧΛॻ͘ͷ͕໘౗ • ϓϩΩγཱͯͯϓϩΩγͱ௨৴͢ΔΫϥΠΞ

    ϯτΛ֤αʔόʹ഑ஔ͠Α͏ʂ
  35. Πϝʔδ

  36. slackboard • A slack proxy server in Go • https://github.com/cubicdaiya/slackboard

    • ߏ੒ϓϩάϥϜ • slackboardʙSlackϓϩΩγαʔόʙ • slackboard-cliʙΫϥΠΞϯτ for slackboardʙ • slackboard-logʙΫϥΠΞϯτ like cronlogʙ
  37. slackboard-cliͰSlack΁௨஌ $ echo mercari | \ slackboard-cli \ -c tech-test

    \ -s slackboard-server:29800 TMBDLCPBSE POST /notify-directly Slack΁ϙετ
  38. slackboard-logͰSlack΁௨஌ $ ls $ slackboard-log \ -c tech-test \ -s

    slackboard-server:29800 -- ls hoge TMBDLCPBSE POST /notify-directly Slack΁ϙετ
  39. θϩμ΢ϯλΠϜσϓϩΠ

  40. "1* "QBDIF NPE@QIQ ҎલͷσϓϩΠखॱ "1* "QBDIF NPE@QIQ "1* "QBDIF NPE@QIQ

    ChatOps with Slack yes EFQMPZCPU rsync rsync rsync
  41. Pros / Cons • Pros • ؆୯Ͱͦͦ͜͜଎͍(appαʔό͸30୆͘Β͍) • Cons •

    ϊϯΤϥʔͰσϓϩΠͰ͖ͳ͍ • Too many 500 errors!!! • ࣌ʑΦϖίʔυΩϟογϡ͕յΕΔʢ࠶ىಈ͕ඞཁʣ
  42. ɾɾɾ ɾɾɾ "1* "QBDIF NPE@QIQ OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY 

    OHY@EZOBNJD@VQTUSFBN HTTPS or SPDY OHJOY  OHY@EZOBNJD@VQTUSFBN HTTP Mercari frontend "1* "QBDIF NPE@QIQ "1* "QBDIF NPE@QIQ ɾɾɾ ɾɾɾ
  43. ɾɾɾ ɾɾɾ "1* "QBDIF NPE@QIQ OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY 

    OHY@EZOBNJD@VQTUSFBN HTTPS or SPDY OHJOY  OHY@EZOBNJD@VQTUSFBN HTTP Mercari frontend "1* "QBDIF NPE@QIQ "1* "QBDIF NPE@QIQ ɾɾɾ ɾɾɾ
  44. ngx_dynamic_upstream • Dynamic Upstream for nginx • https://github.com/cubicdaiya/ngx_dynamic_upstream • nginxͷΞοϓετϦʔϜΛ

    HTTPϕʔεͷAPIͰಈతʹมߋͰ͖Δ • rsyncͷલޙͰΞοϓετϦʔϜΛॻ͖׵͑
  45. "1* "QBDIF NPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack

    yes EFQMPZCPU OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN "1* "QBDIF NPE@QIQ
  46. "1* "QBDIF NPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack

    yes EFQMPZCPU OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN "1* "QBDIF NPE@QIQ ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream
  47. "1* "QBDIF NPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack

    yes EFQMPZCPU OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN "1* "QBDIF NPE@QIQ ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream
  48. θϩμ΢ϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY

     OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN "1* "QBDIF NPE@QIQ ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down "1* "QBDIF NPE@QIQ
  49. "1* "QBDIF NPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack

    yes EFQMPZCPU OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN "1* "QBDIF NPE@QIQ rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream
  50. θϩμ΢ϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY

     OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN "1* "QBDIF NPE@QIQ ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream up "1* "QBDIF NPE@QIQ
  51. "1* "QBDIF NPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack

    yes EFQMPZCPU OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN "1* "QBDIF NPE@QIQ ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream
  52. θϩμ΢ϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY

     OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down "1* "QBDIF NPE@QIQ "1* "QBDIF NPE@QIQ
  53. "1* "QBDIF NPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack

    yes EFQMPZCPU OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN "1* "QBDIF NPE@QIQ rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream
  54. θϩμ΢ϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY

     OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream up "1* "QBDIF NPE@QIQ "1* "QBDIF NPE@QIQ
  55. "1* "QBDIF NPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack

    yes OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN "1* "QBDIF NPE@QIQ ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream EFQMPZCPU
  56. θϩμ΢ϯλΠϜσϓϩΠ ChatOps with Slack yes EFQMPZCPU OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY

     OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down "1* "QBDIF NPE@QIQ "1* "QBDIF NPE@QIQ "1* "QBDIF NPE@QIQ
  57. "1* "QBDIF NPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack

    yes OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN "1* "QBDIF NPE@QIQ rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream EFQMPZCPU
  58. θϩμ΢ϯλΠϜσϓϩΠ ChatOps with Slack yes EFQMPZCPU OHJOY  OHY@EZOBNJD@VQTUSFBN OHJOY

     OHY@EZOBNJD@VQTUSFBN OHJOY  OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream up "1* "QBDIF NPE@QIQ "1* "QBDIF NPE@QIQ "1* "QBDIF NPE@QIQ
  59. ϓογϡ௨஌ج൫

  60. 1೥લ

  61. ൒೥લ

  62. ݱࡏ

  63. Gaurun • A general push notification server in Go •

    https://github.com/mercari/gaurun • APNS΍GCM΁ͷϓογϡ௨஌ϦΫΤετΛϓϩΩγ • Q4M+php-Parallel-Preforkͩͱੑೳग़ͳ͔ͬͨͷͰ ϑϧεΫϥονͰॻ͍ͨ • ଞͷδϣϒϫʔΧʔͷҰ෦΋GoͰϦϥΠτத
  64. ϝϧΧϦͷ੒௕Λࢧ͑Δϩά෼ੳج൫

  65. 200GB / day ϩά༰ྔ

  66. ֤αʔόͰऩूͨ͠ϩάσʔλΛFluentdͰసૹ "QQ 8PSLFS #BUDI ,JCBOB #JH2VFSZ /PSJLSB .BDLFSFM

  67. Mercari\DataTrack • Various purpose logging framework for PHP • αʔόαΠυͰى͜ΔΠϕϯτΛॊೈʹϩΪϯά

    • ϝʔϧɺϓογϡͷૹ৴ • A/Bςετ • Πϕϯτͷܦա • Pascal(ޙड़)ͱૄʹ࿈ܞ
  68. /PSJLSB ϩάσʔλసૹ ఆظతͳΠϕϯτͷऔಘ 3ඵຖ <match access_log> type norikra norikra norikra-server:26571

    target_string access_log </match> <source> type norikra norikra localhost:26571 <fetch> method sweep target target_name tag query_name tag_prefix norikra.query interval 3s </fetch> </source> SQLΛొ࿥ /** * access.log͔ΒHTTPεςʔλείʔυͷׂ߹(1෼ؒ)Λूܭ */ SELECT COUNT(1, status like "5%")/COUNT(1)*100 AS rate_5xx, COUNT(1, status like "4%")/COUNT(1)*100 AS rate_4xx, COUNT(1, status like "3%")/COUNT(1)*100 AS rate_3xx, COUNT(1, status like "2%")/COUNT(1)*100 AS rate_2xx FROM access_log.win:time_batch(1 min) NorikraʹΑΔϩάετϦʔϛϯάॲཧ
  69. Πϕϯτͷऔಘ fluent-plugin-norikra Mackerel΁సૹ ֤छϝτϦΫεΛάϥϑԽ ྫ:APIͷฏۉϨεϙϯελΠϜ౳ ݕ஌ͨ͠Τϥʔ಺༰Λ௨஌ Slack΁సૹ /PSJLSB SQLͰநग़ͨ࣌͠ܥྻσʔλΛ͞Βʹసૹ

  70. σʔλͷूܭɺՄࢹԽɺϨϙʔςΟϯά • Ϩϙʔτϝʔϧ • Web UIʹΑΔ౷ܭμογϡϘʔυ • ֎෦ͷ෼ੳπʔϧར༻

  71. ϩάσʔλ෼ੳ্ͷ՝୊ • ֤छKPI΍෼ੳʹඞཁͳݩσʔλ͕෼ࢄ • ϩά͕෼ੳ༻ʹઃܭ͞Εͯͳ͍ͷͰɺ෼ੳ͢ Δͷʹ͸޲͍ͯͳͯ͘࢖͍ͮΒ͍(ίπ͕ཁΔ) • ֎෦ͷ෼ੳπʔϧͩͱखܰʹूܭɾՄࢹԽͰ ͖Δ͕ɺଞͷσʔλ΍πʔϧͱ૊Έ߹ΘͤΔ ͷ͕೉͍͠

  72. ͱ͍͏Θ͚Ͱɺ • ෼ੳʹదͨ͠ϩάΛҰ͔Βઃܭɾूܭ • ͦͷ্Ͱ෼ੳπʔϧͱ૊Έ߹Θͤͯ࢖͑ΔΑ ͏ʹ͠Α͏ ৽͍͠ϩά෼ੳج൫Λߏங͢Δ͜ͱʹ

  73. PascalʙMercari analysis baseʙ 0QFO3FTUZ 0QFO3FTUZ 0QFO3FTUZ (PPHMF#JH2VFSZ Developer Data Sientist

    Analyze by SQL send events send events send events Powered by cookpad/puree-(ios|android) utilize events transfer utilize events utilize events transfer
  74. Puree • ΫοΫύου͕ࣾ։ൃɾެ։͍ͯ͠ΔεϚϗ ΞϓϦ޲͚ͷϩάίϨΫλϥΠϒϥϦ • https://github.com/cookpad/puree-android • https://github.com/cookpad/puree-ios • Android,

    iOSͰ΄΅ಉ͡৚݅ͰϩάσʔλΛ औಘՄೳ
  75. OpenResty • nginxϕʔεͷWebΞϓϦέʔγϣϯϑϨʔϜϫʔΫ • bundled ngx_lua, Lua, LuaJIT, ͦͷଞศརͳ Ϟδϡʔϧ

    • ͋Δҙຯnginxͷεʔύηοτ
  76. PascalʙMercari analysis baseʙ • جຊ͸BigQueryͰΞυϗοΫʹ෼ੳ • ϩάͷϑΥʔϚοτ͕෼ੳʹదͨ͠ܗʹ • Chartio౳ͷ෼ੳπʔϧͱ΋૊Έ߹Θͤͯར༻ •

    ΞϓϦ಺ΞΫγϣϯ΍A/Bςετ෼ੳΛ͸͡Ί ద༻ൣғΛ֦େத
  77. ·ͱΊ • ৭ʑ޻෉ͯ͠PHPͰߴ଎ͳAPIαʔόΛ࣮ݱ͍ͯ͠·͢ • དྷ೥͸7ରԠʁ • όοΫΤϯυͷํͰ͸GoΛ࠾༻͢Δ৔໘͕૿͑ͯ·͢ • γεςϜͷύϑΥʔϚϯε΍ࣗಈԽɺσϓϩΠɺϩά෼ ੳͳͲٸ੒௕Λࢧ͑ΔͨΊͷج൫࡞Γʹۈ͠ΜͰ͍·͢

  78. We are hiring!