フリマアプリ「メルカリ」の急成長を支えるエンジニアリング

by Tatsuhiko Kubo

Slide 1

Slide 1 text

ϑϦϚΞϓϦ ʮϝϧΧϦʯͷ ٸ੒௕Λࢧ͑Δ ΤϯδχΞϦϯά גࣜձࣾϝϧΧϦ ΤϯδχΞ ٱอୡ඙ [email protected] PHPΧϯϑΝϨϯε 2015

Slide 2

Slide 2 text

@cubicdaiya / Tatsuhiko Kubo • Software Engineer @ Mercari, Inc. • Infrastructure Engineering • Skills • C, Go, Lua, nginx,… • OSS developer • ngx_small_light, ngx_dynamic_upstream, nginx-build, slackboard, cachectl, gaurun, …

Slide 3

Slide 3 text

ISUCON5༧બ 2ҐͰಥഁ͠·ͨ͠ ݴޠ͸PerlͰ͢ νʔϜGoBold @kazeburo @shmorimo @cubicdaiya

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

ओཁKPIσʔλ μ΢ϯϩʔυ਺ ߪೖֹۚ ग़඼਺ ਺஋ 2000ສDL(JP+US) ݄ؒ਺ेԯԁ 1೔਺ेສ඼Ҏ্

Slide 8

Slide 8 text

4ਓ 30ਓऑ ΠϯϑϥνʔϜ ૯ΤϯδχΞ਺

Slide 9

Slide 9 text

100+ 13,000+ αʔό୆਺ ඵؒϦΫΤετ਺ (ϐʔΫ࣌) JP only

Slide 10

Slide 10 text

Πϯϑϥ͸͘͞ΒͱAWSͷϋΠϒϦουߏ੒ • ͘͞Βͷઐ༻αʔό • ͘͞ΒͷΫϥ΢υ • Amazon Web Services • EC2, S3, SES౳

Slide 11

Slide 11 text

ઐ༻αʔόͱΫϥ΢υͷ࢖͍෼͚ • ઐ༻αʔό͸ίετύϑΥʔϚϯε͕ߴ͍ • 1୆͋ͨΓͷੑೳ͕ߴ͍ͷͰ୆਺ΛগͳΊʹͰ͖Δ • MySQL on ioDrive • Ϋϥ΢υ͸ॊೈੑ͕ߴ͍ • ಥൃతͳෛՙ΁ͷରԠ • ࢼݧɾ࢖͍ࣺͯ༻ʹαʔόΛ֬อ͠΍͍͢

Slide 12

Slide 12 text

PHPͰߴ଎ͳAPIαʔόΛ࣮ݱ

Slide 13

Slide 13 text

PHPͰߴ଎ͳAPIαʔόΛ࣮ݱ • ֤ॴͰΧϦΧϦʹνϡʔχϯά • ܰྔͳϑϨʔϜϫʔΫΛར༻(dietcake) • Ωϟογϡ(memcached, Redis) • ඇಉظॲཧ(Q4M + php-Parallel-Prefork) • ۚͷ஄ؙMySQL on ioDrive

Slide 14

Slide 14 text

http://dietcake.github.io/

Slide 15

Slide 15 text

php-Parallel-Prefork • prefork server framework for PHP • https://github.com/travail/php-parallel-prefork • ඇಉظॲཧʹར༻ • Q4M͔ΒδϣϒΛσΩϡʔɺϚϧνϓϩηεͰฒྻʹॲཧ

Slide 16

Slide 16 text

New RelicʹΑΔϞχλϦϯά

Slide 17

Slide 17 text

PHP5.3 -> 5.6ʢ2015೥7݄ʣ

Slide 18

Slide 18 text

nginxʹΑΔϋΠύϑΥʔϚϯε ωοτϫʔΩϯά

Slide 19

Slide 19 text

TLSλʔϛωʔγϣϯɺSPDYήʔτ΢ΣΠ ɾɾɾ ɾɾɾ "1* "QBDIFNPE@QIQ OHJOY OHJOY HTTPS or SPDY HTTP "1* "QBDIFNPE@QIQ "1* "QBDIFNPE@QIQ ɾɾɾ ɾɾɾ OHJOY

Slide 20

Slide 20 text

HTTPSऴ୺αʔόͱͯ͠ͷnginx • େྔͷಉ࣌઀ଓ(਺ສ)Λܰշʹࡹ͚Δ • APIαʔόͷϩʔυόϥϯε΋݉ͶΔ • ߴ଎ͳHTTPS௨৴ͷఏڙ • TLS Session (Cache|Tickets), OCSP Stapling • SPDY/3.1ɺHTTP/2ͷήʔτ΢ΣΠ • ϝϧΧϦ͸SPDY/3.1Λར༻

Slide 21

Slide 21 text

L7(HTTP)ϩʔυόϥϯε "1* "QBDIF 4PMS search "QBDIF 4PMS "QBDIF 4PMS OHJOY "1* "1* search

Slide 22

Slide 22 text

ݕࡧ݁ՌͷΩϟογϡ "1* OHJOY "QBDIF 4PMS search

Slide 23

Slide 23 text

ݕࡧ݁ՌͷΩϟογϡ(Ωϟογϡ͕ͳ͍৔߹) "1* OHJOY "QBDIF 4PMS 1:search 2:proxy 3:response 5:response 4:cached response

Slide 24

Slide 24 text

ݕࡧ݁ՌͷΩϟογϡ(Ωϟογϡ͕͋Δ৔߹) "1* OHJOY "QBDIF 4PMS 1:search 2:response

Slide 25

Slide 25 text

https://speakerdeck.com/cubicdaiya/nginxfalsepahuomansutiyuningu ଓ͖͸WebͰ

Slide 26

Slide 26 text

SlackͰChatOps

Slide 27

Slide 27 text

SlackͰͰ͖Δ͜ͱʢϝϧΧϦͷ৔߹ʣ • σϓϩΠ • ۈଵ؅ཧ • Ξϥʔτ௨஌ • CIͷखಈ࣮ߦ • etc…

Slide 28

Slide 28 text

σϓϩΠ

Slide 29

Slide 29 text

ۈଵ؅ཧ

Slide 30

Slide 30 text

Ξϥʔτ௨஌

Slide 31

Slide 31 text

CIͷखಈ࣮ߦ

Slide 32

Slide 32 text

ࣦഊ͢Δͱɺ

Slide 33

Slide 33 text

SlackϓϩΩγαʔόܦ༝Ͱ Ͳ͔͜ΒͰ΋ϙετ

Slide 34

Slide 34 text

͍ΖΜͳαʔό͔ΒSlackʹ௨஌͢Δࡍͷ໰୊఺ • ֤αʔό্ͷΫϥΠΞϯτ͕Incomming WebhooksͷURLΛ஌͍ͬͯΔඞཁ͕͋Δ • ΫϥΠΞϯτͷར༻ݴޠ͕όϥόϥ • ௨஌ॲཧΛॻ͘ͷ͕໘౗ • ϓϩΩγཱͯͯϓϩΩγͱ௨৴͢ΔΫϥΠΞ ϯτΛ֤αʔόʹ഑ஔ͠Α͏ʂ

Slide 35

Slide 35 text

Πϝʔδ

Slide 36

Slide 36 text

slackboard • A slack proxy server in Go • https://github.com/cubicdaiya/slackboard • ߏ੒ϓϩάϥϜ • slackboardʙSlackϓϩΩγαʔόʙ • slackboard-cliʙΫϥΠΞϯτ for slackboardʙ • slackboard-logʙΫϥΠΞϯτ like cronlogʙ

Slide 37

Slide 37 text

slackboard-cliͰSlack΁௨஌ $ echo mercari | \ slackboard-cli \ -c tech-test \ -s slackboard-server:29800 TMBDLCPBSE POST /notify-directly Slack΁ϙετ

Slide 38

Slide 38 text

slackboard-logͰSlack΁௨஌ $ ls $ slackboard-log \ -c tech-test \ -s slackboard-server:29800 -- ls hoge TMBDLCPBSE POST /notify-directly Slack΁ϙετ

Slide 39

Slide 39 text

θϩμ΢ϯλΠϜσϓϩΠ

Slide 40

Slide 40 text

"1* "QBDIFNPE@QIQ ҎલͷσϓϩΠखॱ "1* "QBDIFNPE@QIQ "1* "QBDIFNPE@QIQ ChatOps with Slack yes EFQMPZCPU rsync rsync rsync

Slide 41

Slide 41 text

Pros / Cons • Pros • ؆୯Ͱͦͦ͜͜଎͍(appαʔό͸30୆͘Β͍) • Cons • ϊϯΤϥʔͰσϓϩΠͰ͖ͳ͍ • Too many 500 errors!!! • ࣌ʑΦϖίʔυΩϟογϡ͕յΕΔʢ࠶ىಈ͕ඞཁʣ

Slide 42

Slide 42 text

ɾɾɾ ɾɾɾ "1* "QBDIFNPE@QIQ OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN HTTPS or SPDY OHJOY OHY@EZOBNJD@VQTUSFBN HTTP Mercari frontend "1* "QBDIFNPE@QIQ "1* "QBDIFNPE@QIQ ɾɾɾ ɾɾɾ

Slide 43

Slide 43 text

ɾɾɾ ɾɾɾ "1* "QBDIFNPE@QIQ OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN HTTPS or SPDY OHJOY OHY@EZOBNJD@VQTUSFBN HTTP Mercari frontend "1* "QBDIFNPE@QIQ "1* "QBDIFNPE@QIQ ɾɾɾ ɾɾɾ

Slide 44

Slide 44 text

ngx_dynamic_upstream • Dynamic Upstream for nginx • https://github.com/cubicdaiya/ngx_dynamic_upstream • nginxͷΞοϓετϦʔϜΛ HTTPϕʔεͷAPIͰಈతʹมߋͰ͖Δ • rsyncͷલޙͰΞοϓετϦʔϜΛॻ͖׵͑

Slide 45

Slide 45 text

"1* "QBDIFNPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1* "QBDIFNPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIFNPE@QIQ

Slide 46

Slide 46 text

"1* "QBDIFNPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1* "QBDIFNPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIFNPE@QIQ ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream

Slide 47

Slide 47 text

Slide 48

Slide 48 text

θϩμ΢ϯλΠϜσϓϩΠ "1* "QBDIFNPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIFNPE@QIQ ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down "1* "QBDIFNPE@QIQ

Slide 49

Slide 49 text

"1* "QBDIFNPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1* "QBDIFNPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIFNPE@QIQ rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream

Slide 50

Slide 50 text

Slide 51

Slide 51 text

Slide 52

Slide 52 text

θϩμ΢ϯλΠϜσϓϩΠ "1* "QBDIFNPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down "1* "QBDIFNPE@QIQ "1* "QBDIFNPE@QIQ

Slide 53

Slide 53 text

"1* "QBDIFNPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1* "QBDIFNPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIFNPE@QIQ rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream

Slide 54

Slide 54 text

Slide 55

Slide 55 text

"1* "QBDIFNPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1* "QBDIFNPE@QIQ ChatOps with Slack yes OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIFNPE@QIQ ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream EFQMPZCPU

Slide 56

Slide 56 text

θϩμ΢ϯλΠϜσϓϩΠ ChatOps with Slack yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down "1* "QBDIFNPE@QIQ "1* "QBDIFNPE@QIQ "1* "QBDIFNPE@QIQ

Slide 57

Slide 57 text

"1* "QBDIFNPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1* "QBDIFNPE@QIQ ChatOps with Slack yes OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIFNPE@QIQ rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream EFQMPZCPU

Slide 58

Slide 58 text

θϩμ΢ϯλΠϜσϓϩΠ ChatOps with Slack yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream up "1* "QBDIFNPE@QIQ "1* "QBDIFNPE@QIQ "1* "QBDIFNPE@QIQ

Slide 59

Slide 59 text

ϓογϡ௨஌ج൫

Slide 60

Slide 60 text

1೥લ

Slide 61

Slide 61 text

൒೥લ

Slide 62

Slide 62 text

ݱࡏ

Slide 63

Slide 63 text

Gaurun • A general push notiﬁcation server in Go • https://github.com/mercari/gaurun • APNS΍GCM΁ͷϓογϡ௨஌ϦΫΤετΛϓϩΩγ • Q4M+php-Parallel-Preforkͩͱੑೳग़ͳ͔ͬͨͷͰ ϑϧεΫϥονͰॻ͍ͨ • ଞͷδϣϒϫʔΧʔͷҰ෦΋GoͰϦϥΠτத

Slide 64

Slide 64 text

ϝϧΧϦͷ੒௕Λࢧ͑Δϩά෼ੳج൫

Slide 65

Slide 65 text

200GB / day ϩά༰ྔ

Slide 66

Slide 66 text

֤αʔόͰऩूͨ͠ϩάσʔλΛFluentdͰసૹ "QQ 8PSLFS #BUDI ,JCBOB #JH2VFSZ /PSJLSB .BDLFSFM

Slide 67

Slide 67 text

Mercari\DataTrack • Various purpose logging framework for PHP • αʔόαΠυͰى͜ΔΠϕϯτΛॊೈʹϩΪϯά • ϝʔϧɺϓογϡͷૹ৴ • A/Bςετ • Πϕϯτͷܦա • Pascal(ޙड़)ͱૄʹ࿈ܞ

Slide 68

Slide 68 text

/PSJLSB ϩάσʔλసૹ ఆظతͳΠϕϯτͷऔಘ 3ඵຖ type norikra norikra norikra-server:26571 target_string access_log type norikra norikra localhost:26571 method sweep target target_name tag query_name tag_prefix norikra.query interval 3s SQLΛొ࿥ /** * access.log͔ΒHTTPεςʔλείʔυͷׂ߹(1෼ؒ)Λूܭ */ SELECT COUNT(1, status like "5%")/COUNT(1)*100 AS rate_5xx, COUNT(1, status like "4%")/COUNT(1)*100 AS rate_4xx, COUNT(1, status like "3%")/COUNT(1)*100 AS rate_3xx, COUNT(1, status like "2%")/COUNT(1)*100 AS rate_2xx FROM access_log.win:time_batch(1 min) NorikraʹΑΔϩάετϦʔϛϯάॲཧ

Slide 69

Slide 69 text

Πϕϯτͷऔಘ ﬂuent-plugin-norikra Mackerel΁సૹ ֤छϝτϦΫεΛάϥϑԽ ྫ:APIͷฏۉϨεϙϯελΠϜ౳ ݕ஌ͨ͠Τϥʔ಺༰Λ௨஌ Slack΁సૹ /PSJLSB SQLͰநग़ͨ࣌͠ܥྻσʔλΛ͞Βʹసૹ

Slide 70

Slide 70 text

σʔλͷूܭɺՄࢹԽɺϨϙʔςΟϯά • Ϩϙʔτϝʔϧ • Web UIʹΑΔ౷ܭμογϡϘʔυ • ֎෦ͷ෼ੳπʔϧར༻

Slide 71

Slide 71 text

ϩάσʔλ෼ੳ্ͷ՝୊ • ֤छKPI΍෼ੳʹඞཁͳݩσʔλ͕෼ࢄ • ϩά͕෼ੳ༻ʹઃܭ͞Εͯͳ͍ͷͰɺ෼ੳ͢ Δͷʹ͸޲͍ͯͳͯ͘࢖͍ͮΒ͍(ίπ͕ཁΔ) • ֎෦ͷ෼ੳπʔϧͩͱखܰʹूܭɾՄࢹԽͰ ͖Δ͕ɺଞͷσʔλ΍πʔϧͱ૊Έ߹ΘͤΔ ͷ͕೉͍͠

Slide 72

Slide 72 text

ͱ͍͏Θ͚Ͱɺ • ෼ੳʹదͨ͠ϩάΛҰ͔Βઃܭɾूܭ • ͦͷ্Ͱ෼ੳπʔϧͱ૊Έ߹Θͤͯ࢖͑ΔΑ ͏ʹ͠Α͏ ৽͍͠ϩά෼ੳج൫Λߏங͢Δ͜ͱʹ

Slide 73

Slide 73 text

PascalʙMercari analysis baseʙ 0QFO3FTUZ 0QFO3FTUZ 0QFO3FTUZ (PPHMF#JH2VFSZ Developer Data Sientist Analyze by SQL send events send events send events Powered by cookpad/puree-(ios|android) utilize events transfer utilize events utilize events transfer

Slide 74

Slide 74 text

Puree • ΫοΫύου͕ࣾ։ൃɾެ։͍ͯ͠ΔεϚϗ ΞϓϦ޲͚ͷϩάίϨΫλϥΠϒϥϦ • https://github.com/cookpad/puree-android • https://github.com/cookpad/puree-ios • Android, iOSͰ΄΅ಉ͡৚݅ͰϩάσʔλΛ औಘՄೳ

Slide 75

Slide 75 text

OpenResty • nginxϕʔεͷWebΞϓϦέʔγϣϯϑϨʔϜϫʔΫ • bundled ngx_lua, Lua, LuaJIT, ͦͷଞศརͳ Ϟδϡʔϧ • ͋Δҙຯnginxͷεʔύηοτ

Slide 76

Slide 76 text

PascalʙMercari analysis baseʙ • جຊ͸BigQueryͰΞυϗοΫʹ෼ੳ • ϩάͷϑΥʔϚοτ͕෼ੳʹదͨ͠ܗʹ • Chartio౳ͷ෼ੳπʔϧͱ΋૊Έ߹Θͤͯར༻ • ΞϓϦ಺ΞΫγϣϯ΍A/Bςετ෼ੳΛ͸͡Ί ద༻ൣғΛ֦େத

Slide 77

Slide 77 text

·ͱΊ • ৭ʑ޻෉ͯ͠PHPͰߴ଎ͳAPIαʔόΛ࣮ݱ͍ͯ͠·͢ • དྷ೥͸7ରԠʁ • όοΫΤϯυͷํͰ͸GoΛ࠾༻͢Δ৔໘͕૿͑ͯ·͢ • γεςϜͷύϑΥʔϚϯε΍ࣗಈԽɺσϓϩΠɺϩά෼ ੳͳͲٸ੒௕Λࢧ͑ΔͨΊͷج൫࡞Γʹۈ͠ΜͰ͍·͢

Slide 78

Slide 78 text

We are hiring!