Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Introduction to Data Science for PHP Users
Search
Sotaro Karasawa
September 14, 2013
Technology
5
14k
Introduction to Data Science for PHP Users
PHPカンファレンス2013「PHPerのためのデータサイエンス入門」 #phpcon2013
Sotaro Karasawa
September 14, 2013
Tweet
Share
More Decks by Sotaro Karasawa
See All by Sotaro Karasawa
ソフトウェアxスタートアップから見た飲食と配送の世界 / The World of Food Deliverlies and Restaurant Businesses from a Software and Startup Perspective
sotarok
1
1.1k
CTO 3度目の正直 / My 3rd CTO Career
sotarok
19
9.4k
Introduction to the Corporate Solutions Engineering at MTC2018
sotarok
1
35k
Mercari meetup for Corporate Engineering #1 / What is "Corporate Engineering"?
sotarok
2
2.3k
Markdown and WYSIWYG
sotarok
1
5.7k
20 Jan 2017 / Moving Beyond Borders - Mercari DAY
sotarok
8
15k
PHPBLT の心得 / PHPBLT #5 @ペパボ
sotarok
5
3.3k
Wiki についての今昔物語 / Crowi
sotarok
5
14k
PHPでダイエット / Dietcube #phpblt
sotarok
2
4.2k
Other Decks in Technology
See All in Technology
推しは推せるときに推せ! プロダクトにフィードバックしていこう
nakasho
0
460
地理空間データ可視化・解析・活用ソリューション Pacific Spatial Solutions (PSS)
pacificspatialsolutions
0
330
ワールドカフェI /チューターを改良する / World Café I and Improving the Tutors
ks91
PRO
0
150
Google Cloud Next '24 Recap(Cloud Run/k8s)
mokocm
0
330
いつか使うかも貯金してたらめちゃめちゃ機能が増えてた話
riyaamemiya
0
610
認知症フレンドリーテックとスタックチャン
naokiuc
0
140
AWS学習者向けにAzureの解説スライドを作成した話
handy
3
190
VSCodeの拡張機能を作っている話
ebarakazuhiro
1
790
リテール金融(キャッシュレス・ネット銀行・ネット証券)の競争環境と経済圏
8maki
0
1.5k
いいたいことちゃんという
tkengo
0
230
Azure Container Apps + Bicep 〜 こんな感じで運用しています
kaz29
3
620
生産性向上チームの紹介
cybozuinsideout
PRO
1
910
Featured
See All Featured
Optimising Largest Contentful Paint
csswizardry
12
2.4k
Web development in the modern age
philhawksworth
203
10k
Docker and Python
trallard
35
2.7k
For a Future-Friendly Web
brad_frost
172
9k
Happy Clients
brianwarren
92
6.4k
Gamification - CAS2011
davidbonilla
77
4.6k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
323
20k
What the flash - Photography Introduction
edds
64
11k
Reflections from 52 weeks, 52 projects
jeffersonlam
345
19k
Bash Introduction
62gerente
605
210k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
275
13k
Fireside Chat
paigeccino
22
2.6k
Transcript
Crocos, Inc. Sotaro Karasawa @sotarok http://facebook.com/sotarok 1)1FSͷͨΊͷ σʔλαΠΤϯεೖ QIQDPO 1)1ΧϯϑΝϨϯε
ࣗݾհ 4PUBSP,BSBTBXB!TPUBSPL ฑ૱ଠ EIBUFOBOFKQTPUBSPL גࣜձࣾΫϩίε$SPDPT*OD 1)1 (JU 5% 3FE#VMM
ύʔϑΣΫτ1)1 ٕज़ධࣾ વΈͳ͞Μ࣋ͬͯ·͢ΑͶʂʁ ˡ
σʔλαΠΤϯε
ৄ͍͜͠ͱ σʔλαΠΤϯςΟετ ཆಡຊ ٕज़ධࣾ IUUQXXXBNB[PODPKQEQ
σʔλαΠΤϯε ۀཧղ σʔλཧղ σʔλநग़ σʔλՃ ϞσϦϯά ޮՌݕূ αʔϏε࣮ Ҿ༻σʔλαΠΤϯςΟετཆಡຊ 1ୈষσʔλαΠΤϯεͷϓϩηε
σʔλαΠΤϯε ੵ͞ΕͨσʔλΛੳɾϞσϦϯάͯ͠ ϏδωεΛߦ͢ΔͨΊʹॏཁͳ ࢦඪΛಘΔ Λ܁Γฦ͢
σʔλαΠΤϯε ੵ͞ΕͨσʔλΛੳɾϞσϦϯάͯ͠ ϏδωεΛߦ͢ΔͨΊʹॏཁͳ ࢦඪΛಘΔ Λ܁Γฦ͢ Βͳ͚Ε͍͚ͳ͍͜ͱ͕ଟ͍ ࣝͷྖҬɾ෯͕͍
࠷ݶͷͱ͜Ζ͔Β खܰʹ࢝ΊΒΕΔͱ͜Ζ͔Β ࠷ॳͷาΛ;Έͩͦ͏
σʔλαΠΤϯε ۀཧղ σʔλཧղ σʔλநग़ σʔλՃ ϞσϦϯά ޮՌݕূ αʔϏε࣮ Ҿ༻σʔλαΠΤϯςΟετཆಡຊ 1ୈষσʔλαΠΤϯεͷϓϩηε
1)1FS 8FCΞϓϦέʔγϣϯʹͱͬͯ σʔλͱԿ͔
1)1FS 8FCΞϓϦέʔγϣϯʹͱͬͯ σʔλͱԿ͔ σʔλϕʔε ϩά
ࠓճϩάͷ͓
େྔͷΞϓϦέʔγϣϯϩάΛ ͍͔ʹऩू͠ ͲͷΑ͏ʹूܭ͢Δ͔
ͦΕΛ౿·͑ͯ ࠓͷΞδΣϯμ ϩάऩूͱੳͷΈ 1)1ΞϓϦέʔγϣϯͷϩάऩू ੳ
ϩάͷऩूͱੳͷΈ
Έͷਚ͖ͳ͍ ϩάͷऩूͱੳ େྔͷσʔλ Ͳ͏ूΊΔ Ͳ͜ʹஷΊΔ Ͳ͏औΓग़͢ Ͳ͏ूܭ͢Δ
Έͷਚ͖ͳ͍ ϩάͷऩूͱੳ େྔͷσʔλ Ͳ͏ूΊΔ Ͳ͜ʹஷΊΔ Ͳ͏औΓग़͢ Ͳ͏ूܭ͢Δ ωοτϫʔΫଳҬ σΟεΫ༰ྔ Ϗοάσʔλॲཧܥ
ॲཧ࣌ؒ
IUUQXXXUSFBTVSFEBUBDPN
TD Web Server Web Server fluentd S3 Hadoop Client Hive
MySQL etc... Result
TD Web Server Web Server fluentd S3 Hadoop Client Hive
MySQL etc... Result ͋ͬͪଆʹσʔλ͕ஷ·ΓɺΫΤ ϦΛ͛Δͱ͋ͬͪͰ)BEPPQ ͕ىಈͯ݁͠ՌΛฦͯ͘͠ΕΔ
ϩάੳΛਐΊΔʹ͋ͨΓ հͳɺσʔλͷऩूɾੵɾσʔλॲཧ ɹˠ5%͕ͬͯ͘ΕΔ ຊ࣭తͳۀ ɾͲͷΑ͏ͳσʔλ ɾͲͷΑ͏ʹूܭ ͷઃܭɾ࣮ʹίϛοτͰ͖Δʂ
$SPDPTʹ͓͚Δϩάͷ׆༻ wΞϓϦέʔγϣϯϩά w'BDFCPPLͷଐੑใʹجͮ͘ੳ wओཁͳΞΫγϣϯͷ࣮ߦ࣮ߦ࣌ؒ wτϥϯβΫγϣϯɾଐੑผɾܦ࿏ผ wΠϕϯτϩά wιʔγϟϧͷγΣΞ w.PEBMͷ։ดFUD wͦͷଞΖΖ
1)1ΞϓϦέʔγϣϯͷ ϩάऩू
ͲΜͳΞϓϦέʔγϣϯϩά جຊతͳϩάઃܭ
ͲΜͳϩάΛूΊͯΔʁ
8FCαʔόͷϩά
ϩάͱ͍͑ 8FCαʔόʔͷϩά 5SFBTVSF%BUBͷνϡʔτϦ Ξϧ"QBDIFͷϩά http://docs.treasure-data.com/articles/quickstart
͚ͩͲຊʹཉ͍͠ͷ
ͲΜͳϢʔβʔ͕ʁ ͲΜͳͰʁͲ͔͜Βʁ ͍ͭԿΛͨ͠ͷ͔ʁ ͲΜͳϘλϯΛΫϦοΫͨ͠ ͷ͔ʁλοϓͨ͠ͷ͔ʁ
ΞϓϦέʔγϣϯϩά
ͲΜͳϢʔβʔ͕ʁ ɹˠϢʔβʔొใ ͲΜͳͰʁͲ͔͜Βʁ ɹˠ6"(&0 ͍ͭԿΛͨ͠ͷ͔ʁ ɹˠ63*ΞΫγϣϯ
ΞϓϦέʔγϣϯϩάΛ Ͳ͏ूΊΔ͔
ͦͷલʹ ܰ͘εΩʔϚϨεϩάʹ͍ͭͯ
εΩʔϚϨεϩάͱʁ εΩʔϚͷແ͍ϩά
ϩάͷεΩʔϚ ͜Ε·Ͱ ˠྫ͑547
ΧϥϜUJNF ΧϥϜTUBUVT ΧϥϜVSJ ΧϥϜVTFS@JE IPHF εΩʔϚ
foreach (file('app.log') as $line) { $column = explode("\t", trim($line)); $time
= $column[0]; $status = $column[1]; ... } ˞࣮ࡍʹ1)1ͳΜ͔ͰͬͯΒΕͳ͍ͷͰTFEBXLͰ
߲ͷΘ͔ΓͮΒ͞ εΩʔϚมߋͷ͠͞ ੳऀͱऩूऀͷೝࣝࠩҟʹ ΑΔࣄނ
5%ͷϩά ͱ͍͏͔qVFOUE +40/ { "time":1373876885, "status":200, "uri":"/52495/facebook", "session_id":"kn6avn2fuh21r25a65mgm3rjh3", "fb_id":"7c40c5dd2e55cde37a8c40ed80e1", ...
}
ϩάͷ1045
qVFOUQIQMPHHFS use Fluent\Logger\FluentLogger; $logger = new FluentLogger("localhost","24224"); $logger->post( "debug.test", array("hello"=>"world")
); IUUQTHJUIVCDPNqVFOUqVFOUMPHHFSQIQ
جຊతͳϩάઃܭ
ΞΫηεϨίʔυͱͳΔΑ ͏ʹه͢Δ
Ϩεϙϯεʹͻ͔͚ͬΔ ϑϨʔϜϫʔΫʹ͍͍ͩͨ ϨεϙϯεΠϕϯτͷϑοΫϙΠϯτ͋ΔΑͶʁ 4ZNGPOZͳΒ PO,FSOFM3FTQPOTF
tags: - { name: kernel.event_listener, event: kernel.response } public function
onKernelResponse(FilterResponseEvent $event) { $request = $event->getRequest(); $response = $event->getResponse(); // ͳΜ͔ྻͭͬͯ͘ $data = $this->onAccess($request, $response); // log data $this->logger->post("access",$data); } ˞࣮ࡍʹͬͱෳͷ-JTUFOFS-PHHFS͕ొͰ͖ΔΑ͏ʹͯ͋͠Γ·͕͢
جຊతͳεΩʔϚΛܾΊΔ
εΩʔϚϨεͱ͍ͬͯ Ͳ͏͍͏ϩάΛѻ͍ͬͯΔͷ͔ ֤ϨίʔυͰҙຯ͕ҧͬͯҙ ຯ͕ແ͍
جຊతͳεΩʔϚΛܾΊΔ UJNF TUBUVT VSJ VB SFGFSSFS LTSVͬΆ໊͍લʹ߹Θͤͯ ͓͘ͱΘ͔Γ͍͔͢
8FCαʔόʹ͋Δϩά ͚ͩͰͳ͘ BQQ SPVUF DPOUSPMMFS QSPDFTT@UJNF EFWJDF ϑϨʔϜϫʔΫͰͷ ϧʔςΟϯά໊ͱ͔ɺ
ίϯτϩʔϥ໊ͱ͔ (uri ʹϊΠζ͕͋ͬͯ routing ໊ͰूܭͰ͖Δ)
ΞϓϦέʔγϣϯͷΓ͏Δ ଐੑΛඇਖ਼نԽͯ͠Ϩίʔυ ʹؚΊΔ
ඇਖ਼نԽ͞ΕͨϨίʔυ TFTTJPO@JE VTFS@JE HFOEFS BHF EFWJDF
ͳͥඇਖ਼نԽ͔ͷϝϦοτ +0*/ͤͣʹूܭؔʹ͔ΔͨΊ )BEPPQͰ+0*/Ͱ͖Δ͕ɺ ͜͏͓ͯ͘͠ͱఔ͕ݮΔ͔Β ͍ˍγϯϓϧ
ͪͳΈʹ VTFS@JE TFTTJPO@JE ͳͲIBTIԽ͓ͯ͘͠ͱྑ͍ ˞ສҰͷͱ͖ͷϓϥΠόγʔʹ ྀ
·ͱΊΔͱ ΞΫηεϨίʔυͱͳΔΑ͏ ʹه͢Δ جຊతͳεΩʔϚΛܾΊΔ ΞϓϦέʔγϣϯͷΓ͏Δଐ ੑΛඇਖ਼نԽͯ͠ϨίʔυʹؚΊΔ
͜͜·ͰདྷΔͱɺ͏ੳ͕Մೳ
ੳͷྫ SELECT AVG(v['process_time']) FROM access WHERE v['route'] = 'crocos_index'
ੳͷྫ SELECT v['gender'], COUNT(*) FROM access GROUP BY v['gender'] ඇਖ਼نԽ͓͍ͯ͠
ͯΑ͔ͬͨʂ
ੳͷྫ Τϥʔͷௐࠪʹ SELECT v['route'], v['status'], v['ua'] FROM access WHERE v['user_id']
= 'xxx'
˞͘ͳΔͷͰؔ࿈ͷॲཧলུͯ͠·͢ ɹຊผʹ(3061#:ͨ͠Γ8&)&3۟ͰߜͬͨΓ
εΩʔϚϨεϩάͷ׆༻ྫ τϥϯβΫγϣϯ
ͯ͞ جຊతͳεΩʔϚΛ࣋ͭ ϩά͕ͨ·Γ࢝Ί·ͨ͠
ಛผͳҙຯΛ࣋ͭ ΞΫγϣϯͷޭͳͲΛ ه͍ͨ͠
τϥϯβΫγϣϯ uri route: ϦΫΤετ͕དྷͨ͜ͱΘ͔Δ ͔͠͠ɺຊʹޭ͔ͨ͠ɺ ΞϓϦέʔγϣϯͰ͔͠Θ͔Β ͳ͍
εΩʔϚϨεͷग़൪
جຊతͳεΩʔϚ ՃͷεΩʔϚ UJNF TUBUVT VSJ VB SFGFSSFS ͳΜͪΌΒ ͔ΜͪΌΒ
ಛఆͷϨίʔυʹɺಛผ ͳҙຯΛͨͤΔ͜ͱ͕Ͱ ͖Δʂ ͔͠ଞͷϨίʔυʹӨڹ Λ͋ͨ͑Δ͜ͱͳ͘ɻ
τϥϯβΫγϣϯ key_action key_attr_*
τϥϯβΫγϣϯ key_action shop:buy:completed ΞϓϦ:ಈ࡞:ঢ়گ ※͜ͷྫʮߪೖྃʯ
τϥϯβΫγϣϯ key_attr_* τϥϯβΫγϣϯʹؔΘΔՃ తͳใΛͭͬ͜Ή εΩʔϚɺkey_action ͝ͱʹ ҟͳΔ
τϥϯβΫγϣϯྫ key_action = shop:buy:completed key_attr_item_id = xxxxx key_attr_ref = fb_share
τϥϯβΫγϣϯੳͷྫ SELECT item_id, ref, COUNT(*) FROM access WHERE key_action =
'shop:buy:completed' GROUP BY item_id, ref ˞จࣈͷ্ؔW<>ল͍ͯΔ
τϥϯβΫγϣϯੳ ׆༻ྫ: ࢪࡦผʹΞΫηεݩΛه τϥϯβΫγϣϯޭ͔Β ࠷ޮՌతͳࢪࡦΛݟ͚ͭΔ
/&9545&1
ूܭ݁Ռ͔Β ɾ౷ܭతղੳख๏ ɾϞσϦϯά Ϗδωεʹରͯ͠ΫϦςΟΧϧͳࢦඪ ͷࢉग़ͱվળϓϩηεͷཱ֬
·ͱΊ
ϩάΛूΊͨΓੳͨ͠Γ͢Δͷେม ɹ→ Fluentd Hadoop ͏ ɹ→ Treasure Data ͏
Ͳ͏͍͏ϩάΛूΊΕ͍͍ͷ͔ ɹ→ 1ΞΫηε1Ϩίʔυඇਖ਼نԽϩά ɹ→ ϩάϑΥʔϚοτࣗମͷઃܭ ɹ→ εΩʔϚϨεͷ׆༻
࠷ޙʹ 8FBSFIJSJOH ύʔϑΣΫτ1)1ஶऀਓ ݩ1)1ΧϯϑΝϨϯεҕһਓ ݩඇϞςਓ ݩυϥ່ਓ ͱಇ͚Δͷ$SPDPT͚ͩ
None