Introduction to Data Science for PHP Users

Slide 1

Slide 1 text

Crocos, Inc. Sotaro Karasawa @sotarok http://facebook.com/sotarok 1)1FSͷͨΊͷ σʔλαΠΤϯεೖ໳ QIQDPO 1)1ΧϯϑΝϨϯε

Slide 2

Slide 2 text

ࣗݾ঺հ 4PUBSP,BSBTBXB!TPUBSPL ฑ୔૱ଠ࿠ EIBUFOBOFKQTPUBSPL גࣜձࣾΫϩίε$SPDPT*OD 1)1 (JU 5% 3FE#VMM

Slide 3

Slide 3 text

ύʔϑΣΫτ1)1 ٕज़ධ࿦ࣾ ౰વΈͳ͞Μ࣋ͬͯ·͢ΑͶʂʁ ˡ

Slide 4

Slide 4 text

σʔλαΠΤϯε

Slide 5

Slide 5 text

ৄ͍͜͠ͱ͸ σʔλαΠΤϯςΟετ ཆ੒ಡຊ ٕज़ධ࿦ࣾ IUUQXXXBNB[PODPKQEQ

Slide 6

Slide 6 text

σʔλαΠΤϯε ۀ຿ཧղ σʔλཧղ σʔλநग़ σʔλՃ޻ ϞσϦϯά ޮՌݕূ αʔϏε࣮૷ Ҿ༻σʔλαΠΤϯςΟετཆ੒ಡຊ 1ୈষσʔλαΠΤϯεͷϓϩηε

Slide 7

Slide 7 text

σʔλαΠΤϯε ஝ੵ͞ΕͨσʔλΛ෼ੳɾϞσϦϯάͯ͠ ϏδωεΛ਱ߦ͢ΔͨΊʹॏཁͳ ࢦඪΛಘΔ Λ܁Γฦ͢

Slide 8

Slide 8 text

σʔλαΠΤϯε ஝ੵ͞ΕͨσʔλΛ෼ੳɾϞσϦϯάͯ͠ ϏδωεΛ਱ߦ͢ΔͨΊʹॏཁͳ ࢦඪΛಘΔ Λ܁Γฦ͢ ΍Βͳ͚Ε͹͍͚ͳ͍͜ͱ͕ଟ͍ ஌ࣝͷྖҬɾ෯͕޿͍

Slide 9

Slide 9 text

࠷௿ݶͷͱ͜Ζ͔Β खܰʹ࢝ΊΒΕΔͱ͜Ζ͔Β ࠷ॳͷาΛ;Έͩͦ͏

Slide 10

Slide 10 text

σʔλαΠΤϯε ۀ຿ཧղ σʔλཧղ σʔλநग़ σʔλՃ޻ ϞσϦϯά ޮՌݕূ αʔϏε࣮૷ Ҿ༻σʔλαΠΤϯςΟετཆ੒ಡຊ 1ୈষσʔλαΠΤϯεͷϓϩηε

Slide 11

Slide 11 text

1)1FS 8FCΞϓϦέʔγϣϯʹͱͬͯ σʔλͱ͸Կ͔

Slide 12

Slide 12 text

1)1FS 8FCΞϓϦέʔγϣϯʹͱͬͯ σʔλͱ͸Կ͔ σʔλϕʔε ϩά

Slide 13

Slide 13 text

ࠓճ͸ϩάͷ͓࿩

Slide 14

Slide 14 text

େྔͷΞϓϦέʔγϣϯϩάΛ ͍͔ʹऩू͠ ͲͷΑ͏ʹूܭ͢Δ͔

Slide 15

Slide 15 text

ͦΕΛ౿·͑ͯ ࠓ೔ͷΞδΣϯμ ϩάऩूͱ෼ੳͷ೰Έ 1)1ΞϓϦέʔγϣϯͷϩάऩू ෼ੳ

Slide 16

Slide 16 text

ϩάͷऩूͱ෼ੳͷ೰Έ

Slide 17

Slide 17 text

೰Έͷਚ͖ͳ͍ ϩάͷऩूͱ෼ੳ େྔͷσʔλ Ͳ͏ूΊΔ Ͳ͜ʹஷΊΔ Ͳ͏औΓग़͢ Ͳ͏ूܭ͢Δ

Slide 18

Slide 18 text

೰Έͷਚ͖ͳ͍ ϩάͷऩूͱ෼ੳ େྔͷσʔλ Ͳ͏ूΊΔ Ͳ͜ʹஷΊΔ Ͳ͏औΓग़͢ Ͳ͏ूܭ͢Δ ωοτϫʔΫଳҬ σΟεΫ༰ྔ Ϗοάσʔλॲཧܥ ॲཧ࣌ؒ

Slide 19

Slide 19 text

IUUQXXXUSFBTVSFEBUBDPN

Slide 20

Slide 20 text

TD Web Server Web Server ﬂuentd S3 Hadoop Client Hive MySQL etc... Result

Slide 21

Slide 21 text

TD Web Server Web Server ﬂuentd S3 Hadoop Client Hive MySQL etc... Result ͋ͬͪଆʹσʔλ͕ஷ·ΓɺΫΤ ϦΛ౤͛Δͱ͋ͬͪͰ)BEPPQ ͕ىಈͯ݁͠ՌΛฦͯ͘͠ΕΔ

Slide 22

Slide 22 text

ϩά෼ੳΛਐΊΔʹ͋ͨΓ ໽հͳɺσʔλͷऩूɾ஝ੵɾσʔλॲཧ ɹˠ5%͕΍ͬͯ͘ΕΔ ຊ࣭తͳۀ຿ ɾͲͷΑ͏ͳσʔλ ɾͲͷΑ͏ʹूܭ ͷઃܭɾ࣮૷ʹίϛοτͰ͖Δʂ

Slide 23

Slide 23 text

$SPDPTʹ͓͚Δϩάͷ׆༻ wΞϓϦέʔγϣϯϩά w'BDFCPPLͷଐੑ৘ใʹجͮ͘෼ੳ wओཁͳΞΫγϣϯͷ࣮ߦ਺΍࣮ߦ࣌ؒ wτϥϯβΫγϣϯ਺ɾଐੑผɾܦ࿏ผ wΠϕϯτϩά wιʔγϟϧ΁ͷγΣΞ w.PEBMͷ։ดFUD wͦͷଞ΋Ζ΋Ζ

Slide 24

Slide 24 text

1)1ΞϓϦέʔγϣϯͷ ϩάऩू

Slide 25

Slide 25 text

ͲΜͳΞϓϦέʔγϣϯϩά جຊతͳϩάઃܭ

Slide 26

Slide 26 text

ͲΜͳϩάΛूΊͯΔʁ

Slide 27

Slide 27 text

8FCαʔόͷϩά

Slide 28

Slide 28 text

ϩάͱ͍͑͹ 8FCαʔόʔͷϩά 5SFBTVSF%BUBͷνϡʔτϦ Ξϧ΋"QBDIFͷϩά http://docs.treasure-data.com/articles/quickstart

Slide 29

Slide 29 text

͚ͩͲຊ౰ʹཉ͍͠ͷ͸

Slide 30

Slide 30 text

ͲΜͳϢʔβʔ͕ʁ ͲΜͳ୺຤ͰʁͲ͔͜Βʁ ͍ͭԿΛͨ͠ͷ͔ʁ ͲΜͳϘλϯΛΫϦοΫͨ͠ ͷ͔ʁλοϓͨ͠ͷ͔ʁ

Slide 31

Slide 31 text

ΞϓϦέʔγϣϯϩά

Slide 32

Slide 32 text

ͲΜͳϢʔβʔ͕ʁ ɹˠϢʔβʔొ࿥৘ใ ͲΜͳ୺຤ͰʁͲ͔͜Βʁ ɹˠ6"(&0 ͍ͭԿΛͨ͠ͷ͔ʁ ɹˠ63*ΞΫγϣϯ

Slide 33

Slide 33 text

ΞϓϦέʔγϣϯϩάΛ Ͳ͏ूΊΔ͔

Slide 34

Slide 34 text

ͦͷલʹ ܰ͘εΩʔϚϨεϩάʹ͍ͭͯ

Slide 35

Slide 35 text

εΩʔϚϨεϩάͱ͸ʁ εΩʔϚͷແ͍ϩά

Slide 36

Slide 36 text

ϩάͷεΩʔϚ ͜Ε·Ͱ ˠྫ͑͹547

Slide 37

Slide 37 text

ΧϥϜ໨UJNF ΧϥϜ໨TUBUVT ΧϥϜ໨VSJ ΧϥϜ໨VTFS@JE IPHF εΩʔϚ

Slide 38

Slide 38 text

foreach (file('app.log') as $line) { $column = explode("\t", trim($line)); $time = $column[0]; $status = $column[1]; ... } ˞࣮ࡍʹ͸1)1ͳΜ͔Ͱ΍ͬͯΒΕͳ͍ͷͰTFE΍BXLͰ

Slide 39

Slide 39 text

߲໨ͷΘ͔ΓͮΒ͞ εΩʔϚมߋͷ೉͠͞ ෼ੳऀͱऩूऀͷೝࣝࠩҟʹ ΑΔࣄނ

Slide 40

Slide 40 text

5%ͷϩά ͱ͍͏͔qVFOUE +40/ { "time":1373876885, "status":200, "uri":"/52495/facebook", "session_id":"kn6avn2fuh21r25a65mgm3rjh3", "fb_id":"7c40c5dd2e55cde37a8c40ed80e1", ... }

Slide 41

Slide 41 text

ϩάͷ1045

Slide 42

Slide 42 text

qVFOUQIQMPHHFS use Fluent\Logger\FluentLogger; $logger = new FluentLogger("localhost","24224"); $logger->post( "debug.test", array("hello"=>"world") ); IUUQTHJUIVCDPNqVFOUqVFOUMPHHFSQIQ

Slide 43

Slide 43 text

جຊతͳϩάઃܭ

Slide 44

Slide 44 text

ΞΫηεϨίʔυͱͳΔΑ ͏ʹه࿥͢Δ

Slide 45

Slide 45 text

Ϩεϙϯεʹͻ͔͚ͬΔ ϑϨʔϜϫʔΫʹ͍͍ͩͨ ϨεϙϯεΠϕϯτ΁ͷϑοΫϙΠϯτ͋ΔΑͶʁ 4ZNGPOZͳΒ PO,FSOFM3FTQPOTF

Slide 46

Slide 46 text

tags: - { name: kernel.event_listener, event: kernel.response } public function onKernelResponse(FilterResponseEvent $event) { $request = $event->getRequest(); $response = $event->getResponse(); // ͳΜ͔഑ྻͭͬͯ͘ $data = $this->onAccess($request, $response); // log data $this->logger->post("access",$data); } ˞࣮ࡍʹ͸΋ͬͱෳ਺ͷ-JTUFOFS΍-PHHFS͕ొ࿥Ͱ͖ΔΑ͏ʹͯ͋͠Γ·͕͢

Slide 47

Slide 47 text

جຊతͳεΩʔϚΛܾΊΔ

Slide 48

Slide 48 text

εΩʔϚϨεͱ͍ͬͯ΋ Ͳ͏͍͏ϩάΛѻ͍ͬͯΔͷ͔ ֤ϨίʔυͰҙຯ͕ҧͬͯ͸ҙ ຯ͕ແ͍

Slide 49

Slide 49 text

جຊతͳεΩʔϚΛܾΊΔ UJNF TUBUVT VSJ VB SFGFSSFS LTSVͬΆ໊͍લʹ߹Θͤͯ ͓͘ͱΘ͔Γ΍͍͔͢΋

Slide 50

Slide 50 text

8FCαʔόʹ͋Δϩά ͚ͩͰͳ͘ BQQ SPVUF DPOUSPMMFS QSPDFTT@UJNF EFWJDF ϑϨʔϜϫʔΫ಺Ͱͷ ϧʔςΟϯά໊ͱ͔ɺ ίϯτϩʔϥ໊ͱ͔ (uri ʹϊΠζ͕͋ͬͯ΋ routing ໊ͰूܭͰ͖Δ)

Slide 51

Slide 51 text

ΞϓϦέʔγϣϯͷ஌Γ͏Δ ଐੑΛඇਖ਼نԽͯ͠Ϩίʔυ ʹؚΊΔ

Slide 52

Slide 52 text

ඇਖ਼نԽ͞ΕͨϨίʔυ TFTTJPO@JE VTFS@JE HFOEFS BHF EFWJDF

Slide 53

Slide 53 text

ͳͥඇਖ਼نԽ͔ͷϝϦοτ +0*/ͤͣʹूܭؔ਺ʹ͔ΔͨΊ )BEPPQͰ΋+0*/͸Ͱ͖Δ͕ɺ ͜͏͓ͯ͘͠ͱ޻ఔ͕ݮΔ͔Β ଎͍ˍγϯϓϧ

Slide 54

Slide 54 text

ͪͳΈʹ VTFS@JE TFTTJPO@JE ͳͲ͸IBTIԽ͓ͯ͘͠ͱྑ͍ ˞ສҰͷͱ͖ͷϓϥΠόγʔʹ ഑ྀ

Slide 55

Slide 55 text

·ͱΊΔͱ ΞΫηεϨίʔυͱͳΔΑ͏ ʹه࿥͢Δ جຊతͳεΩʔϚΛܾΊΔ ΞϓϦέʔγϣϯͷ஌Γ͏Δଐ ੑΛඇਖ਼نԽͯ͠ϨίʔυʹؚΊΔ

Slide 56

Slide 56 text

͜͜·ͰདྷΔͱɺ΋͏෼ੳ͕Մೳ

Slide 57

Slide 57 text

෼ੳͷྫ SELECT AVG(v['process_time']) FROM access WHERE v['route'] = 'crocos_index'

Slide 58

Slide 58 text

෼ੳͷྫ SELECT v['gender'], COUNT(*) FROM access GROUP BY v['gender'] ඇਖ਼نԽ͓͍ͯ͠ ͯΑ͔ͬͨʂ

Slide 59

Slide 59 text

෼ੳͷྫ Τϥʔͷௐࠪʹ΋ SELECT v['route'], v['status'], v['ua'] FROM access WHERE v['user_id'] = 'xxx'

Slide 60

Slide 60 text

˞௕͘ͳΔͷͰ೔෇ؔ࿈ͷॲཧ͸লུͯ͠·͢ ɹຊ౰͸೔ผʹ(3061#:ͨ͠Γ8&)&3۟ͰߜͬͨΓ

Slide 61

Slide 61 text

εΩʔϚϨεϩάͷ׆༻ྫ τϥϯβΫγϣϯ

Slide 62

Slide 62 text

ͯ͞ جຊతͳεΩʔϚΛ࣋ͭ ϩά͕ͨ·Γ࢝Ί·ͨ͠

Slide 63

Slide 63 text

ಛผͳҙຯΛ࣋ͭ ΞΫγϣϯͷ੒ޭͳͲΛ ه࿥͍ͨ͠

Slide 64

Slide 64 text

τϥϯβΫγϣϯ uri ΍ route: ϦΫΤετ͕དྷͨ͜ͱ͸Θ͔Δ ͔͠͠ɺຊ౰ʹ੒ޭ͔ͨ͠͸ɺ ΞϓϦέʔγϣϯͰ͔͠Θ͔Β ͳ͍

Slide 65

Slide 65 text

εΩʔϚϨεͷग़൪

Slide 66

Slide 66 text

جຊతͳεΩʔϚ ௥ՃͷεΩʔϚ UJNF TUBUVT VSJ VB SFGFSSFS ͳΜͪΌΒ ͔ΜͪΌΒ ಛఆͷϨίʔυʹɺಛผ ͳҙຯΛ΋ͨͤΔ͜ͱ͕Ͱ ͖Δʂ ͔͠΋ଞͷϨίʔυʹӨڹ Λ͋ͨ͑Δ͜ͱͳ͘ɻ

Slide 67

Slide 67 text

τϥϯβΫγϣϯ key_action key_attr_*

Slide 68

Slide 68 text

τϥϯβΫγϣϯ key_action shop:buy:completed ΞϓϦ:ಈ࡞:ঢ়گ ※͜ͷྫ͸ʮߪೖ׬ྃʯ

Slide 69

Slide 69 text

τϥϯβΫγϣϯ key_attr_* τϥϯβΫγϣϯʹؔΘΔ෇Ճ తͳ৘ใΛͭͬ͜Ή εΩʔϚ͸ɺkey_action ͝ͱʹ ҟͳΔ

Slide 70

Slide 70 text

τϥϯβΫγϣϯྫ key_action = shop:buy:completed key_attr_item_id = xxxxx key_attr_ref = fb_share

Slide 71

Slide 71 text

τϥϯβΫγϣϯ෼ੳͷྫ SELECT item_id, ref, COUNT(*) FROM access WHERE key_action = 'shop:buy:completed' GROUP BY item_id, ref ˞จࣈ਺ͷؔ܎্W<>ল͍ͯΔ

Slide 72

Slide 72 text

τϥϯβΫγϣϯ෼ੳ ׆༻ྫ: ࢪࡦผʹΞΫηεݩΛه࿥ τϥϯβΫγϣϯ੒ޭ਺͔Β ࠷΋ޮՌతͳࢪࡦΛݟ͚ͭΔ

Slide 73

Slide 73 text

/&9545&1

Slide 74

Slide 74 text

ूܭ݁Ռ͔Β ɾ౷ܭతղੳख๏ ɾϞσϦϯά Ϗδωεʹରͯ͠ΫϦςΟΧϧͳࢦඪ ͷࢉग़ͱվળϓϩηεͷཱ֬

Slide 75

Slide 75 text

·ͱΊ

Slide 76

Slide 76 text

ϩάΛूΊͨΓ෼ੳͨ͠Γ͢Δͷ͸େม ɹ→ Fluentd ΍ Hadoop ࢖͏ ɹ→ Treasure Data ࢖͏ Ͳ͏͍͏ϩάΛूΊΕ͹͍͍ͷ͔ ɹ→ 1ΞΫηε1Ϩίʔυඇਖ਼نԽϩά ɹ→ ϩάϑΥʔϚοτࣗମͷઃܭ ɹ→ εΩʔϚϨεͷ׆༻

Slide 77

Slide 77 text

࠷ޙʹ 8FBSFIJSJOH ύʔϑΣΫτ1)1ஶऀਓ ݩ1)1ΧϯϑΝϨϯεҕһ௕ਓ ݩඇϞςਓ ݩυϥ່ਓ ͱಇ͚Δͷ͸$SPDPT͚ͩ

Slide 78

Slide 78 text

No content