Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
GCPではじめるスモールスタートなデータ活用
Takashi Nishibayashi
September 06, 2016
Technology
2
2.2k
GCPではじめるスモールスタートなデータ活用
2016-09-06
bq_sushi #4での発表資料です
Takashi Nishibayashi
September 06, 2016
Tweet
Share
More Decks by Takashi Nishibayashi
See All by Takashi Nishibayashi
hagino3000
0
350
hagino3000
18
14k
hagino3000
8
4k
hagino3000
2
1.6k
hagino3000
3
3k
hagino3000
15
13k
Other Decks in Technology
See All in Technology
kentaro
1
360
nihonbuson
2
1.5k
masakick
0
110
hirosys
0
120
unifa_dev
0
310
kawaguti
0
110
hacker2202
0
750
kanaugust
PRO
0
210
hsano
0
120
satoryu
0
2k
takem001
0
870
yosuke_matsuura
PRO
0
3.2k
Featured
See All Featured
mojombo
358
62k
thoeni
4
550
shlominoach
176
7.4k
bryan
30
3.3k
shpigford
368
42k
reverentgeek
168
7.1k
danielanewman
200
19k
bkeepers
408
57k
geeforr
332
29k
paulrobertlloyd
71
1.4k
cassininazir
347
20k
addyosmani
1348
190k
Transcript
1 GCPͰ͡ΊΔ εϞʔϧελʔτͳσʔλ׆༻ #bq_sushi ver. bq_sushi #4 2016-09-06 Takashi Nishibayashi
2 Takashi Nishibayashi Software Engineer Zucks AdNetwork, Zucks Inc. Data
analysis team ݱࡏ৴ޮͷ࠷దԽ ೖࡳՁ֨ࣗಈௐϩδοΫɺ৴αʔ όʔͷࠂબϩδοΫͷ։ൃʹैࣄ @hagino3000
3 3 ͜ΕԿ͔ ಉͷGCP NEXT TOKYOͷࣄྫհηογϣ ϯͰൃදͨ͠༰ͷॖখ൛Ͱ͢
4 4 Zucks AdNetwork ͷσʔλ׆༻ͷมભ
5 5 5 ϓϩδΣΫτ։࢝࣌ͷཧͱݱ࣮
6 6 6 ࢦ͢ॴ(Ծ) ࠂ৴αʔόʔͰΠϯϓϨογϣϯຖʹػցֶशϞσϧʹΑΔίϯ όʔδϣϯ༧ଌɺΫϦοΫ༧ଌΛߦͳ͍৴ޮΛΞοϓ ݱ࣮ େྔͷϩάϑΝΠϧ͕༷ʑͳϑΥʔϚοτͰAWS S3ʹஔ͔Ε͍ͯΔ ϚελσʔλMySQLʹ֨ೲ͞Ε͍ͯΔ
Elastic SearchʹೖͬͯΔͷۙ2िؒ
7 7 7
8 8 8 ͍͖ͳΓ౸ୡͰ͖ͳ͍
9 1ظ: ·ͣσʔλαΠΤϯςΟετ͕ར༻Ͱ͖ΔΑ͏ʹ ü ωοτࠂۀքͰػցֶश͕ྲྀߦ͍ͬͯΔͱ͍͑ɺࣗαʔϏεͷ σʔλͰͦΕ͕Մೳͳͷ͔ݕূ͍ͨ͠ ü ࣮ݧԾઆݕূͷͨΊʹਓ͕ؒσʔλΛखܰʹར༻͍ͨ͠ ü ݶΒΕͨਓ͕ؒΫΤϦूܭΛ࣮ߦͰ͖Εྑ͍
ü ඦϛϦඵͷԠੑೳٻΊͳ͍ ü σʔλετΞͷཧʹख͕͔͔ؒΒͳ͍ࣄ͕ॏཁ ü σʔλྔ 600GByte/day ఔ͕ͩɺ·ͩ·ͩ૿͑ͦ͏
10 1ظ: ·ͣσʔλαΠΤϯςΟετ͕ར༻Ͱ͖ΔΑ͏ʹ ² ࠂͷ৴ϩάΛBigQueryʹྲྀ͠ࠐΜͩ ² MySQLͷϚελσʔλBigQueryʹಉظ ² WebUIPandasɺBigQuery Pythonܦ༝Ͱར༻
² BigQueryͰαϒαϯϓϦϯάͯ͠ϩʔΧϧϚγϯͰֶश ² AWS EMRୀ ² Elastic Searchୀ ² Cloud Datalab betaʹඈͼ͍ͭͯരࢮ (20161݄)
11 2ظ: όονॲཧ͔Βར༻Ͱ͖ΔΑ͏ʹ ü ܧଓతʹճ͍࣮ͨ͠ݧɺ༧ଌॲཧͷόονΛcronͰΒ͍ͤͨ ü ੳλεΫʹݶΒͣɺ৴γεςϜଆͷόονॲཧ͍͍ͨ ü ػೳຖͷ༻ঢ়گ(ΫΤϦίετ)Ѳ͍ͨ͠
12 2ظ: όονॲཧ͔Βར༻Ͱ͖ΔΑ͏ʹ ² CloudLoggingͷઃఆͰBigQueryͷࠪϩάΛBigQueryʹΤΫεϙʔτ ² ػೳຖʹαʔϏεΞΧϯτΛ͍ग़ͯ͠ɺ༻ঢ়گΛѲ ² ίετ͕ͶͨΒ௨ ²
ೖࡳ୯Ձࣗಈௐόονɺෆਖ਼ΫϦοΫఆόον͕Քಈ ² ϧʔϧϕʔεɺҟৗݕϕʔεͷࣝผλεΫSQLͰॻ͚Δ ² ࣮ݧ݁ՌCloud Storage/BigQueryʹอଘ
13
14
15 Audit Logͷ༻్ ² ػೳຖͷΫΤϦίετ ² ຖͷΫΤϦίετ ² ςετ༻ͷςʔϒϧ࡞ऀௐࠪ ²
ΘΕ͍ͯͳ͍ςʔϒϧௐࠪ
16 3ظ: ͯ͢ͷ৬छͷϝϯόʔ͕σʔλΛར༻Ͱ͖ΔΑ͏ʹ ü ఆܕͷௐࠪλεΫΤϯδχΞ๊͕͑ͨ͘ͳ͍ ü ίετ͕രൃ͠ͳ͍Α͏ʹར༻ऀΛ૿͍ͨ͠ ü SQLॻ͚Δਓ͕૿͑Δͱྑ͍ײ͡ʹͳΔͷͰ
17 3ظ: ͯ͢ͷ৬छͷϝϯόʔ͕σʔλΛར༻Ͱ͖ΔΑ͏ʹ ² re:dashͰΫΤϦͰ͖ΔΑ͏ʹͨ͠ ² ΤϯδχΞ͕ཁΛݩʹςϯϓϨʔτͷΫΤϦΛ࡞ ² Ϩϙʔτը໘ͷϓϩτλΠϓʹ ²
ΫΤϦ୯ҐͷίετϦϛοτઃఆ(re:dashͷػೳ)ͰߴֹΫΤϦ࣮ߦ Λࢭ
18 ཁٻ͞ΕΔσʔλ࣭ϨϕϧมΘΔ ü Ϣʔεέʔε͕૿͑Δͱσʔλ࣭͕՝ʹ ü 23࣌ͷϩάऔΓࠐΈ͕ऴͬͨޙʹॲཧΛΒ͍ͤͨΜ͚ͩͲ? ² Stream Insert, Batch
Insert, ΫΤϦશͯϦτϥΠػߏඞਢ ² ݄ʹ1BigQueryͷௐࢠͷѱ͍͕͋Δ ² σʔλͷऔΓࠐΈ࿙ΕɺॏෳऔΓࠐΈνΣοΫͷόονΛՔಇ ² σʔλͷऔΓࠐΈঢ়گ͕֎෦͔Β֬ೝͰ͖ΔΈ
19 ෭࣍తՌ • ΤϯδχΞ͕͍ͭͰ৴ϩάͷௐ͕ࠪՄೳʹ • MySQLͰѻ͑ͳ͔ͬͨαΠζͷσʔλΛݩʹͨ͠ҙࢥܾఆ͕Մೳʹ • ༷ʑͳόονॲཧ͕σʔλΛར༻Մೳʹ • SQLΛॻ͚ͩ͘ͰϨϙʔτ͕ࣗ༝ʹ࡞Մೳʹ
• ϓϩδΣΫτͷϝϯόʔશһ͕σʔλʹΞΫηεՄೳʹ
20 ͦͷଞ • ΦϯϥΠϯͰσʔλΛࢀর͢ΔΑ͏ͳॲཧʹBigQuery͔ͳ͍ • Key-ValueͰҾ͚ΔΑ͏ʹͯ͠BigtableΛͬͨํ͕͍͍ • BigQueryͷલʹΩϟογϡϨΠϠΛ༻ҙ͢Δࣄྫ • Cloud
Dataproc or Cloud Dataflow…… • SpotifySparkෳࡶ͗ͯ͑͢ͳ͍ͱͷࣄͰDataflowΛscala͔Βར༻ • https://github.com/spotify/scio • Cloud Datalab͕৽͘͠ͳͬͨͦ͏ͳͷͰظ • Jupyter NotebookͷΫϥυ൛
21 ·ͱΊ • ͍͖ͳΓ͍͠ॴΛૂ͏ͱՌ͕ग़Δ·Ͱ͕͔͔࣌ؒΔͨΊɺͳΒ͠Λ͠ ͳ͕Βσʔλ׆༻ΛਐΊ͍ͯΔ • SQLͰهड़Ͱ͖Δϧʔϧϕʔεҟৗݕϕʔεͷॲཧػցֶशͱൺֱ͢ Δͱૣ͘Ռ͕ग़ͤΔ • Cloud
Storage, Cloud Logging, Cloud Dataprocͱͷ࿈ܞ͕ڧԽ͞Εɺ BigQueryͷϢʔεέʔε͕૿͑ͨ • ඦmsecͷԠੑೳɺಉ࣌ΫΤϦ࣮ߦɺ҆ఆੑΛٻΊͳ͚ΕBigQuery Ϧʔζφϒϧʹ͑Δ
22 ิ BigQueryͰ౷ܭྔΛग़࣌͢ʹ͏ΫΤϦϝϞ http://qiita.com/hagino3000/items/e9ed62638ebe54391188
23 23 Thank You