2016-09-06 bq_sushi #4での発表資料です
1 GCPͰ͡ΊΔεϞʔϧελʔτͳσʔλ׆༻#bq_sushi ver. bq_sushi #4 2016-09-06 Takashi Nishibayashi
View Slide
2 Takashi Nishibayashi Software Engineer Zucks AdNetwork, Zucks Inc. Data analysis team ݱࡏ৴ޮͷ࠷దԽ ೖࡳՁ֨ࣗಈௐϩδοΫɺ৴αʔόʔͷࠂબϩδοΫͷ։ൃʹैࣄ @hagino3000
3 3 ͜ΕԿ͔ಉͷGCP NEXT TOKYOͷࣄྫհηογϣϯͰൃදͨ͠༰ͷॖখ൛Ͱ͢
4 4 Zucks AdNetwork ͷσʔλ׆༻ͷมભ
5 5 5 ϓϩδΣΫτ։࢝࣌ͷཧͱݱ࣮
6 6 6 ࢦ͢ॴ(Ծ)ࠂ৴αʔόʔͰΠϯϓϨογϣϯຖʹػցֶशϞσϧʹΑΔίϯόʔδϣϯ༧ଌɺΫϦοΫ༧ଌΛߦͳ͍৴ޮΛΞοϓݱ࣮େྔͷϩάϑΝΠϧ͕༷ʑͳϑΥʔϚοτͰAWS S3ʹஔ͔Ε͍ͯΔϚελσʔλMySQLʹ֨ೲ͞Ε͍ͯΔElastic SearchʹೖͬͯΔͷۙ2िؒ
7 7 7
8 8 8 ͍͖ͳΓ౸ୡͰ͖ͳ͍
9 1ظ: ·ͣσʔλαΠΤϯςΟετ͕ར༻Ͱ͖ΔΑ͏ʹ ü ωοτࠂۀքͰػցֶश͕ྲྀߦ͍ͬͯΔͱ͍͑ɺࣗαʔϏεͷσʔλͰͦΕ͕Մೳͳͷ͔ݕূ͍ͨ͠ ü ࣮ݧԾઆݕূͷͨΊʹਓ͕ؒσʔλΛखܰʹར༻͍ͨ͠ ü ݶΒΕͨਓ͕ؒΫΤϦूܭΛ࣮ߦͰ͖Εྑ͍ ü ඦϛϦඵͷԠੑೳٻΊͳ͍ ü σʔλετΞͷཧʹख͕͔͔ؒΒͳ͍ࣄ͕ॏཁ ü σʔλྔ 600GByte/day ఔ͕ͩɺ·ͩ·ͩ૿͑ͦ͏
10 1ظ: ·ͣσʔλαΠΤϯςΟετ͕ར༻Ͱ͖ΔΑ͏ʹ ² ࠂͷ৴ϩάΛBigQueryʹྲྀ͠ࠐΜͩ ² MySQLͷϚελσʔλBigQueryʹಉظ ² WebUIPandasɺBigQuery Pythonܦ༝Ͱར༻ ² BigQueryͰαϒαϯϓϦϯάͯ͠ϩʔΧϧϚγϯͰֶश ² AWS EMRୀ ² Elastic Searchୀ ² Cloud Datalab betaʹඈͼ͍ͭͯരࢮ (20161݄)
11 2ظ: όονॲཧ͔Βར༻Ͱ͖ΔΑ͏ʹ ü ܧଓతʹճ͍࣮ͨ͠ݧɺ༧ଌॲཧͷόονΛcronͰΒ͍ͤͨ ü ੳλεΫʹݶΒͣɺ৴γεςϜଆͷόονॲཧ͍͍ͨ ü ػೳຖͷ༻ঢ়گ(ΫΤϦίετ)Ѳ͍ͨ͠
12 2ظ: όονॲཧ͔Βར༻Ͱ͖ΔΑ͏ʹ ² CloudLoggingͷઃఆͰBigQueryͷࠪϩάΛBigQueryʹΤΫεϙʔτ² ػೳຖʹαʔϏεΞΧϯτΛ͍ग़ͯ͠ɺ༻ঢ়گΛѲ ² ίετ͕ͶͨΒ௨ ² ೖࡳ୯Ձࣗಈௐόονɺෆਖ਼ΫϦοΫఆόον͕Քಈ ² ϧʔϧϕʔεɺҟৗݕϕʔεͷࣝผλεΫSQLͰॻ͚Δ ² ࣮ݧ݁ՌCloud Storage/BigQueryʹอଘ
13
14
15 Audit Logͷ༻్ ² ػೳຖͷΫΤϦίετ ² ຖͷΫΤϦίετ ² ςετ༻ͷςʔϒϧ࡞ऀௐࠪ ² ΘΕ͍ͯͳ͍ςʔϒϧௐࠪ
16 3ظ: ͯ͢ͷ৬छͷϝϯόʔ͕σʔλΛར༻Ͱ͖ΔΑ͏ʹ ü ఆܕͷௐࠪλεΫΤϯδχΞ๊͕͑ͨ͘ͳ͍ü ίετ͕രൃ͠ͳ͍Α͏ʹར༻ऀΛ૿͍ͨ͠ü SQLॻ͚Δਓ͕૿͑Δͱྑ͍ײ͡ʹͳΔͷͰ
17 3ظ: ͯ͢ͷ৬छͷϝϯόʔ͕σʔλΛར༻Ͱ͖ΔΑ͏ʹ ² re:dashͰΫΤϦͰ͖ΔΑ͏ʹͨ͠² ΤϯδχΞ͕ཁΛݩʹςϯϓϨʔτͷΫΤϦΛ࡞² Ϩϙʔτը໘ͷϓϩτλΠϓʹ² ΫΤϦ୯ҐͷίετϦϛοτઃఆ(re:dashͷػೳ)ͰߴֹΫΤϦ࣮ߦΛࢭ
18 ཁٻ͞ΕΔσʔλ࣭ϨϕϧมΘΔ ü Ϣʔεέʔε͕૿͑Δͱσʔλ࣭͕՝ʹ ü 23࣌ͷϩάऔΓࠐΈ͕ऴͬͨޙʹॲཧΛΒ͍ͤͨΜ͚ͩͲ? ² Stream Insert, Batch Insert, ΫΤϦશͯϦτϥΠػߏඞਢ ² ݄ʹ1BigQueryͷௐࢠͷѱ͍͕͋Δ ² σʔλͷऔΓࠐΈ࿙ΕɺॏෳऔΓࠐΈνΣοΫͷόονΛՔಇ ² σʔλͷऔΓࠐΈঢ়گ͕֎෦͔Β֬ೝͰ͖ΔΈ
19 ෭࣍తՌ • ΤϯδχΞ͕͍ͭͰ৴ϩάͷௐ͕ࠪՄೳʹ • MySQLͰѻ͑ͳ͔ͬͨαΠζͷσʔλΛݩʹͨ͠ҙࢥܾఆ͕Մೳʹ • ༷ʑͳόονॲཧ͕σʔλΛར༻Մೳʹ • SQLΛॻ͚ͩ͘ͰϨϙʔτ͕ࣗ༝ʹ࡞Մೳʹ • ϓϩδΣΫτͷϝϯόʔશһ͕σʔλʹΞΫηεՄೳʹ
20 ͦͷଞ • ΦϯϥΠϯͰσʔλΛࢀর͢ΔΑ͏ͳॲཧʹBigQuery͔ͳ͍ • Key-ValueͰҾ͚ΔΑ͏ʹͯ͠BigtableΛͬͨํ͕͍͍ • BigQueryͷલʹΩϟογϡϨΠϠΛ༻ҙ͢Δࣄྫ • Cloud Dataproc or Cloud Dataflow…… • SpotifySparkෳࡶ͗ͯ͑͢ͳ͍ͱͷࣄͰDataflowΛscala͔Βར༻ • https://github.com/spotify/scio • Cloud Datalab͕৽͘͠ͳͬͨͦ͏ͳͷͰظ • Jupyter NotebookͷΫϥυ൛
21 ·ͱΊ • ͍͖ͳΓ͍͠ॴΛૂ͏ͱՌ͕ग़Δ·Ͱ͕͔͔࣌ؒΔͨΊɺͳΒ͠Λ͠ͳ͕Βσʔλ׆༻ΛਐΊ͍ͯΔ • SQLͰهड़Ͱ͖Δϧʔϧϕʔεҟৗݕϕʔεͷॲཧػցֶशͱൺֱ͢Δͱૣ͘Ռ͕ग़ͤΔ • Cloud Storage, Cloud Logging, Cloud Dataprocͱͷ࿈ܞ͕ڧԽ͞ΕɺBigQueryͷϢʔεέʔε͕૿͑ͨ • ඦmsecͷԠੑೳɺಉ࣌ΫΤϦ࣮ߦɺ҆ఆੑΛٻΊͳ͚ΕBigQueryϦʔζφϒϧʹ͑Δ
22 ิ BigQueryͰ౷ܭྔΛग़࣌͢ʹ͏ΫΤϦϝϞ http://qiita.com/hagino3000/items/e9ed62638ebe54391188
23 23 Thank You