Introduction of GCP Dataflow

(In Japanese)

Transcript

1. Introduction of GCP Dataflow 2018-04 @soymsk
2. Dataflowͬͯͳʹʁ
4. Կ͕Ͱ͖Δͷʁ
5. ෼ࢄฒྻॲཧ͕Ͱ͖·͢
6. ੈͷதʹ෼ࢄ͍ͨ͠΋ͷ͸ͨ͘͞Μ͋Δ • σʔλूܭ: Sum, Average, etc.. • σʔλՃ޻: JSONύʔεɺMapMatch • I/O: ෳ਺Ϧιʔεͷೖग़ྗ • ߦྻԋࢉ • ػցֶश:෼ࢄֶश ( CV, RL)
7. ͜Μͳ͓೰Έ͕͋Γ·ͤΜ͔ʁ • ෼ࢄॲཧͷ࣮૷͕೉͍͠ɻɻ
 • ࠓ͸όονॲཧ͍͚ͨ͠Ͳɺকདྷ͸ετϦʔϛϯάॲ ཧ͍ͨ͠ʂ
 • ෼ࢄॲཧج൫(Hadoop, Yarn, Spark)ͷӡ༻͕ͭΒ ͍ɻɻ
8. ͜Μͳ͓೰Έ͕͋Γ·ͤΜ͔ʁ • ෼ࢄॲཧͷ࣮૷͕೉͍͠ɻɻ • →PythonͰαΫοͱ(※ʣ͔͚·͢ • ࠓ͸όονॲཧ͍͚ͨ͠Ͳɺকདྷ͸ϦΞϧλΠϜʹॲཧ ͍ͨ͠ʂ • →ಉҰίʔυ(※) Ͱ όον/ετϦʔϛϯά྆ํಈ͔ͤ ·͢ɻ • ෼ࢄॲཧج൫(Hadoop, Yarn, Spark)ͷӡ༻͕ͭΒ͍ɻɻ • → Fully managed ͳͷͰӡ༻ෆཁ(※) ※: ΄΅
9. ෼ࢄ vs ฒྻ • ※ఆٛ͸໌֬ʹ͸ܾ·͍ͬͯ·ͤΜ • ෼ࢄॲཧΛࣗલͰ࣮૷͢Δ͜ͱ͸ෆՄೳʹ͍ۙ • ނোɺ௨৴ःஅ͕͋Γ͏ΔΫϥελ্ͰॲཧΛҡ࣋͢Δʹ͸ɾɾɾʁ • ࠶࣮ߦʁ • ҟͳΔεϖοΫͷෳ਺Ϛγϯ্Ͱɺద੾ʹෛՙΛ෼ࢄͤ͞Δʹ͸ɾɾɾʁ • ద੾ͳεέδϡʔϦϯά • ୭্͕هΛ΍Δͷ͔ɾɾɾʁ • Ϛελʔ ←ɹϚελʔ͕͓ͪͨΒʁ • ϚϧνϚελʔʁ • Ϛελʔબग़ΞϧΰϦζϜɺίϯηϯαεΞϧΰϦζϜ(Raft, Paxos) • http://block-chain.jp/blockchain/distributed-consensus-algorithm-protocol/ ฒྻ 1BSBMMFM ෼ࢄʢ%JTUSJCVUFE ఆٛ˞ ୯ҰϚγϯ্Ͱฒྻॲཧ ෳ਺Ϛγϯ্Ͱฒྻॲཧ ࣮ݱํ๏ ɾϚϧνϓϩηεԽ
   ɾϚϧνεϨουԽ
   ɾ$16໋ྩ
   ɾࣗલͰ࣮૷
   ɾʮ෼ࢄॲཧج൫ʯ্Ͱ࣮ ૷
10. ෼ࢄॲཧج൫ ετϦʔϛϯά όον
11. ෼ࢄॲཧج൫ • ಘҙͳ͜ͱ͕ͦΕͧΕҟͳΔ • όον͔ετϦʔϛϯά͔ • ࣮૷ݴޠ(Java, Python) • ͲͷϨΠϠʔΛ୲౰͢Δ͔ • Ϋϥελ؅ཧ·ͰҰ؏ͯ͠΍Δ • Ϋϥελ؅ཧ͸ผͷίϯϙʔωϯτʹҕৡ͢Δ( on YARN) • ϑϨʔϜϫʔΫ͝ͱʹ࣮૷ํ๏͕ҟͳΔͷͰɺֶशίετ͕͔͔Γɺ ίʔυͷՄൖੑ΋ͳ͍ɻ • ӡ༻͕େม • ίʔυͷσόοά͢Βେมͳͷʹɺج൫ͷϝϯς΋ඞཁ
12. ͦ͜Ͱ
13. Dataflow(Apache Beam)ͱ͸ • GCP্ͷϑϧϚωʔδυ෼ࢄॲཧαʔϏε • PythonɺJavaɺ(Scala)ɺ(Go)Ͱ࣮૷Մೳ • Implement once , Run on any runtimes. • ϑϨʔϜϫʔΫࣗମ͸Apache Beamͱ͍͏໊લͰ࣮૷͕ ެ։͞Ε͍ͯΔɻ • Apache BeamͰ࣮૷͢Ε͹ɺ͍ΖΜͳϥϯλΠϜ্Ͱ࣮ ߦՄೳ • Dataflow͸ɺ࣮͸ʮApache Beam͕ಈ࡞͢ΔϥϯλΠ Ϝʯͷ̍ͭɺͱ͍͏Ґஔ͚ͮ
14. Apache Beam • Apache BeamͰ͸ɺॲཧ಺༰Λఆٛ͢Δ͚ͩ • ॲཧͷ࣮ߦͷ࢓ํ͸ϥϯλΠϜͷ੹೚ • Apache Beam͸ϥϯλΠϜ΍ೖྗσʔλΛந৅Խͯ͘͠ ΕΔ
15. Apache Beam Concept • ॲཧΛؔ਺ͱͯ͠ఆٛ͠ɺ਺चͭͳ͗ʹ࣮૷ • PCollection: ύΠϓϥΠϯؒσʔλΛϥοϓͨ͠Φϒ δΣΫτ • Transform: σʔλॲཧ(Pythonؔ਺ʣɻ෼ࢄͷ࠷খ୯Ґ
16. όονॲཧΛྫʹղઆ͠·͢
17. Hands-on
18. I/O Language File-based Messaging Database Java Beam Java supports Apache HDFS, Amazon S3, Google Cloud Storage, and local filesystems. FileIO (general-purpose reading, writing, and matching of files) AvroIO TextIO TFRecordIO XmlIO TikaIO Amazon Kinesis AMQP Apache Kafka Google Cloud PubSub JMS MQTT Apache Cassandra Apache Hadoop InputFormat Apache HBase Apache Hive (HCatalog) Apache Solr Elasticsearch (v2.x and v5.x) Google BigQuery Google Cloud Bigtable Google Cloud Datastore Google Cloud Spanner JDBC MongoDB Redis Python Beam Python supports Google Cloud Storage and local filesystems. avroio textio tfrecordio vcfio Google BigQuery Google Cloud Datastore
19. Transforms %BUB ParDo(Map) %BUB GroupBy %BUB %BUB %BUB Side Input %BUB %BUB 1 1 N M Multiple Output %BUB %BUB %BUB Combine %BUB %BUB N 1
20. Streaming? • ೖྗΛMessagingαʔϏε(PubSub, Kafka)ʹม͑Δͩ ͚ͰɺࣗಈతʹετϦʔϛϯάॲཧͱͯ͠ಈ࡞͢Δɻ • Transform͸όονॲཧͱશ͘ಉ͡ͰOK • ೖग़ྗ͕1:1ͷॲཧ͸͜Ε͚ͩ • → Aggregation(GroupBy)͢Δʹ͸ɻɻʁ
21. Windowing • StreamingͱBatchͷॲཧΛந৅Խ͢Δ࢓૊Έͱͯ͠ ʮBounded dataʯʮUnbounded dataʯ
 ͱ͍͏֓೦͕͋Δ • Bounded … σʔλͷ࠷ॳͱ࠷ޙ͕ఆٛ͞Ε͍ͯΔ(όο νʣ • Unbounded …࠷ޙ͕ະఆʢετϦʔϛϯά) • Boundedͳσʔλ͸ GroupByͰ͖Δɻ
22. Windowing • Unbounded data͸Bounded dataʹม׵͢Ε͹Α͍ɻ • σʔλΛ۠෼͚ͯ͠Bounded ʹ͢Δ࢓૊Έ͕ ʮWindowingʯ • ࣮͸ɺόονॲཧ͸ɺσʔλΛ͢΂ͯ
 ʮGlobalWindowʯͱ͍͏̍ͭͷWindow಺ͷσʔλ
 ʹର͢ΔετϦʔϛϯάͱಉٛͱߟ͑Δ͜ͱ͕Ͱ͖Δɻ
23. Windowing • Sliding time windowͰ͍͍ͩͨ΍Γ͍ͨ͜ͱ͸Ͱ͖Δ • 5෼ຖʹɺ30෼ؒͷσʔλͷฏۉ஋Λग़͢ɺͳͲɻ window (෯:60s, Φϑηοτ: 30s)
24. DataflowͷϋϚΓͲ͜Ζ
25. ᶃϩʔΧϧͰಈ͍ͨͷʹɺ Dataflow্Ͱಈ͔ͳ͍
26. ߟ͑ΒΕΔͱ͜Ζ • ෼ࢄ؀ڥͰ͸ɺϝϞϦۭؒ͸Ұ੾ڞ༗͞Εͳ͍ • εςʔτϑϧͳϩδοΫ͸޻෉͕ඞཁ • ϫʔΧʔؒͷঢ়ଶ؅ཧ͸֎෦ίϯϙʔωϯτ (memcached, GCS ͳͲ)Λܦ༝͢Δ • ϫʔΧϦιʔεෆ଍ • Transform͸1Ϩίʔυ࠷௿1ճ࣮ߦɻ • ϫʔΧʔ͕མͪͨͱ͖ͳͲɺ࠶࣮ߦ͞ΕΔՄೳੑɻ • ֎෦DBʹϨίʔυॲཧճ਺ΛΠϯΫϦϝϯτͯͨ͠Γ ͢ΔͱɺೖྗϨίʔυͱҰக͠ͳ͍Մೳੑ͕͋Δɻ
27. ߟ͑ΒΕΔͱ͜Ζ2 • Ϟδϡʔϧґଘؔ܎ • σϑΥϧτͰ͸ϫʔΧʔʹ͸جຊతͳύοέʔδ͔͠ೖͬͯͳ͍ • pipͰೖΔ΋ͷ͸requirements.txtॻ͍ͯΦϓγϣϯʹࢦఆͯ͠σ ϓϩΠ͢Δ͚ͩɻ • ࣗ෼Ͱ࡞ͬͨϞδϡʔϧ͸setup.pyΛ༻ҙ͠ɺύοέʔδԽ͕ඞ ཁɻ • όΠφϦ(lib)ͷ௥Ճ͸Ἒͷಓ • DataflowͷϫʔΧʔ͸ChromeOSΆ͍(Ubuntuͷѥछ) • apt͸͔ͭ͑Δ͕ɺखݩͷlib͸ಈ͔ͳ͍ • buildίʔυΛsetupʹ࢓ࠐΉ
28. ᶄͳ͔ͥ஗͍
29. Dataflowνϡʔχϯά • νϡʔχϯά߲໨͸ଟ͘ͳ͍ • Ϧʔδϣϯ(σʔλ, Πϯελϯε͸ͳΔ΂ۙ͘͘) • ΠϯελϯελΠϓ x ʮ࠷େϫʔΧʔ਺ʯ • جຊɺUIΛݟͳ͕Β٧·͍ͬͯΔͱ͜ΖͷίʔυΛվળɻ • ೖྗ/ग़ྗ͕ϘτϧωοΫʹͳΔ͔΋ʁ • PubSubͷQuotaΛ͋͛Δ • GCSͷϑΝΠϧΛେྔʹಡΈࠐΉͱ஗͔ͬͨɻ • Combine͔ͯ͠ΒಡΈࠐΉ • ࠷ۙ͸վળ͞Εͨʁ
30. νϡʔχϯά - Fusion - • ༥߹(Fusion) • Dataflow͸ɺύΠϓϥΠϯͷߏ଄Λղੳͯ͠ɺෳ਺ͷ TransformΛ༥߹ͯ̍ͭ͠ͷTransformͱ࣮ͯ͠ߦ͢Δ࢓૊Έ ͕ೖ͍ͬͯΔɻ • Fusion͞ΕͨTransform͸ɺϥούʔͷPCollection͕࣮ମԽ ͞ΕͣɺೖྗσʔλΛͦͷ··࣍ͷTransformʹྲྀ͢ͷͰɺύ ϑΥʔϚϯε্͕͕Δɻ • Fusion͞ΕͨҰ࿈ͷTransform͸ɺ̍ͭͷTransformͱ࣮ͯ͠ ߦ͞ΕΔɻ
31. νϡʔχϯά - Fusion - Word Countͷྫ ֤εςοϓ͝ͱʹதؒσʔλʢ1$PMMFDUJPO ͕ ࣮ମԽ͞ΕΔͷͰߴίετ εςοϓΛ·ͱΊͯɺதؒσʔλͷ࣮ମԽΛল͍ͯߴ଎Խ
32. νϡʔχϯά - Fusion - • جຊతʹFusion͸ѱ͞Λ͠ͳ͍͕ɺ
 ʮೖྗϨίʔυ਺ʯ<<< ʮग़ྗϨίʔυ਺ʯͳParDo ("high fan- out” ParDo)
 ͕ೖΔͱɺޙଓͷTransformͰॲཧ͕ͭ·Δɺͱ͍͏ݱ৅͕͋Δɻ • Fusion͞Εͨ݁ՌɺதؒͷTransform͕෼ࢄ͞Εͳ͍ɻ ※ೖྗϨίʔυ਺ʹΑͬͯ୲౰worker਺͕൑அ͞ΕΔ 5SBOTGPSN 5SBOTGPSN 5SBOTGPSN 5SBOTGPSN 'VTJPOFE
    5SBOTGPSN ຊདྷҙਤͨ͠෼ࢄ …. record: 1 record: 10000 record: 1
33. νϡʔχϯά - Fusion - • ྫ͑͹ɺϑΝΠϧ໊ͷ഑ྻΛೖྗͱ͠ɺϑΝΠϧͷத ਎ͷϨίʔυΛग़ྗ͢ΔΑ͏ͳParDo ͸ةݥ(සग़)ɹ • ճආࡦ • GroupBy -> UnGroup ͱ͍͏ॲཧΛhigh fan-outͷ ParDoͷ௚ޙʹ࢓ࠐΉ • εςοϓͰग़ྗ͞ΕͨPCollectionΛ࣍ͷεςοϓͷʮSide InputʯʹೖΕ ΔʢSide Input͸ඞ࣮ͣମԽ͞ΕΔʣ
34. Prevent Fusion • ૊ΈࠐΈͷϑΝΠϧಡΈࠐΈॲཧʹ͸͢ͰʹରࡦࡁΈ {
35. ᶅσόοάͭΒ͍
36. ෼ࢄฒྻॲཧͷ໋॓Ͱ͢
37. Dataflow σόοά • ྫ֎࣌ͷελοΫτϨʔε౳͸ग़ͳ͍ɻ • ֤εςοϓͷதͰCatchͯࣗ͠ྗͰग़͢ • ࣦഊ࣌ͷೖྗσʔλ͸GCSͳͲ΁ୀආͤ͞Δ • ҆қʹlogging͢Δͱlog͕େྔʹ(GB୯Ґ)Ͱग़ͯ དྷΔ •
38. ·ͱΊ • Dataflow(Apache Beam)͸Google͕ɺཚཱ͢Δ෼ࢄॲཧج൫Λ౷ Ұ͢ΔͨΊʹऔΓ૊ΜͰ͍ΔΒ͍͠ɻ • ۤ࿑΋͋Δ͕ɺΦʔτεέʔϧͳͲϚωʔδυ෼ࢄॲཧج൫ͱͯ͠ ͸΄΅།Ұͷબ୒ࢶʁ • ϦΞϧλΠϜϩάऩूͱͯ͠͸PubSub->(DF)->BQ͕σϑΝΫτʹ ͳΓͭͭ͋Γͦ͏ɻ • Python, Go, JavaͰ࣮૷Մೳ • ͨͩ͠ɺJava͕Ұ൪҆ఆ͍ͯ͠Δɻ • ಛʹετϦʔϛϯάϞʔυ͸׬੒౓ͷ໘ͰJavaʹ͓͍ͯͨ͠΄ ͏͕Α͍ɻ