Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Introduction of GCP Dataflow
Search
soymsk
April 12, 2018
Technology
260
1
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Introduction of GCP Dataflow
(In Japanese)
soymsk
April 12, 2018
Transcript
38 slides
Introduction of GCP Dataflow 2018-04 @soymsk
Dataflowͬͯͳʹʁ
None
Կ͕Ͱ͖Δͷʁ
ࢄฒྻॲཧ͕Ͱ͖·͢
ੈͷதʹࢄ͍ͨ͠ͷͨ͘͞Μ͋Δ • σʔλूܭ: Sum, Average, etc.. • σʔλՃ: JSONύʔεɺMapMatch •
I/O: ෳϦιʔεͷೖग़ྗ • ߦྻԋࢉ • ػցֶश:ࢄֶश ( CV, RL)
͜Μͳ͓Έ͕͋Γ·ͤΜ͔ʁ • ࢄॲཧͷ࣮͕͍͠ɻɻ • ࠓόονॲཧ͍͚ͨ͠ͲɺকདྷετϦʔϛϯάॲ ཧ͍ͨ͠ʂ • ࢄॲཧج൫(Hadoop, Yarn, Spark)ͷӡ༻͕ͭΒ
͍ɻɻ
͜Μͳ͓Έ͕͋Γ·ͤΜ͔ʁ • ࢄॲཧͷ࣮͕͍͠ɻɻ • →PythonͰαΫοͱ(※ʣ͔͚·͢ • ࠓόονॲཧ͍͚ͨ͠ͲɺকདྷϦΞϧλΠϜʹॲཧ ͍ͨ͠ʂ • →ಉҰίʔυ(※)
Ͱ όον/ετϦʔϛϯά྆ํಈ͔ͤ ·͢ɻ • ࢄॲཧج൫(Hadoop, Yarn, Spark)ͷӡ༻͕ͭΒ͍ɻɻ • → Fully managed ͳͷͰӡ༻ෆཁ(※) ※: ΄΅
ࢄ vs ฒྻ • ※ఆٛ໌֬ʹܾ·͍ͬͯ·ͤΜ • ࢄॲཧΛࣗલͰ࣮͢Δ͜ͱෆՄೳʹ͍ۙ • ނোɺ௨৴ःஅ͕͋Γ͏ΔΫϥελ্ͰॲཧΛҡ࣋͢Δʹɾɾɾʁ •
࠶࣮ߦʁ • ҟͳΔεϖοΫͷෳϚγϯ্ͰɺదʹෛՙΛࢄͤ͞Δʹɾɾɾʁ • దͳεέδϡʔϦϯά • ୭্͕هΛΔͷ͔ɾɾɾʁ • Ϛελʔ ←ɹϚελʔ͕͓ͪͨΒʁ • ϚϧνϚελʔʁ • Ϛελʔબग़ΞϧΰϦζϜɺίϯηϯαεΞϧΰϦζϜ(Raft, Paxos) • http://block-chain.jp/blockchain/distributed-consensus-algorithm-protocol/ ฒྻ 1BSBMMFM ࢄʢ%JTUSJCVUFE ఆٛ˞ ୯ҰϚγϯ্Ͱฒྻॲཧ ෳϚγϯ্Ͱฒྻॲཧ ࣮ݱํ๏ ɾϚϧνϓϩηεԽ ɾϚϧνεϨουԽ ɾ$16໋ྩ ɾࣗલͰ࣮ ɾʮࢄॲཧج൫ʯ্Ͱ࣮
ࢄॲཧج൫ ετϦʔϛϯά όον
ࢄॲཧج൫ • ಘҙͳ͜ͱ͕ͦΕͧΕҟͳΔ • όον͔ετϦʔϛϯά͔ • ࣮ݴޠ(Java, Python) • ͲͷϨΠϠʔΛ୲͢Δ͔
• Ϋϥελཧ·ͰҰ؏ͯ͠Δ • Ϋϥελཧผͷίϯϙʔωϯτʹҕৡ͢Δ( on YARN) • ϑϨʔϜϫʔΫ͝ͱʹ࣮ํ๏͕ҟͳΔͷͰɺֶशίετ͕͔͔Γɺ ίʔυͷՄൖੑͳ͍ɻ • ӡ༻͕େม • ίʔυͷσόοά͢Βେมͳͷʹɺج൫ͷϝϯςඞཁ
ͦ͜Ͱ
Dataflow(Apache Beam)ͱ • GCP্ͷϑϧϚωʔδυࢄॲཧαʔϏε • PythonɺJavaɺ(Scala)ɺ(Go)Ͱ࣮Մೳ • Implement once ,
Run on any runtimes. • ϑϨʔϜϫʔΫࣗମApache Beamͱ͍͏໊લͰ࣮͕ ެ։͞Ε͍ͯΔɻ • Apache BeamͰ࣮͢Εɺ͍ΖΜͳϥϯλΠϜ্Ͱ࣮ ߦՄೳ • Dataflowɺ࣮ʮApache Beam͕ಈ࡞͢ΔϥϯλΠ Ϝʯͷ̍ͭɺͱ͍͏Ґஔ͚ͮ
Apache Beam • Apache BeamͰɺॲཧ༰Λఆٛ͢Δ͚ͩ • ॲཧͷ࣮ߦͷํϥϯλΠϜͷ • Apache BeamϥϯλΠϜೖྗσʔλΛநԽͯ͘͠
ΕΔ
Apache Beam Concept • ॲཧΛؔͱͯ͠ఆٛ͠ɺचͭͳ͗ʹ࣮ • PCollection: ύΠϓϥΠϯؒσʔλΛϥοϓͨ͠Φϒ δΣΫτ •
Transform: σʔλॲཧ(Pythonؔʣɻࢄͷ࠷খ୯Ґ
όονॲཧΛྫʹղઆ͠·͢
Hands-on
I/O Language File-based Messaging Database Java Beam Java supports Apache
HDFS, Amazon S3, Google Cloud Storage, and local filesystems. FileIO (general-purpose reading, writing, and matching of files) AvroIO TextIO TFRecordIO XmlIO TikaIO Amazon Kinesis AMQP Apache Kafka Google Cloud PubSub JMS MQTT Apache Cassandra Apache Hadoop InputFormat Apache HBase Apache Hive (HCatalog) Apache Solr Elasticsearch (v2.x and v5.x) Google BigQuery Google Cloud Bigtable Google Cloud Datastore Google Cloud Spanner JDBC MongoDB Redis Python Beam Python supports Google Cloud Storage and local filesystems. avroio textio tfrecordio vcfio Google BigQuery Google Cloud Datastore
Transforms %BUB ParDo(Map) %BUB GroupBy %BUB %BUB %BUB Side Input
%BUB %BUB 1 1 N M Multiple Output %BUB %BUB %BUB Combine %BUB %BUB N 1
Streaming? • ೖྗΛMessagingαʔϏε(PubSub, Kafka)ʹม͑Δͩ ͚ͰɺࣗಈతʹετϦʔϛϯάॲཧͱͯ͠ಈ࡞͢Δɻ • Transformόονॲཧͱશ͘ಉ͡ͰOK • ೖग़ྗ͕1:1ͷॲཧ͜Ε͚ͩ •
→ Aggregation(GroupBy)͢Δʹɻɻʁ
Windowing • StreamingͱBatchͷॲཧΛநԽ͢ΔΈͱͯ͠ ʮBounded dataʯʮUnbounded dataʯ ͱ͍͏֓೦͕͋Δ • Bounded …
σʔλͷ࠷ॳͱ࠷ޙ͕ఆٛ͞Ε͍ͯΔ(όο νʣ • Unbounded …࠷ޙ͕ະఆʢετϦʔϛϯά) • Boundedͳσʔλ GroupByͰ͖Δɻ
Windowing • Unbounded dataBounded dataʹม͢ΕΑ͍ɻ • σʔλΛ͚۠ͯ͠Bounded ʹ͢ΔΈ͕ ʮWindowingʯ •
࣮ɺόονॲཧɺσʔλΛͯ͢ ʮGlobalWindowʯͱ͍͏̍ͭͷWindowͷσʔλ ʹର͢ΔετϦʔϛϯάͱಉٛͱߟ͑Δ͜ͱ͕Ͱ͖Δɻ
Windowing • Sliding time windowͰ͍͍ͩͨΓ͍ͨ͜ͱͰ͖Δ • 5ຖʹɺ30ؒͷσʔλͷฏۉΛग़͢ɺͳͲɻ window (෯:60s, Φϑηοτ:
30s)
DataflowͷϋϚΓͲ͜Ζ
ᶃϩʔΧϧͰಈ͍ͨͷʹɺ Dataflow্Ͱಈ͔ͳ͍
ߟ͑ΒΕΔͱ͜Ζ • ࢄڥͰɺϝϞϦۭؒҰڞ༗͞Εͳ͍ • εςʔτϑϧͳϩδοΫ͕ඞཁ • ϫʔΧʔؒͷঢ়ଶཧ֎෦ίϯϙʔωϯτ (memcached, GCS ͳͲ)Λܦ༝͢Δ
• ϫʔΧϦιʔεෆ • Transform1Ϩίʔυ࠷1ճ࣮ߦɻ • ϫʔΧʔ͕མͪͨͱ͖ͳͲɺ࠶࣮ߦ͞ΕΔՄೳੑɻ • ֎෦DBʹϨίʔυॲཧճΛΠϯΫϦϝϯτͯͨ͠Γ ͢ΔͱɺೖྗϨίʔυͱҰக͠ͳ͍Մೳੑ͕͋Δɻ
ߟ͑ΒΕΔͱ͜Ζ2 • Ϟδϡʔϧґଘؔ • σϑΥϧτͰϫʔΧʔʹجຊతͳύοέʔδ͔͠ೖͬͯͳ͍ • pipͰೖΔͷrequirements.txtॻ͍ͯΦϓγϣϯʹࢦఆͯ͠σ ϓϩΠ͢Δ͚ͩɻ • ࣗͰ࡞ͬͨϞδϡʔϧsetup.pyΛ༻ҙ͠ɺύοέʔδԽ͕ඞ
ཁɻ • όΠφϦ(lib)ͷՃἚͷಓ • DataflowͷϫʔΧʔChromeOSΆ͍(Ubuntuͷѥछ) • apt͔ͭ͑Δ͕ɺखݩͷlibಈ͔ͳ͍ • buildίʔυΛsetupʹࠐΉ
ᶄͳ͔͍ͥ
Dataflowνϡʔχϯά • νϡʔχϯά߲ଟ͘ͳ͍ • Ϧʔδϣϯ(σʔλ, ΠϯελϯεͳΔۙ͘͘) • ΠϯελϯελΠϓ x ʮ࠷େϫʔΧʔʯ
• جຊɺUIΛݟͳ͕Β٧·͍ͬͯΔͱ͜ΖͷίʔυΛվળɻ • ೖྗ/ग़ྗ͕ϘτϧωοΫʹͳΔ͔ʁ • PubSubͷQuotaΛ͋͛Δ • GCSͷϑΝΠϧΛେྔʹಡΈࠐΉͱ͔ͬͨɻ • Combine͔ͯ͠ΒಡΈࠐΉ • ࠷ۙվળ͞Εͨʁ
νϡʔχϯά - Fusion - • ༥߹(Fusion) • DataflowɺύΠϓϥΠϯͷߏΛղੳͯ͠ɺෳͷ TransformΛ༥߹ͯ̍ͭ͠ͷTransformͱ࣮ͯ͠ߦ͢ΔΈ ͕ೖ͍ͬͯΔɻ
• Fusion͞ΕͨTransformɺϥούʔͷPCollection͕࣮ମԽ ͞ΕͣɺೖྗσʔλΛͦͷ··࣍ͷTransformʹྲྀ͢ͷͰɺύ ϑΥʔϚϯε্͕͕Δɻ • Fusion͞ΕͨҰ࿈ͷTransformɺ̍ͭͷTransformͱ࣮ͯ͠ ߦ͞ΕΔɻ
νϡʔχϯά - Fusion - Word Countͷྫ ֤εςοϓ͝ͱʹதؒσʔλʢ1$PMMFDUJPO ͕ ࣮ମԽ͞ΕΔͷͰߴίετ εςοϓΛ·ͱΊͯɺதؒσʔλͷ࣮ମԽΛল͍ͯߴԽ
νϡʔχϯά - Fusion - • جຊతʹFusionѱ͞Λ͠ͳ͍͕ɺ ʮೖྗϨίʔυʯ<<< ʮग़ྗϨίʔυʯͳParDo ("high fan-
out” ParDo) ͕ೖΔͱɺޙଓͷTransformͰॲཧ͕ͭ·Δɺͱ͍͏ݱ͕͋Δɻ • Fusion͞Εͨ݁ՌɺதؒͷTransform͕ࢄ͞Εͳ͍ɻ ※ೖྗϨίʔυʹΑͬͯ୲worker͕அ͞ΕΔ 5SBOTGPSN 5SBOTGPSN 5SBOTGPSN 5SBOTGPSN 'VTJPOFE5SBOTGPSN ຊདྷҙਤͨ͠ࢄ …. record: 1 record: 10000 record: 1
νϡʔχϯά - Fusion - • ྫ͑ɺϑΝΠϧ໊ͷྻΛೖྗͱ͠ɺϑΝΠϧͷத ͷϨίʔυΛग़ྗ͢ΔΑ͏ͳParDo ةݥ(සग़)ɹ • ճආࡦ
• GroupBy -> UnGroup ͱ͍͏ॲཧΛhigh fan-outͷ ParDoͷޙʹࠐΉ • εςοϓͰग़ྗ͞ΕͨPCollectionΛ࣍ͷεςοϓͷʮSide InputʯʹೖΕ ΔʢSide Inputඞ࣮ͣମԽ͞ΕΔʣ
Prevent Fusion • ΈࠐΈͷϑΝΠϧಡΈࠐΈॲཧʹ͢ͰʹରࡦࡁΈ {
ᶅσόοάͭΒ͍
ࢄฒྻॲཧͷ໋॓Ͱ͢
Dataflow σόοά • ྫ֎࣌ͷελοΫτϨʔεग़ͳ͍ɻ • ֤εςοϓͷதͰCatchͯࣗ͠ྗͰग़͢ • ࣦഊ࣌ͷೖྗσʔλGCSͳͲୀආͤ͞Δ • ҆қʹlogging͢Δͱlog͕େྔʹ(GB୯Ґ)Ͱग़ͯ
དྷΔ •
·ͱΊ • Dataflow(Apache Beam)Google͕ɺཚཱ͢Δࢄॲཧج൫Λ౷ Ұ͢ΔͨΊʹऔΓΜͰ͍ΔΒ͍͠ɻ • ۤ࿑͋Δ͕ɺΦʔτεέʔϧͳͲϚωʔδυࢄॲཧج൫ͱͯ͠ ΄΅།Ұͷબࢶʁ • ϦΞϧλΠϜϩάऩूͱͯ͠PubSub->(DF)->BQ͕σϑΝΫτʹ
ͳΓͭͭ͋Γͦ͏ɻ • Python, Go, JavaͰ࣮Մೳ • ͨͩ͠ɺJava͕Ұ൪҆ఆ͍ͯ͠Δɻ • ಛʹετϦʔϛϯάϞʔυͷ໘ͰJavaʹ͓͍ͯͨ͠΄ ͏͕Α͍ɻ
More Decks by soymsk
See All by soymsk
[SUSTEN 勉強会]マイナンバーカードの仕組み
soymsk
0
270
Google_Cloud_Next_19_AI_ML_Summary_public.pdf
soymsk
7
2k
DeNAにおけるデータ活用事例 〜移動体データ活用によるサービス創出とその基盤 / Data Driven Service in Taxi hiring app MOV
soymsk
0
430
wavenet
soymsk
0
100
Other Decks in Technology
See All in Technology
20261113_JAWS_Osaka
takuyay0ne
0
530
OpenTelemetryで変わった 新時代のCloudWatchメトリクス監視
tsukuboshi
0
300
Claude Codeで新規事業を伸ばせるか? Zennの試行錯誤の記録
wadayusuke
2
210
Kiro 入門 (1 つのエージェントを、どの画面からでも) | 目指せ Kiro マスター 01
inariku
PRO
6
1.9k
ぼくのClaude Codeの使い方
falken
PRO
5
7.2k
「任されビリティ」で引き受けて「よしなりてぃ」に頼らず任せる / Accept tasks with "makasarebility" and hand them over without relying on "yoshinarily"
pauli
1
670
変更差分を読むために環境を整備しよう
pipepasta
1
380
Kiro Web 入門 (サインインから、頼み方、PR の受け取り、クラウドの決まりまで) | 目指せ Kiro マスター 04
inariku
PRO
2
130
目を覚ませ!僕らの認証情報が何者かに侵略されてるぞ
tkito
2
330
狙われているAPIの脆弱性!今こそ手をつけるべき根本的な対策とは?
nagix
0
140
Shared Signals Framework 仕様解説
oidfj
0
450
AWS Emulator を評価する: 手元で動く AWS の API の仕組み、始め方、確かめ方
inariku
PRO
1
700
Featured
See All Featured
A Tale of Four Properties
chriscoyier
163
24k
Fireside Chat
paigeccino
43
4.1k
Utilizing Notion as your number one productivity tool
mfonobong
4
640
Color Theory Basics | Prateek | Gurzu
gurzu
1
520
A brief & incomplete history of UX Design for the World Wide Web: 1989–2019
jct
2
560
The SEO Collaboration Effect
kristinabergwall1
1
620
svc-hook: hooking system calls on ARM64 by binary rewriting
retrage
2
680
How Fast Is Fast Enough? [PerfNow 2025]
tammyeverts
3
980
Visual Storytelling: How to be a Superhuman Communicator
reverentgeek
2
750
Ecommerce SEO: The Keys for Success Now & Beyond - #SERPConf2024
aleyda
1
2.3k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
25
2.2k
How To Speak Unicorn (iThemes Webinar)
marktimemedia
1
640