Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scalaで実装した 分散処理システムの超概要
Search
Sponsored
·
Your Podcast. Everywhere. Effortlessly.
Share. Educate. Inspire. Entertain. You do you. We'll handle the rest.
→
Takashi Funato
April 11, 2018
Technology
0
200
Scalaで実装した 分散処理システムの超概要
Takashi Funato
April 11, 2018
Tweet
Share
More Decks by Takashi Funato
See All by Takashi Funato
AWS をちょっとわかった気になる / Kinda feel the AWS
tfunato
0
98
人に優しいAWS運用
tfunato
0
80
OSI reference model Layer I
tfunato
0
53
Goで作るWebアプリケーション Echo 編 といいつつGoの説明多い
tfunato
0
110
Docker勉強会
tfunato
1
160
GISとして活用するElasticsearch
tfunato
6
7.9k
Other Decks in Technology
See All in Technology
Embedded SREの終わりを設計する 「なんとなく」から計画的な自立支援へ
sansantech
PRO
3
2.6k
Agent Skils
dip_tech
PRO
0
140
今日から始めるAmazon Bedrock AgentCore
har1101
4
420
Webhook best practices for rock solid and resilient deployments
glaforge
2
310
Context Engineeringの取り組み
nutslove
0
380
GitHub Issue Templates + Coding Agentで簡単みんなでIaC/Easy IaC for Everyone with GitHub Issue Templates + Coding Agent
aeonpeople
1
260
SchooでVue.js/Nuxtを技術選定している理由
yamanoku
3
210
ECS障害を例に学ぶ、インシデント対応に備えたAIエージェントの育て方 / How to develop AI agents for incident response with ECS outage
iselegant
4
460
CDK対応したAWS DevOps Agentを試そう_20260201
masakiokuda
1
450
今こそ学びたいKubernetesネットワーク ~CNIが繋ぐNWとプラットフォームの「フラッと」な対話
logica0419
5
550
登壇駆動学習のすすめ — CfPのネタの見つけ方と書くときに意識していること
bicstone
3
130
OpenShiftでllm-dを動かそう!
jpishikawa
0
140
Featured
See All Featured
Game over? The fight for quality and originality in the time of robots
wayneb77
1
120
Site-Speed That Sticks
csswizardry
13
1.1k
Technical Leadership for Architectural Decision Making
baasie
2
250
Why Our Code Smells
bkeepers
PRO
340
58k
WENDY [Excerpt]
tessaabrams
9
36k
Building an army of robots
kneath
306
46k
AI in Enterprises - Java and Open Source to the Rescue
ivargrimstad
0
1.1k
A designer walks into a library…
pauljervisheath
210
24k
Lessons Learnt from Crawling 1000+ Websites
charlesmeaden
PRO
1
1.1k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
35
3.4k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
16
1.8k
Breaking role norms: Why Content Design is so much more than writing copy - Taylor Woolridge
uxyall
0
170
Transcript
ScalaͰ࣮ͨ͠ ࢄॲཧγεςϜͷ֓ཁ 2018.04.11 ΏΊΈ ᢠށ ོ
հ • tfunato • αʔόαΠυͬͯ·͢ • PHPɺNode.jsɺJavaɺetc
Agenda • ࢄॲཧ • Sparkͷઆ໌ • ͬͨ͜ͱ • ·ͱΊ
ࢄॲཧ
ࢄॲཧ • ̍ͭͷॲཧΛࢄͯ͠ߦ͏͜ͱ
ͱ͍͑ • ॲཧͷ߹ੑ • ͦΕͧΕͷܭࢉ݁Ռͷ߹ੑ • ॲཧͷޮԽ • ॲཧͷׂ •
ॲཧରσʔλͷɺऩूɺूܭ • ௨৴ϓϩτίϧεέʔϥϏϦςΟɺোੑ…. • ઐ͡Όͳ͍ͷͰৄ͘͠…
ॲཧܥ • ѻ͏σʔλ͍ΘΏΔϏοάσʔλ • ेςϥόΠτ ʙ ϖλόΠτ • ηϯαʔɺυΩϡϝϯτɺϩάɺఱจֶɺେؾԽֶɺήϊϜ etc
• ҰൠతͳRDBͰॲཧ͖͠Εͳ͘ͳͬͨ • ઐ༻ͷϋʔυΣΞιϑτΣΞ • ඇৗʹߴՁ • Google͕MapReduceΛൃද • ͦΕΛͱʹHadoop͕࡞ΒΕOSSԽ
Hadoop͔ΒSpark • ॲཧ͕͍ • ͍Θ͚Ͱͳ͍͕ɺޮ͕ѱ͍ • IOΛฒྻԽͯ͠εϧʔϓοτΛ্͍͛ͯΔ͕ɺٯʹIO ͕ϘτϧωοΫʹͳ͖ͬͯͨ • αʔόߏ͕ෳࡶɺ͔ͭӡ༻͕େม
• NameNodeɺJobTrackerɺDataNodeɺTaskTracker
Hadoop͔ΒSpark • SparkHadoopͷMapReduceΛஔ͖͑Δ • SparkॲཧΛۃྗϝϞϦʔʹͤͯߦ͏ • ඞཁʹͳͬͨΒσΟεΫIO͢Δ • HadoopͷMapReduceΑΓ100ഒૣ͍ͱ͔
Spark
Sparkͷઆ໌ - ࣮ • Scala • Java • Python •
R
Sparkͷઆ໌ - σʔλೖग़ྗ • ༷ʑͳσʔλͷೖग़ྗʹରԠ • Hadoop Distributed File System(HDFS)
• Cassandra • MongoDB • Couchbase • Amazon S3 • RDBʢJDBCͰଓͰ͖ΔͷͰ͋Εʣ • IOपΓͷࣗ࡞
Sparkͷઆ໌ - ѻ͏σʔλܗࣜ • ༷ʑͳσʔλܗࣜʹରԠ • CSVʢTSVʣ • JSON •
Text • ParquetɺORC • ΧϥϜφϑΥʔϚοτʢྻํσʔλʣ • ReadɺWriteΛࣗ࡞
Sparkͷઆ໌ - ॲཧͷجຊ1 • ॲཧͷجຊ • Read • σʔλΛಡΈࠐΉ෦ •
Write • σʔλΛॻ͖ࠐΉ෦
Sparkͷઆ໌ - ॲཧͷجຊ2 • SQLϥΠΫͳΠϯλʔϑΣʔε͕༻ҙ͞Ε͍ͯΔ • Spark SQL • Readͨ͠σʔλʹରͯ͠Ճ
• FilterɺGroupByɺAvgɺOrderByɺMaxɺMinɺCount • JoinɺUnion • ෳͷಡΈࠐΜͩσʔλʹରͯ͠ • https://spark.apache.org/docs/latest/api/scala/ index.html#org.apache.spark.sql.Dataset
Sparkͷઆ໌ - ॲཧͷجຊ̐ • Write • ॻ͖ࠐΈઌͱϑΥʔϚοτΛࢦఆ͢Δ͚ͩ
ͬͨ͜ͱ
ΞʔΩςΫνϟ Amazon Redshift Amazon EMR Amazon S3 104σʔλ 4QBSL
ϩʔΧϧ։ൃڥ • Redshiftͱޓੑͷ͋ΔPostgreSQLΛDocker Ͱىಈ • IntelliJ IDEA͔ΒSparkΛ࣮ߦ • local࣮ߦϞʔυ͕͋Δ •
ϩʔΧϧʹϑΝΠϧग़ྗ
ͬͨ͜ͱ • Redshift͔ΒSQLΛ͛ͯରσʔλऔಘ • ඞཁ࠷খݶͷσʔλΛऔಘ͢Δ • POSσʔλΛूܭͯ͠ϙΠϯτΛ༩͢Δ • ݁ՌΛCSVͱͯ͠S3ʹอଘ
ͬͨ͜ͱ • JenkinsΛδϣϒεέδϡʔϥͱͯ͠༻ • ͔ͭϩάΛCloudWatchʹసૹͯ͠ࢹ • ͔̍̍͠ಈ͔ͳ͍ͷͰৗ࣌ىಈͷΫϥελͱ͠ͳ͍ͰɺऴΘͬ ͨΒΫϥελΛআ • ݁ߏେ͖͍ΠϯελϯεΛ༻ͯ͠େֹͨۚ͠ʹͳΒͳ͍
• ϩάS3ʹग़ྗ͞ΕΔͷͰ֬ೝ͍͢͠ • sbtʹcryptɺdecryptλεΫΛՃͯ͠ڥใΛ҉߸Խ
·ͱΊ
·ͱΊ • ΦʔόʔεϖοΫ͔ͩͬͨ͠Εͳ͍ • ͏ͪΐͬͱςετΛॻ͖͍ߏʹ͖͢ ͩͬͨ • Spark͍͍Α • Scala͍͍Α