Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Machine Learning with Clojure and Apache Spark
Search
Eric Weinstein
October 25, 2016
Technology
1
410
Machine Learning with Clojure and Apache Spark
Slides for my EuroClojure 2016 talk on machine learning.
Eric Weinstein
October 25, 2016
Tweet
Share
More Decks by Eric Weinstein
See All by Eric Weinstein
Interview Them Where They Are
ericqweinstein
0
120
Value Your Types!
ericqweinstein
0
94
Being Good: An Introduction to Robo- and Machine Ethics
ericqweinstein
1
1.9k
What If...?: Ruby 3
ericqweinstein
1
210
Infinite State Machine
ericqweinstein
1
130
Do Androids Dream of Electronic Dance Music?
ericqweinstein
1
110
Machine Learning with Elixir and Phoenix
ericqweinstein
1
960
Domo Arigato, Mr. Roboto: Machine Learning with Ruby
ericqweinstein
1
1.5k
A Nil Device, A Lonely Operator, and a Voyage to the Void Star
ericqweinstein
1
1k
Other Decks in Technology
See All in Technology
Claude Codeを駆使した初めてのiOSアプリ開発 ~ゼロから3週間でグローバルハッカソンで入賞するまで~
oikon48
10
4.1k
いまからでも遅くない!SSL/TLS証明書超入門(It's not too late to start! SSL/TLS Certificates: The Absolute Beginner's Guide)
norimuraz
0
240
今この時代に技術とどう向き合うべきか
gree_tech
PRO
2
1.8k
Simplifying Cloud Native app testing across environments with Dapr and Microcks
salaboy
0
160
HR Force における DWH の併用事例 ~ サービス基盤としての BigQuery / 分析基盤としての Snowflake ~@Cross Data Platforms Meetup #2「BigQueryと愉快な仲間たち」
ryo_suzuki
0
220
Geospatialの世界最前線を探る [2025年版]
dayjournal
1
220
20251007: What happens when multi-agent systems become larger? (CyberAgent, Inc)
ornew
1
290
サイバーエージェント流クラウドコスト削減施策「みんなで金塊堀太郎」
kurochan
3
1.9k
プレーリーカードを活用しよう❗❗デジタル名刺交換からはじまるイベント会場交流のススメ
tsukaman
0
160
能登半島地震で見えた災害対応の課題と組織変革の重要性
ditccsugii
0
970
AgentCon Accra: Ctrl + Alt + Assist: AI Agents Edition
bethany
0
110
スタートアップにおけるこれからの「データ整備」
shomaekawa
2
480
Featured
See All Featured
A better future with KSS
kneath
239
18k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
55
3k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.5k
Why You Should Never Use an ORM
jnunemaker
PRO
59
9.6k
Measuring & Analyzing Core Web Vitals
bluesmoon
9
620
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
27k
A designer walks into a library…
pauljervisheath
209
24k
The Pragmatic Product Professional
lauravandoore
36
6.9k
How to Think Like a Performance Engineer
csswizardry
27
2k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
35
6.1k
Thoughts on Productivity
jonyablonski
70
4.9k
Become a Pro
speakerdeck
PRO
29
5.5k
Transcript
Machine Learning with Clojure and Apache Spark ;; Eric Weinstein
;; EuroClojure 2016 ;; Bratislava, Slovakia ;; 25 October 2016
for Joshua
Part 0: Hello!
About Me (def eric-weinstein {:employer "Hulu" :github "ericqweinstein" :twitter "ericqweinstein"
:website "ericweinste.in"}) 30% off with EURORUBY30!
Agenda • Machine learning • Apache Spark • Flambo vs.
Sparkling • DL4J, deep learning, and convolutional neural networks
Part 1: ⚡✨
What’s machine learning?
In a word:
Generalization
What’s Supervised Learning? Classification or regression, generalizing from labeled data
to unlabeled data
What’s Apache Spark? Apache Spark is an open-source cluster computing
framework; its parallelism makes it ideal for processing large data sets, and in ML, the more data, the better!
Some Spark Terminology • RDD: Resilient Distributed Dataset • Dataset:
RDD + Spark SQL execution engine • DataFrame: Dataset organized into named columns
Our Data • Police stop data for the city of
Los Angeles, California in 2015 • 4 features, ~600,000 instances • http://bit.ly/2f9jVwn
Features && Labels • Sex (Male | Female) • Race
(American Indian | Asian | Black | Hispanic | White | Other) • Stop type (Pedestrian | Vehicle) • Post-stop activity (Yes | No)
Features && Labels • Sex (Male | Female) • Race
(American Indian | Asian | Black | Hispanic | White | Other) • Stop type (Pedestrian | Vehicle) • Post-stop activity (Yes | No)
Decision Trees X[0] <= 0.5 gini = 0.4033 samples =
139572 value = [100477, 39095] X[1] <= 5.5 gini = 0.4318 samples = 102419 value = [70118, 32301] True X[1] <= 5.5 gini = 0.2989 samples = 37153 value = [30359, 6794] False X[1] <= 4.5 gini = 0.4399 samples = 96665 value = [65083, 31582] gini = 0.2187 samples = 5754 value = [5035, 719] X[1] <= 3.5 gini = 0.4483 samples = 78400 value = [51805, 26595] gini = 0.397 samples = 18265 value = [13278, 4987] X[1] <= 2.5 gini = 0.4324 samples = 51662 value = [35328, 16334] gini = 0.473 samples = 26738 value = [16477, 10261] X[1] <= 0.5 gini = 0.4406 samples = 48927 value = [32894, 16033] gini = 0.1959 samples = 2735 value = [2434, 301] gini = 0.4658 samples = 65 value = [41, 24] gini = 0.4406 samples = 48862 value = [32853, 16009] X[1] <= 3.5 gini = 0.3067 samples = 34817 value = [28234, 6583] gini = 0.1643 samples = 2336 value = [2125, 211] X[1] <= 2.5 gini = 0.2796 samples = 15786 value = [13133, 2653] X[1] <= 4.5 gini = 0.3277 samples = 19031 value = [15101, 3930] X[1] <= 0.5 gini = 0.2921 samples = 13985 value = [11501, 2484] gini = 0.1701 samples = 1801 value = [1632, 169] gini = 0.426 samples = 26 value = [18, 8] gini = 0.2918 samples = 13959 value = [11483, 2476] gini = 0.3747 samples = 9522 value = [7144, 2378] gini = 0.2732 samples = 9509 value = [7957, 1552]
Part 2: A Tale of Two DSLs vs. ✨✨ Image
credit: Adventure Time
Flambo Example (defn make-spark-context "Creates the Apache Spark context using
the Flambo DSL." [] (-> (conf/spark-conf) (conf/master "local") (conf/app-name "euroclojure") (f/spark-context)))
Sparkling Example (defn make-spark-context "Creates the Apache Spark context using
the Sparkling DSL." [] (-> (conf/spark-conf) (conf/master "local") (conf/app-name "euroclojure") (spark/spark-context)))
Straight Spark (def model (DecisionTree/trainClassifier training 2 categorical-features- info "gini"
5 32)) ; max depth: 5, max leaves: 32 (defn predict [p] ; LabeledPoint (let [prediction (.predict model (.features p))] [(.label p) prediction]))
Accuracy: 0.77352
Part 3: Deep Learning
What’s Deep Learning? • Neural networks (computational architecture modeled after
the human brain) • Neural networks with many layers (> 1 hidden layer, but in practice, can be hundreds) • The vanishing/exploding gradient problem
Vanishing && Gradients
Image credit for all ConvNet images: https://deeplearning4j.org/convolutionalnets
Max Pooling/Downsampling
Alternating Layers
Our Data Image credit: http://digitalmedia.fws.gov/cdm/
What’s DL4J? • DL4J == Deep Learning 4 Java, a
library (for Java, unsurprisingly) • Examples on GitHub: https://github.com/ deeplearning4j/deeplearning4j • ConvNet worked example: http://bit.ly/2eBM8ss
DL4J Example (def nn-conf (-> (NeuralNetConfiguration$Builder.) ;; Some values omitted
for space (.activation "relu") (.learningRate 0.0001) (.weightInit (WeightInit/XAVIER)) (.optimizationAlgo OptimizationAlgorithm/STOCHASTIC_GRADIENT_DESCENT) (.updater Updater/RMSPROP) (.momentum 0.9) (.list) (.layer 0 conv-init) (.layer 1 (max-pool "maxpool1" (int-array [2 2]))) (.layer 2 (conv-5x5 "cnn2" 100 (int-array [5 5]) (int-array [1 1]) 0)) (.layer 3 (max-pool "maxpool2" (int-array [2 2]))) (.layer 4 (fully-connected 500)) (.layer 5 output-layer) (.build)))
How’d We Do? • Accuracy: 0.375 • Precision: 0.3333 •
Recall: 0.375 • F1 Score: 0.3529
Summary • Clojure + Spark = • Flambo and Sparkling
are roughly equally powerful • Deep learning is super doable with Clojure (though Java interop is kind of a pain)
Takeaways (TL;DPA) • Contribute to Flambo and/or Sparkling! • Let’s
build or contribute to a nicer DSL for DL4J • https://github.com/ericqweinstein/euroclojure
None