Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Machine Learning with Clojure and Apache Spark
Search
Eric Weinstein
October 25, 2016
Technology
1
400
Machine Learning with Clojure and Apache Spark
Slides for my EuroClojure 2016 talk on machine learning.
Eric Weinstein
October 25, 2016
Tweet
Share
More Decks by Eric Weinstein
See All by Eric Weinstein
Interview Them Where They Are
ericqweinstein
0
120
Value Your Types!
ericqweinstein
0
86
Being Good: An Introduction to Robo- and Machine Ethics
ericqweinstein
1
1.8k
What If...?: Ruby 3
ericqweinstein
1
200
Infinite State Machine
ericqweinstein
1
120
Do Androids Dream of Electronic Dance Music?
ericqweinstein
1
100
Machine Learning with Elixir and Phoenix
ericqweinstein
1
950
Domo Arigato, Mr. Roboto: Machine Learning with Ruby
ericqweinstein
1
1.5k
A Nil Device, A Lonely Operator, and a Voyage to the Void Star
ericqweinstein
1
980
Other Decks in Technology
See All in Technology
Strands Agents & Bedrock AgentCoreを1分でおさらい
minorun365
PRO
6
230
マルチモーダル基盤モデルに基づく動画と音の解析技術
lycorptech_jp
PRO
4
500
データエンジニアがクラシルでやりたいことの現在地
gappy50
3
850
AIに目を奪われすぎて、周りの困っている人間が見えなくなっていませんか?
cap120
1
430
2025新卒研修・HTML/CSS #弁護士ドットコム
bengo4com
3
13k
2時間で300+テーブルをデータ基盤に連携するためのAI活用 / FukuokaDataEngineer
sansan_randd
0
130
データ基盤の管理者からGoogle Cloud全体の管理者になっていた話
zozotech
PRO
0
330
VLMサービスを用いた請求書データ化検証 / SaaSxML_Session_1
sansan_randd
0
220
Foundation Model × VisionKit で実現するローカル OCR
sansantech
PRO
0
290
S3 Glacier のデータを Athena からクエリしようとしたらどうなるのか/try-to-query-s3-glacier-from-athena
emiki
0
180
SRE新規立ち上げ! Hubbleインフラのこれまでと展望
katsuya0515
0
160
バクラクによるコーポレート業務の自動運転 #BetAIDay
layerx
PRO
1
830
Featured
See All Featured
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
34
3.1k
The Language of Interfaces
destraynor
158
25k
Why You Should Never Use an ORM
jnunemaker
PRO
58
9.5k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
Faster Mobile Websites
deanohume
308
31k
Six Lessons from altMBA
skipperchong
28
3.9k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
44
2.4k
Side Projects
sachag
455
43k
How to Think Like a Performance Engineer
csswizardry
25
1.8k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
3k
Why Our Code Smells
bkeepers
PRO
337
57k
Transcript
Machine Learning with Clojure and Apache Spark ;; Eric Weinstein
;; EuroClojure 2016 ;; Bratislava, Slovakia ;; 25 October 2016
for Joshua
Part 0: Hello!
About Me (def eric-weinstein {:employer "Hulu" :github "ericqweinstein" :twitter "ericqweinstein"
:website "ericweinste.in"}) 30% off with EURORUBY30!
Agenda • Machine learning • Apache Spark • Flambo vs.
Sparkling • DL4J, deep learning, and convolutional neural networks
Part 1: ⚡✨
What’s machine learning?
In a word:
Generalization
What’s Supervised Learning? Classification or regression, generalizing from labeled data
to unlabeled data
What’s Apache Spark? Apache Spark is an open-source cluster computing
framework; its parallelism makes it ideal for processing large data sets, and in ML, the more data, the better!
Some Spark Terminology • RDD: Resilient Distributed Dataset • Dataset:
RDD + Spark SQL execution engine • DataFrame: Dataset organized into named columns
Our Data • Police stop data for the city of
Los Angeles, California in 2015 • 4 features, ~600,000 instances • http://bit.ly/2f9jVwn
Features && Labels • Sex (Male | Female) • Race
(American Indian | Asian | Black | Hispanic | White | Other) • Stop type (Pedestrian | Vehicle) • Post-stop activity (Yes | No)
Features && Labels • Sex (Male | Female) • Race
(American Indian | Asian | Black | Hispanic | White | Other) • Stop type (Pedestrian | Vehicle) • Post-stop activity (Yes | No)
Decision Trees X[0] <= 0.5 gini = 0.4033 samples =
139572 value = [100477, 39095] X[1] <= 5.5 gini = 0.4318 samples = 102419 value = [70118, 32301] True X[1] <= 5.5 gini = 0.2989 samples = 37153 value = [30359, 6794] False X[1] <= 4.5 gini = 0.4399 samples = 96665 value = [65083, 31582] gini = 0.2187 samples = 5754 value = [5035, 719] X[1] <= 3.5 gini = 0.4483 samples = 78400 value = [51805, 26595] gini = 0.397 samples = 18265 value = [13278, 4987] X[1] <= 2.5 gini = 0.4324 samples = 51662 value = [35328, 16334] gini = 0.473 samples = 26738 value = [16477, 10261] X[1] <= 0.5 gini = 0.4406 samples = 48927 value = [32894, 16033] gini = 0.1959 samples = 2735 value = [2434, 301] gini = 0.4658 samples = 65 value = [41, 24] gini = 0.4406 samples = 48862 value = [32853, 16009] X[1] <= 3.5 gini = 0.3067 samples = 34817 value = [28234, 6583] gini = 0.1643 samples = 2336 value = [2125, 211] X[1] <= 2.5 gini = 0.2796 samples = 15786 value = [13133, 2653] X[1] <= 4.5 gini = 0.3277 samples = 19031 value = [15101, 3930] X[1] <= 0.5 gini = 0.2921 samples = 13985 value = [11501, 2484] gini = 0.1701 samples = 1801 value = [1632, 169] gini = 0.426 samples = 26 value = [18, 8] gini = 0.2918 samples = 13959 value = [11483, 2476] gini = 0.3747 samples = 9522 value = [7144, 2378] gini = 0.2732 samples = 9509 value = [7957, 1552]
Part 2: A Tale of Two DSLs vs. ✨✨ Image
credit: Adventure Time
Flambo Example (defn make-spark-context "Creates the Apache Spark context using
the Flambo DSL." [] (-> (conf/spark-conf) (conf/master "local") (conf/app-name "euroclojure") (f/spark-context)))
Sparkling Example (defn make-spark-context "Creates the Apache Spark context using
the Sparkling DSL." [] (-> (conf/spark-conf) (conf/master "local") (conf/app-name "euroclojure") (spark/spark-context)))
Straight Spark (def model (DecisionTree/trainClassifier training 2 categorical-features- info "gini"
5 32)) ; max depth: 5, max leaves: 32 (defn predict [p] ; LabeledPoint (let [prediction (.predict model (.features p))] [(.label p) prediction]))
Accuracy: 0.77352
Part 3: Deep Learning
What’s Deep Learning? • Neural networks (computational architecture modeled after
the human brain) • Neural networks with many layers (> 1 hidden layer, but in practice, can be hundreds) • The vanishing/exploding gradient problem
Vanishing && Gradients
Image credit for all ConvNet images: https://deeplearning4j.org/convolutionalnets
Max Pooling/Downsampling
Alternating Layers
Our Data Image credit: http://digitalmedia.fws.gov/cdm/
What’s DL4J? • DL4J == Deep Learning 4 Java, a
library (for Java, unsurprisingly) • Examples on GitHub: https://github.com/ deeplearning4j/deeplearning4j • ConvNet worked example: http://bit.ly/2eBM8ss
DL4J Example (def nn-conf (-> (NeuralNetConfiguration$Builder.) ;; Some values omitted
for space (.activation "relu") (.learningRate 0.0001) (.weightInit (WeightInit/XAVIER)) (.optimizationAlgo OptimizationAlgorithm/STOCHASTIC_GRADIENT_DESCENT) (.updater Updater/RMSPROP) (.momentum 0.9) (.list) (.layer 0 conv-init) (.layer 1 (max-pool "maxpool1" (int-array [2 2]))) (.layer 2 (conv-5x5 "cnn2" 100 (int-array [5 5]) (int-array [1 1]) 0)) (.layer 3 (max-pool "maxpool2" (int-array [2 2]))) (.layer 4 (fully-connected 500)) (.layer 5 output-layer) (.build)))
How’d We Do? • Accuracy: 0.375 • Precision: 0.3333 •
Recall: 0.375 • F1 Score: 0.3529
Summary • Clojure + Spark = • Flambo and Sparkling
are roughly equally powerful • Deep learning is super doable with Clojure (though Java interop is kind of a pain)
Takeaways (TL;DPA) • Contribute to Flambo and/or Sparkling! • Let’s
build or contribute to a nicer DSL for DL4J • https://github.com/ericqweinstein/euroclojure
None