Alice and the Mad Hatter: Predict or not to predict

"Alice and Mad Hatter should solve a mystical riddle that will help them to find their way back home. Alice is a big data engineer and Mad Hatter is a skilled data scientist. In order to solve the riddle, they need to build a prediction using a machine learning model. Will knowledge of Scala help Alice to find a solution? And will she be able to collaborate with the Mad Hatter? You will find out in this talk."
Co-presented at Scale By The Bay 2021 - virtually with Marianna Diachuk

Transcript

1. Alice and the Mad Hatter: Predict or not to predict
2. Marianna Diachuk • Data Scientist at Restream • Women Who Code Kyiv Data Science Lead • Speaker, writer
3. Roksolana Diachuk • Big Data Developer at Captify • Diversity & Inclusion ambassador at Captify • Women Who Code Kyiv Data Engineering Lead • Speaker
4. In previous episodes talks…
5. Functional forest
6. magic-db- cluster-0
7. 2 years later
8. magic-db- cluster-0 Long time no see! !
9. NAME READY STATUS AGE launcher-crd 1/1 Running 33s magic-db- cluster-0
10. 3 months passed
11. { “predictions”: [ [ “B-natural phenomenon”, “O”, “B-geographical entity”, “B-time indicator”, ] ] }
12. { “instances”: [Pods and higher- order functions are in danger]} Test
13. 5 Days passed
15. Alice! Alice!
16. “Long time no see!” “Happy to see you too, Alice!”
17. “Sorry, Alice! We don’t have enough time for stories”
18. “I was just waiting for you!” “What for?”
19. “Oh, don’t be rude, Alice!
 I will only entertain you a bit with my riddle” “Like it explains anything”
20. “Soon it will all make sense” “I hope so”
21. What was lost can be found only in the anomalies of this world
22. “Any ideas?” “Oh yes,I have a pretty clear understanding of this riddle”
23. “Anomalies?” “We need to fi nd anomalies in the data”
24. “Anomaly is some kind of deviation from the standard”
25. Cat Not a cat Supervised learning Unsupervised learning
26. K-means
27. “Wait! You’re talking about ML models” “Exactly!”
28. Data ingestion Data analysis Data transformation Model evaluation and validation Serving ML model building
29. ML model Source data Anomalies detected
30. “How are we going to solve it together?”
31. Data ingestion Data analysis Data transformation Model evaluation and validation Serving ML model building
32. “Let’s solve it in Scala!”
33. def loadData(rawData: RDD[String]) : RDD[Vector] = { val dataAndLabel = rawData.map { line = > val buffer = ArrayBuffer[String]( ) val label = buffer.remove(buffer.length-1 ) val vector = Vectors.dense(buffer.map(_.toDouble).toArray) (vector, label ) … }
34. def loadData(rawData: RDD[String]) : RDD[Vector] = { … val data = dataAndLabel.map(_._1).cache( ) normalization(data ) }
35. “We also need to normalise the data” “What’s data normalisation?”
36. def normalization(dataArray: Array[Double]): RDD[Vector] = { val sums = dataArray.reduce((a, b) => a.zip(b).map(t => t._1 + t._2) ) val sumSquares = dataArray.fold(new Array[Double](numCols)) ( (a,b) => a.zip(b).map(t => t._1 + t._2 * t._2))
 val stdevs = sumSquares.zip(sums).map { cas e (sumSq, sum) => math.sqrt(n * sumSq - sum * sum) / count } va l val means = sums.map(_ / dataArray.count() ) … }
37. “Let’s work on the pipeline” “First part is done. What’s next?”
38. object AnomalyDetection { def main(args: Array[String]) { val sc = new SparkContext(new SparkCon fi g() ) val normalizedData = loadData(sc ) val model = trainModel(normalizedData ) val centroid = model.clusterCenters(0).toStrin g … }
39. object AnomalyDetection { def main(args: Array[String]) { … val distances = normalizedData.map(d => distToCentroid(d, model) ) val threshold = distances.top(2000).las t }
40. “Now we need to actually train a model”
41. def trainModel(normalizedData: RDD[Vector]): KMeansModel = { val kmeans = new KMeans( ) kmeans.setK(1)
 kmeans.setRuns(10)
 val model = kmeans.run(normalizedData ) mode l }
42. “Let’s dive deeper into distToCentroid function”
43. def distToCentroid(data: Vector, model: KMeansModel): Double = { val centroid = model.clusterCenters(model.predict(data) ) Vectors.sqdist(data, centroid ) }
44. “Ok, how are we going to run this job?” “We’ll use Kubernetes”
45. “Kubernetes is always a good idea”
46. apiserver scheduler Client Spark-submit Spark driver Executor 1 Executor 2 Executor 3
47. ./spark-submit \ --master k8s://https://host:port \ --deploy-mode cluster \ --name anomaly-detection \ --class anomaly.AnomalyDetection \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ --conf spark.kubernetes.container.image=spark:latest \ local:///spark/jars/anomaly-detection-3.0.1.jar
48. 01010100 01101000 01100101 00100000 01110000 01100001 01110011 …
49. “Oh, they are not random at all” “Those are just some random numbers”
50. “I’ll give you just a small hint. Those numbers are binary” “We can see that”
51. “I think I know how to…”
54. The past is the key to the future
55. To be continued…
58. Contact info dead_flowers22 roksolanad dark_matter_88 marianna-d