Alice and travelling back in time

"Alice found out that her friends from the world of pods and higher-order functions are in danger. The key to their happy future is their turbulent past. Alice is going to use the power of Delta Lake to explore the records of the past and find a way to save her friends."
Presented at Scala Love 2022 (virtual)

Transcript

1. Alice and travelling back in time
2. Roksolana Diachuk • Big Data Developer at Captify • Diversity & Inclusion ambassador at Captify • Women Who Code Kyiv Data Engineering Lead • Speaker
3. In previous episodes talks…
4. Functional forest
5. magic-db- cluster-0
6. 2 years later
7. magic-db- cluster-0 Long time no see! !
8. NAME READY STATUS AGE launcher-crd 1/1 Running 33s magic-db- cluster-0
9. 3 months passed
10. { “predictions”: [ [ “B-natural phenomenon”, “O”, “B-geographical entity”, “B-time indicator”, ] ] }
11. { “instances”: [Pods and higher- order functions are in danger]} Test
12. 5 Days passed
13. “Oh, don’t be rude, Alice!
 I will only entertain you a bit with my riddle” “Like it explains anything”
14. What was lost can be found only in the anomalies of this world
15. ML model Source data Anomalies detected
16. 01010100 01101000 01100101 00100000 01110000 01100001 01110011 …
18. The past is the key to the future
19. 1 Day LAter
20. Alice kept thinking of the note she got and the journey which led her to it. Was someone helping her? And what this note actually means?
21. “I need to check magic-db”
22. alice% _
23. alice% _ You’re now connected to the magic-db
24. magicdb event_date = 20140608 event_hour = 1402185600 event_hour = 1402196720 … event_date = 20140609 … event_date = 20211031
25. “Wait, there’re some tools that allow to look into data versions”
27. Cloud storage Delta lake storage layer Apache Spark services
28. Data fi les Metadata Storage layer
29. Ingestion tables Re fi ned tables Agg data store
30. ACID Atomicity Consistency Isolation Durability
31. Transactions issue in Spark Failure during write —> data loss Data update —> data is not consistent
32. ACID transactions Schema enforcement Metadata handling Upserts and deletes Time travel
33. Time travel “That’s exactly what I need”
34. magicdb event_date = 20140608 event_hour = 1402185600 event_hour = 1402196720 … event_date = 20140609 … _metadata
35. “Hm, there are already metadata fi les available”
36. magicdb _last_checkpoint 00000000000000005536 … _metadata 00000000000000005537 00000000000000005538 00000000000000005539
37. Delta lake setup libraryDependencies += “io.delta” %% “delta-core” % “1.1.0” val spark = SparkSession .builder()
 .con fi g(“spark.sql.extensions”, “io.delta.sql.DeltaSparkSessionExtension”)
 .con fi g(“spark.sql.catalog.spark_catalog”, “spark.sql.delta.catalog.DeltaCatalog”) .getOrCreate()
38. Timestamp Entity_name Entity_type Belongs to 2014-06-07 factory-worker-0 Pod StatefulSet 2014-06-07 factory-worker-1 Pod StatefulSet 2014-06-07 factory-worker-2 Pod StatefulSet … spark.read.format(“delta”).load(“/magic-db”)
39. “It’s possible to look directly into the history of the data!”
40. Version Timestamp User Operation Comment 1 2014-06-07 Architect CREATE Default name space city 2 2014-06-08 Architect CREATE Factory … val deltaTable = DeltaTable.forPath(spark, “/magic-db”) deltaTable.history()
41. 0. World creation
42. 1. Default namespace city creation
43. 2. Factory creation
44. 3. Forest discovery
45. 4. The fi rst human’s visit
46. 5. Pod lost NAME READY STATUS AGE pod/magic-db-cluster-0 1/1 Error 15m pod/magic-db-cluster-1 1/1 Running 15m pod/magic-db-cluster-2 1/1 Running 15m
47. spark.read .format(“delta”) .option(“versionAsOf”, “4”) .load(“/magic-db”) . fi lter($“entity_type” == “Pod”) .count Result: 10400
48. spark.read .format(“delta”) .option(“versionAsOf”, “5”) .load(“/magic-db”) . fi lter($“entity_type” == “Pod”) .count Result: 10399
49. 6. Anomalies PodLost PodLost PodLost PodLost PodLost PodLost
50. spark.read .format(“delta”) .option(“versionAsOf”, “6”) .load(“/magic-db”) . fi lter($“entity_type” == “Pod”) .count Result: 6130
51. “What happened to the pods?”
52. 7. Second human’s visit
53. 8. Factory failure
55. “Although there was this mysterious visitor…”
56. Alice was sitting at the cafe. Even her favourite cheesecake could not console her. She knew s h e h a d t o d o something to save the pods and higher-order functions.
57. “I can just write them back into the database”
58. val newPods = Seq( (“2021-11-01”, “magic-db-0”, “Pod”, “CRD”), (“2021-11-01”, “magic-db-1”, “Pod”, “CRD”), (“2021-11-01”, “magic-db-2”, “Pod”, “CRD”), ).toDF("timestamp", “entity_name”, “entity_type”, “belongs_to”)
59. newPods.repartition(1)
 .write .format(“delta”) .mode(SaveMode.Append) .save(“/magic-db”) spark.read.format(“delta”) .load(“/magic-db”)
 . fi lter($“entity_type” == “Pod”) .orderBy($“timestamp”.desc)
60. Timestamp Entity_name Entity_type Belongs to 2021-11-01 magic-db-0 Pod CRD 2021-11-01 magic-db-1 Pod CRD 2021-11-01 magic-db-2 Pod CRD …
61. val deltaTable = DeltaTable.forPath(“/magic-db”)
 deltaTable.vacuum(3)
62. val deltaTable = DeltaTable.forPath(“/magic-db”)
 deltaTable.vacuum(3)
63. Version Timestamp User Operation Comment 9 2021-11-01 Alice CREATE New pods created 10 2021-11-01 System DELETE All pods lost val deltaTable = DeltaTable.forPath(spark, “/magic-db”) deltaTable.history()
64. 10. All pods lost
65. spark.read .format(“delta”) .option(“versionAsOf”, “10”) .load(“/magic-db”)
 . fi lter($“entity_type” == “Pod”) .count Result: 0
66. “Hm, there was nothing about the functional forest”
67. spark.read .format(“delta”) .option(“versionAsOf”, “4”) .load(“/magic-db”)
 .select(“entity_type”).distinct Result: Pod, ReplicaSet, Deployment, CRD…
68. “What if the architect could help me?”
69. “I should fi nd a way to help them”
70. “I need to get back to the world of pods and higher-order functions”
71. To be continued…
72. Thank you for attention
73. My contact info dead_flowers22 roksolana-d roksolanadiachuk roksolanad