War stories of lighting a Spark in the Kubernetes sea

"Big data, Spark, Kubernetes - you may have heard these buzzwords far too many times. Have you ever thought of combining all of them? Running spark jobs on Kubernetes - sounds intriguing! But sailing the unknown see always goes with a challenge, and not even one. And who knows what is hiding deep in the water? If you are ready, let’s embark on this dangerous yet exciting journey."

Presented at OSDN (Kyiv, Ukraine) and Scala Berlin User Group meetup
Video recording from OSDN (Kyiv, Ukraine) : https://youtu.be/uQtjAUzglEY

Transcript

1. War stories of lighting a Spark in the Kubernetes sea
2. •Big data developer at Captif y •Diversity&Inclusion ambassador for Captify Kyiv offic e •Women Who Code Kyiv Data Engineering Lead and Mento r •Speaker and traveller Roksolana Diachuk
3. 01 02 03 How to run Spark on Kubernetes Research stage Project development stage Agenda 04 Conclusions
5. What big data engineers do?
8. Spark jobs cluster managers
10. Kubernetes objects landscape Deployment Pod StatefulSet ReplicaSet Volume DaemonSet Service Namespace Custom object
11. Kubernetes operator etcd Postgres Operator Postgres Deploymen t /StatefulSet Postgres Deploymen t /StatefulSet Custom automation for work fl ow actions State kubectl apply
12. Spark Kubernetes operator Controllers Submission runner Spark Pod Monitor Mutating Admission Webhook
13. Spark Kubernetes operator Controllers Submission runner Spark Pod Monitor Mutating Admission Webhook API Server / Scheduler kubectl spark-app.yaml Spark Application Object Spark Application Pod Events Driver Executor Executor
15. Volume 1. Research
17. Documentation
18. Yaml file launch Bash script launch Story №1
19. Yaml file launch Bash script launch Story №1
20. NAME READY STATUS RESTARTS AG E spark-driver 0/1 Pending 0 0 s spark-driver 0/1 Init:0/1 0 0 s spark-driver 0/1 Init:Error 0 3s Story №1
21. Problem Solution Remove restart policy Story №1. API updates Custom objects API update ¯\_(ツ)_/¯
22. apiVersion: sparkoperator.k8s.io/v1alpha1 kind: SparkApplicatio n metadata : name: spark-p i namespace: defaul t spec : type: Scal a image: gcr.io/ynli-k8s/spark:v2.4. 0 mainClass: org.apache.spark.examples.SparkP i mainApplicationFile: local:///tmp/jars/spark_example.jar mode: cluste r deps: { } Spark-app.yaml
23. driver : coreLimit: 1000 m cores: 0. 1 labels : version: 2.4. 0 memory: 1024 m serviceAccount: spar k executor : cores: 1 instances: 1 labels: version: 2.4. 0 memory: 1024 m imagePullPolicy: Neve r restartPolicy: Never Spark-app.yaml
24. Story №2 Docker images
25. NAME READY STATUS RESTARTS AG E sparkoperator 0/1 Pending 0 0 s sparkoperator 0/1 ContainerCreating 0 0 s sparkoperator 0/1 Error 0 5 s sparkoperator 0/1 CrashLoopBackOff 0 9s Story №2. Docker images
26. Problem Solution Docker image upgrade, images storage policies Story №2. Docker images Outdated Docker image version
27. Story №3. Library choice Spark Application Executor Executor Driver Integration tests
28. Story №3. Library choice
29. Problem Solution Limited library functionality More detailed research and discussion Story №3. Library choice
30. Story №3. Library choice
31. Story №4. Subresources
32. The server could not find the requested resource message at http://host:port/apis/ sparkoperator.k8s.io/v1alpha1/ namespaces/default/sparkapplications/ spark-example/status REST API call Story №4. Subresources
33. Problem Solution Spark operator bug Statuses added to spark-operator-crds.yaml Story №4. Subresources
34. apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata : name: sparkapplications.sparkoperator.k8s.io … subresources: status: {} spark-operator-crds.yaml
35. Lessons learned • Thorough research and results discussion • Read the documentation carefully • Community is everything
36. Volume 2. Project development
37. Project initiation
38. Story №1. Expertise
39. Problem Solution Lack of expertise with big data stack on k8s Constant discussions and team education Consequence CI/CD creation took months Story №1. Expertise
40. Story №2.Logging and monitoring
41. spec: driver: javaOptions: -Dlog4j.configuration = /path/to/log4j.properties executor: javaOptions: -Dlog4j.configuration = /path/to/log4j.properties Story №2.Logging and monitoring
42. Problem Solution log4j file configuration is not picked up Building a config map and mounting it into Spark custom object Story №2.Logging and monitoring
43. Story №3.Infrastructure support
44. Story №3.Infrastructure support Problem Solution Shared development cluster Agreements about the policies Consequence Data loss and constant infrastructure changes
45. Lessons learned • Keep in mind challenges while choosing the tech stack • Make sure there’s enough expertise for the project development • Communicate a lot
46. • Expertise development in the departmen t • 1 production-level project with missed deadline s • …lots of sleepless nights \_(ツ)_/ Results
47. Building big data infrastructures on top of Kubernetes is very challenging but do not give up, it is fun!   (may produce headaches and eye twitching)
48. Resources • Running Spark on Kubernetes documentation • Kubernetes documentatio n • K. Hightower. Kubernetes: Up & Runnin g • G. Kim. Project Phoenix
49. https://github.com/GoogleCloudPlatform/ spark-on-k8s-operator Spark-k8s operator repo
50. github.com/ kubernetes-client/java github.com/fabric8io/ kubernetes-client Java-k8s client Fabric8io
51. dead_flowers22 roksolana-d My contact info roksolanadiachuk roksolanad
52. Thank you for attention!