War stories of lighting a Spark in the Kubernetes sea

Slide 1

Slide 1 text

Slide 2

Slide 2 text

•Big data developer at Captif y •Diversity&Inclusion ambassador for Captify Kyiv offic e •Women Who Code Kyiv Data Engineering Lead and Mento r •Speaker and traveller Roksolana Diachuk

Slide 3

Slide 3 text

01 02 03 How to run Spark on Kubernetes Research stage Project development stage Agenda 04 Conclusions

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

What big data engineers do?

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

Spark jobs cluster managers

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

Kubernetes objects landscape Deployment Pod StatefulSet ReplicaSet Volume DaemonSet Service Namespace Custom object

Slide 11

Slide 11 text

Kubernetes operator etcd Postgres Operator Postgres Deploymen t /StatefulSet Postgres Deploymen t /StatefulSet Custom automation for work fl ow actions State kubectl apply

Slide 12

Slide 12 text

Spark Kubernetes operator Controllers Submission runner Spark Pod Monitor Mutating Admission Webhook

Slide 13

Slide 13 text

Spark Kubernetes operator Controllers Submission runner Spark Pod Monitor Mutating Admission Webhook API Server / Scheduler kubectl spark-app.yaml Spark Application Object Spark Application Pod Events Driver Executor Executor

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

Volume 1. Research

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

Documentation

Slide 18

Slide 18 text

Yaml file launch Bash script launch Story №1

Slide 19

Slide 19 text

Yaml file launch Bash script launch Story №1

Slide 20

Slide 20 text

NAME READY STATUS RESTARTS AG E spark-driver 0/1 Pending 0 0 s spark-driver 0/1 Init:0/1 0 0 s spark-driver 0/1 Init:Error 0 3s Story №1

Slide 21

Slide 21 text

Problem Solution Remove restart policy Story №1. API updates Custom objects API update ¯\_(ツ)_/¯

Slide 22

Slide 22 text

apiVersion: sparkoperator.k8s.io/v1alpha1 kind: SparkApplicatio n metadata : name: spark-p i namespace: defaul t spec : type: Scal a image: gcr.io/ynli-k8s/spark:v2.4. 0 mainClass: org.apache.spark.examples.SparkP i mainApplicationFile: local:///tmp/jars/spark_example.jar mode: cluste r deps: { } Spark-app.yaml

Slide 23

Slide 23 text

driver : coreLimit: 1000 m cores: 0. 1 labels : version: 2.4. 0 memory: 1024 m serviceAccount: spar k executor : cores: 1 instances: 1 labels: version: 2.4. 0 memory: 1024 m imagePullPolicy: Neve r restartPolicy: Never Spark-app.yaml

Slide 24

Slide 24 text

Story №2 Docker images

Slide 25

Slide 25 text

NAME READY STATUS RESTARTS AG E sparkoperator 0/1 Pending 0 0 s sparkoperator 0/1 ContainerCreating 0 0 s sparkoperator 0/1 Error 0 5 s sparkoperator 0/1 CrashLoopBackOff 0 9s Story №2. Docker images

Slide 26

Slide 26 text

Problem Solution Docker image upgrade, images storage policies Story №2. Docker images Outdated Docker image version

Slide 27

Slide 27 text

Story №3. Library choice Spark Application Executor Executor Driver Integration tests

Slide 28

Slide 28 text

Story №3. Library choice

Slide 29

Slide 29 text

Problem Solution Limited library functionality More detailed research and discussion Story №3. Library choice

Slide 30

Slide 30 text

Story №3. Library choice

Slide 31

Slide 31 text

Story №4. Subresources

Slide 32

Slide 32 text

The server could not find the requested resource message at http://host:port/apis/ sparkoperator.k8s.io/v1alpha1/ namespaces/default/sparkapplications/ spark-example/status REST API call Story №4. Subresources

Slide 33

Slide 33 text

Problem Solution Spark operator bug Statuses added to spark-operator-crds.yaml Story №4. Subresources

Slide 34

Slide 34 text

apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata : name: sparkapplications.sparkoperator.k8s.io … subresources: status: {} spark-operator-crds.yaml

Slide 35

Slide 35 text

Lessons learned • Thorough research and results discussion • Read the documentation carefully • Community is everything

Slide 36

Slide 36 text

Volume 2. Project development

Slide 37

Slide 37 text

Project initiation

Slide 38

Slide 38 text

Story №1. Expertise

Slide 39

Slide 39 text

Problem Solution Lack of expertise with big data stack on k8s Constant discussions and team education Consequence CI/CD creation took months Story №1. Expertise

Slide 40

Slide 40 text

Story №2.Logging and monitoring

Slide 41

Slide 41 text

spec: driver: javaOptions: -Dlog4j.configuration = /path/to/log4j.properties executor: javaOptions: -Dlog4j.configuration = /path/to/log4j.properties Story №2.Logging and monitoring

Slide 42

Slide 42 text

Problem Solution log4j file configuration is not picked up Building a config map and mounting it into Spark custom object Story №2.Logging and monitoring

Slide 43

Slide 43 text

Story №3.Infrastructure support

Slide 44

Slide 44 text

Story №3.Infrastructure support Problem Solution Shared development cluster Agreements about the policies Consequence Data loss and constant infrastructure changes

Slide 45

Slide 45 text

Lessons learned • Keep in mind challenges while choosing the tech stack • Make sure there’s enough expertise for the project development • Communicate a lot

Slide 46

Slide 46 text

• Expertise development in the departmen t • 1 production-level project with missed deadline s • …lots of sleepless nights \_(ツ)_/ Results

Slide 47

Slide 47 text

Building big data infrastructures on top of Kubernetes is very challenging but do not give up, it is fun! (may produce headaches and eye twitching)

Slide 48

Slide 48 text

Resources • Running Spark on Kubernetes documentation • Kubernetes documentatio n • K. Hightower. Kubernetes: Up & Runnin g • G. Kim. Project Phoenix

Slide 49

Slide 49 text

https://github.com/GoogleCloudPlatform/ spark-on-k8s-operator Spark-k8s operator repo

Slide 50

Slide 50 text

github.com/ kubernetes-client/java github.com/fabric8io/ kubernetes-client Java-k8s client Fabric8io