Alice and the return to the world of pods and higher-order functions

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Roksolana Diachuk • Big Data Developer at Captify • Women Who Code Kyiv Data Engineering Lead • Member of Ukrainian Scala community • Speaker

Slide 3

Slide 3 text

In previous episodes talks…

Slide 4

Slide 4 text

Functional forest

Slide 5

Slide 5 text

magic-db- cluster-0

Slide 6

Slide 6 text

2 years later

Slide 7

Slide 7 text

NAME READY STATUS AGE launcher-crd 1/1 Running 33s magic-db- cluster-0

Slide 8

Slide 8 text

3 months passed

Slide 9

Slide 9 text

{ “predictions”: [ [ “B-natural phenomenon”, “O”, “B-geographical entity”, “B-time indicator”, ] ] }

Slide 10

Slide 10 text

{ “instances”: [Pods and higher- order functions are in danger]} Test

Slide 11

Slide 11 text

5 Days passed

Slide 12

Slide 12 text

“Oh, don’t be rude, Alice!  I will only entertain you a bit with my riddle” “Like it explains anything”

Slide 13

Slide 13 text

The past is the key to the future

Slide 14

Slide 14 text

1 Day LAter

Slide 15

Slide 15 text

Version Timestamp User Operation Comment 9 2021-11-01 Alice CREATE New pods created 10 2021-11-01 System DELETE All pods lost val deltaTable = DeltaTable.forPath(spark, “/magic-db”) deltaTable.history()

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

3 DayS LAter

Slide 18

Slide 18 text

Alice has been thinking of the ways to help the pods but nothing came up.She decided to distract herself from anxious thoughts by the work tasks.

Slide 19

Slide 19 text

anonym% There’s a way to help your friends Alice% What? How? anonym% You need to travel to their world Alice% But I don’t know how to get there anonym% There’s a passage close to you anonym% Follow my instructions Alice% But I don’t know how to get there anonym% You don’t Alice% How will I know that you’re telling the truth?

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

“What is that?”

Slide 22

Slide 22 text

“Oh my …”

Slide 23

Slide 23 text

kafka-1 kafka-0 “Hello, Alice!”

Slide 24

Slide 24 text

kafka-1 kafka-0 “What happened here?” “It’s a long story. Follow us”

Slide 25

Slide 25 text

A few months ago new moment got started psql-0 “Freedom to the pods!!” And then pods started to disappear

Slide 26

Slide 26 text

On one of the days someone set the factory on fi re Some pods found shelter in functional forest kafka-1 kafka-0

Slide 27

Slide 27 text

“And what can I do from here?” “Actually, a lot!” kafka-1 kafka-2

Slide 28

Slide 28 text

“Never used it, but I can fi gure it out” “You’ll need to use Zio-k8s” kafka-1 kafka-2

Slide 29

Slide 29 text

Code generator support for custom resources Client for the Kubernetes API Support for operators

Slide 30

Slide 30 text

Hand-written base code Generated code ZIO-k8s internals

Slide 31

Slide 31 text

libraryDependencies ++= Seq(   "com.coralogix" %% "zio-k8s-client" % “1.4.8”, "com.softwaremill.sttp.client3" %% "async-http-client-backend-zio" % “3.1.1", “com.softwaremill.sttp.client3" %% "slf4j-backend" % “3.1.1" ) ZIO-k8s setup “Looks easy”

Slide 32

Slide 32 text

case class Config(k8s: K8sClusterConfig) val configDesc = ConfigDescriptor.nested(“k8s") (clusterConfigDescriptor).to[Config] val config = TypesafeConfig.fromDefaultLoader[Config](configDesc) val pods = (client ++ cluster) >>> Pods.live

Slide 33

Slide 33 text

def getPod(namespace: String, name: String): IO[ContextInfoFailure, Pod] = { for { ns <- namespace maybePodName <- system .env(name) .mapError(reason => ContextInfoFailure.PodNameMissing(Some(reason))) … } } yield result }

Slide 34

Slide 34 text

…. result <- maybePodName match { case Some(podName) => pods .get(podName, ns) case None => ZIO.fail(ContextInfoFailure.PodNameMissing(None)) } } yield result }

Slide 35

Slide 35 text

getPod("default", “magic-db-cluster-1") “Let’s get this pod!”

Slide 36

Slide 36 text

“But where is this pod?” “We think they’re in another namespace” kafka-1 kafka-2

Slide 37

Slide 37 text

getPod(“pods-freedom“, "magic-db-cluster-1") “Hmm, what about this namespace?”

Slide 38

Slide 38 text

getPod(“pods-freedom“, "magic-db-cluster-1") NAME READY STATUS AGE pod/magic-db-cluster-1 1/1 Running 2y10d

Slide 39

Slide 39 text

private def tailLogs(podName: String, containerName: Option[String]): ZIO[Pods with Console, K8sFailure, Unit] = pods .getLog(podName, K8sNamespace.pods-freedom, containerName, follow = Some(true)) .tap { line => console.putStrLn(line).ignore } .runDrain

Slide 40

Slide 40 text

tailLogs(“magic-db-cluster-1", Some("magic-db")) 5 Nov 17:23:41 * ERROR! Can’t fi nd data storage location  5 Nov 17:25:13 * ERROR! Can’t fi nd data storage location 5 Nov 17:28:03 * ERROR! Can’t fi nd data storage location 5 Nov 17:31:22 * ERROR! Can’t fi nd data storage location

Slide 41

Slide 41 text

“What if we can move this pod back?”

Slide 42

Slide 42 text

val updatedPod = magicDbPod .modify(_.metadata.each.namespace).setTo(“default") println(updatedPod)

Slide 43

Slide 43 text

Code generation! Les boilerplate code Access control

Slide 44

Slide 44 text

“Then let’s build something to move the pod” “???” kafka-1 kafka-2

Slide 45

Slide 45 text

Custom object Service Namespace Volume Deployment StatefulSet ReplicaSet DaemonSet Pod

Slide 46

Slide 46 text

apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: nstrains.transport namespace: default spec: group: transport versions: - name: v1 nstrain.yaml

Slide 47

Slide 47 text

properties: spec: properties: image: magic-train replicas: 2 scope: Namespaced names: plural: nstrains singular: nstrain kind: NsTrain nstrain.yaml

Slide 48

Slide 48 text

//sbt.plugins addSbtPlugin("com.coralogix" %% "zio-k8s-crd" % "1.4.8") //build.sbt externalCustomResourceDefinitions := Seq(file("crds/nstrain.yaml")) enablePlugins(K8sCustomResourceCodegenPlugin) CRD setup

Slide 49

Slide 49 text

import com.coralogix.zio.k8s.client.com.example.stable.v1.nstrain nstrain.get("namespace-train", K8sNamespace.pods-freedom) “This should work!”

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

“What’s happening?” kafka-1 kafka-2 “Some kind of blackout”

Slide 52

Slide 52 text

import com.coralogix.zio.k8s.client.com.example.stable.v1.nstrain nstrain.get("namespace-train", K8sNamespace.pods-freedom)