Writing Kubernetes Operator in Cats-Effect Alexey Novakov, Ultra Tendency 

ABOUT ME ➤ Solution Architect at Ultra Tendency, Germany ➤ Experience: ➤ Scala - 5yrs. ➤ Java - 10 yrs. ➤ Big Data, Kubernetes ➤ My interests: ➤ 8 string guitars ➤ Astronomy ➤ FP ➤ Rust 

RISE OF OPERATORS (~LAST 3 YEARS) ➤ Spark Operator from Google ➤ Confluent Kafka Operator ➤ Strimzi Kafka Operator from RedHat ➤ Dozens of others (Postgres, Cassandra, Jenkins, Prometheus) ➤ Frameworks and libraries in ➤ Go ➤ Rust ➤ Java ➤ Scala ? Kubernetes v1.16 (sep 2019) # apiVersion: apiextensions.k8s.io/v1beta1 apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition 

KUBERNETES COMPONENTS https://kubernetes.io/docs/concepts/overview/components/ api-server 

OPERATOR PATTERN kube-api-server 1. create custom resource custom controller 2. watch custom resource 3. create native resource your application aka. Operator actual state desired state 

KUBERNETES RESOURCES apiVersion: io.kafka-ctl/v1 kind: KafkaCluster metadata: name: my-cluster spec: broker: 3 zookeeper: 3 security: sasl: realm: MY-ORG.COM service: kafka tls: CN: my-org.com apiVersion: v1 kind: Service metadata: labels: app: {{ .Values.KAFKA_NAME }} release: {{ .Values.KAFKA_NAME }} name: {{ .Values.KAFKA_NAME }} apiVersion: apps/v1 kind: StatefulSet metadata: labels: app: {{ .Values.KAFKA_NAME }} release: {{ .Values.KAFKA_NAME }} name: {{ .Values.KAFKA_NAME }} apiVersion: v1 kind: ConfigMap metadata: labels: app: {{ .Values.ZOOKEEPER_NAME } } release: {{ .Values.KAFKA_NAME }} apiVersion: extensions/v1beta1 kind: Ingress metadata: labels: app: {{ .Values.KAFKA_NAME }} … Custom Resource Native Resources x 6 x 2 x 3 x 3 + Secrets, etc. 

BUILDING YOUR OPERATOR ingredients 

MINIMUM SET CustomResourceDefinition manifest Structural Schema (OpenAPIv3) Kubernetes Client (REST) Operator manifests (deployment, RBAC) Kubernetes Cluster Operator Image Development Deployment YAML/JSON Scala library YAML/JSON Docker 

WORKFLOW custom controller 1.a. watch for CR updates (push) 2. parse JSON to model (case class) 3. CRUD necessary native resources 4.* update CR status 1.b. list current CRs (pull) 

USE CASE MIT Kerberos Operator: deployes KDC, Kadmin creates K8s Secrets: credentials, keytabs creates principals MIT Kerberos Operator + gets Input Result 

YAML SPEC apiVersion: io.github.novakov-alexey/v1 kind: Krb metadata: name: my-krb spec: realm: EXAMPLE.COM principals: - name: client1 password: type: static value: mypass keytab: cluster.keytab secret: type: Keytab name: cluster-keytab - name: user2 keytab: cluster.keytab secret: type: KeytabAndPassword name: cluster-keytab Creates KDC/Kadmin Pod, Service Creates principals, Secrets 

Implementation 

REQUIREMENTS ➤ Parse custom resources into Scala case class(es) ➤ Make as functional & generic as possible ➤ Use Cats-Effect and IOApp to watch for resource CRUD events Scala code 

HIGH-LEVEL Event(type, T) F[U] K8s Client CRD Environment: Reconcile on interval watch Reconnect Plugable JSON Codecs T - Spec U - Status 

CRD SPEC: YAML -> SCALA final case class Krb(realm: String, principals: List[Principal]) sealed trait Password final case class Static(value: String) extends Password case object Random extends Password sealed trait Secret final case class Keytab(name: String) extends Secret final case class KeytabAndPassword(name: String) extends Secret final case class Principal( name: String, password: Password = Random, keytab: String, secret: Secret ) ADTs 

CRD STATUS: YAML -> SCALA final case class Status( processed: Boolean, lastPrincipalCount: Int, totalPrincipalCount: Int, error: String = "" ) kubectl describe krb Status: Error: Last Principal Count: 0 Processed: true Total Principal Count: 5 openAPIV3Schema: type: object properties: spec: type: object …… status: type: object …… 

KUBERNETES WATCH API GET /api/v1/namespaces/test/pods?watch=1&resourceVersion=10245 --- 200 OK Transfer-Encoding: chunked Content-Type: application/json { "type": "ADDED", "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "10596", ...}, ...} } { "type": "MODIFIED", "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "11020", ...}, ...} } { "type": "DELETED", ... 

CONTROLLER abstract class Controller[F[_]: Sync, T, U] { def onAdd(res: CustomResource[T, U]): F[NewStatus[U]] def onModify(res: CustomResource[T, U]): F[NewStatus[U]] def onDelete(res: CustomResource[T, U]): F[Unit] } type NewStatus[U] = Option[U] case class CustomResource[T, U]( meta: Metadata, spec: T, status: NewStatus[U] ) 

CONSTRUCT OPERATOR class KrbController[F[_]: Sync](client: KubernetesClient) extends Controller[F, Krb, Status] { override def onAdd(r: CustomResource[Krb, Status]): F[NewStatus[Status]] = ??? override def onModify(r: CustomResource[Krb, Status]): F[NewStatus[Status]] = ??? override def onDelete(r: CustomResource[Krb, Status]): F[Unit] = ??? } val client = Sync[F].delay(new DefaultKubernetesClient) val controller = (c: KubernetesClient) => new KrbController(c) val cfg = CrdConfig(Namespace("test"), "io.github.novakov-alexey") val operator = Operator.ofCrd [F, Krb, Status](cfg, client, controller) Effect Spec Status 

CATS IO APP object Main extends IOApp { override def run(args: List[String]): IO[ExitCode] = operator.run } class Operator[F[_], T: Reader, U] private ( ….. // some parameters )(implicit F: ConcurrentEffect[F], T: Timer[F]) { def run: F[ExitCode] = ??? } 

K8s Client Challenges 

KUBERNETES CLIENT def getPod(client: KubernetesClient)( namespace: String, labelKey: String, labelValue: String ): Option[Pod] = client .pods() .inNamespace(namespace) .withLabel(labelKey, labelValue) .list() .getItems .asScala .find(p => Option(p.getMetadata.getDeletionTimestamp).isEmpty) "io.fabric8" % “kubernetes-client" % fabric8Version "io.fabric8" % "kubernetes-model" % fabric8Version // be ready for nulls and Exceptions 

FABRIC8 WATCH API crdClient.watch(new Watcher[AnyCustomResource]() { override def eventReceived( action: Watcher.Action, cr: AnyCustomResource): Unit = ??? override def onClose(e: KubernetesClientException): Unit = ??? }) val crd: CustomResourceDefinition = ??? val crdClient = client.customResources(crd, classOf[AnyCustomResource], classOf[AnyCrList], classOf[AnyCrDoneable] ).inAnyNamespace First problem to solve! 

DESERIALIZATION import io.fabric8.kubernetes.client.CustomResource class AnyCustomResource extends CustomResource { private var spec: AnyRef = _ private var status: AnyRef = _ def getSpec: AnyRef = spec def setSpec(spec: AnyRef): Unit = this.spec = spec def getStatus: AnyRef = status def setStatus(status: AnyRef): Unit = this.status = status } Second problem to solve! 

Push-API or void problem 

PUSH-API / UNIT TYPE PROBLEM def putAction( namespace: String, action: Either[OperatorError, WatcherAction[T, U]] ): F[Unit] = consumer.putAction(namespace, action) val cr: AnyCustomResource = ??? val converted: Either[OperatorError, CustomResource[T, U]] = convertCr(cr).leftMap[OperatorError] { case (t, resource) => ParseResourceError(action, t, resource) } putAction(namespace, cr) // F[Unit] :-( Can kubernetes-client evaluate the effect on its own? void watch { } 

CATS EFFECT TYPE CLASS def runSync[A](f: F[A]): A = F.toIO(f).unsafeRunSync() def putActionBlocking( namespace: String, action: Either[OperatorError, WatcherAction[T, U]] ): Unit = runSync(consumer.putAction(namespace, action)) implicit F: Effect[F] /** * Convert to an IO[A]. * * The law is that toIO(liftIO(ioa)) is the same as ioa */ def toIO[A](fa: F[A]): IO[A] = Effect.toIOFromRunAsync(fa)(this) 

JSON Deserialization problem 

JACKON-DATA-BIND import io.fabric8.kubernetes.client.CustomResource class AnyCustomResource extends CustomResource { private var spec: AnyRef = _ private var status: AnyRef = _ …. 

class StringPropertyDeserializer extends StdDeserializer[StringProperty]( classOf [StringProperty]) { override def deserialize(jp: JsonParser, ctxt: DeserializationContext): StringProperty = { val node = jp.getCodec.readTree[ObjectNode](jp) StringProperty(node.toString) }} DESERIALISE TO STRING @JsonDeserialize(using = classOf[StringPropertyDeserializer]) final case class StringProperty(value: String) class AnyCustomResource extends CustomResource { private var spec: StringProperty = _ … } Wrapper for spec and status properties reading to Jackson Object AST 

def parseProperty[T: JsonReader](property: StringProperty, name: String) = { val reader = implicitly [JsonReader[T]] val parsed = reader.fromString(property.value) } DESERIALISE TO [T] val cr: AnyCustomResource = ??? val spec = parseProperty[Krb](cr.getSpec, “spec") val status = parseProperty[Status](cr.getStatus, "status") Now use Scala Json library to parse to type T 

JSON PARSER API import io.circe.Decoder import io.circe.parser.decode trait JsonReader[T] { def fromString(json: String): Either[Throwable, T] } implicit def circeRead[T: Decoder]: JsonReader[T] = new JsonReader[T] { override def fromString(json: String): Either[Throwable, T] = decode[T](json) } 

CIRCE DECODER: SPEC trait Codecs { implicit val genConfig: Configuration = Configuration.default.withDiscriminator("type").withDefaults implicit val decodePassword: Decoder[Password] = List [Decoder[Password]]( Decoder[Static].widen, Decoder.const(Random).widen).reduceLeft(_.or(_)) implicit val decodeSecret: Decoder[Secret] = List [Decoder[Secret]]( Decoder[Keytab].widen, Decoder[KeytabAndPassword].widen).reduceLeft(_.or(_)) } 

Final API 

object KrbOperator extends IOApp with CirceCodecs { implicit val cs: ContextShift[IO] = contextShift override def run(args: List[String]): IO[ExitCode] = { … Operator .ofCrd [IO, Kerb, Status](cfg, client, controller) .withReconciler(60.seconds) .withRestart(Times(maxRetries = 3, delay = 1.seconds, multiplier = 2)) } } 

RECONCILER THREAD class Reconciler[F[_], T, U]( …. currentResources: F[Either[Throwable, ResourcesList[T, U]]] )(implicit F: Effect[F], T: Timer[F]) { def run: F[ReconcilerExitCode] = F.suspend { for { _ <- T.sleep(delay) r <- currentResources _ <- publish(r) ec <- run } yield ec }.recoverWith { case e => F.delay(logger.error("Failed in reconciling loop", e)) *> ExitCodes.ReconcileExitCode.pure[F] } loop 

RESTARTABLE OPERATOR def withRestart(retry: Retry = Infinite())(implicit T: Timer[F]): F[ExitCode] = run.flatMap(loop(_, retry)) calls loop when operator fails private def loop(ec: ExitCode, retry: Retry)(implicit T: Timer[F]): F[ExitCode] = { val (canRestart, delay, nextRetry, remaining) = ??? // logic to calculate current retry params if (canRestart) for { _ <- T.sleep(delay) r <- nextRetry code <- withRestart(r) } yield code else ec.pure[F] } 

https://novakov-alexey.github.io/freya/ 

FREYA LIBRARY Core: "io.github.novakov-alexey" %% "freya-core" % “0.2.3" Circe: "io.github.novakov-alexey" %% "freya-circe" % “0.2.3" Jackson: "io.github.novakov-alexey" %% "freya-jackson" % “0.2.3" 

EVENT-DISPATCHING - Dynamic queue per namespace creation - Handling events concurrently preserving original order per namespace 

BLOCKING QUEUE class BlockingQueue[F[_]: Sync, A]( size: Int, queue: Ref[F, Queue[A]], signal: MVar[F, Unit] ) { def produce(a: A): F[Unit] = ??? } def consume(c: A => F[Boolean]): F[Unit] = ??? 

CATS-EFFECT https://typelevel.org/cats-effect/concurrency/ref.html 

CATS-EFFECT https://typelevel.org/cats-effect/concurrency/mvar.html 

NAMESPACE EVENTS BUFFER class BlockingQueue[F[_]: Sync, A]( size: Int, queue: Ref[F, Queue[A]], signal: MVar[F, Unit] ) { def produce(a: A): F[Unit] = for { (added, length) <- queue.modify { q => if (q.length < capacity) (q.enqueue(a), true) else (q, false) } _ <- if (added) signal.tryPut(()).void else signal.put(()) *> produce(a) } yield () used by Event Watcher // notify consumer 

BUFFER NAMESPACE EVENTS def consume(f: A => F[Boolean]): F[Unit] = for { elem <- queue.modify( q => q.dequeueOption.map { case (a, q) => (q, a.some) }.getOrElse((q, None)) ) _ <- elem match { case Some(e) => f(e).flatMap(continue => signal.tryTake *> F.whenA(continue)(consume(c))) case _ => signal.take *> consume(c) } } yield () } used by Controller … 

OPERATOR CONFIGURATION CrdConfig( namespace = Namespace(“test"), eventQueueSize = 10, concurrentController = true, prefix = "io.myorg.kerboperator", customKind = Some("Kerberos"), deployCrd = true, shortNames = List("kr"), pluralName = "kerbs", additionalPrinterColumns = List( AdditionalPrinterColumn(name = "realm", columnType = "string", jsonPath = “realm") ) ) 

CONFIGMAP OPERATOR override def run(args: List[String]): IO[ExitCode] = Operator .ofConfigMap[IO, Kerb]( cmCfg, client, controller ).run apiVersion: v1 kind: ConfigMap metadata: name: my-krb1 namespace: test labels: io.myorg.kerboperator/kind: Kerb data: config: | realm: EXAMPLE.COM principals: - name: client1 password: static value: mypass - name: user2 password: static value: mypass2 

EXAMPLE OPERATOR https://github.com/novakov-alexey/krb-operator For testing purposes: - SPNEGO authentication - Keytab as Secret creation 

PURE FUNCTIONAL CLIENT? https://github.com/joan38/kubernetes-client - based on http4s, cats-effect, circe - supports core APIs: Deployments, Pods, Services, etc. - was missing some K8s APIs, but not anymore ;-) Kubernetes Client for Scala 

Alexey Novakov email: - alexey.novakov at ultratendency.com - novakov.alex at gmail.com Blog: https://novakov-alexey.github.io/ https://medium.com/se-notes-by-alexey-novakov Code: https://github.com/novakov-alexey/freya Twitter: @alexey_novakov Thank you! Questions?