Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Writing Kubernetes Operators using Cats-Effect

Writing Kubernetes Operators using Cats-Effect

Use Scala to write a Kubernetes-Operator

7a04b88e1469561db6da3818348d4b8f?s=128

Alexey Novakov

June 18, 2020
Tweet

Transcript

  1. Writing Kubernetes Operator in Cats-Effect Alexey Novakov, Ultra Tendency

  2. ABOUT ME ➤ Solution Architect at Ultra Tendency, Germany ➤

    Experience: ➤ Scala - 5yrs. ➤ Java - 10 yrs. ➤ Big Data, Kubernetes ➤ My interests: ➤ 8 string guitars ➤ Astronomy ➤ FP ➤ Rust
  3. RISE OF OPERATORS (~LAST 3 YEARS) ➤ Spark Operator from

    Google ➤ Confluent Kafka Operator ➤ Strimzi Kafka Operator from RedHat ➤ Dozens of others (Postgres, Cassandra, Jenkins, Prometheus) ➤ Frameworks and libraries in ➤ Go ➤ Rust ➤ Java ➤ Scala ? Kubernetes v1.16 (sep 2019) # apiVersion: apiextensions.k8s.io/v1beta1 apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition
  4. KUBERNETES COMPONENTS https://kubernetes.io/docs/concepts/overview/components/ api-server

  5. OPERATOR PATTERN kube-api-server 1. create custom resource custom controller 2.

    watch custom resource 3. create native resource your application aka. Operator actual state desired state
  6. KUBERNETES RESOURCES apiVersion: io.kafka-ctl/v1 kind: KafkaCluster metadata: name: my-cluster spec:

    broker: 3 zookeeper: 3 security: sasl: realm: MY-ORG.COM service: kafka tls: CN: my-org.com apiVersion: v1 kind: Service metadata: labels: app: {{ .Values.KAFKA_NAME }} release: {{ .Values.KAFKA_NAME }} name: {{ .Values.KAFKA_NAME }} apiVersion: apps/v1 kind: StatefulSet metadata: labels: app: {{ .Values.KAFKA_NAME }} release: {{ .Values.KAFKA_NAME }} name: {{ .Values.KAFKA_NAME }} apiVersion: v1 kind: ConfigMap metadata: labels: app: {{ .Values.ZOOKEEPER_NAME } } release: {{ .Values.KAFKA_NAME }} apiVersion: extensions/v1beta1 kind: Ingress metadata: labels: app: {{ .Values.KAFKA_NAME }} … Custom Resource Native Resources x 6 x 2 x 3 x 3 + Secrets, etc.
  7. BUILDING YOUR OPERATOR ingredients

  8. MINIMUM SET CustomResourceDefinition manifest Structural Schema (OpenAPIv3) Kubernetes Client (REST)

    Operator manifests (deployment, RBAC) Kubernetes Cluster Operator Image Development Deployment YAML/JSON Scala library YAML/JSON Docker
  9. WORKFLOW custom controller 1.a. watch for CR updates (push) 2.

    parse JSON to model (case class) 3. CRUD necessary native resources 4.* update CR status 1.b. list current CRs (pull)
  10. USE CASE MIT Kerberos Operator: deployes KDC, Kadmin creates K8s

    Secrets: credentials, keytabs creates principals MIT Kerberos Operator + gets Input Result
  11. YAML SPEC apiVersion: io.github.novakov-alexey/v1 kind: Krb metadata: name: my-krb spec:

    realm: EXAMPLE.COM principals: - name: client1 password: type: static value: mypass keytab: cluster.keytab secret: type: Keytab name: cluster-keytab - name: user2 keytab: cluster.keytab secret: type: KeytabAndPassword name: cluster-keytab Creates KDC/Kadmin Pod, Service Creates principals, Secrets
  12. Implementation

  13. REQUIREMENTS ➤ Parse custom resources into Scala case class(es) ➤

    Make as functional & generic as possible ➤ Use Cats-Effect and IOApp to watch for resource CRUD events Scala code
  14. HIGH-LEVEL Event(type, T) F[U] K8s Client CRD Environment: Reconcile on

    interval watch Reconnect Plugable JSON Codecs T - Spec U - Status
  15. CRD SPEC: YAML -> SCALA final case class Krb(realm: String,

    principals: List[Principal]) sealed trait Password final case class Static(value: String) extends Password case object Random extends Password sealed trait Secret final case class Keytab(name: String) extends Secret final case class KeytabAndPassword(name: String) extends Secret final case class Principal( name: String, password: Password = Random, keytab: String, secret: Secret ) ADTs
  16. CRD STATUS: YAML -> SCALA final case class Status( processed:

    Boolean, lastPrincipalCount: Int, totalPrincipalCount: Int, error: String = "" ) kubectl describe krb <cr-name> Status: Error: Last Principal Count: 0 Processed: true Total Principal Count: 5 openAPIV3Schema: type: object properties: spec: type: object …… status: type: object ……
  17. KUBERNETES WATCH API GET /api/v1/namespaces/test/pods?watch=1&resourceVersion=10245 --- 200 OK Transfer-Encoding: chunked

    Content-Type: application/json { "type": "ADDED", "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "10596", ...}, ...} } { "type": "MODIFIED", "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "11020", ...}, ...} } { "type": "DELETED", ...
  18. CONTROLLER abstract class Controller[F[_]: Sync, T, U] { def onAdd(res:

    CustomResource[T, U]): F[NewStatus[U]] def onModify(res: CustomResource[T, U]): F[NewStatus[U]] def onDelete(res: CustomResource[T, U]): F[Unit] } type NewStatus[U] = Option[U] case class CustomResource[T, U]( meta: Metadata, spec: T, status: NewStatus[U] )
  19. CONSTRUCT OPERATOR class KrbController[F[_]: Sync](client: KubernetesClient) extends Controller[F, Krb, Status]

    { override def onAdd(r: CustomResource[Krb, Status]): F[NewStatus[Status]] = ??? override def onModify(r: CustomResource[Krb, Status]): F[NewStatus[Status]] = ??? override def onDelete(r: CustomResource[Krb, Status]): F[Unit] = ??? } val client = Sync[F].delay(new DefaultKubernetesClient) val controller = (c: KubernetesClient) => new KrbController(c) val cfg = CrdConfig(Namespace("test"), "io.github.novakov-alexey") val operator = Operator.ofCrd [F, Krb, Status](cfg, client, controller) Effect Spec Status
  20. CATS IO APP object Main extends IOApp { override def

    run(args: List[String]): IO[ExitCode] = operator.run } class Operator[F[_], T: Reader, U] private ( ….. // some parameters )(implicit F: ConcurrentEffect[F], T: Timer[F]) { def run: F[ExitCode] = ??? }
  21. K8s Client Challenges

  22. KUBERNETES CLIENT def getPod(client: KubernetesClient)( namespace: String, labelKey: String, labelValue:

    String ): Option[Pod] = client .pods() .inNamespace(namespace) .withLabel(labelKey, labelValue) .list() .getItems .asScala .find(p => Option(p.getMetadata.getDeletionTimestamp).isEmpty) "io.fabric8" % “kubernetes-client" % fabric8Version "io.fabric8" % "kubernetes-model" % fabric8Version // be ready for nulls and Exceptions
  23. FABRIC8 WATCH API crdClient.watch(new Watcher[AnyCustomResource]() { override def eventReceived( action:

    Watcher.Action, cr: AnyCustomResource): Unit = ??? override def onClose(e: KubernetesClientException): Unit = ??? }) val crd: CustomResourceDefinition = ??? val crdClient = client.customResources(crd, classOf[AnyCustomResource], classOf[AnyCrList], classOf[AnyCrDoneable] ).inAnyNamespace First problem to solve!
  24. DESERIALIZATION import io.fabric8.kubernetes.client.CustomResource class AnyCustomResource extends CustomResource { private var

    spec: AnyRef = _ private var status: AnyRef = _ def getSpec: AnyRef = spec def setSpec(spec: AnyRef): Unit = this.spec = spec def getStatus: AnyRef = status def setStatus(status: AnyRef): Unit = this.status = status } Second problem to solve!
  25. Push-API or void problem

  26. PUSH-API / UNIT TYPE PROBLEM def putAction( namespace: String, action:

    Either[OperatorError, WatcherAction[T, U]] ): F[Unit] = consumer.putAction(namespace, action) val cr: AnyCustomResource = ??? val converted: Either[OperatorError, CustomResource[T, U]] = convertCr(cr).leftMap[OperatorError] { case (t, resource) => ParseResourceError(action, t, resource) } putAction(namespace, cr) // F[Unit] :-( Can kubernetes-client evaluate the effect on its own? void watch { }
  27. CATS EFFECT TYPE CLASS def runSync[A](f: F[A]): A = F.toIO(f).unsafeRunSync()

    def putActionBlocking( namespace: String, action: Either[OperatorError, WatcherAction[T, U]] ): Unit = runSync(consumer.putAction(namespace, action)) implicit F: Effect[F] /** * Convert to an IO[A]. * * The law is that toIO(liftIO(ioa)) is the same as ioa */ def toIO[A](fa: F[A]): IO[A] = Effect.toIOFromRunAsync(fa)(this)
  28. JSON Deserialization problem

  29. JACKON-DATA-BIND import io.fabric8.kubernetes.client.CustomResource class AnyCustomResource extends CustomResource { private var

    spec: AnyRef = _ private var status: AnyRef = _ ….
  30. class StringPropertyDeserializer extends StdDeserializer[StringProperty]( classOf [StringProperty]) { override def deserialize(jp:

    JsonParser, ctxt: DeserializationContext): StringProperty = { val node = jp.getCodec.readTree[ObjectNode](jp) StringProperty(node.toString) }} DESERIALISE TO STRING @JsonDeserialize(using = classOf[StringPropertyDeserializer]) final case class StringProperty(value: String) class AnyCustomResource extends CustomResource { private var spec: StringProperty = _ … } Wrapper for spec and status properties reading to Jackson Object AST
  31. def parseProperty[T: JsonReader](property: StringProperty, name: String) = { val reader

    = implicitly [JsonReader[T]] val parsed = reader.fromString(property.value) } DESERIALISE TO [T] val cr: AnyCustomResource = ??? val spec = parseProperty[Krb](cr.getSpec, “spec") val status = parseProperty[Status](cr.getStatus, "status") Now use Scala Json library to parse to type T
  32. JSON PARSER API import io.circe.Decoder import io.circe.parser.decode trait JsonReader[T] {

    def fromString(json: String): Either[Throwable, T] } implicit def circeRead[T: Decoder]: JsonReader[T] = new JsonReader[T] { override def fromString(json: String): Either[Throwable, T] = decode[T](json) }
  33. CIRCE DECODER: SPEC trait Codecs { implicit val genConfig: Configuration

    = Configuration.default.withDiscriminator("type").withDefaults implicit val decodePassword: Decoder[Password] = List [Decoder[Password]]( Decoder[Static].widen, Decoder.const(Random).widen).reduceLeft(_.or(_)) implicit val decodeSecret: Decoder[Secret] = List [Decoder[Secret]]( Decoder[Keytab].widen, Decoder[KeytabAndPassword].widen).reduceLeft(_.or(_)) }
  34. Final API

  35. object KrbOperator extends IOApp with CirceCodecs { implicit val cs:

    ContextShift[IO] = contextShift override def run(args: List[String]): IO[ExitCode] = { … Operator .ofCrd [IO, Kerb, Status](cfg, client, controller) .withReconciler(60.seconds) .withRestart(Times(maxRetries = 3, delay = 1.seconds, multiplier = 2)) } }
  36. RECONCILER THREAD class Reconciler[F[_], T, U]( …. currentResources: F[Either[Throwable, ResourcesList[T,

    U]]] )(implicit F: Effect[F], T: Timer[F]) { def run: F[ReconcilerExitCode] = F.suspend { for { _ <- T.sleep(delay) r <- currentResources _ <- publish(r) ec <- run } yield ec }.recoverWith { case e => F.delay(logger.error("Failed in reconciling loop", e)) *> ExitCodes.ReconcileExitCode.pure[F] } loop
  37. RESTARTABLE OPERATOR def withRestart(retry: Retry = Infinite())(implicit T: Timer[F]): F[ExitCode]

    = run.flatMap(loop(_, retry)) calls loop when operator fails private def loop(ec: ExitCode, retry: Retry)(implicit T: Timer[F]): F[ExitCode] = { val (canRestart, delay, nextRetry, remaining) = ??? // logic to calculate current retry params if (canRestart) for { _ <- T.sleep(delay) r <- nextRetry code <- withRestart(r) } yield code else ec.pure[F] }
  38. https://novakov-alexey.github.io/freya/

  39. FREYA LIBRARY Core: "io.github.novakov-alexey" %% "freya-core" % “0.2.3" Circe: "io.github.novakov-alexey"

    %% "freya-circe" % “0.2.3" Jackson: "io.github.novakov-alexey" %% "freya-jackson" % “0.2.3"
  40. EVENT-DISPATCHING - Dynamic queue per namespace creation - Handling events

    concurrently preserving original order per namespace
  41. BLOCKING QUEUE class BlockingQueue[F[_]: Sync, A]( size: Int, queue: Ref[F,

    Queue[A]], signal: MVar[F, Unit] ) { def produce(a: A): F[Unit] = ??? } def consume(c: A => F[Boolean]): F[Unit] = ???
  42. CATS-EFFECT https://typelevel.org/cats-effect/concurrency/ref.html

  43. CATS-EFFECT https://typelevel.org/cats-effect/concurrency/mvar.html

  44. NAMESPACE EVENTS BUFFER class BlockingQueue[F[_]: Sync, A]( size: Int, queue:

    Ref[F, Queue[A]], signal: MVar[F, Unit] ) { def produce(a: A): F[Unit] = for { (added, length) <- queue.modify { q => if (q.length < capacity) (q.enqueue(a), true) else (q, false) } _ <- if (added) signal.tryPut(()).void else signal.put(()) *> produce(a) } yield () used by Event Watcher // notify consumer
  45. BUFFER NAMESPACE EVENTS def consume(f: A => F[Boolean]): F[Unit] =

    for { elem <- queue.modify( q => q.dequeueOption.map { case (a, q) => (q, a.some) }.getOrElse((q, None)) ) _ <- elem match { case Some(e) => f(e).flatMap(continue => signal.tryTake *> F.whenA(continue)(consume(c))) case _ => signal.take *> consume(c) } } yield () } used by Controller …
  46. OPERATOR CONFIGURATION CrdConfig( namespace = Namespace(“test"), eventQueueSize = 10, concurrentController

    = true, prefix = "io.myorg.kerboperator", customKind = Some("Kerberos"), deployCrd = true, shortNames = List("kr"), pluralName = "kerbs", additionalPrinterColumns = List( AdditionalPrinterColumn(name = "realm", columnType = "string", jsonPath = “realm") ) )
  47. CONFIGMAP OPERATOR override def run(args: List[String]): IO[ExitCode] = Operator .ofConfigMap[IO,

    Kerb]( cmCfg, client, controller ).run apiVersion: v1 kind: ConfigMap metadata: name: my-krb1 namespace: test labels: io.myorg.kerboperator/kind: Kerb data: config: | realm: EXAMPLE.COM principals: - name: client1 password: static value: mypass - name: user2 password: static value: mypass2
  48. EXAMPLE OPERATOR https://github.com/novakov-alexey/krb-operator For testing purposes: - SPNEGO authentication -

    Keytab as Secret creation
  49. PURE FUNCTIONAL CLIENT? https://github.com/joan38/kubernetes-client - based on http4s, cats-effect, circe

    - supports core APIs: Deployments, Pods, Services, etc. - was missing some K8s APIs, but not anymore ;-) Kubernetes Client for Scala
  50. Alexey Novakov email: - alexey.novakov at ultratendency.com - novakov.alex at

    gmail.com Blog: https://novakov-alexey.github.io/ https://medium.com/se-notes-by-alexey-novakov Code: https://github.com/novakov-alexey/freya Twitter: @alexey_novakov Thank you! Questions?