Writing Kubernetes Operators using Cats-Effect

Use Scala to write a Kubernetes-Operator

Transcript

1. Writing
   Kubernetes Operator
   in Cats-Effect
   Alexey Novakov, Ultra Tendency
   

   View Slide

2. ABOUT ME
   ➤ Solution Architect at Ultra Tendency,
   Germany
   ➤ Experience:
   ➤ Scala - 5yrs.
   ➤ Java - 10 yrs.
   ➤ Big Data, Kubernetes
   ➤ My interests:
   ➤ 8 string guitars
   ➤ Astronomy
   ➤ FP
   ➤ Rust
   

   View Slide

3. RISE OF OPERATORS (~LAST 3 YEARS)
   ➤ Spark Operator from Google
   ➤ Confluent Kafka Operator
   ➤ Strimzi Kafka Operator from RedHat
   ➤ Dozens of others (Postgres, Cassandra,
   Jenkins, Prometheus)
   ➤ Frameworks and libraries in
   ➤ Go
   ➤ Rust
   ➤ Java
   ➤ Scala ?
   Kubernetes v1.16 (sep 2019)
   # apiVersion: apiextensions.k8s.io/v1beta1
   apiVersion: apiextensions.k8s.io/v1
   kind: CustomResourceDefinition
   

   View Slide

4. KUBERNETES COMPONENTS
   https://kubernetes.io/docs/concepts/overview/components/
   api-server
   

   View Slide

5. OPERATOR PATTERN
   kube-api-server
   1. create custom
   resource
   custom controller
   2. watch custom
   resource
   3. create native
   resource
   your
   application aka.
   Operator
   actual state
   desired state
   

   View Slide

6. KUBERNETES RESOURCES
   apiVersion: io.kafka-ctl/v1
   kind: KafkaCluster
   metadata:
   name: my-cluster
   spec:
   broker: 3
   zookeeper: 3
   security:
   sasl:
   realm: MY-ORG.COM
   service: kafka
   tls:
   CN: my-org.com
   apiVersion: v1
   kind: Service
   metadata:
   labels:
   app:
   {{ .Values.KAFKA_NAME }}
   release:
   {{ .Values.KAFKA_NAME }}
   name:
   {{ .Values.KAFKA_NAME }}
   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
   labels:
   app:
   {{ .Values.KAFKA_NAME }}
   release:
   {{ .Values.KAFKA_NAME }}
   name:
   {{ .Values.KAFKA_NAME }}
   apiVersion: v1
   kind: ConfigMap
   metadata:
   labels:
   app:
   {{ .Values.ZOOKEEPER_NAME }
   }
   release:
   {{ .Values.KAFKA_NAME }}
   apiVersion: extensions/v1beta1
   kind: Ingress
   metadata:
   labels:
   app:
   {{ .Values.KAFKA_NAME }}
   …
   Custom Resource Native Resources
   x 6
   x 2
   x 3
   x 3
   + Secrets, etc.
   

   View Slide

7. BUILDING
   YOUR
   OPERATOR
   ingredients
   

   View Slide

8. MINIMUM SET
   CustomResourceDefinition
   manifest
   Structural Schema
   (OpenAPIv3)
   Kubernetes Client
   (REST)
   Operator manifests
   (deployment, RBAC)
   Kubernetes Cluster
   Operator Image
   Development Deployment
   YAML/JSON
   Scala library
   YAML/JSON
   Docker
   

   View Slide

9. WORKFLOW
   custom controller
   1.a. watch for CR updates (push)
   2. parse JSON to model (case class)
   3. CRUD necessary native resources
   4.* update CR status
   1.b. list current CRs (pull)
   

   View Slide

10. USE CASE
    MIT Kerberos Operator:
    deployes KDC, Kadmin
    creates K8s Secrets: credentials, keytabs
    creates principals
    MIT Kerberos Operator
    +
    gets
    Input
    Result
    

    View Slide

11. YAML SPEC
    apiVersion: io.github.novakov-alexey/v1
    kind: Krb
    metadata:
    name: my-krb
    spec:
    realm: EXAMPLE.COM
    principals:
    - name: client1
    password:
    type: static
    value: mypass
    keytab: cluster.keytab
    secret:
    type: Keytab
    name: cluster-keytab
    - name: user2
    keytab: cluster.keytab
    secret:
    type: KeytabAndPassword
    name: cluster-keytab
    Creates KDC/Kadmin
    Pod, Service
    Creates principals,
    Secrets
    

    View Slide

12. Implementation
    

    View Slide

13. REQUIREMENTS
    ➤ Parse custom resources
    into Scala case
    class(es)
    ➤ Make as functional &
    generic as possible
    ➤ Use Cats-Effect and
    IOApp to watch for
    resource CRUD events
    Scala code
    

    View Slide

14. HIGH-LEVEL
    Event(type, T) F[U]
    K8s
    Client
    CRD
    Environment:
    Reconcile on interval
    watch
    Reconnect Plugable JSON Codecs
    T - Spec
    U - Status
    

    View Slide

15. CRD SPEC: YAML -> SCALA
    final case class Krb(realm: String, principals: List[Principal])
    sealed trait Password
    final case class Static(value: String) extends Password
    case object Random extends Password
    sealed trait Secret
    final case class Keytab(name: String) extends Secret
    final case class KeytabAndPassword(name: String) extends Secret
    final case class Principal(
    name: String, password: Password = Random,
    keytab: String, secret: Secret
    )
    ADTs
    

    View Slide

16. CRD STATUS: YAML -> SCALA
    final case class Status(
    processed: Boolean,
    lastPrincipalCount: Int,
    totalPrincipalCount: Int,
    error: String = ""
    )
    kubectl describe krb
    Status:
    Error:
    Last Principal Count: 0
    Processed: true
    Total Principal Count: 5
    openAPIV3Schema:
    type: object
    properties:
    spec:
    type: object
    ……
    status:
    type: object
    ……
    

    View Slide

17. KUBERNETES WATCH API
    GET /api/v1/namespaces/test/pods?watch=1&resourceVersion=10245
    ---
    200 OK
    Transfer-Encoding: chunked
    Content-Type: application/json
    {
    "type": "ADDED",
    "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "10596",
    ...}, ...}
    }
    {
    "type": "MODIFIED",
    "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "11020",
    ...}, ...}
    }
    {
    "type": "DELETED",
    ...
    

    View Slide

18. CONTROLLER
    abstract class Controller[F[_]: Sync, T, U] {
    def onAdd(res: CustomResource[T, U]): F[NewStatus[U]]
    def onModify(res: CustomResource[T, U]): F[NewStatus[U]]
    def onDelete(res: CustomResource[T, U]): F[Unit]
    }
    type NewStatus[U] = Option[U]
    case class CustomResource[T, U](
    meta: Metadata, spec: T, status: NewStatus[U]
    )
    

    View Slide

19. CONSTRUCT OPERATOR
    class KrbController[F[_]: Sync](client: KubernetesClient)
    extends Controller[F, Krb, Status] {
    override def onAdd(r: CustomResource[Krb, Status]): F[NewStatus[Status]] = ???
    override def onModify(r: CustomResource[Krb, Status]): F[NewStatus[Status]] = ???
    override def onDelete(r: CustomResource[Krb, Status]): F[Unit] = ???
    }
    val client = Sync[F].delay(new DefaultKubernetesClient)
    val controller = (c: KubernetesClient) => new KrbController(c)
    val cfg = CrdConfig(Namespace("test"), "io.github.novakov-alexey")
    val operator = Operator.ofCrd [F, Krb, Status](cfg, client, controller)
    Effect Spec Status
    

    View Slide

20. CATS IO APP
    object Main extends IOApp {
    override def run(args: List[String]): IO[ExitCode] =
    operator.run
    }
    class Operator[F[_], T: Reader, U] private (
    ….. // some parameters
    )(implicit F: ConcurrentEffect[F], T: Timer[F]) {
    def run: F[ExitCode] = ???
    }
    

    View Slide

21. K8s Client
    Challenges
    

    View Slide

22. KUBERNETES CLIENT
    def getPod(client: KubernetesClient)(
    namespace: String, labelKey: String, labelValue: String
    ): Option[Pod] =
    client
    .pods()
    .inNamespace(namespace)
    .withLabel(labelKey, labelValue)
    .list()
    .getItems
    .asScala
    .find(p =>
    Option(p.getMetadata.getDeletionTimestamp).isEmpty)
    "io.fabric8" % “kubernetes-client" % fabric8Version
    "io.fabric8" % "kubernetes-model" % fabric8Version
    // be ready for nulls and Exceptions
    

    View Slide

23. FABRIC8 WATCH API
    crdClient.watch(new Watcher[AnyCustomResource]() {
    override def eventReceived(
    action: Watcher.Action, cr: AnyCustomResource): Unit = ???
    override def onClose(e: KubernetesClientException): Unit = ???
    })
    val crd: CustomResourceDefinition = ???
    val crdClient = client.customResources(crd,
    classOf[AnyCustomResource],
    classOf[AnyCrList],
    classOf[AnyCrDoneable]
    ).inAnyNamespace
    First problem
    to solve!
    

    View Slide

24. DESERIALIZATION
    import io.fabric8.kubernetes.client.CustomResource
    class AnyCustomResource extends CustomResource {
    private var spec: AnyRef = _
    private var status: AnyRef = _
    def getSpec: AnyRef = spec
    def setSpec(spec: AnyRef): Unit =
    this.spec = spec
    def getStatus: AnyRef = status
    def setStatus(status: AnyRef): Unit =
    this.status = status
    }
    Second problem
    to solve!
    

    View Slide

25. Push-API
    or void problem
    

    View Slide

26. PUSH-API / UNIT TYPE PROBLEM
    def putAction(
    namespace: String,
    action: Either[OperatorError, WatcherAction[T, U]]
    ): F[Unit] =
    consumer.putAction(namespace, action)
    val cr: AnyCustomResource = ???
    val converted: Either[OperatorError, CustomResource[T, U]] =
    convertCr(cr).leftMap[OperatorError] {
    case (t, resource) => ParseResourceError(action, t, resource)
    }
    putAction(namespace, cr) // F[Unit]
    :-( Can kubernetes-client evaluate the effect on its own?
    void watch {
    }
    

    View Slide

27. CATS EFFECT TYPE CLASS
    def runSync[A](f: F[A]): A =
    F.toIO(f).unsafeRunSync()
    def putActionBlocking(
    namespace: String,
    action: Either[OperatorError, WatcherAction[T, U]]
    ): Unit =
    runSync(consumer.putAction(namespace, action))
    implicit F: Effect[F]
    /**
    * Convert to an IO[A].
    *
    * The law is that toIO(liftIO(ioa)) is the same as ioa
    */
    def toIO[A](fa: F[A]): IO[A] =
    Effect.toIOFromRunAsync(fa)(this)
    

    View Slide

28. JSON
    Deserialization
    problem
    

    View Slide

29. JACKON-DATA-BIND
    import io.fabric8.kubernetes.client.CustomResource
    class AnyCustomResource extends CustomResource {
    private var spec: AnyRef = _
    private var status: AnyRef = _
    ….
    

    View Slide

30. class StringPropertyDeserializer extends StdDeserializer[StringProperty](
    classOf [StringProperty]) {
    override def deserialize(jp: JsonParser, ctxt: DeserializationContext): StringProperty = {
    val node = jp.getCodec.readTree[ObjectNode](jp)
    StringProperty(node.toString)
    }}
    DESERIALISE TO STRING
    @JsonDeserialize(using = classOf[StringPropertyDeserializer])
    final case class StringProperty(value: String)
    class AnyCustomResource extends CustomResource {
    private var spec: StringProperty = _
    …
    }
    Wrapper for spec and
    status properties
    reading to Jackson Object AST
    

    View Slide

31. def parseProperty[T: JsonReader](property: StringProperty, name: String) = {
    val reader = implicitly [JsonReader[T]]
    val parsed = reader.fromString(property.value)
    }
    DESERIALISE TO [T]
    val cr: AnyCustomResource = ???
    val spec = parseProperty[Krb](cr.getSpec, “spec")
    val status = parseProperty[Status](cr.getStatus, "status")
    Now use Scala Json library to parse to type T
    

    View Slide

32. JSON PARSER API
    import io.circe.Decoder
    import io.circe.parser.decode
    trait JsonReader[T] {
    def fromString(json: String): Either[Throwable, T]
    }
    implicit def circeRead[T: Decoder]: JsonReader[T] = new JsonReader[T] {
    override def fromString(json: String): Either[Throwable, T] =
    decode[T](json)
    }
    

    View Slide

33. CIRCE DECODER: SPEC
    trait Codecs {
    implicit val genConfig: Configuration =
    Configuration.default.withDiscriminator("type").withDefaults
    implicit val decodePassword: Decoder[Password] =
    List [Decoder[Password]](
    Decoder[Static].widen,
    Decoder.const(Random).widen).reduceLeft(_.or(_))
    implicit val decodeSecret: Decoder[Secret] =
    List [Decoder[Secret]](
    Decoder[Keytab].widen,
    Decoder[KeytabAndPassword].widen).reduceLeft(_.or(_))
    }
    

    View Slide

34. Final API
    

    View Slide

35. object KrbOperator extends IOApp with CirceCodecs {
    implicit val cs: ContextShift[IO] = contextShift
    override def run(args: List[String]): IO[ExitCode] = {
    …
    Operator
    .ofCrd [IO, Kerb, Status](cfg, client, controller)
    .withReconciler(60.seconds)
    .withRestart(Times(maxRetries = 3, delay = 1.seconds, multiplier = 2))
    }
    }
    

    View Slide

36. RECONCILER THREAD
    class Reconciler[F[_], T, U](
    ….
    currentResources: F[Either[Throwable, ResourcesList[T, U]]]
    )(implicit F: Effect[F], T: Timer[F]) {
    def run: F[ReconcilerExitCode] =
    F.suspend {
    for {
    _ <- T.sleep(delay)
    r <- currentResources
    _ <- publish(r)
    ec <- run
    } yield ec
    }.recoverWith {
    case e =>
    F.delay(logger.error("Failed in reconciling loop", e)) *> ExitCodes.ReconcileExitCode.pure[F]
    }
    loop
    

    View Slide

37. RESTARTABLE OPERATOR
    def withRestart(retry: Retry = Infinite())(implicit T: Timer[F]): F[ExitCode] =
    run.flatMap(loop(_, retry)) calls loop when operator fails
    private def loop(ec: ExitCode, retry: Retry)(implicit T: Timer[F]): F[ExitCode] = {
    val (canRestart, delay, nextRetry, remaining) = ??? // logic to calculate current retry params
    if (canRestart)
    for {
    _ <- T.sleep(delay)
    r <- nextRetry
    code <- withRestart(r)
    } yield code
    else ec.pure[F]
    }
    

    View Slide

38. https://novakov-alexey.github.io/freya/
    

    View Slide

39. FREYA LIBRARY
    Core:
    "io.github.novakov-alexey" %% "freya-core" % “0.2.3"
    Circe:
    "io.github.novakov-alexey" %% "freya-circe" % “0.2.3"
    Jackson:
    "io.github.novakov-alexey" %% "freya-jackson" % “0.2.3"
    

    View Slide

40. EVENT-DISPATCHING
    - Dynamic queue per
    namespace creation
    - Handling events
    concurrently
    preserving original
    order per namespace
    

    View Slide

41. BLOCKING QUEUE
    class BlockingQueue[F[_]: Sync, A](
    size: Int,
    queue: Ref[F, Queue[A]],
    signal: MVar[F, Unit]
    ) {
    def produce(a: A): F[Unit] = ???
    }
    def consume(c: A => F[Boolean]): F[Unit] = ???
    

    View Slide

42. CATS-EFFECT
    https://typelevel.org/cats-effect/concurrency/ref.html
    

    View Slide

43. CATS-EFFECT
    https://typelevel.org/cats-effect/concurrency/mvar.html
    

    View Slide

44. NAMESPACE EVENTS BUFFER
    class BlockingQueue[F[_]: Sync, A](
    size: Int,
    queue: Ref[F, Queue[A]],
    signal: MVar[F, Unit]
    ) {
    def produce(a: A): F[Unit] =
    for {
    (added, length) <- queue.modify { q =>
    if (q.length < capacity)
    (q.enqueue(a), true)
    else
    (q, false)
    }
    _ <- if (added) signal.tryPut(()).void
    else signal.put(()) *> produce(a)
    } yield ()
    used by Event Watcher
    // notify consumer
    

    View Slide

45. BUFFER NAMESPACE EVENTS
    def consume(f: A => F[Boolean]): F[Unit] =
    for {
    elem <- queue.modify(
    q =>
    q.dequeueOption.map {
    case (a, q) => (q, a.some)
    }.getOrElse((q, None))
    )
    _ <- elem match {
    case Some(e) => f(e).flatMap(continue => signal.tryTake *> F.whenA(continue)(consume(c)))
    case _ => signal.take *> consume(c)
    }
    } yield ()
    }
    used by Controller
    …
    

    View Slide

46. OPERATOR CONFIGURATION
    CrdConfig(
    namespace = Namespace(“test"),
    eventQueueSize = 10,
    concurrentController = true,
    prefix = "io.myorg.kerboperator",
    customKind = Some("Kerberos"),
    deployCrd = true,
    shortNames = List("kr"),
    pluralName = "kerbs",
    additionalPrinterColumns = List(
    AdditionalPrinterColumn(name = "realm", columnType = "string", jsonPath = “realm")
    )
    )
    

    View Slide

47. CONFIGMAP OPERATOR
    override def run(args: List[String]): IO[ExitCode] =
    Operator
    .ofConfigMap[IO, Kerb](
    cmCfg, client, controller
    ).run
    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: my-krb1
    namespace: test
    labels:
    io.myorg.kerboperator/kind: Kerb
    data:
    config: |
    realm: EXAMPLE.COM
    principals:
    - name: client1
    password: static
    value: mypass
    - name: user2
    password: static
    value: mypass2
    

    View Slide

48. EXAMPLE OPERATOR
    https://github.com/novakov-alexey/krb-operator
    For testing purposes:
    - SPNEGO authentication
    - Keytab as Secret creation
    

    View Slide

49. PURE FUNCTIONAL CLIENT?
    https://github.com/joan38/kubernetes-client
    - based on http4s, cats-effect, circe
    - supports core APIs: Deployments, Pods, Services, etc.
    - was missing some K8s APIs, but not anymore ;-)
    Kubernetes Client for Scala
    

    View Slide

50. Alexey Novakov
    email:
    - alexey.novakov at ultratendency.com
    - novakov.alex at gmail.com
    Blog:
    https://novakov-alexey.github.io/
    https://medium.com/se-notes-by-alexey-novakov
    Code: https://github.com/novakov-alexey/freya
    Twitter: @alexey_novakov
    Thank you! Questions?
    

    View Slide