Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Writing Kubernetes Operators using Cats-Effect

Writing Kubernetes Operators using Cats-Effect

Use Scala to write a Kubernetes-Operator

Alexey Novakov

June 18, 2020
Tweet

More Decks by Alexey Novakov

Other Decks in Programming

Transcript

  1. Writing
    Kubernetes Operator
    in Cats-Effect
    Alexey Novakov, Ultra Tendency

    View full-size slide

  2. ABOUT ME
    ➤ Solution Architect at Ultra Tendency,
    Germany
    ➤ Experience:
    ➤ Scala - 5yrs.
    ➤ Java - 10 yrs.
    ➤ Big Data, Kubernetes
    ➤ My interests:
    ➤ 8 string guitars
    ➤ Astronomy
    ➤ FP
    ➤ Rust

    View full-size slide

  3. RISE OF OPERATORS (~LAST 3 YEARS)
    ➤ Spark Operator from Google
    ➤ Confluent Kafka Operator
    ➤ Strimzi Kafka Operator from RedHat
    ➤ Dozens of others (Postgres, Cassandra,
    Jenkins, Prometheus)
    ➤ Frameworks and libraries in
    ➤ Go
    ➤ Rust
    ➤ Java
    ➤ Scala ?
    Kubernetes v1.16 (sep 2019)
    # apiVersion: apiextensions.k8s.io/v1beta1
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition

    View full-size slide

  4. KUBERNETES COMPONENTS
    https://kubernetes.io/docs/concepts/overview/components/
    api-server

    View full-size slide

  5. OPERATOR PATTERN
    kube-api-server
    1. create custom
    resource
    custom controller
    2. watch custom
    resource
    3. create native
    resource
    your
    application aka.
    Operator
    actual state
    desired state

    View full-size slide

  6. KUBERNETES RESOURCES
    apiVersion: io.kafka-ctl/v1
    kind: KafkaCluster
    metadata:
    name: my-cluster
    spec:
    broker: 3
    zookeeper: 3
    security:
    sasl:
    realm: MY-ORG.COM
    service: kafka
    tls:
    CN: my-org.com
    apiVersion: v1
    kind: Service
    metadata:
    labels:
    app:
    {{ .Values.KAFKA_NAME }}
    release:
    {{ .Values.KAFKA_NAME }}
    name:
    {{ .Values.KAFKA_NAME }}
    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
    labels:
    app:
    {{ .Values.KAFKA_NAME }}
    release:
    {{ .Values.KAFKA_NAME }}
    name:
    {{ .Values.KAFKA_NAME }}
    apiVersion: v1
    kind: ConfigMap
    metadata:
    labels:
    app:
    {{ .Values.ZOOKEEPER_NAME }
    }
    release:
    {{ .Values.KAFKA_NAME }}
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
    labels:
    app:
    {{ .Values.KAFKA_NAME }}

    Custom Resource Native Resources
    x 6
    x 2
    x 3
    x 3
    + Secrets, etc.

    View full-size slide

  7. BUILDING
    YOUR
    OPERATOR
    ingredients

    View full-size slide

  8. MINIMUM SET
    CustomResourceDefinition
    manifest
    Structural Schema
    (OpenAPIv3)
    Kubernetes Client
    (REST)
    Operator manifests
    (deployment, RBAC)
    Kubernetes Cluster
    Operator Image
    Development Deployment
    YAML/JSON
    Scala library
    YAML/JSON
    Docker

    View full-size slide

  9. WORKFLOW
    custom controller
    1.a. watch for CR updates (push)
    2. parse JSON to model (case class)
    3. CRUD necessary native resources
    4.* update CR status
    1.b. list current CRs (pull)

    View full-size slide

  10. USE CASE
    MIT Kerberos Operator:
    deployes KDC, Kadmin
    creates K8s Secrets: credentials, keytabs
    creates principals
    MIT Kerberos Operator
    +
    gets
    Input
    Result

    View full-size slide

  11. YAML SPEC
    apiVersion: io.github.novakov-alexey/v1
    kind: Krb
    metadata:
    name: my-krb
    spec:
    realm: EXAMPLE.COM
    principals:
    - name: client1
    password:
    type: static
    value: mypass
    keytab: cluster.keytab
    secret:
    type: Keytab
    name: cluster-keytab
    - name: user2
    keytab: cluster.keytab
    secret:
    type: KeytabAndPassword
    name: cluster-keytab
    Creates KDC/Kadmin
    Pod, Service
    Creates principals,
    Secrets

    View full-size slide

  12. Implementation

    View full-size slide

  13. REQUIREMENTS
    ➤ Parse custom resources
    into Scala case
    class(es)
    ➤ Make as functional &
    generic as possible
    ➤ Use Cats-Effect and
    IOApp to watch for
    resource CRUD events
    Scala code

    View full-size slide

  14. HIGH-LEVEL
    Event(type, T) F[U]
    K8s
    Client
    CRD
    Environment:
    Reconcile on interval
    watch
    Reconnect Plugable JSON Codecs
    T - Spec
    U - Status

    View full-size slide

  15. CRD SPEC: YAML -> SCALA
    final case class Krb(realm: String, principals: List[Principal])
    sealed trait Password
    final case class Static(value: String) extends Password
    case object Random extends Password
    sealed trait Secret
    final case class Keytab(name: String) extends Secret
    final case class KeytabAndPassword(name: String) extends Secret
    final case class Principal(
    name: String, password: Password = Random,
    keytab: String, secret: Secret
    )
    ADTs

    View full-size slide

  16. CRD STATUS: YAML -> SCALA
    final case class Status(
    processed: Boolean,
    lastPrincipalCount: Int,
    totalPrincipalCount: Int,
    error: String = ""
    )
    kubectl describe krb
    Status:
    Error:
    Last Principal Count: 0
    Processed: true
    Total Principal Count: 5
    openAPIV3Schema:
    type: object
    properties:
    spec:
    type: object
    ……
    status:
    type: object
    ……

    View full-size slide

  17. KUBERNETES WATCH API
    GET /api/v1/namespaces/test/pods?watch=1&resourceVersion=10245
    ---
    200 OK
    Transfer-Encoding: chunked
    Content-Type: application/json
    {
    "type": "ADDED",
    "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "10596",
    ...}, ...}
    }
    {
    "type": "MODIFIED",
    "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "11020",
    ...}, ...}
    }
    {
    "type": "DELETED",
    ...

    View full-size slide

  18. CONTROLLER
    abstract class Controller[F[_]: Sync, T, U] {
    def onAdd(res: CustomResource[T, U]): F[NewStatus[U]]
    def onModify(res: CustomResource[T, U]): F[NewStatus[U]]
    def onDelete(res: CustomResource[T, U]): F[Unit]
    }
    type NewStatus[U] = Option[U]
    case class CustomResource[T, U](
    meta: Metadata, spec: T, status: NewStatus[U]
    )

    View full-size slide

  19. CONSTRUCT OPERATOR
    class KrbController[F[_]: Sync](client: KubernetesClient)
    extends Controller[F, Krb, Status] {
    override def onAdd(r: CustomResource[Krb, Status]): F[NewStatus[Status]] = ???
    override def onModify(r: CustomResource[Krb, Status]): F[NewStatus[Status]] = ???
    override def onDelete(r: CustomResource[Krb, Status]): F[Unit] = ???
    }
    val client = Sync[F].delay(new DefaultKubernetesClient)
    val controller = (c: KubernetesClient) => new KrbController(c)
    val cfg = CrdConfig(Namespace("test"), "io.github.novakov-alexey")
    val operator = Operator.ofCrd [F, Krb, Status](cfg, client, controller)
    Effect Spec Status

    View full-size slide

  20. CATS IO APP
    object Main extends IOApp {
    override def run(args: List[String]): IO[ExitCode] =
    operator.run
    }
    class Operator[F[_], T: Reader, U] private (
    ….. // some parameters
    )(implicit F: ConcurrentEffect[F], T: Timer[F]) {
    def run: F[ExitCode] = ???
    }

    View full-size slide

  21. K8s Client
    Challenges

    View full-size slide

  22. KUBERNETES CLIENT
    def getPod(client: KubernetesClient)(
    namespace: String, labelKey: String, labelValue: String
    ): Option[Pod] =
    client
    .pods()
    .inNamespace(namespace)
    .withLabel(labelKey, labelValue)
    .list()
    .getItems
    .asScala
    .find(p =>
    Option(p.getMetadata.getDeletionTimestamp).isEmpty)
    "io.fabric8" % “kubernetes-client" % fabric8Version
    "io.fabric8" % "kubernetes-model" % fabric8Version
    // be ready for nulls and Exceptions

    View full-size slide

  23. FABRIC8 WATCH API
    crdClient.watch(new Watcher[AnyCustomResource]() {
    override def eventReceived(
    action: Watcher.Action, cr: AnyCustomResource): Unit = ???
    override def onClose(e: KubernetesClientException): Unit = ???
    })
    val crd: CustomResourceDefinition = ???
    val crdClient = client.customResources(crd,
    classOf[AnyCustomResource],
    classOf[AnyCrList],
    classOf[AnyCrDoneable]
    ).inAnyNamespace
    First problem
    to solve!

    View full-size slide

  24. DESERIALIZATION
    import io.fabric8.kubernetes.client.CustomResource
    class AnyCustomResource extends CustomResource {
    private var spec: AnyRef = _
    private var status: AnyRef = _
    def getSpec: AnyRef = spec
    def setSpec(spec: AnyRef): Unit =
    this.spec = spec
    def getStatus: AnyRef = status
    def setStatus(status: AnyRef): Unit =
    this.status = status
    }
    Second problem
    to solve!

    View full-size slide

  25. Push-API
    or void problem

    View full-size slide

  26. PUSH-API / UNIT TYPE PROBLEM
    def putAction(
    namespace: String,
    action: Either[OperatorError, WatcherAction[T, U]]
    ): F[Unit] =
    consumer.putAction(namespace, action)
    val cr: AnyCustomResource = ???
    val converted: Either[OperatorError, CustomResource[T, U]] =
    convertCr(cr).leftMap[OperatorError] {
    case (t, resource) => ParseResourceError(action, t, resource)
    }
    putAction(namespace, cr) // F[Unit]
    :-( Can kubernetes-client evaluate the effect on its own?
    void watch {
    }

    View full-size slide

  27. CATS EFFECT TYPE CLASS
    def runSync[A](f: F[A]): A =
    F.toIO(f).unsafeRunSync()
    def putActionBlocking(
    namespace: String,
    action: Either[OperatorError, WatcherAction[T, U]]
    ): Unit =
    runSync(consumer.putAction(namespace, action))
    implicit F: Effect[F]
    /**
    * Convert to an IO[A].
    *
    * The law is that toIO(liftIO(ioa)) is the same as ioa
    */
    def toIO[A](fa: F[A]): IO[A] =
    Effect.toIOFromRunAsync(fa)(this)

    View full-size slide

  28. JSON
    Deserialization
    problem

    View full-size slide

  29. JACKON-DATA-BIND
    import io.fabric8.kubernetes.client.CustomResource
    class AnyCustomResource extends CustomResource {
    private var spec: AnyRef = _
    private var status: AnyRef = _
    ….

    View full-size slide

  30. class StringPropertyDeserializer extends StdDeserializer[StringProperty](
    classOf [StringProperty]) {
    override def deserialize(jp: JsonParser, ctxt: DeserializationContext): StringProperty = {
    val node = jp.getCodec.readTree[ObjectNode](jp)
    StringProperty(node.toString)
    }}
    DESERIALISE TO STRING
    @JsonDeserialize(using = classOf[StringPropertyDeserializer])
    final case class StringProperty(value: String)
    class AnyCustomResource extends CustomResource {
    private var spec: StringProperty = _

    }
    Wrapper for spec and
    status properties
    reading to Jackson Object AST

    View full-size slide

  31. def parseProperty[T: JsonReader](property: StringProperty, name: String) = {
    val reader = implicitly [JsonReader[T]]
    val parsed = reader.fromString(property.value)
    }
    DESERIALISE TO [T]
    val cr: AnyCustomResource = ???
    val spec = parseProperty[Krb](cr.getSpec, “spec")
    val status = parseProperty[Status](cr.getStatus, "status")
    Now use Scala Json library to parse to type T

    View full-size slide

  32. JSON PARSER API
    import io.circe.Decoder
    import io.circe.parser.decode
    trait JsonReader[T] {
    def fromString(json: String): Either[Throwable, T]
    }
    implicit def circeRead[T: Decoder]: JsonReader[T] = new JsonReader[T] {
    override def fromString(json: String): Either[Throwable, T] =
    decode[T](json)
    }

    View full-size slide

  33. CIRCE DECODER: SPEC
    trait Codecs {
    implicit val genConfig: Configuration =
    Configuration.default.withDiscriminator("type").withDefaults
    implicit val decodePassword: Decoder[Password] =
    List [Decoder[Password]](
    Decoder[Static].widen,
    Decoder.const(Random).widen).reduceLeft(_.or(_))
    implicit val decodeSecret: Decoder[Secret] =
    List [Decoder[Secret]](
    Decoder[Keytab].widen,
    Decoder[KeytabAndPassword].widen).reduceLeft(_.or(_))
    }

    View full-size slide

  34. object KrbOperator extends IOApp with CirceCodecs {
    implicit val cs: ContextShift[IO] = contextShift
    override def run(args: List[String]): IO[ExitCode] = {

    Operator
    .ofCrd [IO, Kerb, Status](cfg, client, controller)
    .withReconciler(60.seconds)
    .withRestart(Times(maxRetries = 3, delay = 1.seconds, multiplier = 2))
    }
    }

    View full-size slide

  35. RECONCILER THREAD
    class Reconciler[F[_], T, U](
    ….
    currentResources: F[Either[Throwable, ResourcesList[T, U]]]
    )(implicit F: Effect[F], T: Timer[F]) {
    def run: F[ReconcilerExitCode] =
    F.suspend {
    for {
    _ <- T.sleep(delay)
    r <- currentResources
    _ <- publish(r)
    ec <- run
    } yield ec
    }.recoverWith {
    case e =>
    F.delay(logger.error("Failed in reconciling loop", e)) *> ExitCodes.ReconcileExitCode.pure[F]
    }
    loop

    View full-size slide

  36. RESTARTABLE OPERATOR
    def withRestart(retry: Retry = Infinite())(implicit T: Timer[F]): F[ExitCode] =
    run.flatMap(loop(_, retry)) calls loop when operator fails
    private def loop(ec: ExitCode, retry: Retry)(implicit T: Timer[F]): F[ExitCode] = {
    val (canRestart, delay, nextRetry, remaining) = ??? // logic to calculate current retry params
    if (canRestart)
    for {
    _ <- T.sleep(delay)
    r <- nextRetry
    code <- withRestart(r)
    } yield code
    else ec.pure[F]
    }

    View full-size slide

  37. https://novakov-alexey.github.io/freya/

    View full-size slide

  38. FREYA LIBRARY
    Core:
    "io.github.novakov-alexey" %% "freya-core" % “0.2.3"
    Circe:
    "io.github.novakov-alexey" %% "freya-circe" % “0.2.3"
    Jackson:
    "io.github.novakov-alexey" %% "freya-jackson" % “0.2.3"

    View full-size slide

  39. EVENT-DISPATCHING
    - Dynamic queue per
    namespace creation
    - Handling events
    concurrently
    preserving original
    order per namespace

    View full-size slide

  40. BLOCKING QUEUE
    class BlockingQueue[F[_]: Sync, A](
    size: Int,
    queue: Ref[F, Queue[A]],
    signal: MVar[F, Unit]
    ) {
    def produce(a: A): F[Unit] = ???
    }
    def consume(c: A => F[Boolean]): F[Unit] = ???

    View full-size slide

  41. CATS-EFFECT
    https://typelevel.org/cats-effect/concurrency/ref.html

    View full-size slide

  42. CATS-EFFECT
    https://typelevel.org/cats-effect/concurrency/mvar.html

    View full-size slide

  43. NAMESPACE EVENTS BUFFER
    class BlockingQueue[F[_]: Sync, A](
    size: Int,
    queue: Ref[F, Queue[A]],
    signal: MVar[F, Unit]
    ) {
    def produce(a: A): F[Unit] =
    for {
    (added, length) <- queue.modify { q =>
    if (q.length < capacity)
    (q.enqueue(a), true)
    else
    (q, false)
    }
    _ <- if (added) signal.tryPut(()).void
    else signal.put(()) *> produce(a)
    } yield ()
    used by Event Watcher
    // notify consumer

    View full-size slide

  44. BUFFER NAMESPACE EVENTS
    def consume(f: A => F[Boolean]): F[Unit] =
    for {
    elem <- queue.modify(
    q =>
    q.dequeueOption.map {
    case (a, q) => (q, a.some)
    }.getOrElse((q, None))
    )
    _ <- elem match {
    case Some(e) => f(e).flatMap(continue => signal.tryTake *> F.whenA(continue)(consume(c)))
    case _ => signal.take *> consume(c)
    }
    } yield ()
    }
    used by Controller

    View full-size slide

  45. OPERATOR CONFIGURATION
    CrdConfig(
    namespace = Namespace(“test"),
    eventQueueSize = 10,
    concurrentController = true,
    prefix = "io.myorg.kerboperator",
    customKind = Some("Kerberos"),
    deployCrd = true,
    shortNames = List("kr"),
    pluralName = "kerbs",
    additionalPrinterColumns = List(
    AdditionalPrinterColumn(name = "realm", columnType = "string", jsonPath = “realm")
    )
    )

    View full-size slide

  46. CONFIGMAP OPERATOR
    override def run(args: List[String]): IO[ExitCode] =
    Operator
    .ofConfigMap[IO, Kerb](
    cmCfg, client, controller
    ).run
    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: my-krb1
    namespace: test
    labels:
    io.myorg.kerboperator/kind: Kerb
    data:
    config: |
    realm: EXAMPLE.COM
    principals:
    - name: client1
    password: static
    value: mypass
    - name: user2
    password: static
    value: mypass2

    View full-size slide

  47. EXAMPLE OPERATOR
    https://github.com/novakov-alexey/krb-operator
    For testing purposes:
    - SPNEGO authentication
    - Keytab as Secret creation

    View full-size slide

  48. PURE FUNCTIONAL CLIENT?
    https://github.com/joan38/kubernetes-client
    - based on http4s, cats-effect, circe
    - supports core APIs: Deployments, Pods, Services, etc.
    - was missing some K8s APIs, but not anymore ;-)
    Kubernetes Client for Scala

    View full-size slide

  49. Alexey Novakov
    email:
    - alexey.novakov at ultratendency.com
    - novakov.alex at gmail.com
    Blog:
    https://novakov-alexey.github.io/
    https://medium.com/se-notes-by-alexey-novakov
    Code: https://github.com/novakov-alexey/freya
    Twitter: @alexey_novakov
    Thank you! Questions?

    View full-size slide