Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Writing Kubernetes Operators using Cats-Effect

Writing Kubernetes Operators using Cats-Effect

Use Scala to write a Kubernetes-Operator

Alexey Novakov

June 18, 2020
Tweet

More Decks by Alexey Novakov

Other Decks in Programming

Transcript

  1. Writing
    Kubernetes Operator
    in Cats-Effect
    Alexey Novakov, Ultra Tendency

    View Slide

  2. ABOUT ME
    ➤ Solution Architect at Ultra Tendency,
    Germany
    ➤ Experience:
    ➤ Scala - 5yrs.
    ➤ Java - 10 yrs.
    ➤ Big Data, Kubernetes
    ➤ My interests:
    ➤ 8 string guitars
    ➤ Astronomy
    ➤ FP
    ➤ Rust

    View Slide

  3. RISE OF OPERATORS (~LAST 3 YEARS)
    ➤ Spark Operator from Google
    ➤ Confluent Kafka Operator
    ➤ Strimzi Kafka Operator from RedHat
    ➤ Dozens of others (Postgres, Cassandra,
    Jenkins, Prometheus)
    ➤ Frameworks and libraries in
    ➤ Go
    ➤ Rust
    ➤ Java
    ➤ Scala ?
    Kubernetes v1.16 (sep 2019)
    # apiVersion: apiextensions.k8s.io/v1beta1
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition

    View Slide

  4. KUBERNETES COMPONENTS
    https://kubernetes.io/docs/concepts/overview/components/
    api-server

    View Slide

  5. OPERATOR PATTERN
    kube-api-server
    1. create custom
    resource
    custom controller
    2. watch custom
    resource
    3. create native
    resource
    your
    application aka.
    Operator
    actual state
    desired state

    View Slide

  6. KUBERNETES RESOURCES
    apiVersion: io.kafka-ctl/v1
    kind: KafkaCluster
    metadata:
    name: my-cluster
    spec:
    broker: 3
    zookeeper: 3
    security:
    sasl:
    realm: MY-ORG.COM
    service: kafka
    tls:
    CN: my-org.com
    apiVersion: v1
    kind: Service
    metadata:
    labels:
    app:
    {{ .Values.KAFKA_NAME }}
    release:
    {{ .Values.KAFKA_NAME }}
    name:
    {{ .Values.KAFKA_NAME }}
    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
    labels:
    app:
    {{ .Values.KAFKA_NAME }}
    release:
    {{ .Values.KAFKA_NAME }}
    name:
    {{ .Values.KAFKA_NAME }}
    apiVersion: v1
    kind: ConfigMap
    metadata:
    labels:
    app:
    {{ .Values.ZOOKEEPER_NAME }
    }
    release:
    {{ .Values.KAFKA_NAME }}
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
    labels:
    app:
    {{ .Values.KAFKA_NAME }}

    Custom Resource Native Resources
    x 6
    x 2
    x 3
    x 3
    + Secrets, etc.

    View Slide

  7. BUILDING
    YOUR
    OPERATOR
    ingredients

    View Slide

  8. MINIMUM SET
    CustomResourceDefinition
    manifest
    Structural Schema
    (OpenAPIv3)
    Kubernetes Client
    (REST)
    Operator manifests
    (deployment, RBAC)
    Kubernetes Cluster
    Operator Image
    Development Deployment
    YAML/JSON
    Scala library
    YAML/JSON
    Docker

    View Slide

  9. WORKFLOW
    custom controller
    1.a. watch for CR updates (push)
    2. parse JSON to model (case class)
    3. CRUD necessary native resources
    4.* update CR status
    1.b. list current CRs (pull)

    View Slide

  10. USE CASE
    MIT Kerberos Operator:
    deployes KDC, Kadmin
    creates K8s Secrets: credentials, keytabs
    creates principals
    MIT Kerberos Operator
    +
    gets
    Input
    Result

    View Slide

  11. YAML SPEC
    apiVersion: io.github.novakov-alexey/v1
    kind: Krb
    metadata:
    name: my-krb
    spec:
    realm: EXAMPLE.COM
    principals:
    - name: client1
    password:
    type: static
    value: mypass
    keytab: cluster.keytab
    secret:
    type: Keytab
    name: cluster-keytab
    - name: user2
    keytab: cluster.keytab
    secret:
    type: KeytabAndPassword
    name: cluster-keytab
    Creates KDC/Kadmin
    Pod, Service
    Creates principals,
    Secrets

    View Slide

  12. Implementation

    View Slide

  13. REQUIREMENTS
    ➤ Parse custom resources
    into Scala case
    class(es)
    ➤ Make as functional &
    generic as possible
    ➤ Use Cats-Effect and
    IOApp to watch for
    resource CRUD events
    Scala code

    View Slide

  14. HIGH-LEVEL
    Event(type, T) F[U]
    K8s
    Client
    CRD
    Environment:
    Reconcile on interval
    watch
    Reconnect Plugable JSON Codecs
    T - Spec
    U - Status

    View Slide

  15. CRD SPEC: YAML -> SCALA
    final case class Krb(realm: String, principals: List[Principal])
    sealed trait Password
    final case class Static(value: String) extends Password
    case object Random extends Password
    sealed trait Secret
    final case class Keytab(name: String) extends Secret
    final case class KeytabAndPassword(name: String) extends Secret
    final case class Principal(
    name: String, password: Password = Random,
    keytab: String, secret: Secret
    )
    ADTs

    View Slide

  16. CRD STATUS: YAML -> SCALA
    final case class Status(
    processed: Boolean,
    lastPrincipalCount: Int,
    totalPrincipalCount: Int,
    error: String = ""
    )
    kubectl describe krb
    Status:
    Error:
    Last Principal Count: 0
    Processed: true
    Total Principal Count: 5
    openAPIV3Schema:
    type: object
    properties:
    spec:
    type: object
    ……
    status:
    type: object
    ……

    View Slide

  17. KUBERNETES WATCH API
    GET /api/v1/namespaces/test/pods?watch=1&resourceVersion=10245
    ---
    200 OK
    Transfer-Encoding: chunked
    Content-Type: application/json
    {
    "type": "ADDED",
    "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "10596",
    ...}, ...}
    }
    {
    "type": "MODIFIED",
    "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "11020",
    ...}, ...}
    }
    {
    "type": "DELETED",
    ...

    View Slide

  18. CONTROLLER
    abstract class Controller[F[_]: Sync, T, U] {
    def onAdd(res: CustomResource[T, U]): F[NewStatus[U]]
    def onModify(res: CustomResource[T, U]): F[NewStatus[U]]
    def onDelete(res: CustomResource[T, U]): F[Unit]
    }
    type NewStatus[U] = Option[U]
    case class CustomResource[T, U](
    meta: Metadata, spec: T, status: NewStatus[U]
    )

    View Slide

  19. CONSTRUCT OPERATOR
    class KrbController[F[_]: Sync](client: KubernetesClient)
    extends Controller[F, Krb, Status] {
    override def onAdd(r: CustomResource[Krb, Status]): F[NewStatus[Status]] = ???
    override def onModify(r: CustomResource[Krb, Status]): F[NewStatus[Status]] = ???
    override def onDelete(r: CustomResource[Krb, Status]): F[Unit] = ???
    }
    val client = Sync[F].delay(new DefaultKubernetesClient)
    val controller = (c: KubernetesClient) => new KrbController(c)
    val cfg = CrdConfig(Namespace("test"), "io.github.novakov-alexey")
    val operator = Operator.ofCrd [F, Krb, Status](cfg, client, controller)
    Effect Spec Status

    View Slide

  20. CATS IO APP
    object Main extends IOApp {
    override def run(args: List[String]): IO[ExitCode] =
    operator.run
    }
    class Operator[F[_], T: Reader, U] private (
    ….. // some parameters
    )(implicit F: ConcurrentEffect[F], T: Timer[F]) {
    def run: F[ExitCode] = ???
    }

    View Slide

  21. K8s Client
    Challenges

    View Slide

  22. KUBERNETES CLIENT
    def getPod(client: KubernetesClient)(
    namespace: String, labelKey: String, labelValue: String
    ): Option[Pod] =
    client
    .pods()
    .inNamespace(namespace)
    .withLabel(labelKey, labelValue)
    .list()
    .getItems
    .asScala
    .find(p =>
    Option(p.getMetadata.getDeletionTimestamp).isEmpty)
    "io.fabric8" % “kubernetes-client" % fabric8Version
    "io.fabric8" % "kubernetes-model" % fabric8Version
    // be ready for nulls and Exceptions

    View Slide

  23. FABRIC8 WATCH API
    crdClient.watch(new Watcher[AnyCustomResource]() {
    override def eventReceived(
    action: Watcher.Action, cr: AnyCustomResource): Unit = ???
    override def onClose(e: KubernetesClientException): Unit = ???
    })
    val crd: CustomResourceDefinition = ???
    val crdClient = client.customResources(crd,
    classOf[AnyCustomResource],
    classOf[AnyCrList],
    classOf[AnyCrDoneable]
    ).inAnyNamespace
    First problem
    to solve!

    View Slide

  24. DESERIALIZATION
    import io.fabric8.kubernetes.client.CustomResource
    class AnyCustomResource extends CustomResource {
    private var spec: AnyRef = _
    private var status: AnyRef = _
    def getSpec: AnyRef = spec
    def setSpec(spec: AnyRef): Unit =
    this.spec = spec
    def getStatus: AnyRef = status
    def setStatus(status: AnyRef): Unit =
    this.status = status
    }
    Second problem
    to solve!

    View Slide

  25. Push-API
    or void problem

    View Slide

  26. PUSH-API / UNIT TYPE PROBLEM
    def putAction(
    namespace: String,
    action: Either[OperatorError, WatcherAction[T, U]]
    ): F[Unit] =
    consumer.putAction(namespace, action)
    val cr: AnyCustomResource = ???
    val converted: Either[OperatorError, CustomResource[T, U]] =
    convertCr(cr).leftMap[OperatorError] {
    case (t, resource) => ParseResourceError(action, t, resource)
    }
    putAction(namespace, cr) // F[Unit]
    :-( Can kubernetes-client evaluate the effect on its own?
    void watch {
    }

    View Slide

  27. CATS EFFECT TYPE CLASS
    def runSync[A](f: F[A]): A =
    F.toIO(f).unsafeRunSync()
    def putActionBlocking(
    namespace: String,
    action: Either[OperatorError, WatcherAction[T, U]]
    ): Unit =
    runSync(consumer.putAction(namespace, action))
    implicit F: Effect[F]
    /**
    * Convert to an IO[A].
    *
    * The law is that toIO(liftIO(ioa)) is the same as ioa
    */
    def toIO[A](fa: F[A]): IO[A] =
    Effect.toIOFromRunAsync(fa)(this)

    View Slide

  28. JSON
    Deserialization
    problem

    View Slide

  29. JACKON-DATA-BIND
    import io.fabric8.kubernetes.client.CustomResource
    class AnyCustomResource extends CustomResource {
    private var spec: AnyRef = _
    private var status: AnyRef = _
    ….

    View Slide

  30. class StringPropertyDeserializer extends StdDeserializer[StringProperty](
    classOf [StringProperty]) {
    override def deserialize(jp: JsonParser, ctxt: DeserializationContext): StringProperty = {
    val node = jp.getCodec.readTree[ObjectNode](jp)
    StringProperty(node.toString)
    }}
    DESERIALISE TO STRING
    @JsonDeserialize(using = classOf[StringPropertyDeserializer])
    final case class StringProperty(value: String)
    class AnyCustomResource extends CustomResource {
    private var spec: StringProperty = _

    }
    Wrapper for spec and
    status properties
    reading to Jackson Object AST

    View Slide

  31. def parseProperty[T: JsonReader](property: StringProperty, name: String) = {
    val reader = implicitly [JsonReader[T]]
    val parsed = reader.fromString(property.value)
    }
    DESERIALISE TO [T]
    val cr: AnyCustomResource = ???
    val spec = parseProperty[Krb](cr.getSpec, “spec")
    val status = parseProperty[Status](cr.getStatus, "status")
    Now use Scala Json library to parse to type T

    View Slide

  32. JSON PARSER API
    import io.circe.Decoder
    import io.circe.parser.decode
    trait JsonReader[T] {
    def fromString(json: String): Either[Throwable, T]
    }
    implicit def circeRead[T: Decoder]: JsonReader[T] = new JsonReader[T] {
    override def fromString(json: String): Either[Throwable, T] =
    decode[T](json)
    }

    View Slide

  33. CIRCE DECODER: SPEC
    trait Codecs {
    implicit val genConfig: Configuration =
    Configuration.default.withDiscriminator("type").withDefaults
    implicit val decodePassword: Decoder[Password] =
    List [Decoder[Password]](
    Decoder[Static].widen,
    Decoder.const(Random).widen).reduceLeft(_.or(_))
    implicit val decodeSecret: Decoder[Secret] =
    List [Decoder[Secret]](
    Decoder[Keytab].widen,
    Decoder[KeytabAndPassword].widen).reduceLeft(_.or(_))
    }

    View Slide

  34. Final API

    View Slide

  35. object KrbOperator extends IOApp with CirceCodecs {
    implicit val cs: ContextShift[IO] = contextShift
    override def run(args: List[String]): IO[ExitCode] = {

    Operator
    .ofCrd [IO, Kerb, Status](cfg, client, controller)
    .withReconciler(60.seconds)
    .withRestart(Times(maxRetries = 3, delay = 1.seconds, multiplier = 2))
    }
    }

    View Slide

  36. RECONCILER THREAD
    class Reconciler[F[_], T, U](
    ….
    currentResources: F[Either[Throwable, ResourcesList[T, U]]]
    )(implicit F: Effect[F], T: Timer[F]) {
    def run: F[ReconcilerExitCode] =
    F.suspend {
    for {
    _ <- T.sleep(delay)
    r <- currentResources
    _ <- publish(r)
    ec <- run
    } yield ec
    }.recoverWith {
    case e =>
    F.delay(logger.error("Failed in reconciling loop", e)) *> ExitCodes.ReconcileExitCode.pure[F]
    }
    loop

    View Slide

  37. RESTARTABLE OPERATOR
    def withRestart(retry: Retry = Infinite())(implicit T: Timer[F]): F[ExitCode] =
    run.flatMap(loop(_, retry)) calls loop when operator fails
    private def loop(ec: ExitCode, retry: Retry)(implicit T: Timer[F]): F[ExitCode] = {
    val (canRestart, delay, nextRetry, remaining) = ??? // logic to calculate current retry params
    if (canRestart)
    for {
    _ <- T.sleep(delay)
    r <- nextRetry
    code <- withRestart(r)
    } yield code
    else ec.pure[F]
    }

    View Slide

  38. https://novakov-alexey.github.io/freya/

    View Slide

  39. FREYA LIBRARY
    Core:
    "io.github.novakov-alexey" %% "freya-core" % “0.2.3"
    Circe:
    "io.github.novakov-alexey" %% "freya-circe" % “0.2.3"
    Jackson:
    "io.github.novakov-alexey" %% "freya-jackson" % “0.2.3"

    View Slide

  40. EVENT-DISPATCHING
    - Dynamic queue per
    namespace creation
    - Handling events
    concurrently
    preserving original
    order per namespace

    View Slide

  41. BLOCKING QUEUE
    class BlockingQueue[F[_]: Sync, A](
    size: Int,
    queue: Ref[F, Queue[A]],
    signal: MVar[F, Unit]
    ) {
    def produce(a: A): F[Unit] = ???
    }
    def consume(c: A => F[Boolean]): F[Unit] = ???

    View Slide

  42. CATS-EFFECT
    https://typelevel.org/cats-effect/concurrency/ref.html

    View Slide

  43. CATS-EFFECT
    https://typelevel.org/cats-effect/concurrency/mvar.html

    View Slide

  44. NAMESPACE EVENTS BUFFER
    class BlockingQueue[F[_]: Sync, A](
    size: Int,
    queue: Ref[F, Queue[A]],
    signal: MVar[F, Unit]
    ) {
    def produce(a: A): F[Unit] =
    for {
    (added, length) <- queue.modify { q =>
    if (q.length < capacity)
    (q.enqueue(a), true)
    else
    (q, false)
    }
    _ <- if (added) signal.tryPut(()).void
    else signal.put(()) *> produce(a)
    } yield ()
    used by Event Watcher
    // notify consumer

    View Slide

  45. BUFFER NAMESPACE EVENTS
    def consume(f: A => F[Boolean]): F[Unit] =
    for {
    elem <- queue.modify(
    q =>
    q.dequeueOption.map {
    case (a, q) => (q, a.some)
    }.getOrElse((q, None))
    )
    _ <- elem match {
    case Some(e) => f(e).flatMap(continue => signal.tryTake *> F.whenA(continue)(consume(c)))
    case _ => signal.take *> consume(c)
    }
    } yield ()
    }
    used by Controller

    View Slide

  46. OPERATOR CONFIGURATION
    CrdConfig(
    namespace = Namespace(“test"),
    eventQueueSize = 10,
    concurrentController = true,
    prefix = "io.myorg.kerboperator",
    customKind = Some("Kerberos"),
    deployCrd = true,
    shortNames = List("kr"),
    pluralName = "kerbs",
    additionalPrinterColumns = List(
    AdditionalPrinterColumn(name = "realm", columnType = "string", jsonPath = “realm")
    )
    )

    View Slide

  47. CONFIGMAP OPERATOR
    override def run(args: List[String]): IO[ExitCode] =
    Operator
    .ofConfigMap[IO, Kerb](
    cmCfg, client, controller
    ).run
    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: my-krb1
    namespace: test
    labels:
    io.myorg.kerboperator/kind: Kerb
    data:
    config: |
    realm: EXAMPLE.COM
    principals:
    - name: client1
    password: static
    value: mypass
    - name: user2
    password: static
    value: mypass2

    View Slide

  48. EXAMPLE OPERATOR
    https://github.com/novakov-alexey/krb-operator
    For testing purposes:
    - SPNEGO authentication
    - Keytab as Secret creation

    View Slide

  49. PURE FUNCTIONAL CLIENT?
    https://github.com/joan38/kubernetes-client
    - based on http4s, cats-effect, circe
    - supports core APIs: Deployments, Pods, Services, etc.
    - was missing some K8s APIs, but not anymore ;-)
    Kubernetes Client for Scala

    View Slide

  50. Alexey Novakov
    email:
    - alexey.novakov at ultratendency.com
    - novakov.alex at gmail.com
    Blog:
    https://novakov-alexey.github.io/
    https://medium.com/se-notes-by-alexey-novakov
    Code: https://github.com/novakov-alexey/freya
    Twitter: @alexey_novakov
    Thank you! Questions?

    View Slide