Building a self-service Kafka Platform

Have you ever wondered, what motivates people? According to many authors, autonomy is a key factor to build a sustainable and motivated team. This is as well a core principle of the devops.

If you are in the quest of building a self-service kafka based platform where your users can manage topics, acls and rbac by themself, but still keep change control and a verification process? Gitops and Kafka Topology Builder is here to help you.

In this talk we’re going to present and guide you towards the steps necessary to build up a self-service Apache Kafka using Gitops and Kafka Topology Builder, including a few war stories from people using it in production. As well we introduce the idea of service brokers for k8s and Cloud Foundry.

By the end of this talk, you will take home an automatic Jenkins pipeline, that will make your users autonomous and give you more time to focus on less bureaucratic work and a setup with service brokers in k8s and cf.

Transcript

1. Building a self-service Kafka
   platform
   Pere Urbon-Bayes / Benedikt Linse
   pere@confluent.io / benedikt@confluent.io
   @purbon
   

   View Slide

2. What are we going to talk about today?
   2
   • How to build a self-service and building full automation for an Apache Kafka platform to
   make your users autonomous
   • Present the Kafka Topology Builder as descriptive change agent to manage your
   changes in the Kafka platform
   • Introduce Service brokers as a Kubernetes/Cloud Foundry ways of providing autonomy
   to teams
   

   View Slide

3. The quest to build a self-service
   Kafka platform
   Building a self-service platform
   

   View Slide

4. @purbon
   Common
   questions for
   KafkaOps
   

   View Slide

5. @purbon
   What happens to my logs?
   

   View Slide

6. @purbon
   Automate all the things!
   

   View Slide

7. @purbon
   A declarative method
   

   View Slide

8. @purbon
   A gitops approach for Apache Kafka, and others
   

   View Slide

9. What does Kafka Topology Builder offer you?
   Continuous delivery of changes to Apache
   Kafka and Confluent Platform
   Multitenancy out of the box
   All changes are documented and
   approved. Full control of change
   management for audit
   Declarative approach, users don’t need to
   know low level of Kafka
   Full development team autonomy, without
   hampering operational control
   9
   

   View Slide

10. In detail, what do I get with this process?
    ● API driven solution to integrate with CI/CD practises in house.
    ○ Support for on-prem cluster and Confluent Cloud.
    ● Topic management with:
    ○ creation, delete and update of specific configuration value
    ○ out of the box namespacing based on the project structure and data types.
    ● Multi tenancy enablement out of the box
    ● Access Control management with:
    ○ Predefined roles for Consumers, Producers, Kafka Connect Connectors and Kafka
    Streams applications
    ○ Support for traditional ACLs and Confluent RBAC
    ○ Platform wide ACLs for Schema Registry deployments
    NOTE: For all this futures, the tool managed automatic create, delete and update when
    required.
    10
    

    View Slide

11. To know more about the project
    If you are interested to know more about the project, you can find more information in:
    ● Kafka Topology Builder Github: https://github.com/purbon/kafka-topology-builder
    ● Documentation: https://kafka-topology-builder.readthedocs.io/
    ● Releases: https://github.com/purbon/kafka-topology-builder/releases
    11
    

    View Slide

12. Does anything changes if I do containers?
    12
    

    View Slide

13. Self Service Platforms / Service Brokers
    ● A way to enrich self service platforms with custom services
    ● The Open Service Broker API defines these interactions, and therefore allows software
    providers to offer their services to anyone, regardless of the technology or infrastructure
    those software providers wish to utilise.
    ● The tasks of a Service Broker are the following:
    ● Allow easy use of stateful services by application developers
    ● Provide a catalog of offers
    ● Provisioning new service instances
    ● Dis/connecting applications from services
    ● Deprovision service instances
    13
    

    View Slide

14. Concepts: Apps and Services
    ● App:
    ○ written by developers
    ○ stateless
    ○ comply with 12-factor-app standard
    ○ autoscale or can be manually scaled
    ○ use services to persist data
    ○ run within a container
    ○ deployed through declarative application
    manifests.
    14
    ● Service
    ○ provided by platform operators
    ○ stateful
    ○ can be declaratively provisioned by
    developers in a self-service style
    ○ can be shared or dedicated
    ○ Examples:
    ■ MysqlDB, Redis,Kafka,
    ElasticSearch, DynamoDB,
    BigTable …
    

    View Slide

15. Many ones available, for Kubernetes and CF
    ● A Kafka Service Broker for Cloud Foundry at BOSH
    ● Pivotal - Datastax: Cassandra Service Broker
    ● Service Broker for MongoDB Atlas
    ● Another Kafka Service Broker by 1123
    15
    

    View Slide

16. Service Brokers on Kubernetes
    16
    Source: https://kubernetes.io/docs/concepts/extend-kubernetes/service-catalog
    

    View Slide

17. Kubernetes: Service Catalog
    17
    

    View Slide

18. Kafka Service Broker Workflow on CloudFoundry
    Service Instantiation
    18
    Cloud
    Controller
    Application Runtime
    Kafka Service
    Broker
    1. My App needs access to a topic!
    cf create-service confluent-kafka gold t1
    2. create
    topic
    3. create topic
    Success/Failure
    response
    Success /
    Failure
    response
    Success/
    Failure
    response
    MyApp
    MyApp
    

    View Slide

19. Kafka Service Broker on CloudFoundry
    Service Binding
    19
    Cloud
    Controller
    PWS Application
    Runtime
    Kafka Service
    Broker
    6. create user and
    api key
    7. set ACLs
    4. bind topic to my app
    cf bind-service my-app t1
    5. bindRequest my-app, t1
    8. return user, password
    MyApp
    9. Inject credentials as
    environment variable
    9. Read Credentials from
    Environment upon startup
    MyApp
    

    View Slide

20. Many ones available, for Kubernetes and CF
    ● A Kafka Service Broker for Cloud Foundry at BOSH
    ● Pivotal - Datastax: Cassandra Service Broker
    ● Service Broker for MongoDB Atlas
    ● Kafka Service Broker for Topics by 1123
    20
    

    View Slide

21. Kafka Service Broker: Future Work
    ● Support for SASL/SCRAM users
    ● Support for Role Based Access Control
    ● Support for prefixed ACLs
    ● Support for more topic configurations
    ● Support for more bind methods:
    ○ read only
    ○ write only
    ○ streams
    ○ transactional
    ● Use future CCloud API for managing users
    ● Test Service Instance sharing
    21
    

    View Slide

22. Demo time….
    22
    

    View Slide