Building a self-service Kafka platform Pere Urbon-Bayes / Benedikt Linse pere@confluent.io / benedikt@confluent.io @purbon 

What are we going to talk about today? 2 • How to build a self-service and building full automation for an Apache Kafka platform to make your users autonomous • Present the Kafka Topology Builder as descriptive change agent to manage your changes in the Kafka platform • Introduce Service brokers as a Kubernetes/Cloud Foundry ways of providing autonomy to teams 

The quest to build a self-service Kafka platform Building a self-service platform 

@purbon Common questions for KafkaOps 

@purbon What happens to my logs? 

@purbon Automate all the things! 

@purbon A declarative method 

@purbon A gitops approach for Apache Kafka, and others 

What does Kafka Topology Builder offer you? Continuous delivery of changes to Apache Kafka and Confluent Platform Multitenancy out of the box All changes are documented and approved. Full control of change management for audit Declarative approach, users don’t need to know low level of Kafka Full development team autonomy, without hampering operational control 9 

In detail, what do I get with this process? ● API driven solution to integrate with CI/CD practises in house. ○ Support for on-prem cluster and Confluent Cloud. ● Topic management with: ○ creation, delete and update of specific configuration value ○ out of the box namespacing based on the project structure and data types. ● Multi tenancy enablement out of the box ● Access Control management with: ○ Predefined roles for Consumers, Producers, Kafka Connect Connectors and Kafka Streams applications ○ Support for traditional ACLs and Confluent RBAC ○ Platform wide ACLs for Schema Registry deployments NOTE: For all this futures, the tool managed automatic create, delete and update when required. 10 

To know more about the project If you are interested to know more about the project, you can find more information in: ● Kafka Topology Builder Github: https://github.com/purbon/kafka-topology-builder ● Documentation: https://kafka-topology-builder.readthedocs.io/ ● Releases: https://github.com/purbon/kafka-topology-builder/releases 11 

Does anything changes if I do containers? 12 

Self Service Platforms / Service Brokers ● A way to enrich self service platforms with custom services ● The Open Service Broker API defines these interactions, and therefore allows software providers to offer their services to anyone, regardless of the technology or infrastructure those software providers wish to utilise. ● The tasks of a Service Broker are the following: ● Allow easy use of stateful services by application developers ● Provide a catalog of offers ● Provisioning new service instances ● Dis/connecting applications from services ● Deprovision service instances 13 

Concepts: Apps and Services ● App: ○ written by developers ○ stateless ○ comply with 12-factor-app standard ○ autoscale or can be manually scaled ○ use services to persist data ○ run within a container ○ deployed through declarative application manifests. 14 ● Service ○ provided by platform operators ○ stateful ○ can be declaratively provisioned by developers in a self-service style ○ can be shared or dedicated ○ Examples: ■ MysqlDB, Redis,Kafka, ElasticSearch, DynamoDB, BigTable … 

Many ones available, for Kubernetes and CF ● A Kafka Service Broker for Cloud Foundry at BOSH ● Pivotal - Datastax: Cassandra Service Broker ● Service Broker for MongoDB Atlas ● Another Kafka Service Broker by 1123 15 

Service Brokers on Kubernetes 16 Source: https://kubernetes.io/docs/concepts/extend-kubernetes/service-catalog 

Kubernetes: Service Catalog 17 

Kafka Service Broker Workflow on CloudFoundry Service Instantiation 18 Cloud Controller Application Runtime Kafka Service Broker 1. My App needs access to a topic! cf create-service confluent-kafka gold t1 2. create topic 3. create topic Success/Failure response Success / Failure response Success/ Failure response MyApp MyApp 

Kafka Service Broker on CloudFoundry Service Binding 19 Cloud Controller PWS Application Runtime Kafka Service Broker 6. create user and api key 7. set ACLs 4. bind topic to my app cf bind-service my-app t1 5. bindRequest my-app, t1 8. return user, password MyApp 9. Inject credentials as environment variable 9. Read Credentials from Environment upon startup MyApp 

Many ones available, for Kubernetes and CF ● A Kafka Service Broker for Cloud Foundry at BOSH ● Pivotal - Datastax: Cassandra Service Broker ● Service Broker for MongoDB Atlas ● Kafka Service Broker for Topics by 1123 20 

Kafka Service Broker: Future Work ● Support for SASL/SCRAM users ● Support for Role Based Access Control ● Support for prefixed ACLs ● Support for more topic configurations ● Support for more bind methods: ○ read only ○ write only ○ streams ○ transactional ● Use future CCloud API for managing users ● Test Service Instance sharing 21 

Demo time…. 22