Building a self-service Kafka Platform

Have you ever wondered, what motivates people? According to many authors, autonomy is a key factor to build a sustainable and motivated team. This is as well a core principle of the devops.

If you are in the quest of building a self-service kafka based platform where your users can manage topics, acls and rbac by themself, but still keep change control and a verification process? Gitops and Kafka Topology Builder is here to help you.

In this talk we’re going to present and guide you towards the steps necessary to build up a self-service Apache Kafka using Gitops and Kafka Topology Builder, including a few war stories from people using it in production. As well we introduce the idea of service brokers for k8s and Cloud Foundry.

By the end of this talk, you will take home an automatic Jenkins pipeline, that will make your users autonomous and give you more time to focus on less bureaucratic work and a setup with service brokers in k8s and cf.

Transcript

1. Building a self-service Kafka platform Pere Urbon-Bayes / Benedikt Linse pere@confluent.io / benedikt@confluent.io @purbon
2. What are we going to talk about today? 2 • How to build a self-service and building full automation for an Apache Kafka platform to make your users autonomous • Present the Kafka Topology Builder as descriptive change agent to manage your changes in the Kafka platform • Introduce Service brokers as a Kubernetes/Cloud Foundry ways of providing autonomy to teams
3. The quest to build a self-service Kafka platform Building a self-service platform
4. @purbon Common questions for KafkaOps
5. @purbon What happens to my logs?
6. @purbon Automate all the things!
7. @purbon A declarative method
8. @purbon A gitops approach for Apache Kafka, and others
9. What does Kafka Topology Builder offer you? Continuous delivery of changes to Apache Kafka and Confluent Platform Multitenancy out of the box All changes are documented and approved. Full control of change management for audit Declarative approach, users don’t need to know low level of Kafka Full development team autonomy, without hampering operational control 9
10. In detail, what do I get with this process? • API driven solution to integrate with CI/CD practises in house. ◦ Support for on-prem cluster and Confluent Cloud. • Topic management with: ◦ creation, delete and update of specific configuration value ◦ out of the box namespacing based on the project structure and data types. • Multi tenancy enablement out of the box • Access Control management with: ◦ Predefined roles for Consumers, Producers, Kafka Connect Connectors and Kafka Streams applications ◦ Support for traditional ACLs and Confluent RBAC ◦ Platform wide ACLs for Schema Registry deployments NOTE: For all this futures, the tool managed automatic create, delete and update when required. 10
11. To know more about the project If you are interested to know more about the project, you can find more information in: • Kafka Topology Builder Github: https://github.com/purbon/kafka-topology-builder • Documentation: https://kafka-topology-builder.readthedocs.io/ • Releases: https://github.com/purbon/kafka-topology-builder/releases 11
12. Does anything changes if I do containers? 12
13. Self Service Platforms / Service Brokers • A way to enrich self service platforms with custom services • The Open Service Broker API defines these interactions, and therefore allows software providers to offer their services to anyone, regardless of the technology or infrastructure those software providers wish to utilise. • The tasks of a Service Broker are the following: • Allow easy use of stateful services by application developers • Provide a catalog of offers • Provisioning new service instances • Dis/connecting applications from services • Deprovision service instances 13
14. Concepts: Apps and Services • App: ◦ written by developers ◦ stateless ◦ comply with 12-factor-app standard ◦ autoscale or can be manually scaled ◦ use services to persist data ◦ run within a container ◦ deployed through declarative application manifests. 14 • Service ◦ provided by platform operators ◦ stateful ◦ can be declaratively provisioned by developers in a self-service style ◦ can be shared or dedicated ◦ Examples: ▪ MysqlDB, Redis,Kafka, ElasticSearch, DynamoDB, BigTable …
15. Many ones available, for Kubernetes and CF • A Kafka Service Broker for Cloud Foundry at BOSH • Pivotal - Datastax: Cassandra Service Broker • Service Broker for MongoDB Atlas • Another Kafka Service Broker by 1123 15
16. Service Brokers on Kubernetes 16 Source: https://kubernetes.io/docs/concepts/extend-kubernetes/service-catalog
17. Kubernetes: Service Catalog 17
18. Kafka Service Broker Workflow on CloudFoundry Service Instantiation 18 Cloud Controller Application Runtime Kafka Service Broker 1. My App needs access to a topic! cf create-service confluent-kafka gold t1 2. create topic 3. create topic Success/Failure response Success / Failure response Success/ Failure response MyApp MyApp
19. Kafka Service Broker on CloudFoundry Service Binding 19 Cloud Controller PWS Application Runtime Kafka Service Broker 6. create user and api key 7. set ACLs 4. bind topic to my app cf bind-service my-app t1 5. bindRequest my-app, t1 8. return user, password MyApp 9. Inject credentials as environment variable 9. Read Credentials from Environment upon startup MyApp
20. Many ones available, for Kubernetes and CF • A Kafka Service Broker for Cloud Foundry at BOSH • Pivotal - Datastax: Cassandra Service Broker • Service Broker for MongoDB Atlas • Kafka Service Broker for Topics by 1123 20
21. Kafka Service Broker: Future Work • Support for SASL/SCRAM users • Support for Role Based Access Control • Support for prefixed ACLs • Support for more topic configurations • Support for more bind methods: ◦ read only ◦ write only ◦ streams ◦ transactional • Use future CCloud API for managing users • Test Service Instance sharing 21
22. Demo time…. 22