Zürich Apache Kafka Meetup - Swisscom Firehose

Kafka has been used for several years at Swisscom to stream data from various sources to sinks such as Hadoop. Providing Kafka as a Service to multiple teams in a large company presents governance, security and multi-tenancy challenges.

In this talk we will present how we have built our Swisscom Firehose platform which enables teams to use Kafka internally. We will explain how we have tackled these challenges by describing our governance model, our identity & ACLs management, and our self-service capabilities. We will also present how we leverage Kubernetes and how it simplifies our operations.

Transcript

1. Swisscom Firehose Kafka as a Service @ Swisscom Swisscom Data, Analytics & AI Group Thibaud Chardonnens David Jacot Do not duplicate or distribute without written permission from Swisscom
2. Who are we? David Jacot Head of Big Data Infrastructure @davidjacot Thibaud Chardonnens Big Data Engineer @tbcdns 2 Do not duplicate or distribute without written permission from Swisscom
3. Do not duplicate or distribute without written permission from Swisscom Swisscom’s Streaming Platform 3 KStreams, Spark, … Schema Registry
4. History of Kafka @ Swisscom 4 2014 2015 2016 2017 Stream data into Hadoop < 5 topics < 5 users More data into Hadoop 10+ topics < 5 users Stream Processing is a thing 100+ topics 30+ users Role Based Access Control 1000+ topics 100+ users Do not duplicate or distribute without written permission from Swisscom
5. Scaling challenges (organizational not technical!) 5 Management of all the policies is hard with tools available out of the box Who approves access to topics? RBAC increases the pressure on the administrators Define higher level of abstraction to reason about governance and security Decentralized approval process with well defined owners Self-Service API Do not duplicate or distribute without written permission from Swisscom
6. Simple yet powerful concepts Group of individuals working on something together (e.g. project), characterized by a name. Organization Identity delivered to a Kafka Client, characterized by a name, a network zone, a set of IPs, and a certificate. Identity has RW access to all resources within its space. Identity Kafka Topic characterized by a name, a description, a schema, a classification (C1-C4), and shareable flag. Topic Group of isolated resources (e.g. Topics) belonging to an organization, characterized by a name and a classification. Space Constraints on Classification <-> Network Zone 6 Do not duplicate or distribute without written permission from Swisscom
7. Clear roles and responsibilities Use the resources within his organization. Org. Member Responsible of the resources (usage, legal, approvals, etc.) and the members within his organization. Org. Owner Audit all the organizations to ensure rules are enforced and respected. Auditor Enforce the security and governance rules by technical means. Admin 7 Do not duplicate or distribute without written permission from Swisscom
8. 8 Kafka org1.prod.topic-1 (C2) org1.prod.topic-2 (C3) org1.prod.topic-3 (C4) org1.prod (C4) org3.prod (C4) org3.prod.topic-1 (C2) org3.prod.topic-2 (C3) org3.prod.topic-3 (C4) org2.prod.topic-1 (C2) org2.prod.topic-2 (C2) org2.prod.topic-3 (C1) org2.prod (C2) RW Do not duplicate or distribute without written permission from Swisscom
9. How do we share topics between organizations/spaces? 9 Owner of the topic publishes it in the data catalog Publish Topic Owner of an organization requests access to the topic Request Access Access to the topic is approved by its owner and an administrator Approvals Topic virtually appears in the space and is readable by all identities Access Do not duplicate or distribute without written permission from Swisscom
10. Architecture 10 Servers Runtime Data plane Control plane Schema Registry Connect Navis API UI Do not duplicate or distribute without written permission from Swisscom
11. Why Kubernetes? 11 Scheduler Increase hardware utilisation Network Policies Containers It’s fun! Do not duplicate or distribute without written permission from Swisscom
12. Do not duplicate or distribute without written permission from Swisscom Kafka on Kubernetes 12 1 2 3 Physical servers subnet 10.10.10.0/24 Pod subnet 10.10.11.0/24 Consumer Producer Swisscom Network
13. Navis Design 13 REST API API writes state to Microservices consume state from Kafka topics State is stored in compacted topics POST /organisations Microservices talk to target systems to apply the state Topics management Kafka ACLs Network ACLs Kafka Connect management Connect Kafka topics Do not duplicate or distribute without written permission from Swisscom
14. Kafka and Network ACLs 10.1.2.3 1. Identity created 2. Identity approved 3. Network ACL microservice: - creates Kubernetes network policy 4. Kafka ACL microservice: - adds Read/Write ACL for <org>.<space>.* - adds Read Only ACL for shared topics 5. User generates a certificate and starts to consume/produce to topics belonging to the space Custom authorizer Network ACL Kafka ACL Do not duplicate or distribute without written permission from Swisscom
15. Auto-Dump to HDFS Connectors as a Service Connectors as a Service Automatically dump topics into HDFS Users simply need to activate the feature and select some bucketing options A connect cluster per topic is spawned on Kubernetes Connector images can be scheduled on Kubernetes via the REST API Users need to select or create the connector image and configure the connector Users are responsible to make sure the connector is running We provide access to the connectors logs and alert users if a connector is failing Do not duplicate or distribute without written permission from Swisscom
16. What’s next? 16 Statefulset with Local Persistent Volumes (Kubernetes 1.10) Built-in transformations Quotas for Kafka clients Firehose in second data centre for geo-redundancy Include Hadoop platform in our new governance framework Do not duplicate or distribute without written permission from Swisscom
17. Thank you! Zurich, May 2018 Do not duplicate or distribute without written permission from Swisscom