Role Based Access Control in Real-Time Streaming Data: What, Why and How (Hojjat Jafarpour, DeltaStream) | RTA Summit 2023

Data streaming platforms such as Apache Kafka and AWS Kinesis have become a foundational part of real-time data processing. It is crucial for such systems to ensure security of streaming data as such data plays an increasingly important role in mission critical applications in organizations. Role-Based Access Control (RBAC) is one of the most common ways to provide security for data in motion. Access control privileges that are defined in a RBAC service determine which role can access and perform operations on specific resources. In this talk, we first present the state of the art in Role-Based Access Control for streaming data in platforms such as Apache Kafka and AWS Kinesis. We then discuss the shortcomings of the current solutions and present a novel approach where we bring the same RBAC concepts from relational systems to the data in motion space and explain how it addresses aforementioned shortcomings in the current solutions.

Attendees will learn about the state of the art in security and Role-Based Access Control in data streaming technologies and understand shortcomings and challenges in these approaches. They will also learn a novel approach that they can use in their organizations to secure access to the streaming data regardless of which system is storing the streaming data, whether it is Apache Kafka, AWS Kinesis or a hybrid of these systems.

Transcript

1. Role Based Access Control in Real-Time Streaming Data: What, Why and How Hojjat Jafarpour Founder & CEO @ DeltaStream, Inc. hojjat@deltastream.io @hojjat
2. Streaming Data is Everywhere 2 … IoT InfoSec Machine Generated Data Gaming Mobile ML/AI
3. Streaming Storage 3 • Apache Kafka, AWS Kinesis, Apache Pulsar,... • Backbone of streaming data • Decouple producers and consumers • Make data available in real-time (low latency) Streaming Storage (Kafka, Kinesis, …)
4. Access Control • Access Control Lists (ACLs) ◦ Good for smaller scale 4
5. Access Control • Access Control Lists (ACLs) ◦ Good for smaller scale • Role-based Access Control (RBAC) ◦ Access control privileges ◦ Roles ◦ Resources ◦ Privileges determine which role can access and perform operations on specific resources. 5
6. Example1: Confluent Cloud • Allows access control to ◦ Organization, environment, cluster, granular Kafka resources(topics, consumer groups, and transactional IDs), SR and ksqlDB resources https://docs.confluent.io/cloud/current/access-management/access-control/cloud-rbac.html 6
7. Example1: Confluent Cloud • Allows access control to ◦ Organization, environment, cluster, granular Kafka resources(topics, consumer groups, and transactional IDs), SR and ksqlDB resources • Roles are predefined ◦ As of now there are 13 available roles ▪ OrganizationAdmin, EnvironmentAdmin, CloudClusterAdmin, Operator,... https://docs.confluent.io/cloud/current/access-management/access-control/cloud-rbac.html 7
8. Example1: Confluent Cloud • Allows access control to ◦ Organization, environment, cluster, granular Kafka resources(topics, consumer groups, and transactional IDs), SR and ksqlDB resources • Roles are predefined ◦ As of now there are 13 available roles ▪ OrganizationAdmin, EnvironmentAdmin, CloudClusterAdmin, Operator,... • Each role has View Scope and Admin Scope https://docs.confluent.io/cloud/current/access-management/access-control/cloud-rbac.html 8
9. Example1: Confluent Cloud • Role bindings ◦ Permissions for principals ▪ Which roles are assigned to a given user ▪ e.g., grant permissions to a new user 9
10. Example1: Confluent Cloud • Role bindings ◦ Permissions for principals ▪ Which roles are assigned to a given user ▪ e.g., grant permissions to a new user ◦ Permissions on resources ▪ Which roles can access to a given resource ▪ e.g., grant permissions on a resource such as a new cluster to roles 10
11. Example1: Confluent Cloud • Example role bindings confluent iam rbac role-binding create --principal User:u-a03bcd --role CloudClusterAdmin --environment env-nx5jd --cloud-cluster lkc-xyxmz +--------------+-------------------+ | Principal | User:u-a03bcd | | Role | CloudClusterAdmin | | ResourceType | Cluster | +--------------+-------------------+ 11
12. Example1: Confluent Cloud • Example role bindings confluent iam rbac role-binding create --principal User:u-e03vqq --role ResourceOwner \ --environment env-nx5jd --cloud-cluster lkc-xyxmz --kafka-cluster-id lkc-xyxmz \ --resource Topic:connect-config +----------------+----------------+ | Principal | User:u-e03vqq | | Email | | | Role | ResourceOwner | | Environment | | | CloudCluster | | | ClusterType | | | LogicalCluster | | | ResourceType | Topic | | Name | connect-config | | PatternType | LITERAL | +----------------+----------------+ 12
13. Example1: Confluent Cloud • Need to use Confluent Cloud Console, CLI or API • Confluent only commands • Limited roles available 13
14. Example2: AWS MSK • AWS MSK uses AWS Identity and Access Management(IAM) for authentication and authorization 14
15. Example2: AWS MSK • AWS MSK uses AWS Identity and Access Management(IAM) for authentication and authorization • Uses Authorization Policies ◦ Specifies which actions to allow or deny on a resource for a role 15
16. Example2: AWS MSK • There is a list of available actions ◦ kafka-cluster:Connect, kafka-cluster:DescribeCluster,.. ◦ Some actions depend on others ▪ Actions with dependencies should include those dependencies 16
17. Example2: AWS MSK • There is a list of available actions ◦ kafka-cluster:Connect, kafka-cluster:DescribeCluster,.. ◦ Some actions depend on others ▪ Actions with dependencies should include those dependencies • Four types of resources that can be used in authorization policy ◦ Cluster, Topic, Group and Transactional ID 17
18. Example2: AWS MSK 18 { "Version": "2022-12-16", "Statement": [ { "Effect": "Allow", "Action": [ "kafka-cluster:Connect", "kafka-cluster:AlterCluster", "kafka-cluster:DescribeCluster" ], "Resource": [ "arn:aws:kafka:us-east-1:0123456789012:cluster/MyTestCluster/abcd1234-0123-abcd-5678-1234abcd-1" ] } ] }
19. Example2: AWS MSK • Need to use AWS IAM • AWS IAM requests only • Can be used through AWS Management Console, the API, or the AWS CLI 19
20. 20 Is there an easier way to have RBAC for streaming data?
21. RBAC in Relational Systems 21
22. RBAC in Relational Systems • The most familiar for data at rest 22
23. RBAC in Relational Systems • The most familiar for data at rest • Widely used and large user base ◦ From open source relational databases such as Postgres to commercial cloud data warehouses 23
24. RBAC in Relational Systems • The most familiar for data at rest • Widely used and large user base ◦ From open source relational databases such as Postgres to commercial cloud data warehouses • Why not use the same model for data in motion(streaming data)? 24
25. Relational Model for Streams 25
26. Relational Model for Streams 26 Relational model on streaming data
27. Relational Model for Streams 27 Relational model on streaming data Organize Process Secure
28. Example 28 CC_1 MSK_1 orders customers • Two Kafka clusters, one on Confluent Cloud(CC_1) and one on AWS(MSK_1) • Two topics, orders and customers
29. Hierarchical Namespacing • Same as relational systems ◦ Relation: static or dynamic set of tuples ▪ STREAM ▪ CHANGELOG ▪ MATERIALIZED VIEW ▪ TABLE 29
30. Hierarchical Namespacing • Same as relational systems ◦ Relation: static or dynamic set of tuples ▪ STREAM ▪ CHANGELOG ▪ MATERIALIZED VIEW ▪ TABLE ◦ Schema: a logical grouping of relational objects such as Streams, Changelogs, Materialized Views, and Tables. 30
31. Hierarchical Namespacing • Same as relational systems ◦ Relation: static or dynamic set of tuples ▪ STREAM ▪ CHANGELOG ▪ MATERIALIZED VIEW ▪ TABLE ◦ Schema: a logical grouping of relational objects such as Streams, Changelogs, Materialized Views, and Tables. ◦ Database: a logical grouping of schemas. 31
32. Example 32 online_store(Database) public(Schema) orders(Stream) customers(Changelog)
33. Example 33 CC_1 MSK_1 orders customers • Run continuous queries and build materialized views ◦ Create enriched_orders by joining orders with customers. ◦ Build a materialized view to compute daily order value per customer enriched_orders
34. Example 34 online_store(Database) public(Schema) orders(Stream) customers(Changelog) enriched_orders(Stream) daily_spend(Materialized View)
35. Key Concepts • User: represents an authenticated identity. 35
36. Key Concepts • User: represents an authenticated identity. • Objects (Securable Objects): represents an entity to which access can be granted. 36
37. Key Concepts • User: represents an authenticated identity. • Objects (Securable Objects): represents an entity to which access can be granted. • Privilege: is a defined level of access to an object. 37
38. Key Concepts • User: represents an authenticated identity. • Objects (Securable Objects): represents an entity to which access can be granted. • Privilege: is a defined level of access to an object. • Role: is an entity to which privileges can be granted. 38
39. RBAC for Streaming Data • Use the same SQL syntax as the relational systems ◦ CREATE/DROP/ALTER ◦ GRANT/REVOKE 39
40. Example 40 • Create a new role to access the enriched_orders stream and daily_spent materialized view CREATE ROLE test_role_1; GRANT ROLE test_role_1 TO USER user_1; GRANT SELECT ON RELATION online_store.public.enriched_orders TO ROLE test_role_1; GRANT SELECT ON RELATION online_store.public.daily_spent TO ROLE test_role_1;
41. Example 41 • Create a new role to access all objects in online_store database CREATE ROLE test_role_2; GRANT ROLE test_role_2 TO USER user_2; GRANT USAGE ON DATABASE online_store TO ROLE test_role_2;
42. Relational RBAC for Streaming Data • Works across streaming stores ◦ Apache Kafka, AWS Kinesis, … 42
43. Relational RBAC for Streaming Data • Works across streaming stores ◦ Apache Kafka, AWS Kinesis, … • Familiar syntax ◦ No need to learn new syntax/concepts 43
44. Relational RBAC for Streaming Data • Works across streaming stores ◦ Apache Kafka, AWS Kinesis, … • Familiar syntax ◦ No need to learn new syntax/concepts • Hierarchical namespacing 44
45. Relational RBAC for Streaming Data • Works across streaming stores ◦ Apache Kafka, AWS Kinesis, … • Familiar syntax ◦ No need to learn new syntax/concepts • Hierarchical namespacing • Unified view of all streaming data by abstracting the streaming stores 45
46. Relational RBAC for Streaming Data • Works across streaming stores ◦ Apache Kafka, AWS Kinesis, … • Familiar syntax ◦ No need to learn new syntax/concepts • Hierarchical namespacing • Unified view of all streaming data by abstracting the streaming stores • No limitations on roles ◦ Can build as many custom roles as needed 46
47. We have built exactly this at 47 Try all of these and much more for free at www.deltastream.io