Kubernetes RBAC in microservices

Talk at Mercari Meetup for Microservices Platform: https://mercari.connpass.com/event/92168/

Explained how we (will) use RBAC in microservices architecture at Mercari, inc.

Transcript

1. Mercari Meetup for Microservices Platform, Jul 19, 2018
   Kubernetes RBAC in microservices
   

   View Slide

2. 2
   About me
   @spesnova
   SRE at Mercari Microservices Platform team
   Kubernetes Tokyo Community Organizer
   

   View Slide

3. 3
   Agenda
   1. AuthN/Z in Kubernetes
   2. Design Principles
   3. RBAC Policy
   

   View Slide

4. AuthN/Z in Kubernetes
   

   View Slide

5. What is Kubernetes RBAC?
   

   View Slide

6. 6
   What is Kubernetes RBAC?
   Role Based Access Control
   

   View Slide

7. 7
   a method of regulating access to computer or
   network resources based on the roles of
   individual users within an enterprise
   What is Kubernetes RBAC?
   https://kubernetes.io/docs/reference/access-authn-authz/rbac/
   

   View Slide

8. 8
   Role based?
   ● User based - easy but inefficient
   ● Team based - intermediate
   ● Role based - efficient but not easy
   

   View Slide

9. 9
   4 Objects in Kubernetes RBAC
   Roles
   ClusterRoles
   RoleBindings
   ClusterRoleBindings
   

   View Slide

10. Roles and ClusterRoles
    

    View Slide

11. 11
    Roles and ClusterRoles
    Roles
    ClusterRoles
    Verbs
    Resources
    Verbs
    Resources
    

    View Slide

12. 12
    Verbs
    Verbs
    “create”
    “get”, “list”
    “update”
    “patch”
    “delete”, “deletecollection”
    “watch”
    

    View Slide

13. 13
    Resources
    Resources
    “pods”
    “deployments”
    “configmaps”
    “secrets”
    “namespaces”
    “resourcequotas”
    “nodes”
    

    View Slide

14. 14
    Namespace scoped resources
    Namespace scoped
    resources
    “pods”
    “deployments”
    “configmaps”
    “secrets”
    “namespaces”
    “resourcequotas”
    “nodes”
    

    View Slide

15. 15
    Cluster scoped resources
    Cluster scoped
    resources
    “pods”
    “deployments”
    “configmaps”
    “secrets”
    “namespaces”
    “resourcequotas”
    “nodes”
    

    View Slide

16. 16
    Roles example - a role can read pods in default namespace
    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
    namespace: default
    name: pod-reader
    rules:
    - apiGroups: [""] # "" indicates the core API group
    resources: ["pods"]
    verbs: ["get", "watch", "list"]
    

    View Slide

17. 17
    ClusterRoles example - a clusterrole can read pods in all namespaces
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
    # “namespace” is not required
    name: global-pod-reader
    rules:
    - apiGroups: [""] # "" indicates the core API group
    resources: ["pods"]
    verbs: ["get", "watch", "list"]
    

    View Slide

18. RoleBindingss
    and
    ClusterRoleBindings
    

    View Slide

19. 19
    4 Objects in Kubernetes RBAC
    Roles
    ClusterRoles
    RoleBindings
    ClusterRoleBindings
    

    View Slide

20. 20
    RoleBindings and ClusterRoleBindings
    RoleBindings
    ClusterRoleBindings
    Role/ClusterRole
    Subjects
    ClusterRole
    Subjects
    

    View Slide

21. 21
    Subjects
    Subjects
    “User”
    “Group”
    “ServiceAccount”
    

    View Slide

22. 22
    RoleBinding example - spesnova can read pods in default namespace
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
    name: read-pods
    namespace: default
    subjects:
    - kind: User
    name: spesnova@example.com
    apiGroup: rbac.authorization.k8s.io
    roleRef:
    kind: Role
    name: pod-reader
    apiGroup: rbac.authorization.k8s.io
    

    View Slide

23. 23
    RoleBinding example - spesnova can read pods in default namespace
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
    name: read-pods
    namespace: default
    subjects:
    - kind: User
    name: spesnova@example.com
    apiGroup: rbac.authorization.k8s.io
    roleRef:
    kind: ClusterRole
    name: global-pod-reader
    apiGroup: rbac.authorization.k8s.io
    

    View Slide

24. 24
    RoleBinding example - spesnova can read pods in all namespace
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
    name: read-pods
    # “namespace” is not required
    subjects:
    - kind: User
    name: spesnova@example.com
    apiGroup: rbac.authorization.k8s.io
    roleRef:
    kind: ClusterRole
    name: global-pod-reader
    apiGroup: rbac.authorization.k8s.io
    

    View Slide

25. Default ClusterRoles
    

    View Slide

26. 26
    Default ClusterRoles / admin
    Allows read/write access to most resources
    in a namespace, including the ability to create
    roles and rolebindings within the namespace.
    https://kubernetes.io/docs/reference/access-authn-authz/rbac/
    

    View Slide

27. 27
    Default ClusterRoles / edit
    Allows read/write access to most objects in a
    namespace. It does not allow viewing or
    modifying roles or rolebindings.
    https://kubernetes.io/docs/reference/access-authn-authz/rbac/
    

    View Slide

28. 28
    Default ClusterRoles / view
    Allows read-only access to see most objects
    in a namespace. It does not allow viewing
    roles or rolebindings. It does not allow
    viewing secrets, since those are escalating.
    https://kubernetes.io/docs/reference/access-authn-authz/rbac/
    

    View Slide

29. 29
    How to test RBAC settings?
    $ kubectl auth can-i get pods /
    --namespace=default /
    --as=spesnova@example.com
    yes
    

    View Slide

30. Authentication in GKE
    

    View Slide

31. 31
    Authentication in GKE (skip details because of time limitation)
    Google OAuth2
    (OpenID Connect Tokens authentication)
    

    View Slide

32. Design Principles
    

    View Slide

33. End-to-End Ownership
    

    View Slide

34. 34
    The purpose of microservices
    Getting agility at large scale org&system
    

    View Slide

35. 35
    The purpose of microservices
    How to get agility?
    How to make people high-performer?
    

    View Slide

36. With Great power comes great responsibility
    

    View Slide

37. Great power requires
    great responsibility
    

    View Slide

38. 38
    I was in an organization holding 1000+ engineers
    Small responsibility makes
    people low-performer
    

    View Slide

39. 39
    Great responsibility means...
    End-to-End ownership
    

    View Slide

40. 40
    Our platform mission
    Build system & organization which
    people have E2E ownership
    

    View Slide

41. Hide complexity behind infrastructure
    

    View Slide

42. 42
    Hide complexity behind infrastructure
    Microservices architecture itself
    is already complicated
    

    View Slide

43. 43
    Microservices architecture
    

    View Slide

44. 44
    Microservices architecture
    Each microservices become simple
    but operations become complex
    

    View Slide

45. 45
    Hide complexity behind infrastructure
    Keep each components simple
    Hide complexity behind infrastructure
    

    View Slide

46. 46
    Hide complexity behind infrastructure
    Make each RBAC policy simple
    Handle the complexity with tools
    

    View Slide

47. RBAC Policy
    

    View Slide

48. Namespace/Team per microservices
    

    View Slide

49. 49
    Our microservices conventions
    Team per microservice
    Namespace per microservice
    

    View Slide

50. 50
    Namespace per microservice
    microservice A microservice B microservice C
    Kubernetes Cluster
    

    View Slide

51. 51
    Namespace per microservice
    microservice A microservice B microservice C
    namespace A namespace B namespace C
    Kubernetes Cluster
    

    View Slide

52. 52
    Namespace and Team per microservice
    microservice A microservice B microservice C
    namespace A namespace B namespace C
    Kubernetes Cluster
    team A team B team C
    

    View Slide

53. Give namespace-admin
    

    View Slide

54. 54
    Give namespace-admin = End-to-End ownership
    microservice A microservice B microservice C
    namespace A namespace B namespace C
    Kubernetes Cluster
    team A team B team C
    

    View Slide

55. 55
    Give namespace-admin = End-to-End ownership
    microservice A microservice B microservice C
    namespace A namespace B namespace C
    Kubernetes Cluster
    team A team B team C
    RoleBiding A RoleBiding B RoleBiding C
    

    View Slide

56. 56
    RoleBinding example - hello team members are namespace admin in hello ns
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
    name: service-admins
    namespace: hello
    subjects:
    - kind: User
    name: spesnova@example.com
    apiGroup: rbac.authorization.k8s.io
    roleRef:
    kind: ClusterRole
    name: admin
    apiGroup: rbac.authorization.k8s.io
    

    View Slide

57. Microservice diversity
    

    View Slide

58. 58
    Microservice diversity
    Roles
    namespace A
    Kubernetes Cluster
    team A / service Admins
    RoleBindings
    Secrets
    Deployments
    ConfigMaps
    ClusterRoleBiding admins
    

    View Slide

59. 59
    Some teams want “read-only”
    namespace A
    Kubernetes Cluster
    team A / service Viewers
    Deployments
    ConfigMaps
    ClusterRoleBiding viewers
    

    View Slide

60. 60
    RoleBinding example - some members are namespace view in hello namespace
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
    name: service-viewers
    namespace: hello
    subjects:
    - kind: User
    name: dtan4@example.com
    apiGroup: rbac.authorization.k8s.io
    roleRef:
    kind: ClusterRole
    name: view
    apiGroup: rbac.authorization.k8s.io
    

    View Slide

61. 61
    Some teams want to restrict RBAC editing
    namespace A
    Kubernetes Cluster
    team A / service Editors
    Deployments
    ConfigMaps
    ClusterRoleBiding editors
    Secrets
    

    View Slide

62. 62
    RoleBinding example - some members are namespace edit in hello namespace
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
    name: service-editors
    namespace: hello
    subjects:
    - kind: User
    name: babarot@example.com
    apiGroup: rbac.authorization.k8s.io
    roleRef:
    kind: ClusterRole
    name: edit
    apiGroup: rbac.authorization.k8s.io
    

    View Slide

63. 63
    Using default ClusterRoles
    Accept diversity but keep it simple
    

    View Slide

64. 64
    Microservice diversity
    If required, they can create custom roles
    

    View Slide

65. 65
    But...avoid to create custom role as possible (RBAC rabbit hole)
    $ kubectl get clusterroles/admin -o yaml | wc -l
    457
    457 lines yaml !!
    

    View Slide

66. Handle complexity with tools
    

    View Slide

67. 67
    Our microservices conventions
    Team per microservice
    Namespace per microservice
    

    View Slide

68. 68
    Namespace and Team per microservice
    microservice A microservice B microservice C
    namespace A namespace B namespace C
    Kubernetes Cluster
    team A team B team C
    

    View Slide

69. 69
    Namespace and Team per microservice, then...
    So many namespaces and teams!!
    

    View Slide

70. 70
    We aim to have 200+ microservices
    Impossible to create
    hundreds RoleBindings by hand
    

    View Slide

71. 71
    Too many namespaces and teams?
    No, it is intended
    

    View Slide

72. 72
    Hide complexity behind infrastructure
    Keep each components simple
    Hide complexity behind infrastructure
    

    View Slide

73. 73
    Keep each components simple
    If multiple services/teams in a namespace,
    RBAC must become much more complicated
    

    View Slide

74. 74
    Hide complexity behind infrastructure
    Keep each components simple
    Hide complexity behind infrastructure
    

    View Slide

75. 75
    Microservice Starter Kit
    An internal Terraform module to reduce
    provisioning burdun from developers and
    manage infrastructure as code
    https://speakerdeck.com/b4b4r07/terraform-ops-for-microservices?slide=13
    

    View Slide

76. 76
    Automate RoleBindings creation with microservice-starter-kit (WIP)
    module “hello-service” {
    ...
    service_admins = [
    "spesnova@example.com",
    "tcnksm@example.com",
    "b4b4r07@example.com",
    ]
    ...
    

    View Slide

77. 77
    Automate RoleBindings creation with microservice-starter-kit (WIP)
    …
    service_editors = [
    "dtan4@example.com",
    ]
    …
    service_viewers = [
    "terry@example.com",
    ]
    

    View Slide

78. 78
    Automate RoleBindings creation with microservice-starter-kit (WIP)
    microservice A microservice B microservice C
    namespace A namespace B namespace C
    Kubernetes Cluster
    team A team B team C
    RoleBiding A RoleBiding B RoleBiding C
    

    View Slide

79. 79
    Need to implement kubernetes_rolebinding Terraform resource (WIP)
    No official resource!
    

    View Slide

80. Recap
    

    View Slide

81. 81
    Recap
    Delegation
    Simplicity Tools
    Keep RBAC policy
    simple as possible
    Delegate enough
    permission to teams
    Handle complexity
    with tools
    

    View Slide

82. End
    

    View Slide