Auditing in Kubernetes 101

Have you ever wondered who created particular changes in your cluster, when they created it or what resources were modified? All of such information about “what sequence of events lead to this scenario” can be obtained using the powerful audit logging feature. In this talk, we will first go over what audit logs are and how to leverage them to stay informed with what goes on in your cluster. Keeping both performance impact and accountability in mind, we will then walk through examples of policy configurations to enforce best security practices, detect misuse and make your cluster more compliant. We’ll also do a demo of setting up auditing on a cluster and inspecting the logs. Finally, we will see what future improvements are planned for this feature and how you can provide feedback and get involved.

Transcript

1. Nikhita Raghunath
   Loodse
   Auditing in Kubernetes 101
   

   View Slide

2. WHO AM I
   ● Software Engineer at Loodse
   ● Member of the Kubernetes Steering
   Committee
   ● Technical Lead for SIG Contributor
   Experience
   ● CNCF Ambassador
   Github - nikhita
   Twitter - TheNikhita
   

   View Slide

3. SECRET CONTAINING PASSWORD
   IN YOUR CLUSTER
   

   View Slide

4. SECRET CONTAINING PASSWORD
   IN YOUR CLUSTER
   SECRET GOT UPDATED TO
   MYSTERIOUS VALUE
   

   View Slide

5. SECRET CONTAINING PASSWORD
   IN YOUR CLUSTER
   SECRET GOT UPDATED TO
   MYSTERIOUS VALUE
   LOGS
   

   View Slide

6. Logs from the Pod
   @TheNikhita
   

   View Slide

7. Logs from the Pod
   @TheNikhita
   

   View Slide

8. Logs from the Pod
   Events
   @TheNikhita
   

   View Slide

9. Logs from the Pod
   Events
   @TheNikhita
   

   View Slide

10. Logs from the Pod
    Events
    Apiserver Logs
    @TheNikhita
    

    View Slide

11. Logs from the Pod
    Events
    Apiserver Logs
    @TheNikhita
    

    View Slide

12. AUDIT LOGS!
    

    View Slide

13. {
    "kind": "Event",
    "apiVersion": "audit.k8s.io/v1",
    "level": "Metadata",
    "auditID": "7684b057-7e2d-4188-a6ae-8fc51afd0c9d",
    "stage": "ResponseComplete",
    "requestURI": "/api/v1/namespaces/default/secrets",
    "verb": "create",
    "user": {
    "username": "minikube-user",
    "groups": [
    "system:masters",
    "system:authenticated"
    ]
    },
    "sourceIPs": [
    "X.Y.Z.1"
    ],
    "objectRef": {
    "resource": "secrets",
    "namespace": "default",
    "name": "mysecret",
    "apiVersion": "v1"
    },
    "responseStatus": {
    "metadata": {},
    "code": 201
    },
    "requestReceivedTimestamp": "2020-02-12T18:06:04.577792Z",
    "stageTimestamp": "2020-02-12T18:06:04.584173Z",
    }
    AUDIT
    EVENTS
    @TheNikhita
    

    View Slide

14. WHAT HAPPENED
    "verb": "create",
    @TheNikhita
    

    View Slide

15. ON WHAT DID IT HAPPEN
    "objectRef": {
    "resource": "secrets",
    "namespace": "default",
    "name": "mysecret",
    "apiVersion": "v1"
    },
    @TheNikhita
    

    View Slide

16. WHEN DID IT HAPPEN
    "requestReceivedTimestamp":
    "2020-02-12T18:06:04.577792Z",
    "stageTimestamp":
    "2020-02-12T18:06:04.584173Z",
    @TheNikhita
    

    View Slide

17. WHO DID IT
    "user": {
    "username": "minikube-user",
    "groups": [
    "system:masters",
    "system:authenticated"
    ]
    },
    @TheNikhita
    

    View Slide

18. WHERE WAS IT INITIATED
    "sourceIPs": [
    "1.2.3.4"
    ],
    @TheNikhita
    

    View Slide

19. THAT’S A LOT OF LOGS!
    

    View Slide

20. LET’S CONTROL THE
    VERBOSITY
    

    View Slide

21. LET’S CONTROL THE
    VERBOSITY
    WHAT TO LOG WHEN TO LOG
    

    View Slide

22. LET’S CONTROL THE
    VERBOSITY
    WHAT TO LOG WHEN TO LOG
    YAML
    

    View Slide

23. AUDIT POLICY
    apiVersion: audit.k8s.io/v1
    kind: Policy
    rules:
    - level: Metadata
    omitStages:
    - RequestReceived
    resources:
    - group: ""
    resources:
    - secrets
    @TheNikhita
    

    View Slide

24. AUDIT POLICY
    apiVersion: audit.k8s.io/v1
    kind: Policy
    rules:
    - level: Metadata
    omitStages:
    - RequestReceived
    resources:
    - group: ""
    resources:
    - secrets
    @TheNikhita
    

    View Slide

25. AUDIT POLICY
    apiVersion: audit.k8s.io/v1
    kind: Policy
    rules:
    - level: Metadata
    omitStages:
    - RequestReceived
    resources:
    - group: ""
    resources:
    - secrets
    @TheNikhita
    

    View Slide

26. AUDIT POLICY
    apiVersion: audit.k8s.io/v1
    kind: Policy
    rules:
    - level: Metadata
    omitStages:
    - RequestReceived
    resources:
    - group: ""
    resources:
    - secrets
    @TheNikhita
    

    View Slide

27. WHEN TO LOG
    1. RequestReceived - Audit handler receives
    request
    @TheNikhita
    

    View Slide

28. WHEN TO LOG
    1. RequestReceived - Audit handler receives
    request
    2. ResponseStarted - For long running requests
    @TheNikhita
    

    View Slide

29. WHEN TO LOG
    1. RequestReceived - Audit handler receives
    request
    2. ResponseStarted - For long running requests
    3. ResponseComplete - Response body completed
    @TheNikhita
    

    View Slide

30. WHEN TO LOG
    1. RequestReceived - Audit handler receives
    request
    2. ResponseStarted - For long running requests
    3. ResponseComplete - Response body completed
    4. Panic - Event generated when panic occurs
    @TheNikhita
    

    View Slide

31. RequestReceived
    ResponseComplete
    Response
    ResponseStarted
    Panic
    Request
    Response
    Kube APIserver
    

    View Slide

32. Request
    Kube APIserver
    

    View Slide

33. RequestReceived
    Request
    Kube APIserver
    

    View Slide

34. RequestReceived Response
    Request
    Kube APIserver
    

    View Slide

35. RequestReceived Response
    Panic
    Request
    Kube APIserver
    

    View Slide

36. RequestReceived Response
    ResponseStarted
    Request
    Kube APIserver
    

    View Slide

37. RequestReceived
    ResponseComplete
    Response
    ResponseStarted
    Request
    Kube APIserver
    

    View Slide

38. RequestReceived
    ResponseComplete
    Response
    ResponseStarted
    Request
    Response
    Kube APIserver
    

    View Slide

39. RequestReceived
    ResponseComplete
    Response
    ResponseStarted
    Panic
    Request
    Response
    Kube APIserver
    

    View Slide

40. WHAT TO LOG
    apiVersion: audit.k8s.io/v1
    kind: Policy
    rules:
    - level: Metadata
    omitStages:
    - RequestReceived
    resources:
    - group: ""
    resources:
    - secrets
    GROUP/VERSION
    RESOURCE
    VERBS
    @TheNikhita
    

    View Slide

41. WHAT TO LOG
    apiVersion: audit.k8s.io/v1
    kind: Policy
    rules:
    - level: Metadata
    omitStages:
    - RequestReceived
    resources:
    - group: ""
    resources:
    - secrets
    @TheNikhita
    

    View Slide

42. LEVELS
    1. None - don’t log these requests
    @TheNikhita
    

    View Slide

43. LEVELS
    1. None - don’t log these requests
    2. Metadata - only request metadata
    @TheNikhita
    

    View Slide

44. LEVELS
    1. None - don’t log these requests
    2. Metadata - only request metadata
    3. Request - ,, + request body
    @TheNikhita
    

    View Slide

45. LEVELS
    1. None - don’t log these requests
    2. Metadata - only request metadata
    3. Request - ,, + request body
    4. RequestResponse - ,, + response body
    @TheNikhita
    

    View Slide

46. RECOMMENDATIONS FOR
    WRITING POLICIES
    

    View Slide

47. - level: Metadata
    resources:
    - group: ""
    resources:
    - secrets
    - configmaps
    - group: authentication.k8s.io
    resources:
    - tokenreviews
    Only log at
    Metadata level for
    sensitive resources
    @TheNikhita
    

    View Slide

48. - level: None
    nonResourceURLs:
    - '/healthz*'
    - /version
    - '/swagger*'
    Don’t log read-only URLs
    @TheNikhita
    

    View Slide

49. Log at RequestResponse level for
    critical resources
    Log at atleast Metadata level for all
    resources
    @TheNikhita
    

    View Slide

50. rules:
    - level: RequestResponse
    resources:
    - group: ""
    resources: ["pods"]
    - level: Metadata
    resources:
    - group: ""
    resources: ["pods/log", "pods/status"]
    @TheNikhita
    

    View Slide

51. rules:
    - level: RequestResponse
    resources:
    - group: ""
    resources: ["pods"]
    - level: Metadata
    resources:
    - group: ""
    resources: ["pods/log", "pods/status"]
    Evaluated in
    top-down order
    @TheNikhita
    

    View Slide

52. rules:
    - level: RequestResponse
    resources:
    - group: ""
    resources: ["pods"]
    - level: Metadata
    resources:
    - group: ""
    resources: ["pods/log", "pods/status"]
    Status calls can be large and
    high-volume
    @TheNikhita
    

    View Slide

53. More examples at
    https://github.com/kubernetes/kubernetes/blob/master/cl
    uster/gce/gci/configure-helper.sh
    @TheNikhita
    

    View Slide

54. WHERE DO THESE LOGS GO
    

    View Slide

55. BACKEND
    LOG WEBHOOK
    @TheNikhita
    

    View Slide

56. BACKEND
    LOG WEBHOOK
    ● Writes events to disk ● Sends events to external API
    @TheNikhita
    

    View Slide

57. BACKEND
    LOG WEBHOOK
    ● Writes events to disk
    ● --audit-log-path
    ● Sends events to external API
    ● --audit-webhook-config-file
    @TheNikhita
    

    View Slide

58. BACKEND
    LOG WEBHOOK
    ● Writes events to disk
    ● --audit-log-path
    ● Sends events to external API
    ● --audit-webhook-config-file
    --audit-policy-file
    @TheNikhita
    

    View Slide

59. HOW ARE THESE LOGS
    SENT TO THE BACKEND
    

    View Slide

60. BATCHING
    BATCH BLOCKING BLOCKING-STRICT
    @TheNikhita
    

    View Slide

61. BATCHING
    BATCH BLOCKING BLOCKING-STRICT
    Buffers events
    & processes
    in batches
    Blocks APIserver
    responses to
    process individual
    events
    Failure at
    RequestReceived
    stage leads to
    failure of whole call
    @TheNikhita
    

    View Slide

62. BATCHING
    BATCH BLOCKING BLOCKING-STRICT
    --audit-webhook-mode
    --audit-log-mode
    @TheNikhita
    

    View Slide

63. BATCHING
    BATCH BLOCKING BLOCKING-STRICT
    --audit-webhook-mode
    --audit-log-mode
    @TheNikhita
    

    View Slide

64. UPDATING AUDIT POLICY
    

    View Slide

65. UPDATING AUDIT POLICY
    RESTART OF APISERVER
    

    View Slide

66. UPDATING AUDIT POLICY
    RESTART OF APISERVER
    

    View Slide

67. UPDATING AUDIT POLICY
    

    View Slide

68. UPDATING AUDIT POLICY
    UPDATING A K8S RESOURCE
    

    View Slide

69. UPDATING AUDIT POLICY
    UPDATING A K8S RESOURCE
    

    View Slide

70. DYNAMIC AUDIT CONFIGURATION
    --audit-dynamic-configuration
    --feature-gates=DynamicAuditing=true
    --runtime-config=auditregistration.k8s.io/v1alpha1=true
    @TheNikhita
    

    View Slide

71. DYNAMIC AUDIT CONFIGURATION
    apiVersion: auditregistration.k8s.io/v1alpha1
    kind: AuditSink
    metadata:
    name: mysink
    spec:
    policy:
    level: Metadata
    stages:
    - ResponseComplete
    webhook:
    throttle:
    qps: 10
    burst: 15
    clientConfig:
    url: "https://audit.app"
    @TheNikhita
    

    View Slide

72. DYNAMIC AUDIT CONFIGURATION
    apiVersion: auditregistration.k8s.io/v1alpha1
    kind: AuditSink
    metadata:
    name: mysink
    spec:
    policy:
    level: Metadata
    stages:
    - ResponseComplete
    webhook:
    throttle:
    qps: 10
    burst: 15
    clientConfig:
    url: "https://audit.app"
    @TheNikhita
    

    View Slide

73. DYNAMIC AUDIT CONFIGURATION
    apiVersion: auditregistration.k8s.io/v1alpha1
    kind: AuditSink
    metadata:
    name: mysink
    spec:
    policy:
    level: Metadata
    stages:
    - ResponseComplete
    webhook:
    throttle:
    qps: 10
    burst: 15
    clientConfig:
    url: "https://audit.app"
    @TheNikhita
    

    View Slide

74. DYNAMIC AUDIT CONFIGURATION
    apiVersion: auditregistration.k8s.io/v1alpha1
    kind: AuditSink
    metadata:
    name: mysink
    spec:
    policy:
    level: Metadata
    stages:
    - ResponseComplete
    webhook:
    throttle:
    qps: 10
    burst: 15
    clientConfig:
    url: "https://audit.app"
    @TheNikhita
    

    View Slide

75. SECURITY
    PERFORMANCE
    @TheNikhita
    

    View Slide

76. SECURITY
    PERFORMANCE
    Write access to feature = Read access to all cluster data
    cluster-admin level privilege
    @TheNikhita
    

    View Slide

77. SECURITY
    PERFORMANCE
    Write access to feature = Read access to all cluster data
    cluster-admin level privilege
    Increase in CPU/Memory Usage
    Don’t use too many sinks
    @TheNikhita
    

    View Slide

78. KEP
    https://github.com/kubernetes/enhancements/pull/1259
    #sig-auth slack channel on k8s slack
    @TheNikhita
    

    View Slide

79. View Slide

80. LOG COLLECTOR PATTERNS
    

    View Slide

81. LOG COLLECTOR PATTERNS
    Audit Log File + Fluentd
    @TheNikhita
    

    View Slide

82. LOG COLLECTOR PATTERNS
    Audit Webhook File +
    Logstash
    @TheNikhita
    

    View Slide

83. LOG COLLECTOR PATTERNS
    Audit Webhook File +
    Falco
    @TheNikhita
    

    View Slide

84. HOW ARE AUDIT LOGS
    HELPFUL
    

    View Slide

85. UNDERSTANDING K8S INTERNALS
    Analysing system calls show how
    different components interact
    @TheNikhita
    

    View Slide

86. DETECTING MISCONFIGURATIONS
    “Who deleted this resource?”
    @TheNikhita
    

    View Slide

87. TROUBLESHOOTING ISSUES
    Analysing calls which trigger
    HTTP errors
    @TheNikhita
    

    View Slide

88. PERFORMANCE ISSUES
    “Which app is generating lots of calls”
    @TheNikhita
    

    View Slide

89. CONCLUSION
    ● Audit logs can give us a lot of information of what goes on in our cluster
    ● To control what should be logged, we write audit policies
    ● Recommendations for writing audit policies
    ● Different audit backends
    ● Batching methods
    ● Dynamic Audit Configuration
    ● Log Collector Patterns
    

    View Slide

90. THANK YOU
    

    View Slide