マスター タイトルの書式設定 What are security measures for kubernetes

マスター タイトルの書式設定 Profile ● Name : Hiroki Akiba ● Company : ● Relation with CNCF ： kubernetes Certified Service Provider (Silver Members) 1

マスター タイトルの書式設定 Agenda • Choices to run kubernetes in production • Cloud native security measures • Each security measure of kubernetes • Conclusion 2

マスター タイトルの書式設定 Choices to run kubernetes in production 3

マスター タイトルの書式設定 Commonly used container management tools 4 • This is the top 10 most commonly used container management tools. Reference) https://www.cncf.io/wp-content/uploads/2020/08/CNCF_Survey_Report.pdf

マスター タイトルの書式設定 5 Shared responsibility model of major cloud vendor AWS GCP Azure Reference) https://aws.amazon.com/compliance/shared-responsibility-model/?nc1=h_ls Reference) https://services.google.com/fh/files/misc/google-cloud-security-foundations-guide.pdf Reference) https://docs.microsoft.com/en-us/azure/security/fundamentals/shared-responsibility

マスター タイトルの書式設定 Uses public cloud managed services Not use ECS/EKS Amazon ECS Amazon EKS k8s Open Shift EC2 Fargate EC2 Fargate EC2 ECR Docker hub Orchestration Compute Resource Image Registry etc 6 Choises to use kubernetes environment (e.g. AWS) etc

マスター タイトルの書式設定 7 Mapped to a shared responsibility model (e.g. AWS) Uses public cloud managed services Not use ECS/EKS Amazon ECS Amazon EKS k8s、 Open Shift、 etc EC2 Fargate EC2 Fargate Application Application Application Application Application Container Container Container Container Container Worker Node (data plane) OS・Middle OS・Middle OS・Middle OS・Middle OS・Middle EC2 EC2 EC2 EC2 EC2 HW・NW HW・NW HW・NW HW・NW HW・NW Master Node (control plane) OS・Middle OS・Middle OS・Middle OS・Middle OS・Middle EC2 EC2 EC2 EC2 EC2 HW・NW HW・NW HW・NW HW・NW HW・NW • AWS and user responsibility for security when running container.

マスター タイトルの書式設定 Cloud native security measures 8

マスター タイトルの書式設定 9 Security & Compliance of CNCF Landscape (As of November 2,2020)

マスター タイトルの書式設定 Kubernetes potential security threats 10 • There are various ways that an attacker could attempt to compromise your kubernetes cluster and the applications running on it. • An example of a security threat  External attacks targeting security holes in the control plane  Attacks targeting security holes in containers and nodes  Attacks aimed at leaking administrator credentials  Attacks aimed at mistakes in permission settings  etc Reference) https://kubernetes-security.info/

マスター タイトルの書式設定 11 Points to consider for kubernetes security Authz Workload isolate Vulnerability Encryption Backup Firewall Monitoring Authn Governamce / Compliance Security category Boundary of each component Public Cloud k8s Cluster Worker Node Master Node Namespace Pod Container Pod Container Application Application Secret management

マスター タイトルの書式設定 12 Map countermeasure tools for each security category authz workload isolation firewall authn secret management vulnerability monitoring encryption backup governance/compl iance IAM KMS Secret Manager KMS kubesec Kubernetes- External-secret Sealed- secretes Inspector EC2 K8s dashboard Guard Duty ACM Backup System Manager Cloud watch ES ECR Account VPC NW ACL SG EKS WAF Shield Cognito Directory Macie Security HUb Admission Control kubeaudit Security category e.g.) AWS kubernetes CNCF ecosystem

マスター タイトルの書式設定 Each security measure of kubernetes 13

マスター タイトルの書式設定 14 Authn / Authz / Admission control Public Cloud k8s Cluster Master Node  API Version • Extension • Core • Apps ***  Resources • Deployment • Node • Pod ***  Action • Create • Get • List *** Human Constraint of Namespace For Pod For User Admission Control Authn Authz  k8s cluster Guard rail • ResourceQuota • LimitRange • AlwaysPullImages • NamespaceLifecycle • Priority • Pod Security Policy *** IAM OpenID Connect IAM Role for Service Account IAM aws-auth • For examples, authentication and authorization in AWS EKS.

マスター タイトルの書式設定 15 Workload isolation Public Cloud k8s Cluster Namespace Pod Container Pod ⑤Service K8s Cluster Worker Node Worker Node ③Service ③Service Namespace ②Service ④Service ④Service ②Service ①Service ①Service Container ⑤Service ⑥Service ⑥Service Workload isolation patterm ①Cluster ②Namespace ③Node ④Pod ⑤Container ⑥Volume • Separete each workload to minimize the impact of security threats.

マスター タイトルの書式設定 16 Firewall Public Cloud (e.g. AWS) k8s Cluster WorkerNode Master Node Such as SaaS Client Container registory such as ECR Service outside kubnertes cluster such as RDS Container Container Private subnet Network  Communication type application service operation、release work Maintenance server such as EC2 ELB ｚ • Control network communication between each component.

マスター タイトルの書式設定 Encryption • Use key management tool to encrypt the following stateful data.  Etcd, Persisten storage, Database  Git repogitory, container registory, etc Public Cloud(e.g. AWS) Block NFS Object Persistent storage Key management tool vault KMS Container registory Relational database Git repogitory ripogitory、db、etc worker cluster etc 17

マスター タイトルの書式設定 Secret management 18  Problem • It is not safe to manage kubernetes secret resource with manifest file in Git.  Solution • Use the following secure management tools  Kubesec  Kubernetes-external-secrets  Sealed-secrets  HashiCorp vault  etc apiVersion: v1 kind: Secret metadata: name: service01 namespace: default data: MYSQL_PASSWORD: YWJjZAo= MYSQL_ROOT_PASSWORD: cm9vdDEyMzQ1Njc4Cg== type: Opaque 例：secret.yml The value of 「abcd」and「root12345678」is base64 encoded. “MYSQL_PASSWORD”: “abcd”, "MYSQL_ROOT_PASSWORD": "root12345678" service01 kubesec CLI # kubectl apply -f secret.yml push

マスター タイトルの書式設定 Usage example of sealed-secrets Helm & kubectl CLI # helm install --name sealed-secrets --namespace kube-system stable/sealed-secrets kubeseal-cli # kubeseal --fetch-cert --controller-namespace=kube-system --controller-name=sealed-secrets > pub-cert.pem apiVersion: bitnami.com/v1alpha1 kind: SealedSecret metadata: creationTimestamp: null name: mysql-env namespace: deault spec: encryptedData: MYSQL_PASSWORD: [kubesealで暗号化された値] MYSQL_ROOT_PASSWORD: [kubesealで暗号化された値] template: metadata: creationTimestamp: null name: mysql-env status: {} secret-encrypt.yml push Kubectl-cli pull ① create SealedSecretController CI/CD tool or Namespace：kube-system sealed-secrets sealed-secrets-controller Namespace：default sealedsecret/ mysql-env secret/ mysql-env ② create key for encryption. and encrypt secret file. ③ apply secret-encrypt.yml to kubernetes cluster. then, secret resource is created transparently. # kubeseal --format=yaml --cert=pub-cert.pem < secret.yml > secret-encrypt.yml # kubectl apply -f secret-encrypt.yml Reference) https://github.com/bitnami-labs/sealed-secrets 19

マスター タイトルの書式設定 How to management secret values 20 • Separate from application code and container images. • Make it unique with production and staging and development. • Set the expiration date and rotate the encryption key. • The secret value is specified by the volume, not the environment variable. environment variables have the following risks.  There are cases where environment variables are written to files or logs when a process crashes.  Environment variables can be viewed with the “kubeclt describe pod” command.  Environment variables can be viewed with the “docker inspect” command at the docker runtime.

マスター タイトルの書式設定 21 Backup of runtime resource of kubernetes Public Cloud k8s Cluster Woker Node Backup Backup /Restore  Use case with VELERO • Backup/restore • Disaster recovery • Cluster migration • etc  Backup target • Various reources such as deployment, svc, pv, etc  Backup data storage location • AWS S3 • Azure Blob Storage • Google Cloud Storage • etc など object or block storage backupData backupData

マスター タイトルの書式設定 Managing kubernetes resouces with GitOps • If the workload running in the kubernetes is stateless, all version control with Git for updating or roleback. • Use CICD tooles like GitOps such as flux, argo, Jenkins x. Git repository is single source of true. Gitops tools flux argo Jenkins X developer operator Git code Git config Container registry build Config update cluster Code change Countinuous Integration Countinuous Deployment Deploy (update,roleback) etc 22

マスター タイトルの書式設定 Base container image 23 Vulnerability checking throughout the container life cycle Public Cloud (e.g. AWS) kubernetes Cluster Master/Woker Node Container private registry(ECR, etc) Git repository (codeCommit, etc) Server・OS Container runtime K8s manifest container image workload Development server public container registry such as Github or Quay Build Ship Run CI tool (CodeBuild, S3, etc) Build tool (docker, Bazel, etc) CDツール(CodeDeploy,etc) Penetration test Check artifact container images CIS kubernetes Benchmark test Configuration test • Security measures at shift left are important. Check base container images Unit & Configuration test

マスター タイトルの書式設定 Vulnerability considerations for cluster and node  Updating Version for kubernetes cluster and node. • Update kubernetes version regularly. If used Helm or operator, the same. • Update master and worker node os version regularly.  Managing permissions for node. • Uses optimized host os. e.g)CoreOS, Container-Optimized OS, Bottlerocket • Enable rootless mode. supported in podman, experimental in docker v19.03. • Restrict remote login to host and container. same for sudo. • Prevent local file system mounts on the node. 24

マスター タイトルの書式設定 Vulnerability considerations for container of build and ship phaze  Build phaze • Minimum required package and library tools. e.g) shell,cli,etc • Use the trusted container based image provided by cloud vendors. • Use the latest version and vulnerabilities checked container image.  Ship phaze • Check the container image regularly, not just at build time. • Add container tag like semantic version. not「latest」. e.g) v1.2.3 • Delete old container images regularly. • Use container image signature checking. e.g) DOCKER_CONTENT_TRUST = 1 25

マスター タイトルの書式設定 Vulnerability check of build and ship phase • Check using a security check tool suitable for each layer. terraform cloudformation Infrastructure as code kubernetes manifest conftest kubesec Each layer Check tools Check summary Docker file trivy hadlint Scans container images for known vulnerabilities Checks workload configuration for security issues Detection security risks in cloud infrastructure Specify custom checks for kubernetes resources Checks docker file for best practice etc container image 26

マスター タイトルの書式設定 Vulnerability considerations for container of run phaze  Run phaze • Prohibit promotion to privileged user and process execution. • Prohibit writing to the root file system. • Limit the mount path of volumes on the host. • Use linux capabilities. e.g) CAP_NET_RAW = disable • Limit number of process on the container. e.g) pod-max-pids, podPidsLimit • Disable unused listen port on the container. • Defines the request and limit of cpu and memory resource. • Challenge DevSecOps. 27

マスター タイトルの書式設定 Vulnerability considerations for container of run phaze  Run phaze • Separate the UID and GID of the container OS and host OS. • Limit Input/Output access from container to storage. • Limit input/output network access on container. • Limit number of file descriptor. e.g) ulimit –n 10 • Use the functions of AppArmor or seccomp. • Detecting and blocking suspicious activity. Some of the paid vendor tools. 28

マスター タイトルの書式設定 Introducing some setting of Pod Security Policy Item Filed name Type Summary Example process privileged bool Allow privileged mode false allowPrivilegeEscalation bool Allow promotion to root user false defaultAllowPrivilegeEscalation bool Default value above false runAsUser runAsUserStrategyOptio n Specify the execution UID SecurityContext > runAsNonRoot：true 1000 runAsGroup runAsGroupStrategyOpti on Specify the execution GID 1000 filesystem readOnlyRootFileSystem bool Make the root file system read only true allowedHostPaths []AllowedHostPath Specify a AllowList for hostpath - pathPrefix: "/foo" readOnly: true fsGroup FSGroupStrategyOption Set groups of file systems to allow 1000 capabiliti es requiredDropCapabilities []string List to remove capabilities set in container requiredDropCapabilities: - NET_RAW - ＊＊ allowedCapabilitties []string List of capabilities to allow. If not specifiled, the default set is implicitly granted. allowedCapabilities: - KILL - CHOWN - ＊＊ • Pod security policy is still beta and has not reached GA. There is also a plan to deprecate pod security policy and move to Open Policy Agent. So be careful about future tends. 29

マスター タイトルの書式設定 Vulnerability check of run phase • Check using a security check tool suitable for each layer. gatekeeper Each layer Check tools Check summary trivy Scans container images for known vulnerabilities Checks best practices in workload configuration Detection security risks in cloud infrastructure Specify custom checks for kubernetes resources Checks the cluster against the CIS Benchmark Kube-bench Kube-hunter Checks for cluster and node-level security vulnerabilities terraform Infrastructure as code kubernetes cluster (master/worker node) cloudformation container image Kubernetes workload etc etc Recommends resource limits and requests based on actual resource usage 30

マスター タイトルの書式設定 31 The three pillars of observability 1. Metrics • usage of cpu and mem and disk, etc • number of api request, network traffic, cache hit rate, page load time, etc 2. Logs • syslog of server and container, log of application and middleware • network packet, database query, api request, release operation, audit log, etc 3. Traces • Tracking api call times between microservcies Reference) https://www.elastic.co/blog/observability-with-the-elastic-stack Reference) https://grafana.com/blog/2019/10/21/whats-next-for-observability/

マスター タイトルの書式設定 The CNCF End User Technology Radar 32 • The most commonly adopted tools are open source. (Prometheus, Grafana, Elastic, etc) • Many companies are using multiple tools. half of the companies are using 5 or more tools. Reference) https://radar.cncf.io/

マスター タイトルの書式設定 Logging for audits and investigations 33  Aggregate various logs for each of the following layers, and make them ready for investigation and auditing.  Cloud layer • cloud operation, host node, network communication, etc • Enable auditing and network logging options provided by cloud vendors.  Kubernetes cluster layer • api-server request, master/worker node, pod, etc • Enable audit policy on kubernetes. Reference) https://kubernetes.io/docs/tasks/debug-application-cluster/audit  Container layer • Application logs such as nginx, tomcat, etc • Aggregate the StdOut and StdErr of the container with a logging tool such as fluentd or elastic. • Logging Architecture Reference) https://kubernetes.io/docs/concepts/cluster-administration/logging/

マスター タイトルの書式設定 34 Governance / Compliance Public Cloud k8s Cluster Master Node Woker Node AdmissionController AuthZ Webhook e.g) AWS CloudWatch application Query Data Policy(Rego) Rule AdmissionReview AuthZ gatekeeper Rule For audit extension of AdmissionController gatekeeper offload of authn an authz

マスター タイトルの書式設定 Introducing Falco 35  For example, monitor the behavior of the following containers.  Container runs in privileged mode.  Mount sensitive directory path.  Browsing sensitive files.  Writing device files.  Browsing system binary files.  Out-of-band network connection. etc  Cloud Native Security Hub publishes Falco rules for multiple purposes. Reference) https://securityhub.dev/ Note) Falco cannot detect writes to host paths bound-mounted by processes in the container or indirect writes via symbolic links. Therefore, if you want to detect it, monitor file access with auditd which is a package provided by each Linux distribution.

マスター タイトルの書式設定 Conclusion 36

マスター タイトルの書式設定 Conclusion 37  Set priorities and take continuous measures.  Defense in depth and minimize blast radius.  Leverage security tools such as OSS or paid vendor solutions or cloud.

マスター タイトルの書式設定 Thanks 38