Containerization at Rabobank: Securing the production cluster

Talk delivered at the Dutch OpenShift Usergroup meetup on 14 March 2018 at Devoteam Amsterdam.

Transcript

1. Containerization at Rabobank Securing the production cluster March 14, 2018 Author: Gopal Ramachandran 1
2. Containerization journey: Where are we? 2 OCP 3.0 OCP 3.7 OCP 3.1 (PoC) Q3 2016 Today OCP 3.7 MSP (5 teams on Production) Q2 2017 OCP 3.4 MVP (1st customer Onboarded) 2018 Full self- service PaaS
3. Securing the platform: Why? 3 • Infrastructure platform (on-premise, multi-tenant) • Protect assets • Security for customer applications • Personal information • Comply with laws, standards • New tech and evolving rapidly (opensource based)
4. Securing the platform: Challenge 4 • Security assessment sign-off needed for Production status • Traditional security model vs new technology “System must record all attempts to initiate a communication channel (cryptographically secured or not) or initiate intended data exchanges with other system, or deposit data in intended data storage areas.” Clause-7,Item-68:
5. Our approach to security 5 Agile approach • Evolving baseline security • Sessions with security officer • Continuous input/feedbacks from customers (tenants) • Grant new capabilities to applications only when needed
6. Out-of-the-box security 6 • Infrastructure • RHEL Atomic VMs, SELinux enabled • RBAC • LDAP/ AD integrated, OpenShift Roles and RoleBindings • OpenShift Secrets • Audit logging • Namespace isolation • Container security • Service signing certificates, Network Policies, and more…
7. Overview of the cluster 7 M M I E E E I W W W W W W W W W W W W I I F5 F5 F5 Datacenter 1 Datacenter 2 Prod Test M E I W Master Etcd node Infra node Worker node 50 nodes 4 cores per worker node 30 GB per node (50% mem use)
8. OpenShift Secrets 8 • Sensitive information: keys, certificates, passwords • Separate sensitive information from application pods • Easy access via env variables/volumes • Securely delivered to nodes (TLS) • Centralized management and storage
9. Secrets as code 9 apiVersion: v1 kind: Pod metadata: name: use-secret-pod spec: containers: - name: secret-test-container image: myapp env: - name: SECRET_USERNAME valueFrom: secretKeyRef: name: top-secret key: username restartPolicy: Always apiVersion: v1 kind: Secret metadata: name: top-secret data: username: bmVuYWQ= password: aWtuZXd5b3V3b3VsZHRyeXRoaXM=
10. Secrets encrypted at rest 10 • Added as feature with version 3.6 • Not just secrets, any Kubernetes resource (like configmaps) maybe stored encrypted. • Requires etcd schema v3. • Simple to implement (using an encryption_config.yaml file). • Migration of existing resources possible via oc adm command.
11. Audit logging 11 • Openshift provides for auditing all API calls to the API server on the masters • Enable on the master-config.yaml • Snippet of the audit.log: auditConfig: enabled: true 2017-10-23T13:54:25.651410376Z AUDIT: id="feff384c-c2ab-4f66-8e4e-af1000fb5aa1" ip="10.239.24.14" method="PUT" user="Gopal.Ramachandran@rabobank.nl" groups="\"system_admins\",\"system:authenticated:oauth\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/oapi/v1/groups/system_admins" 2017-10-23T13:54:25.656698821Z AUDIT: id="feff384c-c2ab-4f66-8e4e-af1000fb5aa1" response="200"
12. Auditd overview 12
13. Get alerted when… 13 • Brute force detection based on failed login attempts • Someone gives someone high privilege role • Some plays with etcd data Master audit.log rsyslog daemonset Arcsight (alerting) API server LB
14. Developers • containers Run as root • Use arbitrary user id’s • chown • -R someuser:root /app && chmod –R g+rwX /app From • dockerhub Image control: Secured source • Make RHEL • -based images OpenSCAP • image scanning Container security 14
15. Final thoughts 15 Platform can be secured • Security is a shared responsibility • Principle of least access •
16. Any questions ? 16