Permitting agility while enforcing security - a story of making Docker a win-win proposition

presented @ DevSecCon London 2017

Transcript

1. Join the conversation #DevSecCon BY ALINA RADU Permitting agility while enforcing security - a story of making Docker a win-win proposition
2. Overview • PaaSTA: Yelp’s open source Platform as a service • Microservices in Docker containers • Jenkins • build pipelines • multiple steps • security-check step • runs a suite of security tests • Notification system for failures & runbook • Take aways
3. whoami • Alina • Software Engineer at Yelp London • Security team: Infrastructure security • Politehnica University of Bucharest Alumni • Avid traveller
4. Yelp’s Mission Connecting people with great local businesses.
5. PaaSTA: Platform as a Service Mesos, Marathon & Chronos • Platform as a Service • Mesos • distributed job scheduler • Marathon • Mesos framework for long running tasks • Chronos • Mesos framework to schedule batch jobs
6. PaaSTA: Platform as a Service Mesos, Marathon & Chronos • Mesos • distributed job scheduler • master & agents • offers compute resources to frameworks • frameworks provide task - schedules it on an agent
7. PaaSTA: Platform as a Service Mesos, Marathon & Chronos • Marathon • Mesos framework for long running tasks • upstart at datacenter level
8. PaaSTA: Platform as a Service Mesos, Marathon & Chronos • Chronos • Mesos framework to schedule batch jobs
9. PaaSTA: Platform as a Service • microservices in Docker containers • autoscaling cluster & resource specification • move from: • hundreds of specialized servers to • heterogenous Mesos agents • SOA architecture • operational ownership of individual services • from operations team to the service authors
10. PaaSTA contract • service: 1 git repo and 1 Dockerfile • Docker image • runs the service • same image multiple use cases: worker daemon vs web task • contains all the code necessary for the service • Service: • stateless • filesystem I/O but ephemeral disk • log to external processors (Yelp: Scribe or Kafka) • all checked, all good
11. Build Pipeline of a Service • configuration repository • Jenkins • orchestrates build and deployment • pipelines of sequential steps • security-check step
12. PaaSTA security-check • security status of the service • run a set of tests at every build • high level security health of the service • something changed? • actionable alerts for failures • faster response time from team
13. Security tests • Ubuntu packages up to date • Docker container best practices • Well known vulnerabilities • No secrets in the service repo • Python/Java dependency check
14. Ubuntu packages up to date • Check if the latest packages are installed against our apt repositories • apt-get update && apt-get --simulate dist-upgrade
15. Docker container best practices • container not running as user root • Dockerfile • Yelp maintained Docker images, no public images • Latest images • no packages pinned to certain versions • .dockerignore contains .git
16. Whitelisting • Certain version(s) of a package • Docker images • public images for open source projects • non-standard Yelp images
17. Well known vulnerabilities • bash shellshock • extending the list in the future • heartbleed - server side applications
18. No secrets into the service repo • detect and prevent high entropy strings from entering our code base • assumes the existing code has no secrets • checks only the new code • solution loosely based off truffleHog
19. Package vulnerability checking • look for known vulnerabilities in python/Java packages • Docker container scanning • vulnerabilities in Ubuntu packages • classified by severity • weekly report via email
20. Failures & alerts • Solves: ticket creation to track failures that need to be fixed • security-check failed? • email • Jira ticket • sensu • Runbook
21. How we got here • run bash tests with goss • get around the libc incompatibilities with rspec • moving everything to python • race condition between the Ubuntu packages that are updated upstream and the Docker base images we build daily • some services are not built regularly
22. You can do it too • You don’t have to • run our PaaS (PaaSTA) • use our containerisation solution (Docker) • write high-level security tests • integrate them in your build pipeline • open source our security tests suite next year
23. Take aways • PaaSTA • PaaS • microservices in Docker containers • build pipeline: multiple steps • security-check • runs a set of tests and send notifications • most important - service owners: • more aware of the security of their service • involved in keeping it safe
24. @YelpEngineering fb.com/YelpEngineers engineeringblog.yelp.com github.com/yelp
25. www.yelp.com/careers/ We're Hiring!
26. Join the conversation #DevSecCon Thank you
27. Join the conversation #DevSecCon Q & A