PaaSTA - Yelp's Platform as a Service

A talk given at the Docker Dublin meetup February 24 2016

Transcript

1. PaaSTA - Yelp’s Platform as a Service Rob Johnson @rob_johnson_
2. - Rob Johnson - Operations Team at Yelp - Spend most of my time working on PaaSTA. Who Am I:
3. Yelp’s Mission: Connecting people with great local businesses.
4. Yelp Stats: As of Q4 2015 86M 32 70% 95M
6. Yelp’s Platform as a Service
8. How did we get here?
9. Yelp’s monolith is ~3 million LoC (that’s just the Python).* Increasing number of developers. *as of 28/09/2015
10. - Code deployments become increasingly difficult to coordinate. - Surface area for impact of a bug greatly increases.
11. SOA: Round 1
12. - Statically defined list of hosts to deploy a service on. - Operations handle deciding which hosts to deploy to.
13. - Manually configure Nagios for each service. - Manual deployment system. Lots of rsync wrappers to push code around.
14. Evaluating existing options
15. YOUR PAAS HERE
16. An aside: What do PaaS’s all have in common?
17. Code Delivery
18. Code Delivery Scheduling
19. Code Delivery Scheduling Discovery
20. Code Delivery Scheduling Discovery Monitoring
21. YOUR PAAS HERE
22. Building PaaSTA
23. Code Delivery Scheduling Discovery Monitoring
24. Code Delivery Scheduling Discovery Monitoring
25. Code Delivery Scheduling Discovery Monitoring
26. Code Delivery Scheduling Discovery Monitoring
27. Code Delivery Scheduling Discovery Monitoring
28. Moving Configuration Ownership to Developers
29. We don’t want the operations team to be a blocker on the deployments of services. Service authors should be able to self administer their application, describing how it should be deployed.
30. Configuration is declarative in its nature: it sets expectation of what the deployment of a system should look like. The tools we use should automate matching that expectation.
31. We want configuration to have the same attributes as regular code. By storing configuration in a git repository, we get the same benefits we get in a regular project, and can use the same workflows.
32. robj@dev22-devc:~/yelpsoa-configs (master) $ tree example_service/ example_service/ ├── chronos-mesosstage.yaml ├── deploy.yaml ├── marathon-PROD.yaml ├── marathon-STAGE.yaml ├── monitoring.yaml ├── port ├── service.yaml ├── smartstack.yaml └── status_port
33. robj@dev22-devc:~/yelpsoa-configs (master) $ cat example_service/marathon-PROD.yaml --- canary: cpus: 0.1 deploy_group: prod.canary instances: 1 mem: 500 nerve_ns: main main: cpus: 0.1 deploy_group: prod.non_canary instances: 3 mem: 500
34. robj@dev22-devc:~/yelpsoa-configs (master) $ cat example_service/monitoring.yaml --- team: paasta ticket: true page: false runbook: y/example_service
35. robj@dev22-devc:~/yelpsoa-configs (master) $ cat example_service/smartstack.yaml --- main: advertise: [superregion] discover: superregion healthcheck_timeout_s: 6 proxy_port: 20603
36. Making it Our Own
37. Superregion Region Habitat A quick insight into Yelp’s Infrastructure
38. Optimising for latency vs reliability
39. Each of our slaves have attributes that describe their environment: super region, region, habitat,. By making Mesos aware of these, we can be smarter about how we deploy for higher availability.
40. Service Authors can now make a decision on the priority for their service: - If we discover a ‘bigger’ latency zone, then we’re optimising for availability (all clients in the same latency zone can discover that instance). - If we discover at a ‘smaller’ latency zone, then we’re optimising for latency. (only clients in the same latency zone can discover that instance).
43. Context Aware Monitoring
46. Demo Time!
47. - paasta check http://ascii.yelpcorp.com/a/be5a11 - paasta local-run http://ascii.yelpcorp.com/a/0e3b08 - paasta status http://ascii.yelpcorp.com/a/a1b5cf - paasta metastatus http://ascii.yelpcorp.com/a/08a87d
48. PaaSTA https://github.com/yelp/paasta (Check out the Videos and PaaSTA Principles!) ◦ Sensu: https://sensuapp.org/ ◦ Mesos: http://mesos.apache.org/ ◦ Marathon: https://mesosphere.github.io/marathon/ ◦ Smartstack: http://nerds.airbnb.com/smartstack-service- discovery-cloud/