Agile Application Development for Cloud-native Apps

Talk at Agile Tour London, see http://2016.agiletourlondon.co.uk/Session/agile-application-development-for-cloud-native-apps/

Transcript

1. © 2016 Mesosphere, Inc. All Rights Reserved. AGILE APPLICATION DEVELOPMENT FOR CLOUD-NATIVE APPS 1 Michael Hausenblas, Distributed Systems Jester | 2016-10-21 | Agile Tour London @mhausenblas
2. © 2016 Mesosphere, Inc. All Rights Reserved. sys admin/SRE appops developer architect QA/test engineer data engineer
3. © 2016 Mesosphere, Inc. All Rights Reserved. CHALLENGE 3
4. © 2016 Mesosphere, Inc. All Rights Reserved. CHALLENGE 4
5. © 2016 Mesosphere, Inc. All Rights Reserved. WHAT THE HECK IS CLOUD NATIVE? 5
6. © 2016 Mesosphere, Inc. All Rights Reserved. 6 application orchestration container orchestration job scheduling containerization resource management base provisioning CloudFoundry, Heroku, OpenShift Kubernetes, Marathon, Swarm, Nomad, Firmament Chronos, Kubernetes, Nomad Docker, OCI/runc, rkt/appc, Apache Mesos Apache Mesos Vagrant, Puppet, Chef, Ansible, Salt machine management AWS, Azure, Google Cloud, OpenStack, vSphere
7. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER
 101 7 • Containers are not VMs • app-level dependency management • lightweight (startup time, footprint, average runtime) • isolation & resource consumption/accounting
8. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER
 101 8 • namespaces (resource isolation) • CLONE_NEWNS (Linux 2.4.19) filesystem mount points • CLONE_NEWUTS (Linux 2.6.19) hostname & NIS domain name • CLONE_NEWIPC (Linux 2.6.19) interprocess communication • CLONE_NEWPID (Linux 2.6.24) process ID space • CLONE_NEWNET (Linux 2.6.29) network system resources (network devices, IP routing tables, port numbers, etc.) • CLONE_NEWUSER (Linux 3.8) user & group ID space • cgroups (resource consumption throttling & accounting)
9. © 2016 Mesosphere, Inc. All Rights Reserved. DOCKER 9 https://articles.microservices.com/containers-are-a-lie-2521afda1f81
10. © 2016 Mesosphere, Inc. All Rights Reserved. ARTEFACTS LAYERING 10 examples layer source artifacts .scala, .go, .py, .sh, etc.
 and config files packaging Dockerfile, OCI, rkt runtime Marathon app spec, Kubernetes RC, Docker compose
11. © 2016 Mesosphere, Inc. All Rights Reserved. RELEASE CYCLE 11
12. © 2016 Mesosphere, Inc. All Rights Reserved. RELEASE CYCLE 12 dev CI/CD CI/CD time QA prod dev QA prod release
13. © 2016 Mesosphere, Inc. All Rights Reserved. GOOD PRACTICES 13
14. © 2016 Mesosphere, Inc. All Rights Reserved. CI/CD PIPELINES 14
15. © 2016 Mesosphere, Inc. All Rights Reserved. CI/CD PIPELINES 15
16. © 2016 Mesosphere, Inc. All Rights Reserved. VERSION CONTROL ALL THE THINGZ! 16 • code and infrastructure • ideally: use a DVCS (GitHub, Gitlab) • immutability (see also http://p24e.io) • be careful with sensitive bits (credentials)
17. © 2016 Mesosphere, Inc. All Rights Reserved. HOW TO BUILD STUFF 17 1st generation • Jenkins
 https://jenkins.io/ • Bamboo
 https://www.atlassian.com/software/bamboo • TeamCity
 https://www.jetbrains.com/teamcity/ 2nd generation • Travis
 https://travis-ci.org/ • CircleCI
 https://circleci.com/ • concourse
 https://concourse.ci/
18. © 2016 Mesosphere, Inc. All Rights Reserved. REGISTRIES: WHERE TO PUT YOUR BUILD ARTEFACTS 18 • Docker Hub
 https://hub.docker.com/ • Google Cloud
 https://cloud.google.com/tools/container-registry/ • AWS
 https://aws.amazon.com/ecr/ • CoreOS
 https://quay.io/ • SUSE Portus
 http://port.us.org/ • JFrog Artifactory
 https://www.jfrog.com/artifactory/ • Run your own
 https://docs.docker.com/registry/deploying/
19. © 2016 Mesosphere, Inc. All Rights Reserved. A WORD ON TESTING 19
20. © 2016 Mesosphere, Inc. All Rights Reserved. CONTINUOUS DEPLOYMENT 20
21. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER ORCHESTRATION 21
22. © 2016 Mesosphere, Inc. All Rights Reserved. WHAT DOES A SCHEDULER DO? 22
23. © 2016 Mesosphere, Inc. All Rights Reserved. SERVICE DISCOVERY 23 DNS-based easy to integrate SRV records no health checks TTL Proxy-based no port conflicts fast failover no UDP management of VIPs or service ports Application-aware developer fully in control and full-feature implementation effort requires distributed state management (ZK, etcd or Consul)
24. © 2016 Mesosphere, Inc. All Rights Reserved. ZERO-DOWNTIME DEPLOYMENTS 24 rolling deployment bring up new version 
 and terminate old ones until all old are gone blue-green deployment launch a new stack and switch traffic from old to new when new instances are healthy canary deployment bring up a new version, start by routing a small portion of traffic to the new app, and slowly increase
25. © 2016 Mesosphere, Inc. All Rights Reserved. ZERO-DOWNTIME DEPLOYMENTS 25 https://github.com/mhausenblas/zdd-lab
26. © 2016 Mesosphere, Inc. All Rights Reserved. LOGGING AND MONITORING 26
27. © 2016 Mesosphere, Inc. All Rights Reserved. SCOPES 27 container host & intra-host service (app/business)
28. © 2016 Mesosphere, Inc. All Rights Reserved. MONITORING 28 host container service* *) service: app-level + business-level collectd event router storage dashboard alerting collectd collectd node
29. © 2016 Mesosphere, Inc. All Rights Reserved. MONITORING 29 • Event router • Riemann http://riemann.io/ • fluentd http://www.fluentd.org/ • (Kafka, logstash, Flume) • Storage • KairosDB https://kairosdb.github.io/ • InfluxDB https://influxdata.com/time-series-platform/ influxdb/ • (Local FS, HDFS)
30. © 2016 Mesosphere, Inc. All Rights Reserved. MONITORING 30 • Multi-role tools • Sysdig* https://sysdig.com/blog/monitoring-mesos/ • Prometheus https://prometheus.io/ • DataDog* https://www.datadoghq.com/ • Ruxit* http://www.dynatrace.com/en/ruxit/ • Sensu https://sensuapp.org/ 
 *) available via DC/OS Universe
31. © 2016 Mesosphere, Inc. All Rights Reserved. BUT WHAT ABOUT HUMANS? 31
32. © 2016 Mesosphere, Inc. All Rights Reserved. 32 appops The person who writes an app is also the person responsible for operating the app in prod.
33. © 2016 Mesosphere, Inc. All Rights Reserved. 33 It's not about provisioning
 a VM or installing a DC/OS cluster or replacing a faulty HDD …
 
 … this would be on the infrastructure team. appops
34. © 2016 Mesosphere, Inc. All Rights Reserved. 34 appops speakerdeck.com/charity/devops-for-developers-building-an-effective-ops-org-1
35. © 2016 Mesosphere, Inc. All Rights Reserved. CASE STUDIES 35
36. © 2016 Mesosphere, Inc. All Rights Reserved. ISV: AUTODESK 36 http://cloudengineering.autodesk.com/blog/2016/04/autodesk-is-forging-ahead-with-dcos.html
37. © 2016 Mesosphere, Inc. All Rights Reserved. ONLINE RETAILER: OTTO 37 https://dev.otto.de/2015/09/30/on-monoliths-and-microservices/
38. © 2016 Mesosphere, Inc. All Rights Reserved. LEARNING RESOURCES 38
39. © 2016 Mesosphere, Inc. All Rights Reserved. 39 WHERE CAN
 I LEARN MORE? 39 http://itrevolution.com/books/phoenix-project-devops-book/
40. © 2016 Mesosphere, Inc. All Rights Reserved. 40 WHERE CAN
 I LEARN MORE? 40 https://dcos.io
41. © 2016 Mesosphere, Inc. All Rights Reserved. Q & A 41 • @mhausenblas • mhausenblas.info • michael@dcos.io https://dcos.io