Agile Application Development for Cloud-native Apps

Agile Application Development for Cloud-native Apps

5c3807aaaf0ffefe6c75e3dbbb8588b5?s=128

Michael Hausenblas

October 21, 2016
Tweet

Transcript

  1. © 2016 Mesosphere, Inc. All Rights Reserved. AGILE APPLICATION DEVELOPMENT

    FOR CLOUD-NATIVE APPS 1 Michael Hausenblas, Distributed Systems Jester | 2016-10-21 | Agile Tour London @mhausenblas
  2. © 2016 Mesosphere, Inc. All Rights Reserved. sys admin/SRE appops

    developer architect QA/test engineer data engineer
  3. © 2016 Mesosphere, Inc. All Rights Reserved. CHALLENGE 3

  4. © 2016 Mesosphere, Inc. All Rights Reserved. CHALLENGE 4

  5. © 2016 Mesosphere, Inc. All Rights Reserved. WHAT THE HECK

    IS CLOUD NATIVE? 5
  6. © 2016 Mesosphere, Inc. All Rights Reserved. 6 application orchestration

    container orchestration job scheduling containerization resource management base provisioning CloudFoundry, Heroku, OpenShift Kubernetes, Marathon, Swarm, Nomad, Firmament Chronos, Kubernetes, Nomad Docker, OCI/runc, rkt/appc, Apache Mesos Apache Mesos Vagrant, Puppet, Chef, Ansible, Salt machine management AWS, Azure, Google Cloud, OpenStack, vSphere
  7. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER
 101 7

    • Containers are not VMs • app-level dependency management • lightweight (startup time, footprint, average runtime) • isolation & resource consumption/accounting
  8. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER
 101 8

    • namespaces (resource isolation) • CLONE_NEWNS (Linux 2.4.19) filesystem mount points • CLONE_NEWUTS (Linux 2.6.19) hostname & NIS domain name • CLONE_NEWIPC (Linux 2.6.19) interprocess communication • CLONE_NEWPID (Linux 2.6.24) process ID space • CLONE_NEWNET (Linux 2.6.29) network system resources (network devices, IP routing tables, port numbers, etc.) • CLONE_NEWUSER (Linux 3.8) user & group ID space • cgroups (resource consumption throttling & accounting)
  9. © 2016 Mesosphere, Inc. All Rights Reserved. DOCKER 9 https://articles.microservices.com/containers-are-a-lie-2521afda1f81

  10. © 2016 Mesosphere, Inc. All Rights Reserved. ARTEFACTS LAYERING 10

    examples layer source artifacts .scala, .go, .py, .sh, etc.
 and config files packaging Dockerfile, OCI, rkt runtime Marathon app spec, Kubernetes RC, Docker compose
  11. © 2016 Mesosphere, Inc. All Rights Reserved. RELEASE CYCLE 11

  12. © 2016 Mesosphere, Inc. All Rights Reserved. RELEASE CYCLE 12

    dev CI/CD CI/CD time QA prod dev QA prod release
  13. © 2016 Mesosphere, Inc. All Rights Reserved. GOOD PRACTICES 13

  14. © 2016 Mesosphere, Inc. All Rights Reserved. CI/CD PIPELINES 14

  15. © 2016 Mesosphere, Inc. All Rights Reserved. CI/CD PIPELINES 15

  16. © 2016 Mesosphere, Inc. All Rights Reserved. VERSION CONTROL ALL

    THE THINGZ! 16 • code and infrastructure • ideally: use a DVCS (GitHub, Gitlab) • immutability (see also http://p24e.io) • be careful with sensitive bits (credentials)
  17. © 2016 Mesosphere, Inc. All Rights Reserved. HOW TO BUILD

    STUFF 17 1st generation • Jenkins
 https://jenkins.io/ • Bamboo
 https://www.atlassian.com/software/bamboo • TeamCity
 https://www.jetbrains.com/teamcity/ 2nd generation • Travis
 https://travis-ci.org/ • CircleCI
 https://circleci.com/ • concourse
 https://concourse.ci/
  18. © 2016 Mesosphere, Inc. All Rights Reserved. REGISTRIES: WHERE TO

    PUT YOUR BUILD ARTEFACTS 18 • Docker Hub
 https://hub.docker.com/ • Google Cloud
 https://cloud.google.com/tools/container-registry/ • AWS
 https://aws.amazon.com/ecr/ • CoreOS
 https://quay.io/ • SUSE Portus
 http://port.us.org/ • JFrog Artifactory
 https://www.jfrog.com/artifactory/ • Run your own
 https://docs.docker.com/registry/deploying/
  19. © 2016 Mesosphere, Inc. All Rights Reserved. A WORD ON

    TESTING 19
  20. © 2016 Mesosphere, Inc. All Rights Reserved. CONTINUOUS DEPLOYMENT 20

  21. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER ORCHESTRATION 21

  22. © 2016 Mesosphere, Inc. All Rights Reserved. WHAT DOES A

    SCHEDULER DO? 22
  23. © 2016 Mesosphere, Inc. All Rights Reserved. SERVICE DISCOVERY 23

    DNS-based easy to integrate SRV records no health checks TTL Proxy-based no port conflicts fast failover no UDP management of VIPs or service ports Application-aware developer fully in control and full-feature implementation effort requires distributed state management (ZK, etcd or Consul)
  24. © 2016 Mesosphere, Inc. All Rights Reserved. ZERO-DOWNTIME DEPLOYMENTS 24

    rolling deployment bring up new version 
 and terminate old ones until all old are gone blue-green deployment launch a new stack and switch traffic from old to new when new instances are healthy canary deployment bring up a new version, start by routing a small portion of traffic to the new app, and slowly increase
  25. © 2016 Mesosphere, Inc. All Rights Reserved. ZERO-DOWNTIME DEPLOYMENTS 25

    https://github.com/mhausenblas/zdd-lab
  26. © 2016 Mesosphere, Inc. All Rights Reserved. LOGGING AND MONITORING

    26
  27. © 2016 Mesosphere, Inc. All Rights Reserved. SCOPES 27 container

    host & intra-host service (app/business)
  28. © 2016 Mesosphere, Inc. All Rights Reserved. MONITORING 28 host

    container service* *) service: app-level + business-level collectd event router storage dashboard alerting collectd collectd node
  29. © 2016 Mesosphere, Inc. All Rights Reserved. MONITORING 29 •

    Event router • Riemann http://riemann.io/ • fluentd http://www.fluentd.org/ • (Kafka, logstash, Flume) • Storage • KairosDB https://kairosdb.github.io/ • InfluxDB https://influxdata.com/time-series-platform/ influxdb/ • (Local FS, HDFS)
  30. © 2016 Mesosphere, Inc. All Rights Reserved. MONITORING 30 •

    Multi-role tools • Sysdig* https://sysdig.com/blog/monitoring-mesos/ • Prometheus https://prometheus.io/ • DataDog* https://www.datadoghq.com/ • Ruxit* http://www.dynatrace.com/en/ruxit/ • Sensu https://sensuapp.org/ 
 *) available via DC/OS Universe
  31. © 2016 Mesosphere, Inc. All Rights Reserved. BUT WHAT ABOUT

    HUMANS? 31
  32. © 2016 Mesosphere, Inc. All Rights Reserved. 32 appops The

    person who writes an app is also the person responsible for operating the app in prod.
  33. © 2016 Mesosphere, Inc. All Rights Reserved. 33 It's not

    about provisioning
 a VM or installing a DC/OS cluster or replacing a faulty HDD …
 
 … this would be on the infrastructure team. appops
  34. © 2016 Mesosphere, Inc. All Rights Reserved. 34 appops speakerdeck.com/charity/devops-for-developers-building-an-effective-ops-org-1

  35. © 2016 Mesosphere, Inc. All Rights Reserved. CASE STUDIES 35

  36. © 2016 Mesosphere, Inc. All Rights Reserved. ISV: AUTODESK 36

    http://cloudengineering.autodesk.com/blog/2016/04/autodesk-is-forging-ahead-with-dcos.html
  37. © 2016 Mesosphere, Inc. All Rights Reserved. ONLINE RETAILER: OTTO

    37 https://dev.otto.de/2015/09/30/on-monoliths-and-microservices/
  38. © 2016 Mesosphere, Inc. All Rights Reserved. LEARNING RESOURCES 38

  39. © 2016 Mesosphere, Inc. All Rights Reserved. 39 WHERE CAN


    I LEARN MORE? 39 http://itrevolution.com/books/phoenix-project-devops-book/
  40. © 2016 Mesosphere, Inc. All Rights Reserved. 40 WHERE CAN


    I LEARN MORE? 40 https://dcos.io
  41. © 2016 Mesosphere, Inc. All Rights Reserved. Q & A

    41 • @mhausenblas • mhausenblas.info • michael@dcos.io https://dcos.io