Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud-native Application Stacks: a lightly-opinionated survey (May 2016)

Cloud-native Application Stacks: a lightly-opinionated survey (May 2016)

A survey of the platforms and projects that make up the cloud-native computing landscape, circa May 2016.

Peter Bourgon

May 25, 2016
Tweet

More Decks by Peter Bourgon

Other Decks in Programming

Transcript

  1. Cloud-native application stacks
    From orchestrators to monitoring
    A lightly-opinionated survey

    View Slide

  2. These are just, like,
    my opinions, man.

    View Slide

  3. Amazon Web Services • Google Cloud Platform • Microsoft Azure

    Mesosphere • Apache Mesos • DC/OS • Marathon • Aurora

    Chronos • Kubernetes • Docker • Docker Swarm • Deis

    Rancher • Kismatic • Apprenda • Cloud Foundry • OpenShift

    OpenStack • CoreOS • Project Atomic • Snappy Ubuntu

    Joyent Triton • SmartOS • RancherOS • Vagrant • Terraform

    Consul • Serf • Packer • Nomad • Flocker • Ceph • Flannel

    Weave • Canal • Prometheus • InfluxDB • Nagios • Zabbix

    Elasticsearch • OpenZipkin • LightStep • Appdash

    View Slide

  4. View Slide

  5. The Datacenter as a Computer
    An Introduction to the Design of
    Warehouse-Scale Machines
    http://www.cs.berkeley.edu/~rxin/db-papers/WarehouseScaleComputing.pdf

    View Slide

  6. https://twitter.com/alicemazzy/status/655306196128280576

    View Slide

  7. View Slide

  8. View Slide

  9. View Slide

  10. http://architecturebestinfo.blogspot.de/2015/09/computer-architecture.html

    View Slide

  11. View Slide

  12. CPU Mem Disk Net

    View Slide

  13. CPU Mem Disk Net
    Cooling
    Circuits
    Power
    App
    OS User
    Silicon

    View Slide

  14. CPU Mem Disk Net
    Cooling
    Circuits
    Power
    App
    OS User
    Silicon

    View Slide

  15. CPU Mem Disk Net App
    OS

    View Slide

  16. App
    OS
    CPU Mem Disk Net

    View Slide

  17. CPU Mem Disk Net Data
    Batch
    Biz logic
    Resource mgmt
    Abstraction Index

    View Slide

  18. CPU Mem Disk Net Data
    Batch
    Biz logic
    Resource mgmt
    Abstraction Index

    View Slide

  19. CPU Mem Disk Net Data
    Batch
    Biz logic
    Resource mgmt
    Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Abstraction Index

    View Slide

  20. CPU Mem Disk Net Data
    Batch
    Biz logic
    Resource mgmt
    Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Scheduler
    Service discovery
    Config mgmt
    Abstraction
    Virtualization
    Index

    View Slide

  21. CPU Mem Disk Net Data
    Batch
    Biz logic
    Resource mgmt
    Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Scheduler
    Service discovery
    Config mgmt
    Abstraction
    Virtualization
    Database
    Data
    warehouse
    Biz logic
    Index

    View Slide

  22. Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Scheduler
    Service discovery
    Config mgmt
    Virtualization
    Database
    Data
    warehouse
    Biz logic

    View Slide

  23. Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Virtualization
    Database
    Data
    warehouse
    Biz logic
    Scheduler
    Service discovery
    Config mgmt

    View Slide

  24. Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Virtualization
    Database
    Data
    warehouse
    Biz logic
    Scheduler
    Config mgmt
    Service discovery

    View Slide

  25. Database
    RDBMS AP CP Document Transactional
    Oracle
    MySQL
    Postgres
    Dynamo
    Voldemort
    Cassandra
    Riak
    Chubby
    ZooKeeper
    etcd
    Consul
    Mongo
    Couch
    Redis
    F1
    Cockroach
    Rethink

    View Slide

  26. Data warehouse
    Hadoop RedShift BigQuery
    Spark Mahout
    Hbase Pig ···

    View Slide

  27. Lambda Architecture
    Realtime + Batch
    Storm Kafka Druid

    View Slide

  28. Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Virtualization
    Database
    Data
    warehouse
    Biz logic
    Scheduler
    Config mgmt
    Service discovery

    View Slide

  29. Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Virtualization
    Data Biz logic
    Scheduler
    Service discovery
    Config mgmt

    View Slide

  30. Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Virtualization
    Data Biz logic
    Scheduler
    Service discovery
    Config mgmt

    View Slide

  31. Biz logic
    Monolith

    View Slide

  32. Biz logic
    Foo
    service
    Bar
    service
    Baz
    service
    Users
    service
    Auth
    service
    Search
    service
    Reports
    service
    Audit
    service

    View Slide

  33. Biz logic
    Foo
    service
    Bar
    service
    Baz
    service
    Users
    service
    Auth
    service
    Search
    service
    Reports
    service
    Audit
    service

    View Slide

  34. Biz logic
    Foo
    service
    Bar
    service
    Baz
    service
    Users
    service
    Auth
    service
    Search
    service
    Reports
    service
    Audit
    service
    Continuous
    delivery
    Continuous
    integration
    Monitoring
    Tracing
    Safety
    Application logging Event sourcing

    View Slide

  35. Sean Treadway, SoundCloud

    View Slide

  36. Sean Treadway, SoundCloud

    View Slide

  37. Sean Treadway, SoundCloud

    View Slide

  38. Sean Treadway, SoundCloud

    View Slide

  39. Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Scheduler
    Service discovery
    Config mgmt
    Virtualization
    Data Biz logic

    View Slide

  40. Monitoring
    Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Scheduler
    Service discovery
    Config mgmt
    Virtualization
    Data Biz logic

    View Slide

  41. Virtualization
    Distributed
    storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt Data
    Monitoring
    Compute
    resources

    View Slide

  42. Virtualization
    Distributed
    storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Compute
    resources

    View Slide

  43. Physical
    network
    Virtualization
    Distributed
    storage
    Local
    (scratch
    Software defined

    network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Compute
    resources

    View Slide

  44. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View Slide

  45. Application
    Platform
    Infrastructure
    Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View Slide

  46. http://trapptechnology.com/saas-iaas-paas-decoding-cloud-acronyms/

    View Slide

  47. View Slide

  48. Software defined network
    Monitoring
    Distr.
    storage
    Scheduler
    Config mgmt
    Data
    Service discovery
    Compute
    resources
    Virtualization

    View Slide

  49. ¯\_(ϑ)_/¯
    Software defined network
    Monitoring
    Distr.
    storage
    Scheduler
    Config mgmt
    Data
    Service discovery
    Compute
    resources
    Virtualization

    View Slide

  50. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization
    Our new warehouse computer

    View Slide

  51. Cloud platforms

    View Slide

  52. Virtualization
    OS
    Middleware
    Runtime
    Data
    Applications
    Servers
    Storage
    Networking
    { }

    View Slide

  53. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization
    AWS

    View Slide

  54. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization
    EC2 S3
    RedShift
    VPC
    Route53
    CloudWatch
    CloudFormation
    RDS
    Machine
    Learning
    Kinesis
    Dynamo
    ELB
    ECS
    Lambda

    View Slide

  55. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization
    GCP

    View Slide

  56. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization
    GCE
    GKE GAE
    BigTable
    SQL
    Datastore BigQuery
    Cloud Monitoring
    Data*
    Deployment
    manager
    Cloud
    Storage
    Cloud
    Networking

    View Slide

  57. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization
    Azure?

    View Slide

  58. Newer

    View Slide

  59. Open-source platforms

    View Slide

  60. Virtualization
    OS
    Middleware
    Runtime
    Data
    Applications
    Servers
    Storage
    Networking
    { }

    View Slide

  61. https://mesos.apache.org/ https://mesosphere.com/

    View Slide

  62. https://thecustomizewindows.com/2014/12/mesosphere-data-center-operating-system/

    View Slide

  63. https://aphyr.com/posts/326-call-me-maybe-chronos

    View Slide

  64. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization
    Master
    Slave
    Slave
    Slave
    ZooKeeper
    Marathon Aurora Chronos
    DNS
    Master
    Master

    View Slide

  65. http://kubernetes.io/

    View Slide

  66. http://www.slideshare.net/brendandburns/defrag-2014-41815642

    View Slide

  67. http://kubernetes.io/

    View Slide

  68. Compute
    resources
    Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    API server
    Kubelet
    Kubelet
    Kubelet
    etcd
    SkyDNS
    Various
    Volume mounts
    Controllers

    View Slide

  69. http://devopscube.com/how-to-setup-and-configure-docker-swarm-cluster/

    View Slide

  70. http://blog.daocloud.io/swarm_analysis_part1/

    View Slide

  71. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization
    Docker
    Docker
    Docker
    Swarm
    Scheduler
    Node discovery
    libnetwork
    libkv

    View Slide

  72. View Slide

  73. http://deis.io/ https://www.engineyard.com/

    View Slide

  74. https://rancher.com

    View Slide

  75. http://kismatic.com http://apprenda.com

    View Slide

  76. https://pivotal.io/platform

    View Slide

  77. https://www.openstack.org/

    View Slide

  78. View Slide

  79. https://www.openshift.com

    View Slide

  80. More enterprisey?

    View Slide

  81. Less opinionated

    View Slide

  82. Cloud-native operating systems

    View Slide

  83. }
    Virtualization
    OS
    Middleware
    Runtime
    Data
    Applications
    Servers
    Storage
    Networking

    View Slide

  84. https://coreos.com/

    View Slide

  85. https://en.wikipedia.org/wiki/CoreOS

    View Slide

  86. http://projectatomic.io/

    View Slide

  87. https://access.redhat.com/products/red-hat-atomic-enterprise-platform

    View Slide

  88. https://developer.ubuntu.com/en/snappy/

    View Slide

  89. https://rancher.com
    OS

    View Slide

  90. https://smartos.org/
    https://joyent.com

    View Slide

  91. http://www.hypervizor.com/diagrams/

    View Slide

  92. ???

    View Slide

  93. Open-source components

    View Slide






  94. Virtualization
    OS
    Middleware
    Runtime
    Data
    Applications
    Servers
    Storage
    Networking

    View Slide

  95. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View Slide

  96. Virtualization
    Vagrant
    Docker
    rkt

    View Slide

  97. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View Slide

  98. Schedulers
    Kubernetes
    Swarm
    Nomad
    Marathon
    Aurora Chronos

    View Slide

  99. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View Slide

  100. Config management
    Chubby
    Consul
    etcd
    ZooKeeper

    View Slide

  101. Config management
    Chubby
    Consul
    etcd
    ZooKeeper
    CFEngine
    Chef
    Puppet Salt
    Ansible
    Terraform
    CloudFormation
    Cloud

    Deployment
    Manager

    View Slide

  102. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View Slide

  103. Distributed storage
    EBS
    Ceph
    GlusterFS
    Flocker

    View Slide

  104. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View Slide

  105. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View Slide

  106. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View Slide

  107. Software defined network
    Weave
    Canal
    Open vSwitch

    View Slide

  108. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View Slide

  109. Service discovery
    DNS/SRV
    Eureka
    Consul etcd
    ZooKeeper
    SmartStack

    View Slide

  110. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization
    Proxying &

    Load balancing

    View Slide

  111. Proxying & Load balancing
    HAProxy nginx+
    Traefik
    SmartStack
    Finagle
    Linkerd

    View Slide

  112. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View Slide

  113. Monitoring
    Prometheus
    InfluxData
    Riemann
    Nagios
    Zabbix
    Elasticsearch

    ELK stack
    OpenZipkin
    Lightstep
    AppDash
    Weave Scope
    Sysdig

    View Slide

  114. View Slide

  115. https://cncf.io/

    View Slide

  116. View Slide

  117. Thank you! Questions?
    http://weave.works • @peterbourgon

    View Slide