Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud-native Application Stacks: a lightly-opinionated survey (May 2016)

Cloud-native Application Stacks: a lightly-opinionated survey (May 2016)

A survey of the platforms and projects that make up the cloud-native computing landscape, circa May 2016.

Peter Bourgon

May 25, 2016
Tweet

More Decks by Peter Bourgon

Other Decks in Programming

Transcript

  1. Cloud-native application stacks
    From orchestrators to monitoring
    A lightly-opinionated survey

    View full-size slide

  2. These are just, like,
    my opinions, man.

    View full-size slide

  3. Amazon Web Services • Google Cloud Platform • Microsoft Azure

    Mesosphere • Apache Mesos • DC/OS • Marathon • Aurora

    Chronos • Kubernetes • Docker • Docker Swarm • Deis

    Rancher • Kismatic • Apprenda • Cloud Foundry • OpenShift

    OpenStack • CoreOS • Project Atomic • Snappy Ubuntu

    Joyent Triton • SmartOS • RancherOS • Vagrant • Terraform

    Consul • Serf • Packer • Nomad • Flocker • Ceph • Flannel

    Weave • Canal • Prometheus • InfluxDB • Nagios • Zabbix

    Elasticsearch • OpenZipkin • LightStep • Appdash

    View full-size slide

  4. The Datacenter as a Computer
    An Introduction to the Design of
    Warehouse-Scale Machines
    http://www.cs.berkeley.edu/~rxin/db-papers/WarehouseScaleComputing.pdf

    View full-size slide

  5. https://twitter.com/alicemazzy/status/655306196128280576

    View full-size slide

  6. http://architecturebestinfo.blogspot.de/2015/09/computer-architecture.html

    View full-size slide

  7. CPU Mem Disk Net

    View full-size slide

  8. CPU Mem Disk Net
    Cooling
    Circuits
    Power
    App
    OS User
    Silicon

    View full-size slide

  9. CPU Mem Disk Net
    Cooling
    Circuits
    Power
    App
    OS User
    Silicon

    View full-size slide

  10. CPU Mem Disk Net App
    OS

    View full-size slide

  11. App
    OS
    CPU Mem Disk Net

    View full-size slide

  12. CPU Mem Disk Net Data
    Batch
    Biz logic
    Resource mgmt
    Abstraction Index

    View full-size slide

  13. CPU Mem Disk Net Data
    Batch
    Biz logic
    Resource mgmt
    Abstraction Index

    View full-size slide

  14. CPU Mem Disk Net Data
    Batch
    Biz logic
    Resource mgmt
    Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Abstraction Index

    View full-size slide

  15. CPU Mem Disk Net Data
    Batch
    Biz logic
    Resource mgmt
    Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Scheduler
    Service discovery
    Config mgmt
    Abstraction
    Virtualization
    Index

    View full-size slide

  16. CPU Mem Disk Net Data
    Batch
    Biz logic
    Resource mgmt
    Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Scheduler
    Service discovery
    Config mgmt
    Abstraction
    Virtualization
    Database
    Data
    warehouse
    Biz logic
    Index

    View full-size slide

  17. Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Scheduler
    Service discovery
    Config mgmt
    Virtualization
    Database
    Data
    warehouse
    Biz logic

    View full-size slide

  18. Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Virtualization
    Database
    Data
    warehouse
    Biz logic
    Scheduler
    Service discovery
    Config mgmt

    View full-size slide

  19. Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Virtualization
    Database
    Data
    warehouse
    Biz logic
    Scheduler
    Config mgmt
    Service discovery

    View full-size slide

  20. Database
    RDBMS AP CP Document Transactional
    Oracle
    MySQL
    Postgres
    Dynamo
    Voldemort
    Cassandra
    Riak
    Chubby
    ZooKeeper
    etcd
    Consul
    Mongo
    Couch
    Redis
    F1
    Cockroach
    Rethink

    View full-size slide

  21. Data warehouse
    Hadoop RedShift BigQuery
    Spark Mahout
    Hbase Pig ···

    View full-size slide

  22. Lambda Architecture
    Realtime + Batch
    Storm Kafka Druid

    View full-size slide

  23. Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Virtualization
    Database
    Data
    warehouse
    Biz logic
    Scheduler
    Config mgmt
    Service discovery

    View full-size slide

  24. Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Virtualization
    Data Biz logic
    Scheduler
    Service discovery
    Config mgmt

    View full-size slide

  25. Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Virtualization
    Data Biz logic
    Scheduler
    Service discovery
    Config mgmt

    View full-size slide

  26. Biz logic
    Monolith

    View full-size slide

  27. Biz logic
    Foo
    service
    Bar
    service
    Baz
    service
    Users
    service
    Auth
    service
    Search
    service
    Reports
    service
    Audit
    service

    View full-size slide

  28. Biz logic
    Foo
    service
    Bar
    service
    Baz
    service
    Users
    service
    Auth
    service
    Search
    service
    Reports
    service
    Audit
    service

    View full-size slide

  29. Biz logic
    Foo
    service
    Bar
    service
    Baz
    service
    Users
    service
    Auth
    service
    Search
    service
    Reports
    service
    Audit
    service
    Continuous
    delivery
    Continuous
    integration
    Monitoring
    Tracing
    Safety
    Application logging Event sourcing

    View full-size slide

  30. Sean Treadway, SoundCloud

    View full-size slide

  31. Sean Treadway, SoundCloud

    View full-size slide

  32. Sean Treadway, SoundCloud

    View full-size slide

  33. Sean Treadway, SoundCloud

    View full-size slide

  34. Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Scheduler
    Service discovery
    Config mgmt
    Virtualization
    Data Biz logic

    View full-size slide

  35. Monitoring
    Compute
    resources
    Distributed storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Scheduler
    Service discovery
    Config mgmt
    Virtualization
    Data Biz logic

    View full-size slide

  36. Virtualization
    Distributed
    storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt Data
    Monitoring
    Compute
    resources

    View full-size slide

  37. Virtualization
    Distributed
    storage
    Local
    disk
    Physical
    network
    Software
    defined
    network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Compute
    resources

    View full-size slide

  38. Physical
    network
    Virtualization
    Distributed
    storage
    Local
    (scratch
    Software defined

    network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Compute
    resources

    View full-size slide

  39. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View full-size slide

  40. Application
    Platform
    Infrastructure
    Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View full-size slide

  41. http://trapptechnology.com/saas-iaas-paas-decoding-cloud-acronyms/

    View full-size slide

  42. Software defined network
    Monitoring
    Distr.
    storage
    Scheduler
    Config mgmt
    Data
    Service discovery
    Compute
    resources
    Virtualization

    View full-size slide

  43. ¯\_(ϑ)_/¯
    Software defined network
    Monitoring
    Distr.
    storage
    Scheduler
    Config mgmt
    Data
    Service discovery
    Compute
    resources
    Virtualization

    View full-size slide

  44. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization
    Our new warehouse computer

    View full-size slide

  45. Cloud platforms

    View full-size slide

  46. Virtualization
    OS
    Middleware
    Runtime
    Data
    Applications
    Servers
    Storage
    Networking
    { }

    View full-size slide

  47. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization
    AWS

    View full-size slide

  48. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization
    EC2 S3
    RedShift
    VPC
    Route53
    CloudWatch
    CloudFormation
    RDS
    Machine
    Learning
    Kinesis
    Dynamo
    ELB
    ECS
    Lambda

    View full-size slide

  49. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization
    GCP

    View full-size slide

  50. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization
    GCE
    GKE GAE
    BigTable
    SQL
    Datastore BigQuery
    Cloud Monitoring
    Data*
    Deployment
    manager
    Cloud
    Storage
    Cloud
    Networking

    View full-size slide

  51. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization
    Azure?

    View full-size slide

  52. Open-source platforms

    View full-size slide

  53. Virtualization
    OS
    Middleware
    Runtime
    Data
    Applications
    Servers
    Storage
    Networking
    { }

    View full-size slide

  54. https://mesos.apache.org/ https://mesosphere.com/

    View full-size slide

  55. https://thecustomizewindows.com/2014/12/mesosphere-data-center-operating-system/

    View full-size slide

  56. https://aphyr.com/posts/326-call-me-maybe-chronos

    View full-size slide

  57. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization
    Master
    Slave
    Slave
    Slave
    ZooKeeper
    Marathon Aurora Chronos
    DNS
    Master
    Master

    View full-size slide

  58. http://kubernetes.io/

    View full-size slide

  59. http://www.slideshare.net/brendandburns/defrag-2014-41815642

    View full-size slide

  60. http://kubernetes.io/

    View full-size slide

  61. Compute
    resources
    Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    API server
    Kubelet
    Kubelet
    Kubelet
    etcd
    SkyDNS
    Various
    Volume mounts
    Controllers

    View full-size slide

  62. http://devopscube.com/how-to-setup-and-configure-docker-swarm-cluster/

    View full-size slide

  63. http://blog.daocloud.io/swarm_analysis_part1/

    View full-size slide

  64. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization
    Docker
    Docker
    Docker
    Swarm
    Scheduler
    Node discovery
    libnetwork
    libkv

    View full-size slide

  65. http://deis.io/ https://www.engineyard.com/

    View full-size slide

  66. https://rancher.com

    View full-size slide

  67. http://kismatic.com http://apprenda.com

    View full-size slide

  68. https://pivotal.io/platform

    View full-size slide

  69. https://www.openstack.org/

    View full-size slide

  70. https://www.openshift.com

    View full-size slide

  71. More enterprisey?

    View full-size slide

  72. Less opinionated

    View full-size slide

  73. Cloud-native operating systems

    View full-size slide

  74. }
    Virtualization
    OS
    Middleware
    Runtime
    Data
    Applications
    Servers
    Storage
    Networking

    View full-size slide

  75. https://coreos.com/

    View full-size slide

  76. https://en.wikipedia.org/wiki/CoreOS

    View full-size slide

  77. http://projectatomic.io/

    View full-size slide

  78. https://access.redhat.com/products/red-hat-atomic-enterprise-platform

    View full-size slide

  79. https://developer.ubuntu.com/en/snappy/

    View full-size slide

  80. https://rancher.com
    OS

    View full-size slide

  81. https://smartos.org/
    https://joyent.com

    View full-size slide

  82. http://www.hypervizor.com/diagrams/

    View full-size slide

  83. Open-source components

    View full-size slide






  84. Virtualization
    OS
    Middleware
    Runtime
    Data
    Applications
    Servers
    Storage
    Networking

    View full-size slide

  85. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View full-size slide

  86. Virtualization
    Vagrant
    Docker
    rkt

    View full-size slide

  87. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View full-size slide

  88. Schedulers
    Kubernetes
    Swarm
    Nomad
    Marathon
    Aurora Chronos

    View full-size slide

  89. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View full-size slide

  90. Config management
    Chubby
    Consul
    etcd
    ZooKeeper

    View full-size slide

  91. Config management
    Chubby
    Consul
    etcd
    ZooKeeper
    CFEngine
    Chef
    Puppet Salt
    Ansible
    Terraform
    CloudFormation
    Cloud

    Deployment
    Manager

    View full-size slide

  92. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View full-size slide

  93. Distributed storage
    EBS
    Ceph
    GlusterFS
    Flocker

    View full-size slide

  94. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View full-size slide

  95. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View full-size slide

  96. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View full-size slide

  97. Software defined network
    Weave
    Canal
    Open vSwitch

    View full-size slide

  98. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View full-size slide

  99. Service discovery
    DNS/SRV
    Eureka
    Consul etcd
    ZooKeeper
    SmartStack

    View full-size slide

  100. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization
    Proxying &

    Load balancing

    View full-size slide

  101. Proxying & Load balancing
    HAProxy nginx+
    Traefik
    SmartStack
    Finagle
    Linkerd

    View full-size slide

  102. Compute
    resources Distributed storage
    Software defined network
    Scheduler
    Biz logic
    Service discovery
    Config mgmt
    Data
    Monitoring
    Virtualization

    View full-size slide

  103. Monitoring
    Prometheus
    InfluxData
    Riemann
    Nagios
    Zabbix
    Elasticsearch

    ELK stack
    OpenZipkin
    Lightstep
    AppDash
    Weave Scope
    Sysdig

    View full-size slide

  104. https://cncf.io/

    View full-size slide

  105. Thank you! Questions?
    http://weave.works • @peterbourgon

    View full-size slide