Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Constructive destructiveness for containers

Constructive destructiveness for containers

Talk at ContainerCamp London, see https://container.camp/uk/2016/schedule/

Michael Hausenblas

September 09, 2016
Tweet

More Decks by Michael Hausenblas

Other Decks in Technology

Transcript

  1. © 2016 Mesosphere, Inc. All Rights Reserved.
    CONSTRUCTIVE
    DESTRUCTIVENESS
    FOR CONTAINERS
    1
    Michael Hausenblas | ContainerCamp, London | 2016-09-09

    View Slide

  2. © 2016 Mesosphere, Inc. All Rights Reserved.
    MOTIVATION
    2

    View Slide

  3. © 2016 Mesosphere, Inc. All Rights Reserved.
    WHY WOULD I WANT TO DO THAT?
    3

    View Slide

  4. © 2016 Mesosphere, Inc. All Rights Reserved.
    WHY I USUALLY DON'T DO IT …
    4
    • My code is bug free
    • It costs time/money
    • 'Uncool' work

    View Slide

  5. © 2016 Mesosphere, Inc. All Rights Reserved.
    TOWARDS A
    FRAMEWORK
    5

    View Slide

  6. © 2016 Mesosphere, Inc. All Rights Reserved.
    TERMINOLOGY
    6
    unit testing
    property based testing
    fault injection
    resilience testing
    smoke testing
    stress testing
    canary deployments
    dev prod
    cost-benefit
    language dependent

    View Slide

  7. © 2016 Mesosphere, Inc. All Rights Reserved.
    SCOPES
    7
    container
    host & intra-host
    service (app/business)

    View Slide

  8. © 2016 Mesosphere, Inc. All Rights Reserved.
    FALLACIES OF DISTRIBUTED COMPUTING
    8
    • The network is reliable.
    • Latency is zero.
    • Bandwidth is infinite.
    • The network is secure.
    • Topology doesn't change.
    • There is one administrator.
    • Transport cost is zero.
    • The network is homogeneous.

    View Slide

  9. © 2016 Mesosphere, Inc. All Rights Reserved.
    CHAOS ENGINEERING
    9
    Pioneered by Netflix
    • Define normal behaviour
    • Define expectations
    • Test hypothesis

    View Slide

  10. © 2015 Mesosphere, Inc. All Rights Reserved.
    FAILURE
    MODES
    10
    container
    master
    worker
    container
    container
    container
    container
    master master
    worker worker worker worker worker
    container container
    container
    container
    container
    container
    container
    container

    View Slide

  11. © 2016 Mesosphere, Inc. All Rights Reserved.
    ASPECTS
    11
    • automation (health checks)
    • black box testing (what happens if you cut off the connection to
    environment?)
    • dimensions
    • disk I/O
    • time (skew)
    • network traffic
    • requests per second served
    • cascading failures

    View Slide

  12. © 2016 Mesosphere, Inc. All Rights Reserved.
    STATEFUL
    SERVICES
    12

    View Slide

  13. © 2016 Mesosphere, Inc. All Rights Reserved.
    APPLYING
    RESILIENCE
    TESTING
    13

    View Slide

  14. © 2016 Mesosphere, Inc. All Rights Reserved.
    TOOLING
    14
    • Chaos Monkey

    http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html 

    https://medium.com/production-ready/chaos-monkey-for-fun-and-
    profit-87e2f343db31
    • Wehkamp's Half-Life2/microservices attack tool

    https://www.wehkamplabs.com/blog/2016/06/02/docker-and-zombies/

    View Slide

  15. © 2016 Mesosphere, Inc. All Rights Reserved.
    TOOLING
    15
    • Pumba

    http://blog.terranillius.com/post/pumba_docker_chaos_testing/
    • Mizaru

    https://github.com/giantswarm/mizaru
    • Blockade

    https://github.com/dcm-oss/blockade
    • Kubernetes (sig-testing)

    https://github.com/kubernetes/kubernetes/issues/4548
    • DC/OS: DRAX

    https://github.com/dcos-labs/drax

    View Slide

  16. © 2016 Mesosphere, Inc. All Rights Reserved.
    JEPSEN.IO
    16

    View Slide

  17. © 2016 Mesosphere, Inc. All Rights Reserved.
    RESOURCES
    17

    View Slide

  18. © 2015 Mesosphere, Inc. All Rights Reserved. 18
    Resilience Engineering in Practice: A Guidebook

    View Slide

  19. © 2015 Mesosphere, Inc. All Rights Reserved. 19
    https://www.youtube.com/watch?v=4fFDFbi3toc

    View Slide

  20. © 2015 Mesosphere, Inc. All Rights Reserved. 20
    https://github.com/spacejam
    Big kudos to
    Tyler Neely!

    View Slide

  21. © 2016 Mesosphere, Inc. All Rights Reserved.
    Q & A
    21
    • @mhausenblas
    • mhausenblas.info
    [email protected]
    https://dcos.io

    View Slide