Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Chaos Monkey on my Laptop

Chaos Monkey on my Laptop

Chaos testing/engineering talk about integrating chaos into integration test suites. Original talk done at GothamGO and Velocity NYC 2015. Videos only

Fc19147e89d6521f92ef9e2c1c4dac24?s=128

Matthew Campbell

October 02, 2015
Tweet

Transcript

  1. Chaos Monkey on my Laptop

  2. What is chaos testing?

  3. Matt Campbell

  4. Eikon Messenger

  5. None
  6. Scale 300k financial traders Every major bank around the world

  7. Our testing setup

  8. –Said by no one “Sweet lets just start killing production

    databases”
  9. Lets learn how to crawl first

  10. Vagrant Docker Integration Tests

  11. Continuous Integration

  12. Failure integration tests

  13. Less Mocking

  14. Database failovers

  15. Microservice Problems

  16. Performance Downtime Discovery

  17. TheSlowNess

  18. Problems • Connection pooling in go • Dead tcp connections

    • Hung connections • Network splits / Congestion
  19. Production integration tests

  20. Chaos Gopher

  21. • Unix/GO Style tools • Simple utilities , Slowness, Failover,

    Assassin
  22. Where Next? Better integration test suites Allowing for flappy tests

    Statistically analysis of flapping tests
  23. Other cool projects • Chaos Monkey (the mac daddy, Netflix)

    • Chaos Lemur (from Pivotal)
  24. Questions? Matthew Campbell hyper@hyperworks.nu @kanwisher github.com/mattkanwisher