Dealing with big black boxes

0f698d0c08d4d77f1b981eb449056dac?s=47 Fergus Doyle
September 20, 2014

Dealing with big black boxes

"Full-stack" includes so much more than just client and server these days, with systems building on top of multiple database engines, caching layers, search backends, third party web services and hosting solutions to name but a few.

What does this mean for infrastructure? In this talk we'll explore the impact maintaining so many independent cogs can have on a system as a whole.

We’ll look at third-party components vs. those developed and maintained in-house and their similarities and differences. How can we encapsulate and efficiently test components in such disparate systems? What can we do to minimise the overhead and risk of releasing new code to individual components nested deep within the machine?

0f698d0c08d4d77f1b981eb449056dac?s=128

Fergus Doyle

September 20, 2014
Tweet

Transcript

  1. Dealing with big black boxes How to manage a machine

    with loads of cogs
  2. Introduction Who am I? Distributed systems Services and their dependencies

    Not all integration tests are evil ... but some are
  3. Distributed systems are cool! Separation of concerns Clearer boundaries Logical

    ownership Putting the hardware to good use
  4. Distributed systems can be difficult! Harder to see the big

    picture Visibility and standardisation of tools Integration points between components Atomicity of actions Debugging / Tracebacks / Callstacks
  5. Components you may have Services Web applications Relational DBs NoSQL

    DBs Mongo Solr Caches Data warehouse ...
  6. What does the black box look like to the outside

    world?
  7. None
  8. It either works or it doesn't

  9. Where do we start?

  10. Join the dots Dependency graph Simplify the reporting lines Make

    sure all the cogs fit
  11. Test it Unit Integration, integration and more integration!

  12. The differences between unit and integration tests

  13. Unit tests

  14. Integration tests

  15. Integration tests

  16. Assumption is the mother of all f***ups!

  17. Concentric circles of integration testing

  18. Close to perfection production Always have a realistic test environment

    Always have a staging environment that mirrors production
  19. Swapping out cogs (releasing) Expect the system to evolve Make

    sure you test against the current versions of everything else Re-run the circles prior to release Confidence is key Check the surrounding cogs
  20. Looking inside the box Introspect the box - tooling for

    live systems Monitor - Logstash, StatsD Report - Sentry, pagerduty
  21. What does it all mean Systems grow out of necessity

    - the distributed paradigm is just one way of managing this Don't lose sight of the overall system while working at the smaller scale Classify your tests and be clear on what you're testing Make sure you have tools to do the hard stuff for you
  22. / Fergus Doyle @junkafarian Thank You twitter.com/junkafarian github.com/junkafarian speakerdeck.com/junkafarian We're

    hiring!