Dealing with big black boxes

"Full-stack" includes so much more than just client and server these days, with systems building on top of multiple database engines, caching layers, search backends, third party web services and hosting solutions to name but a few.

What does this mean for infrastructure? In this talk we'll explore the impact maintaining so many independent cogs can have on a system as a whole.

We’ll look at third-party components vs. those developed and maintained in-house and their similarities and differences. How can we encapsulate and efficiently test components in such disparate systems? What can we do to minimise the overhead and risk of releasing new code to individual components nested deep within the machine?

Transcript

1. Dealing with big black boxes How to manage a machine with loads of cogs
2. Introduction Who am I? Distributed systems Services and their dependencies Not all integration tests are evil ... but some are
3. Distributed systems are cool! Separation of concerns Clearer boundaries Logical ownership Putting the hardware to good use
4. Distributed systems can be difficult! Harder to see the big picture Visibility and standardisation of tools Integration points between components Atomicity of actions Debugging / Tracebacks / Callstacks
5. Components you may have Services Web applications Relational DBs NoSQL DBs Mongo Solr Caches Data warehouse ...
6. What does the black box look like to the outside world?
8. It either works or it doesn't
9. Where do we start?
10. Join the dots Dependency graph Simplify the reporting lines Make sure all the cogs fit
11. Test it Unit Integration, integration and more integration!
12. The differences between unit and integration tests
13. Unit tests
14. Integration tests
15. Integration tests
16. Assumption is the mother of all f***ups!
17. Concentric circles of integration testing
18. Close to perfection production Always have a realistic test environment Always have a staging environment that mirrors production
19. Swapping out cogs (releasing) Expect the system to evolve Make sure you test against the current versions of everything else Re-run the circles prior to release Confidence is key Check the surrounding cogs
20. Looking inside the box Introspect the box - tooling for live systems Monitor - Logstash, StatsD Report - Sentry, pagerduty
21. What does it all mean Systems grow out of necessity - the distributed paradigm is just one way of managing this Don't lose sight of the overall system while working at the smaller scale Classify your tests and be clear on what you're testing Make sure you have tools to do the hard stuff for you
22. / Fergus Doyle @junkafarian Thank You twitter.com/junkafarian github.com/junkafarian speakerdeck.com/junkafarian We're hiring!