Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Handing Failure in Microservice Architectures

mattheath
January 15, 2016

Handing Failure in Microservice Architectures

Presented at NDC London on 15th January 2016

Microservice architectures allow us to decompose domain logic into small services with a bounded context, which allows us to gain simplicity within services at the expense of complexity in the interactions between services.

However any distributed system operating at scale will experience failure, and this interaction complexity makes dealing with failure harder. This is especially important when requests may traverse many systems, and failures of a single component may cascade through several more. In this talk we look at a number of common patterns from simple usage of concurrency primitives and timeouts to control and throttle concurrency, to more complex patterns such as the CircuitBreaker which can be used to prevent cascading failures; increasing the reliability of our systems.

mattheath

January 15, 2016
Tweet

More Decks by mattheath

Other Decks in Programming

Transcript

  1. ?

  2. 2012 June - RBS - Batch processing causes 3 day

    outage 2013 December - RBS - Card payments, cash withdrawals 2015 June - RBS lose 600,000 payments 2015 August - HSBC lose 275,000 payments 2015 October - Barclays - Failure of accounts and cards 2016 January - HSBC - 2 day outage of online systems
  3. api webhook api webhook service api webhook api webhook service

    ????? SENTRY error timeout error timeout
  4. api webhook api webhook service api webhook api webhook service

    ????? SENTRY error error error error duplicated errors
  5. api webhook api webhook service api webhook api webhook service

    ????? SENTRY error error error error hash & deduplicate
  6. TELEGRAF w/ STATSD
 PLUGIN SERVICE A UDP SERVICE B UDP

    HOST INSTANCES GRAFANA INFLUXDB TAGGED METRICS
  7. WEBHOOK API LOAD BALANCER HTTP API & ROUTING LAYER WEBHOOK

    SERVICE WEBHOOK SERVICE WEBHOOK SERVICE
  8. WEBHOOK API LOAD BALANCER HTTP API & ROUTING LAYER WEBHOOK

    SERVICE WEBHOOK SERVICE WEBHOOK SERVICE
  9. WEBHOOK API LOAD BALANCER HTTP API & ROUTING LAYER WEBHOOK

    SERVICE WEBHOOK SERVICE WEBHOOK SERVICE WEBHOOK SERVICE
  10. WEBHOOK API LOAD BALANCER HTTP API & ROUTING LAYER WEBHOOK

    SERVICE WEBHOOK SERVICE WEBHOOK SERVICE WEBHOOK SERVICE SLOW / ERRORS
  11. API SERVICE SERVICE A SERVICE B LOAD BALANCER HTTP API

    & ROUTING LAYER SERVICE C SERVICE D E
  12. API SERVICE SERVICE A SERVICE B LOAD BALANCER HTTP API

    & ROUTING LAYER SERVICE C SERVICE D G E F
  13. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed

    apns API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
  14. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed

    apns API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
  15. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed

    apns API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
  16. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed

    apns API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
  17. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed

    apns API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
  18. API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed

    apns API card-api card-processing cards transactions balance transaction-enrichment merchant feed-generator feed apns
  19. ATM: Thomas Hawk
 Bank of Commerce: ABQ Museum Archives IBM

    System/360: IBM Absorbed: Saxbald Photography Orbital Ion Cannon: www.rom.ac Credits