Building a Fault Tolerant API with Hystrix Ryan Dearing Engineering Manager Bodybuilding.com 

100 million API requests every day 

hundreds of servers 

hundreds of dependencies 

Failures Will Happen! 

Resiliency • Adapt automatically to unexpected failures • Maintain performance during failures • Prevent cascading failures • Fail-Fast, Fail Gracefully • Visibility Tardigrade 

Bulkheading isolates failures and latency “A ship’s hull is divided into different watertight bulkheads so that if the hull is compromised, the failure is limited to that bulkhead as opposed to taking the entire ship down. By partitioning your system, you can confine errors to one area as opposed to taking the entire system down.” - John Ragan 

Prevent failures from cascading “Domino Effect” 

Circuit Breaker Pattern Detects failures and prevents system from executing actions that are certain to fail. Retries periodically. 

Fail Fast If a dependency is failing or slow, reduce rate of requests. Prevents overloading the system. 

Hystrix • Timeout slow commands • Rate limit number of concurrent commands • Circuit Breaker • back off of failing dependencies • Attempt a fallback for failures • Real-Time dashboard for visibility 

Hystrix Flowchart 

Hystrix Flowchart 

Hystrix Flowchart 

Hystrix Flowchart ` 

Hystrix Flowchart 

public class CommandHelloWorld extends HystrixCommand {! ! private final String name;! ! public CommandHelloWorld(String name) {! super(HystrixCommandGroupKey.Factory.asKey("ExampleGroup"));! this.name = name;! }! ! @Override! protected String run() {! // a real example would do work like a network call here! return "Hello " + name + "!";! }! }! 

// this blocks! String s = new CommandHelloWorld("World").execute();! //this doesn't! Future fs = new CommandHelloWorld("World").queue();! //this doesn't either! Observable fs = new CommandHelloWorld("World").observe();! 

public class CommandHelloFailure extends HystrixCommand {! ! private final String name;! ! public CommandHelloFailure(String name) {! super(HystrixCommandGroupKey.Factory.asKey("ExampleGroup"));! this.name = name;! }! ! @Override! protected String run() {! throw new RuntimeException("this command always fails");! }! ! @Override! protected String getFallback() {! return "Hello Failure " + name + "!";! }! }! 

public class CommandThatFailsSilently extends HystrixCommand {! ! private final boolean throwException;! ! public CommandThatFailsSilently(boolean throwException) {! super(HystrixCommandGroupKey.Factory.asKey("ExampleGroup"));! this.throwException = throwException;! }! ! @Override! protected String run() {! if (throwException) {! throw new RuntimeException("failure from CommandThatFailsFast");! } else {! return "success";! }! }! ! @Override! protected String getFallback() {! return null;! }! }! 

public class CommandWithStubbedFallback extends HystrixCommand {! ! private final int customerId;! private final String countryCodeFromGeoLookup;! ! protected CommandWithStubbedFallback(int customerId, String countryCodeFromGeoLookup) {! super(HystrixCommandGroupKey.Factory.asKey("ExampleGroup"));! this.customerId = customerId;! this.countryCodeFromGeoLookup = countryCodeFromGeoLookup;! }! ! @Override! protected UserAccount run() {! // fetch UserAccount from remote service! // return UserAccountClient.getAccount(customerId);! throw new RuntimeException("forcing failure for example");! }! ! @Override! protected UserAccount getFallback() {! /**! * Return stubbed fallback with some static defaults, placeholders,! * and an injected value 'countryCodeFromGeoLookup' that we'll use! * instead of what we would have retrieved from the remote service.! */! return new UserAccount(customerId, "Unknown Name",! countryCodeFromGeoLookup, true, true, false);! }! }! 

Request Collapsing 

Hystrix Dashboard Especially useful if we have real-time insight into dependencies and failures 

Hystrix Dashboard 

Hystrix Dashboard Especially useful if we have real-time insight into dependencies and failures 

Hystrix Dashboard System is backing off and adapting dynamically. Some calls still working, others timing out not blocking and cascading. 

Lessons Learned • Organize code for Hystrix from the start • Use Hystrix in API client libraries • Easier to configure if you have historical data about response times, request rate, etc • Command instances should not be reused • Fault vs Non Fault exceptions • Use Hystrix Dashboard 

Thanks • Hystrix: https://github.com/Netflix/Hystrix • Hystrix Dashboard: https://github.com/Netflix/ Hystrix/tree/master/hystrix-dashboard • Slides: https://speakerdeck.com/ryandearing/ building-a-fault-tolerant-api-with-hystrix