Clojure Ireland Talk June 2015

Brief rundown of some Netflix libraries we've found useful when rolling out new backend services at Logentries.

1. BUILDING CLOUD SERVICES WITH NETFLIX OSS
 AT
2. ME Chris Mowforth chris[at]mowforth.com @m0wfo
3. CAVEATS 1. No affiliation with Netflix! 2. Just because it works for us…
4. LOGENTRIES • Founded in ’10 • Massive growth: • Customer base • Volume sent • Volume searched Now we have to make it scale!
5. IN THE BEGINNING Log processing infrastructure Web Tier (Java) (Django/JS) Browser
6. LATER ON Log processing infrastructure Web Tier (Java) (Django/JS) Browser Service x (JVM-based) Service y (JVM-based) Service ? (JVM-based) We went hiring :)
7. LATER STILL… We got some dev-ops guys Log processing infrastructure Web Tier (Java) (Django/JS) Browser Service x (JVM-based) Service y (JVM-based) Service ? (JVM-based) Magic continuous-delivery system
8. CONSTRAINTS 1. Avoid new services incurring
 excessive ops burden 2. Minimise engineering time spent
 developing non-essential* IP
9. NON-ESSENTIAL? People value us for: • Reliable log ingestion • Real-time alerts • Simple, expressive query language • Extending our analytics capabilities
10. NON-ESSENTIAL? This is critical, but not what we’re
 valued for:
 • Configuration • Load-balancing • Fault-tolerance (bulkheading) • Metrics & monitoring • Routing, production debugging, TiP
11. NON-ESSENTIAL? Unless you have a good reason, you should allocate engineering time to the former.
12. NON-ESSENTIAL? A common reason is, “there’s no off-the-shelf FOSS solution” Enter Netflix :)
13. NF@LE Problem Solution Configuration Archaius Async / Reactive APIs Rx{Java/Scala/Clojure/…} Fault-tolerance / Monitoring Hystrix / Turbine
14. #1: CONFIGURATION
15. ARCHAIUS Make config simple
16. ARCHAIUS Take config from anywhere: • Environment vars • Properties file on disk • File at URL • ZooKeeper • JMX • etcd • …
17. ARCHAIUS “Can we schedule some downtime to change value x?” private static final int TIMEOUT = 1000;
18. ARCHAIUS “Nah, we can just change it.” private static final DynamicIntProperty TIMEOUT = DynamicPropertyFactory
 .getInstance()
 .getIntProperty("service.timeout", 1000); TIMEOUT.addCallback(() ->
 System.out.println("value changed!"));
19. #2:
 ASYNC/REACTIVE
20. RxJava "ReactiveX is a combination of the best ideas from the Observer pattern, the Iterator pattern, and functional programming" http://reactivex.io/
21. RxJava Async by default. Even if it’s not :) public interface TagService { List<Tag> getTags(UUID accountId); }
22. RxJava Async by default. public interface TagService { Observable<Tag> getTags(UUID accountId); }
23. RxJava Async by default. List<Tag> tags = tagService.getTags("5bd8bda8-b2ae-4d83- a20f-50a5327a4a13"); List<Tag> publicTags = new ArrayList<>(); for (Tag tag : tags) { if (!tag.isPrivate()) { publicTags.add(tag); } }
24. RxJava Async by default. tagService.getTags("5bd8bda8-b2ae-4d83-a20f-50a5327a4a13") .filter(tag -> !tag.isPrivate()) .subscribe(tag -> System.out.println(tag.getId()));
25. RxJava Async by default. (defn rx-fn [fn] (reify Func1 (call [this obj] (fn obj)))) (defn rx-action [fn] (reify Action1 (call [this e] (fn e)))) (-> (.getTags tagService "5bd8bda8-b2ae-4d83-a20f-50a5327a4a13") (.filter (rx-fn #(not (.isPrivate %)))) (.subscribe (rx-action #(println (.getId %)))))
26. #3:
 FAULT-TOLERANCE / MONITORING
27. HYSTRIX Expect failure Make it easy to react
28. HYSTRIX Bulkheading: • Mock responses- sometimes it's ok • Just die- no point burning a thread if 
 it'll fail anyway
29. HYSTRIX class SomeServiceRequest extends HystrixRequest<String> { @Override public String run() throws Exception { // some fallible API call } @Override public String getFallback() { // return some stale data, if viable } }
30. HYSTRIX
31. END
32. LINKS ★ http://netflix.github.io/#repo ★ http://techblog.netflix.com/2013/02/rxjava-netflix-api.html ★ http://techblog.netflix.com/2012/11/hystrix.html ★ http://techblog.netflix.com/2013/06/announcing-zuul-edge- service-in-cloud.html