Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Clojure Ireland Talk June 2015

Clojure Ireland Talk June 2015

Brief rundown of some Netflix libraries we've found useful when rolling out new backend services at Logentries.

F4e90ce9fdf9e968587c7f9bec94721e?s=128

Chris Mowforth

June 03, 2015
Tweet

Transcript

  1. BUILDING CLOUD SERVICES WITH NETFLIX OSS
 AT

  2. ME Chris Mowforth chris[at]mowforth.com @m0wfo

  3. CAVEATS 1. No affiliation with Netflix! 2. Just because it

    works for us…
  4. LOGENTRIES • Founded in ’10 • Massive growth: • Customer

    base • Volume sent • Volume searched Now we have to make it scale!
  5. IN THE BEGINNING Log processing infrastructure Web Tier (Java) (Django/JS)

    Browser
  6. LATER ON Log processing infrastructure Web Tier (Java) (Django/JS) Browser

    Service x (JVM-based) Service y (JVM-based) Service ? (JVM-based) We went hiring :)
  7. LATER STILL… We got some dev-ops guys Log processing infrastructure

    Web Tier (Java) (Django/JS) Browser Service x (JVM-based) Service y (JVM-based) Service ? (JVM-based) Magic continuous-delivery system
  8. CONSTRAINTS 1. Avoid new services incurring
 excessive ops burden 2.

    Minimise engineering time spent
 developing non-essential* IP
  9. NON-ESSENTIAL? People value us for: • Reliable log ingestion •

    Real-time alerts • Simple, expressive query language • Extending our analytics capabilities
  10. NON-ESSENTIAL? This is critical, but not what we’re
 valued for:


    • Configuration • Load-balancing • Fault-tolerance (bulkheading) • Metrics & monitoring • Routing, production debugging, TiP
  11. NON-ESSENTIAL? Unless you have a good reason, you should allocate

    engineering time to the former.
  12. NON-ESSENTIAL? A common reason is, “there’s no off-the-shelf FOSS solution”

    Enter Netflix :)
  13. NF@LE Problem Solution Configuration Archaius Async / Reactive APIs Rx{Java/Scala/Clojure/…}

    Fault-tolerance / Monitoring Hystrix / Turbine
  14. #1: CONFIGURATION

  15. ARCHAIUS Make config simple

  16. ARCHAIUS Take config from anywhere: • Environment vars • Properties

    file on disk • File at URL • ZooKeeper • JMX • etcd • …
  17. ARCHAIUS “Can we schedule some downtime to change value x?”

    private static final int TIMEOUT = 1000;
  18. ARCHAIUS “Nah, we can just change it.” private static final

    DynamicIntProperty TIMEOUT = DynamicPropertyFactory
 .getInstance()
 .getIntProperty("service.timeout", 1000); TIMEOUT.addCallback(() ->
 System.out.println("value changed!"));
  19. #2:
 ASYNC/REACTIVE

  20. RxJava "ReactiveX is a combination of the best ideas from

    the Observer pattern, the Iterator pattern, and functional programming" http://reactivex.io/
  21. RxJava Async by default. Even if it’s not :) public

    interface TagService { List<Tag> getTags(UUID accountId); }
  22. RxJava Async by default. public interface TagService { Observable<Tag> getTags(UUID

    accountId); }
  23. RxJava Async by default. List<Tag> tags = tagService.getTags("5bd8bda8-b2ae-4d83- a20f-50a5327a4a13"); List<Tag>

    publicTags = new ArrayList<>(); for (Tag tag : tags) { if (!tag.isPrivate()) { publicTags.add(tag); } }
  24. RxJava Async by default. tagService.getTags("5bd8bda8-b2ae-4d83-a20f-50a5327a4a13") .filter(tag -> !tag.isPrivate()) .subscribe(tag ->

    System.out.println(tag.getId()));
  25. RxJava Async by default. (defn rx-fn [fn] (reify Func1 (call

    [this obj] (fn obj)))) (defn rx-action [fn] (reify Action1 (call [this e] (fn e)))) (-> (.getTags tagService "5bd8bda8-b2ae-4d83-a20f-50a5327a4a13") (.filter (rx-fn #(not (.isPrivate %)))) (.subscribe (rx-action #(println (.getId %)))))
  26. #3:
 FAULT-TOLERANCE / MONITORING

  27. HYSTRIX Expect failure Make it easy to react

  28. HYSTRIX Bulkheading: • Mock responses- sometimes it's ok • Just

    die- no point burning a thread if 
 it'll fail anyway
  29. HYSTRIX class SomeServiceRequest extends HystrixRequest<String> { @Override public String run()

    throws Exception { // some fallible API call } @Override public String getFallback() { // return some stale data, if viable } }
  30. HYSTRIX

  31. END

  32. LINKS ★ http://netflix.github.io/#repo ★ http://techblog.netflix.com/2013/02/rxjava-netflix-api.html ★ http://techblog.netflix.com/2012/11/hystrix.html ★ http://techblog.netflix.com/2013/06/announcing-zuul-edge- service-in-cloud.html