A talk for Craftsman Guild on my team's use of Clojure at Netflix. Describes good, bad, and ugly lessons learned from going from a pure-Java codebase to Clojure in production.
Freedom and Responsibility ... “We hire smart people, give them hard problems and get out of their way. We strive to increase the freedom of our employees as we grow, enabling them to move quickly as the industry evolves. With that freedom comes increased responsibility. High performers thrive in that environment and make great choices for Netflix.” http://jobs.netflix.com/who-we-are.html
Our Team ● Netflix Social Infrastructure ○ Social data storage/analysis, stateless web services ○ Support Social APIs used by many Netflix UIs/devices ● 3 engineers ● 1 supportive manager ● 1 medium-size-ish existing Java codebase
Data 1, 3.14, 1/2 ; Numbers "A string" this ; A symbol, used to name things :a-keyword ; Used for enumerations and map keys {:key1 "value1" :key2 "value2"} ; HashMap [:a :vector 1 2 3] ; Like java.util.ArrayList '(this is :a :list 1 2 3) ● All data objects are immutable by default ● Data is composable.
Naming things ; def names a global thing (def pi 3.14) ; Use let to bind values to local ; names (let [r 3.0 c (* 2.0 Math/PI r) a (* Math/PI (* r r))] {:radius r :circumference c :area a}) ;=> {:radius 3.0, :circumference 18.84955592153876, :area 28.274333882308138}
Functions ; Use fn to create an ; anonymous function (fn [x] (* x x)) ; Use defn to define a named function (defn square [x] (* x x)) ; Call a function (square 3) ;=> 9 ; Pass a function as an argument (map square [1 2 3]) ;=> [1 4 9]
Macros ; A macro can control evaluation, ; e.g. short-circuiting and expression (and (even? a) (odd? b)) -> (if (even? a) (if (odd? b) true false) false) ● Clojure code is data ● A macro is a function that takes code and returns new code ● Invoked by the compiler ; A macro can also reduce boilerplate (rx/fn [a] (* 2 a)) -> (reify rx.Func1 (call [this a] (* 2 a)))
Joyspring - “Netflix REPL” More sane/powerful than sh+curl (def s (subscriber/subscriber+ 12345)) ;=> { ... user data from subscriber service ...} ; Check if they're in A/B test 4567 (ab/allocation+ s 4567) ;=> nil ; Forcibly allocate them to test 4567 cell 2 (ab/allocate+ s 4567 2) ; Get social info (social/profile+ s :netflix) ;=> { ... social connection status ...} Separate tool, free from constraints of Netflix platform
Non-Critical Greenfield Service ● What’s a greenfield?! ● Pure Clojure implementation of small, low risk service, in production ● Learn about: app structure, DI issues, testing, build, deploy
Java, meet Clojure What to keep, what to rewrite? ● Take it easy ● No need to throw out working code ● Clojure has good Java interop ● When you need to write new Java, write Clojure instead ● Tastefully add abstractions as you go
Java, meet Clojure ● Escape Java as fast as you can! ● Collect args, call a single entry point Be careful with caching. Breaks interactive model. ; Some Clojure code (ns com.netflix.mine) (defn func-to-call [x] (* 2 x)) // Invoke it from Java final Var require = RT.var("clojure.core", "require"); require.invoke(Symbol.intern("com.netflix.mine")); final Var funcToCall = RT.var("com.netflix.mine", "func-to-call"); assertEquals(198L, funcToCall.invoke(99L));
Java, meet Clojure … but ● Where do tests go? ○ JUnit? ○ clojure.test? ● Data structures? ○ Keep existing objects? ○ Map from Clojure maps and back? ● Classes? What about classes? ● Dependency Injection
Where do tests go? ● We write tests in clojure.test ○ Still a lot of Java so we have some helper macros for Mockito ● Custom JUnit4 test runner that finds and runs Clojure tests ○ Run tests from Eclipse or wherever ○ Tests magically appear in jenkins ● There are many options here. Our approach is pragmatic, bowing to playing nicely within Netflix build infrastructure
Data Structures? ● But I typed in all these dumb Pojos already! ● Again, we’ve been pragmatic ● For existing code, for the most part, stick with existing Java objects ● For new code, use plain maps and simple Clojure types ● Occasional conversion functions where “old meets new”
Dependency Injection? ● AKA Passing parameters around ● IMHO DI doesn’t magically go away in a dynamic language ○ I still need to get the Cassandra Keyspace object to functions that use it ○ It’s still DI even if the dependency is a simple function instead of an object implementing and interface ● We currently take the “big context map” approach. Have other ideas we’d like to try
Refactoring ● One of the few things Java (tooling) is good at, but still isn’t perfect ● … but I have broken prod by moving a Java class ref'd by property files ● 1/10th the code that does 10x as much. Maybe it's not so bad? ● Still a pain point, especially if test coverage is low...
Working with Others ● Clojure code is dense, especially someone else’s clojure code ● Given an undocumented function with arbitrary args, what does it accept/produce? ● Need more discipline about documentation, pre/post-conditions, schemas
java.lang.ClassCastException: java.lang.Long cannot be cast to clojure.lang.IFn at joyspring.main$eval2203.invoke(NO_SOURCE_FILE:1) at clojure.lang.Compiler.eval(Compiler.java:6619) at clojure.lang.Compiler.eval(Compiler.java:6582) at clojure.core$eval.invoke(core.clj:2852) at clojure.main$repl$read_eval_print__6588$fn__6591.invoke(main.clj:259) at clojure.main$repl$read_eval_print__6588.invoke(main.clj:259) at clojure.main$repl$fn__6597.invoke(main.clj:277) at clojure.main$repl.doInvoke(main.clj:277) at clojure.lang.RestFn.invoke(RestFn.java:1096) at clojure.tools.nrepl.middleware.interruptible_eval$evaluate$fn__1610.invoke (interruptible_eval.clj:56) at clojure.lang.AFn.applyToHelper(AFn.java:159) at clojure.lang.AFn.applyTo(AFn.java:151) at clojure.core$apply.invoke(core.clj:617) at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1788) at clojure.lang.RestFn.invoke(RestFn.java:425) at clojure.tools.nrepl.middleware.interruptible_eval$evaluate.invoke(interruptible_eva 41) at clojure.tools.nrepl.middleware.interruptible_eval$interruptible_eval$fn__1651$fn__1 invoke(interruptible_eval.clj:171) at clojure.core$comp$fn__4154.invoke(core.clj:2330) at clojure.tools.nrepl.middleware.interruptible_eval$run_next$fn__1644.invoke (interruptible_eval.clj:138) at clojure.lang.AFn.run(AFn.java:24) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:680) OMG, the Stacktraces!?!?!?!?
Thawing Mocks clojure.core.typed (require '[clojure.core.typed :refer [ann]]) (ann pi Double) ; annotate a constant (def pi "Pi, more or less" 3.14) ; annotate a function (ann area [Double -> Double]) (defn area [r] (* pi (* r r))) https://github.com/clojure/core.typed A la carte static type checking for Clojure Powerful, but invasive
So, Integration Tests ● Exercise module boundaries ○ This is where stuff breaks ● Requires ○ Dedicated test/staging environment ○ Occasional diagnostic endpoints for setup ● Write them in Clojure! ○ clojure.test + Joyspring
So, Integration Tests ● Slower? ○ (but interactive development) ● Brittle? ○ False negatives due to state ○ False negatives due to normal failures ● Occasional Hacks ○ Services tuned for production use. Differs from integration test patterns ○ Caches!
Integration Test Failure Fail (deftest test-link-visitor-to-facebook-id-failure (testing "A failure while linking fb id to customer leaves user in not_connected state" (fixture/ensure-test-user-disconnected) (sabot/inject (at hystrix.linkVisitorToFacebookId eval (throw (RejectedExecutionException. "test-link-visitor-to-facebook-id-failure"))) => (try+ (fixture/ensure-test-user-facebook-connected) (throw (Exception. "Request unexpectedly succeeded.")) (catch (comp #{503} :status) e (let [s (subscriber/subscriber+ (:customer-id fixture/netflix-user))] ; make sure status is restored in subscriber and fb id is removed (is (= "not_connected" (get-in s [:social :connection-status]))))))))) Sabot is a Clojure library for injecting specific, fine-grained failures into a request Magic here!
Web REPL http://blog.jayfields.com/2012/06/clojure-production-web-repl.html ● Explore instance state ● Quickly sanity check function behavior ● Easy back-of-envelope performance characteristics in real world conditions (region!)
Abstraction ● Not about lines of code. About clearly expressing intent ● Take it easy ○ Programmers love to wrap, especially Java ● You'll get it wrong at least once ● Wrapping or abstracting is language design, i.e. hard
Abstracting Hystrix ● https://github.com/Netflix/Hystrix ● Resilience via thread pools, circuit breakers, fallbacks // Define a command in Java public class GetUserCommand extends HystrixCommand { private final HttpClient client; private final long id; public GetUserCommand(HttpClient client, long id) { this.client = client; this.id = id; } @Override protected User run() { return client.get("/user/" + id, User.class); } @Override protected User getFallback() { return User.missing(id); } } // ... and use it .... new GetUserCommand(client, id).execute();
Abstracting Hystrix (defn get-user [client id] (.get client id User)) (require '[com.netflix.hystrix.core :as hystrix]) (hystrix/defcommand get-user {:hystrix/fallback-fn (fn [client id] (User/missing id))} [client id] (.get client id User)) ; ... and use it. OMG, just a fn call!! (get-user client 12345) This is what we’re doing, just defining a function So why does the Hystrix command have to look so different? No boilerplate here...
Functions over Macros Consider combining 3 RxJava Observables An Observable is an asynchronous sequence ; Raw RxJava (Observable/zip (reify rx.util.functions.Func3 (call [this a b c] (+ a b c))) stream-1 stream-2 stream-3) https://github.com/Netflix/RxJava Initially, we use raw Java interop to implement the “Func3” interface. This is tedious.
Functions over Macros Consider combining 3 RxJava Observables An Observable is an asynchronous sequence ; With a macro (Observable/zip (rx/fn [a b c] (+ a b c))) stream-1 stream-2 stream-3) https://github.com/Netflix/RxJava I know! Macros are great for eliminating boilerplate! Enter rx/fn macro
Functions over Macros Consider combining 3 RxJava Observables An Observable is an asynchronous sequence ; With a function (Observable/zip (rx/fn* +) stream-1 stream-2 stream-3) https://github.com/Netflix/RxJava But a function can do better, allowing composition with existing Clojure functions
Separate representation/behavior // This can be expressed more fluently/builder-y keyspace.put() .withRow(1234) .withColumn(“name”) .withValue(“dave”) .withTtl(90) .execute() // Consider a typical method call putColumn(keyspace, 1234, “name”, “dave”, 90); ● No representation of the operation ● Closer, but without a lot of work still isn’t manipulable, introspectable, reusable etc
Emulate Existing Idioms ● Abstraction is language design ● Abstractions that emulate existing idioms in the host language (Clojure) will ○ Look better ○ Be easier to understand without research ○ Play better with existing features ● In Hystrix, defcommand is structurally identical to defn ○ Easy to switch. Easy to understand. ● Pigpen (mostly) has semantics identical to Clojure data pipelines
Emulate Existing Idioms - rx/let-o ● clojure.core/let makes wiring together data transforms easy ● Not so in Rx, especially for expressions with “forks” ● Enter rx/let-o to take care of the details (rx/let-o [?user (get-user-o 123) ?friends (rx/mapcat (fn [u] (map get-friends-o (:friends u))) ?user) ?ab (rx/mapcat get-ab ?user)] (rx/merge ?user ?friends ?ab))
Conclusion ● Take your time ● Clojure can produce big gains in productivity and satisfaction ● Yes, it's scary ● Mental shift required ○ Clojure examples to Clojure “in the large” ○ People are still figuring this out
miciek
piyoppi
goutarouh
techouse
ivargrimstad
nano72mkn
yusukebe
okuzawats
kishida
ivanova1991
.png){kind=avatar}
melonattacker
marktimemedia
csswizardry
bkeepers
shpigford
addyosmani
pauljervisheath
lauravandoore
mza
malarkey
scottboms
zakiyama
philnash