Babashka and GraalVM: taking Clojure to new places @ Clojure NYC meetup

Online presentation given at the Clojure NYC meetup in July 2020.

Transcript

1. and GraalVM; taking Clojure to new places
   Michiel Borkent
   
   @borkdude
   
   2020-07-11
   
   

   View Slide

2. • CLI tools with instant startup! (< 10ms)
   
   • clj-kondo: a linter for Clojure that sparks joy
   
   • jet: convert between JSON, EDN and Transit
   
   • No eval: dynamic classloader not supported!
   +
   

   View Slide

3. DSL -> scripting
   • Added a query DSL to jet, a GraalVM CLI:

   

   $ jet --query '(map :id)' <<< '[{:id 1} {:id 2}]'

   [1 2]
   
   • Extend this DSL to significant subset of Clojure?

   

   View Slide

4. • Native Clojure scripting tool, single binary, no JVM
   
   • Can be used to replace “the grey areas” of bash
   
   • Installable via script, brew (macOS, linux), aur (linux), scoop
   (Windows)
   $ time bb '(+ 1 2 3)'

   6

   0.00s user 0.00s system 67% cpu 0.013 total
   

   View Slide

5. CLJ scripting
   Runtime Impl Startup* Interop Windows Execution Threads
   clojure JVM Java ~1048ms + + Compiled +
   planck JSCore CLJS / JS ~ 728ms + - Compiled -
   joker Native Go ~7ms - + Interpreted -
   babashka Native GraalVM ~10ms + + Interpreted +
   *) measured with multitime -n10 -e '(+ 1 2 3)' on Ubuntu Bionic with Intel i7-3770K CPU @ 3.50GHz
   

   View Slide

6. Babashka goals
   • Fast starting Clojure scripting alternative for JVM Clojure
   
   • Easy installation: README ⟶ grab binary ⟶ run within seconds
   • Familiar: targeted at JVM clojure users
   • Cross-platform: #{linux, macOS, Windows}
   • Interop with commonly used classes (System, File, java.time.*, java.nio.*)
   
   • Multi-threading support (pmap, future, ...)
   
   • Batteries included (tools.cli, cheshire, ...) + external libraries + pods
   

   View Slide

7. Babashka non-goals
   • Performance
   • Long running performance intensive processes: use the JVM
   
   • Provide a mixed Clojure/Bash DSL (rather: be compatible with JVM Clojure)
   
   • Replace existing shells like Bash (rather: play well with them)
   

   View Slide

8. Shell interaction
   $ ls | bb -i '(filter #(-> % io/file .isDirectory) *input*)'
   ("doc" "examples" "logo" ...)
   

   View Slide

9. Predefine functions
   $ export BABASHKA_PRELOADS="(defn is-dir? [f] (-> f io/file .isDirectory))"

   

   $ ls | bb -i '(filter is-dir? *input*)'
   ("doc" "examples" "logo" ...)
   

   View Slide

10. Scripts
    $ pst.clj

    04:58
    #!/usr/bin/env bb
    (def now (java.time.ZonedDateTime/now))
    (def LA-timezone (java.time.ZoneId/of "America/Los_Angeles"))
    (def LA-time (.withZoneSameInstant now LA-timezone))
    (def pattern (java.time.format.DateTimeFormatter/ofPattern "HH:mm"))
    (println (.format LA-time pattern))
    

    View Slide

11. Cross platform!
    

    View Slide

12. Included libs / namespaces
    • clojure.{core, edn, java.shell, java.io, pprint, set, string, test, walk, zip}
    
    • clojure.tools.cli
    
    • clojure.core.async
    
    • clojure.data.csv
    
    • cheshire.core (JSON)
    
    • clojure.xml
    
    • cognitect.transit
    
    • clj-yaml
    
    • babashka.curl
    

    View Slide

13. View Slide

14. Compatibility with JVM
    • No deftype, definterface, reify (defrecord, defprotocol
    added recently)
    
    • Code that does not use these constructs often works
    
    • Possible to load library code using existing mechanisms:
    classpath
    

    View Slide

15. Classpath
    ;; spec.clj

    (require '[spartan.spec :as s])

    (s/explain (s/cat :x int? :y keyword?) [1 #{:foo}])
    $ export BABASHKA_CLASSPATH=$(clojure -Spath ...)
    $ bb spec.clj

    #{:foo} - failed: keyword? in: [1] at: [:y]
    

    View Slide

16. Selection of compatible libs
    • spartan.spec: clojure.spec.alpha (1) for bb
    
    • clj-http-lite: lighter fork of cli-http-lite
    
    • medley: "missing" clojure utility functions
    
    • regal: create regular expressions from EDN/hiccup
    
    • camel-snake-kebab: word case conversions
    
    • aero: library for explicit, intentful configuration.
    
    • nubank/docopt: docopt in Clojure
    

    View Slide

17. $ clojure
    

    View Slide

18. $ deps.clj
    $ deps.clj -Sdeps ...
    C:\>deps.clj -Sdeps ...
    

    View Slide

19. Decomplecting babashka
    Libraries coming out of babashka:
    
    • edamame: EDN/Clojure parser
    
    • sci: a Small Clojure Interpreter
    

    View Slide

20. edamame
    (def parsed (edamame/parse-string "#(+ 1 2 %)" {:fn true}))
    ;;=> (fn* [%1] (+ 1 2 %1))
    (meta parsed)
    ;;=> {:row 1, :col 1, :end-row 1, :end-col 11}
    - EDN/code parser
    - GraalVM compatible (no eval!)
    - location metadata
    - opt-in code-like features
    

    View Slide

21. Small Clojure Interpreter
    (def f (sci/eval-string "#(+ 1 2 %)"))
    (f 1)
    ;;=> 4
    - Clojure interpreter
    - Works on JVM / GraalVM / JS
    - Sandboxing

    - Works in CLJS advanced compiled apps

    - Also available on NPM
    

    View Slide

22. Sci from JavaScript
    https://observablehq.com/@jeroenvandijk/untitled/5
    $ npm install @borkdude/sci

    $ node

    > const { evalString, toJS } = require('@borkdude/sci');

    > x = evalString("(assoc {:a 1} :b 2)")

    > toJS(x)
    { a: 1, b: 2 }
    

    View Slide

23. Malli: serializable schemas
    (def my-schema
    [:and
    [:map
    [:x int?]
    [:y int?]]
    [:fn '(fn [{:keys [x y]}] (> x y))]])
    (m/validate my-schema {:x 1, :y 0})
    ; => true
    (m/validate my-schema {:x 1, :y 2})
    ; => false
    

    View Slide

24. Function CLI args
    $ jet --from json --keywordize '(comp keyword str/upper-case)' \

    <<< '{"a": 1}'
    {:A 1}
    

    View Slide

25. clj-kondo hooks
    • The problem:
    
    • Clj-kondo doesn't recognize syntax of user-defined macros
    
    • Existing solutions:
    
    • Add built-in support
    
    • :lint-as - only adequate for identical syntax
    
    • :unresolved-symbol - works for suppressing, but also
    suppresses useful information
    

    View Slide

26. clj-kondo hooks
    (ns mylib)
    (defmacro with-bound [binding-vector & body] ,,,)
    

    View Slide

27. clj-kondo hooks
    

    View Slide

28. clj-kondo hooks
    (hooks.with-bound/with-bound

    '(my-lib/with-bound [a 1 ...] ...))) 

    => (let [a 1] {:with-bound/setting true} (inc 1))
    

    View Slide

29. Sci: adding libs
    (require '[cheshire.core :as json])

    

    (def sci-opts

    {:namespaces

    {'cheshire.core

    {'generate-string json/generate-string}}})
    

    (sci/eval-string
    "(require '[cheshire.core :as json])
    (json/generate-string {:a 1})"
    sci-opts)

    ;;=> "{\"a\":1}"
    

    View Slide

30. Projects using sci
    • borkdude/babashka
    • borkdude/clj-kondo (Clojure linter)
    
    • borkdude/jet (JSON, EDN, Transit
    conversion)
    
    • epiccastle/spire (Ansible in Clojure)
    
    • metosin/malli (serializable schemas)
    
    • retrogradeorbit/bootleg (static HTML
    generation)
    
    • chlorine (Atom Clojure plugin)
    
    • dundalek/closh (Clojure shell
    replacement, mixed Clojure/Bash DSL)
    
    • liquidz/dad (config management, e.g.
    complex install scripts)
    
    • alekcz/pcp (Clojure PHP replacement)
    
    • theiceshelf/firn (Org-mode static site
    generator)
    

    View Slide

31. Bootleg: static site CLI
    

    View Slide

32. nREPL
    

    View Slide

33. Babashka pods
    Problems:
    
    • Including more libs into bb will cause GraalVM to take
    more RAM, can't build on free CI anymore
    
    • Including more libs makes binary grow, while not all
    libraries are useful to everyone
    
    • How to access functionality from other platforms useful
    for scripting (Rust, Go, Python)?
    

    View Slide

34. • Initial solutions:
    
    • Feature flags (PostgreSQL, HyperSQL, ...). Still
    supported.
    
    • Access native libraries (hard, brittle, too static)
    Babashka pods
    

    View Slide

35. • Standalone CLIs that can act as libraries to babashka
    
    • Can be built in Clojure + GraalVM or any other language
    (Rust, Python)
    
    • Architecture similar to nREPL
    
    • Uses bencode and (JSON or EDN) for RPC
    Babashka pods
    

    View Slide

36. • babashka-sql-pods (PostgreSQL, HyperSQL)
    
    • pod-babashka-etaoin (browser automation)
    
    • bootleg (HTML generation)
    
    • pod-babashka-filewatcher (implemented in Rust)
    
    • pod-babashka-parcera (whitespace preserving Clojure
    parser)
    
    • pod-lispyclouds-docker
    Babashka pods
    

    View Slide

37. Filewatcher pod (Rust)
    

    View Slide

38. PostgreSQL pod (clojure)
    

    View Slide

39. Future
    • Babashka
    
    • book.babashka.org
    
    • bb.deps.edn / .babashka/config.edn
    
    • clojure.spec integration
    
    • Datomic client (optional via feature flag)

    • Sci
    
    • clojure.datafy
    
    • Smaller JS bundle by configuration
    
    • More control over duration / interrupts
    

    View Slide

40. Clojure + GraalVM
    • CLJ-1472: issue with GraalVM and locking macro
    
    • Solved in 1.10.2-alpha1 (thanks Rich and Alex!)
    
    • https://github.com/lread/clj-graal-docs
    
    • https://github.com/BrunoBonacci/graalvm-clojure/
    

    View Slide

41. Companies using babashka / sci
    Github #254
    Add your company to the list:
    

    View Slide

42. View Slide

43. Conclusion
    Clojure might not be the best language for everything, like
    scripting
    
    Clojure is the best language for scripting.
    

    View Slide

44. Thanks to sponsors
    • Sponsors:
    
    • Github sponsors
    
    • OpenCollective
    
    • Ko-fi
    
    • Patreon
    

    View Slide

45. Thank you!
    https://github.com/borkdude/babashka

    https://github.com/borkdude/sci
    And the Small Clojure Interpreter
    

    View Slide