Babashka and GraalVM: taking Clojure to new places @ Clojure NYC meetup

Babashka and GraalVM: taking Clojure to new places @ Clojure NYC meetup

Online presentation given at the Clojure NYC meetup in July 2020.

Bbb17eeeecd3d9d06f4db310ab630a12?s=128

Michiel Borkent

July 11, 2020
Tweet

Transcript

  1. and GraalVM; taking Clojure to new places Michiel Borkent @borkdude

    2020-07-11
  2. • CLI tools with instant startup! (< 10ms) • clj-kondo:

    a linter for Clojure that sparks joy • jet: convert between JSON, EDN and Transit • No eval: dynamic classloader not supported! +
  3. DSL -> scripting • Added a query DSL to jet,

    a GraalVM CLI:
 
 $ jet --query '(map :id)' <<< '[{:id 1} {:id 2}]'
 [1 2] • Extend this DSL to significant subset of Clojure?

  4. • Native Clojure scripting tool, single binary, no JVM •

    Can be used to replace “the grey areas” of bash • Installable via script, brew (macOS, linux), aur (linux), scoop (Windows) $ time bb '(+ 1 2 3)'
 6
 0.00s user 0.00s system 67% cpu 0.013 total
  5. CLJ scripting Runtime Impl Startup* Interop Windows Execution Threads clojure

    JVM Java ~1048ms + + Compiled + planck JSCore CLJS / JS ~ 728ms + - Compiled - joker Native Go ~7ms - + Interpreted - babashka Native GraalVM ~10ms + + Interpreted + *) measured with multitime -n10 <cmd> -e '(+ 1 2 3)' on Ubuntu Bionic with Intel i7-3770K CPU @ 3.50GHz
  6. Babashka goals • Fast starting Clojure scripting alternative for JVM

    Clojure • Easy installation: README ⟶ grab binary ⟶ run within seconds • Familiar: targeted at JVM clojure users • Cross-platform: #{linux, macOS, Windows} • Interop with commonly used classes (System, File, java.time.*, java.nio.*) • Multi-threading support (pmap, future, ...) • Batteries included (tools.cli, cheshire, ...) + external libraries + pods
  7. Babashka non-goals • Performance • Long running performance intensive processes:

    use the JVM • Provide a mixed Clojure/Bash DSL (rather: be compatible with JVM Clojure) • Replace existing shells like Bash (rather: play well with them)
  8. Shell interaction $ ls | bb -i '(filter #(-> %

    io/file .isDirectory) *input*)' ("doc" "examples" "logo" ...)
  9. Predefine functions $ export BABASHKA_PRELOADS="(defn is-dir? [f] (-> f io/file

    .isDirectory))"
 
 $ ls | bb -i '(filter is-dir? *input*)' ("doc" "examples" "logo" ...)
  10. Scripts $ pst.clj
 04:58 #!/usr/bin/env bb (def now (java.time.ZonedDateTime/now)) (def

    LA-timezone (java.time.ZoneId/of "America/Los_Angeles")) (def LA-time (.withZoneSameInstant now LA-timezone)) (def pattern (java.time.format.DateTimeFormatter/ofPattern "HH:mm")) (println (.format LA-time pattern))
  11. Cross platform!

  12. Included libs / namespaces • clojure.{core, edn, java.shell, java.io, pprint,

    set, string, test, walk, zip} • clojure.tools.cli • clojure.core.async • clojure.data.csv • cheshire.core (JSON) • clojure.xml • cognitect.transit • clj-yaml • babashka.curl
  13. None
  14. Compatibility with JVM • No deftype, definterface, reify (defrecord, defprotocol

    added recently) • Code that does not use these constructs often works • Possible to load library code using existing mechanisms: classpath
  15. Classpath ;; spec.clj
 (require '[spartan.spec :as s])
 (s/explain (s/cat :x

    int? :y keyword?) [1 #{:foo}]) $ export BABASHKA_CLASSPATH=$(clojure -Spath ...) $ bb spec.clj
 #{:foo} - failed: keyword? in: [1] at: [:y]
  16. Selection of compatible libs • spartan.spec: clojure.spec.alpha (1) for bb

    • clj-http-lite: lighter fork of cli-http-lite • medley: "missing" clojure utility functions • regal: create regular expressions from EDN/hiccup • camel-snake-kebab: word case conversions • aero: library for explicit, intentful configuration. • nubank/docopt: docopt in Clojure
  17. $ clojure

  18. $ deps.clj $ deps.clj -Sdeps ... C:\>deps.clj -Sdeps ...

  19. Decomplecting babashka Libraries coming out of babashka: • edamame: EDN/Clojure

    parser • sci: a Small Clojure Interpreter
  20. edamame (def parsed (edamame/parse-string "#(+ 1 2 %)" {:fn true}))

    ;;=> (fn* [%1] (+ 1 2 %1)) (meta parsed) ;;=> {:row 1, :col 1, :end-row 1, :end-col 11} - EDN/code parser - GraalVM compatible (no eval!) - location metadata - opt-in code-like features
  21. Small Clojure Interpreter (def f (sci/eval-string "#(+ 1 2 %)"))

    (f 1) ;;=> 4 - Clojure interpreter - Works on JVM / GraalVM / JS - Sandboxing
 - Works in CLJS advanced compiled apps
 - Also available on NPM
  22. Sci from JavaScript https://observablehq.com/@jeroenvandijk/untitled/5 $ npm install @borkdude/sci
 $ node


    > const { evalString, toJS } = require('@borkdude/sci');
 > x = evalString("(assoc {:a 1} :b 2)")
 > toJS(x) { a: 1, b: 2 }
  23. Malli: serializable schemas (def my-schema [:and [:map [:x int?] [:y

    int?]] [:fn '(fn [{:keys [x y]}] (> x y))]]) (m/validate my-schema {:x 1, :y 0}) ; => true (m/validate my-schema {:x 1, :y 2}) ; => false
  24. Function CLI args $ jet --from json --keywordize '(comp keyword

    str/upper-case)' \
 <<< '{"a": 1}' {:A 1}
  25. clj-kondo hooks • The problem: • Clj-kondo doesn't recognize syntax

    of user-defined macros • Existing solutions: • Add built-in support • :lint-as - only adequate for identical syntax • :unresolved-symbol - works for suppressing, but also suppresses useful information
  26. clj-kondo hooks (ns mylib) (defmacro with-bound [binding-vector & body] ,,,)

  27. clj-kondo hooks

  28. clj-kondo hooks (hooks.with-bound/with-bound
 '(my-lib/with-bound [a 1 ...] ...))) 
 =>

    (let [a 1] {:with-bound/setting true} (inc 1))
  29. Sci: adding libs (require '[cheshire.core :as json])
 
 (def sci-opts


    {:namespaces
 {'cheshire.core
 {'generate-string json/generate-string}}}) 
 (sci/eval-string "(require '[cheshire.core :as json]) (json/generate-string {:a 1})" sci-opts)
 ;;=> "{\"a\":1}"
  30. Projects using sci • borkdude/babashka • borkdude/clj-kondo (Clojure linter) •

    borkdude/jet (JSON, EDN, Transit conversion) • epiccastle/spire (Ansible in Clojure) • metosin/malli (serializable schemas) • retrogradeorbit/bootleg (static HTML generation) • chlorine (Atom Clojure plugin) • dundalek/closh (Clojure shell replacement, mixed Clojure/Bash DSL) • liquidz/dad (config management, e.g. complex install scripts) • alekcz/pcp (Clojure PHP replacement) • theiceshelf/firn (Org-mode static site generator)
  31. Bootleg: static site CLI

  32. nREPL

  33. Babashka pods Problems: • Including more libs into bb will

    cause GraalVM to take more RAM, can't build on free CI anymore • Including more libs makes binary grow, while not all libraries are useful to everyone • How to access functionality from other platforms useful for scripting (Rust, Go, Python)?
  34. • Initial solutions: • Feature flags (PostgreSQL, HyperSQL, ...). Still

    supported. • Access native libraries (hard, brittle, too static) Babashka pods
  35. • Standalone CLIs that can act as libraries to babashka

    • Can be built in Clojure + GraalVM or any other language (Rust, Python) • Architecture similar to nREPL • Uses bencode and (JSON or EDN) for RPC Babashka pods
  36. • babashka-sql-pods (PostgreSQL, HyperSQL) • pod-babashka-etaoin (browser automation) • bootleg

    (HTML generation) • pod-babashka-filewatcher (implemented in Rust) • pod-babashka-parcera (whitespace preserving Clojure parser) • pod-lispyclouds-docker Babashka pods
  37. Filewatcher pod (Rust)

  38. PostgreSQL pod (clojure)

  39. Future • Babashka • book.babashka.org • bb.deps.edn / .babashka/config.edn •

    clojure.spec integration • Datomic client (optional via feature flag)
 • Sci • clojure.datafy • Smaller JS bundle by configuration • More control over duration / interrupts
  40. Clojure + GraalVM • CLJ-1472: issue with GraalVM and locking

    macro • Solved in 1.10.2-alpha1 (thanks Rich and Alex!) • https://github.com/lread/clj-graal-docs • https://github.com/BrunoBonacci/graalvm-clojure/
  41. Companies using babashka / sci Github #254 Add your company

    to the list:
  42. None
  43. Conclusion Clojure might not be the best language for everything,

    like scripting Clojure is the best language for scripting.
  44. Thanks to sponsors • Sponsors: • Github sponsors • OpenCollective

    • Ko-fi • Patreon
  45. Thank you! https://github.com/borkdude/babashka
 https://github.com/borkdude/sci And the Small Clojure Interpreter