Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Babashka and GraalVM: taking Clojure to new places @ Clojure NYC meetup

Babashka and GraalVM: taking Clojure to new places @ Clojure NYC meetup

Online presentation given at the Clojure NYC meetup in July 2020.

Michiel Borkent

July 11, 2020
Tweet

More Decks by Michiel Borkent

Other Decks in Programming

Transcript

  1. and GraalVM; taking Clojure to new places Michiel Borkent @borkdude

    2020-07-11
  2. • CLI tools with instant startup! (< 10ms) • clj-kondo:

    a linter for Clojure that sparks joy • jet: convert between JSON, EDN and Transit • No eval: dynamic classloader not supported! +
  3. DSL -> scripting • Added a query DSL to jet,

    a GraalVM CLI:
 
 $ jet --query '(map :id)' <<< '[{:id 1} {:id 2}]'
 [1 2] • Extend this DSL to significant subset of Clojure?

  4. • Native Clojure scripting tool, single binary, no JVM •

    Can be used to replace “the grey areas” of bash • Installable via script, brew (macOS, linux), aur (linux), scoop (Windows) $ time bb '(+ 1 2 3)'
 6
 0.00s user 0.00s system 67% cpu 0.013 total
  5. CLJ scripting Runtime Impl Startup* Interop Windows Execution Threads clojure

    JVM Java ~1048ms + + Compiled + planck JSCore CLJS / JS ~ 728ms + - Compiled - joker Native Go ~7ms - + Interpreted - babashka Native GraalVM ~10ms + + Interpreted + *) measured with multitime -n10 <cmd> -e '(+ 1 2 3)' on Ubuntu Bionic with Intel i7-3770K CPU @ 3.50GHz
  6. Babashka goals • Fast starting Clojure scripting alternative for JVM

    Clojure • Easy installation: README ⟶ grab binary ⟶ run within seconds • Familiar: targeted at JVM clojure users • Cross-platform: #{linux, macOS, Windows} • Interop with commonly used classes (System, File, java.time.*, java.nio.*) • Multi-threading support (pmap, future, ...) • Batteries included (tools.cli, cheshire, ...) + external libraries + pods
  7. Babashka non-goals • Performance • Long running performance intensive processes:

    use the JVM • Provide a mixed Clojure/Bash DSL (rather: be compatible with JVM Clojure) • Replace existing shells like Bash (rather: play well with them)
  8. Shell interaction $ ls | bb -i '(filter #(-> %

    io/file .isDirectory) *input*)' ("doc" "examples" "logo" ...)
  9. Predefine functions $ export BABASHKA_PRELOADS="(defn is-dir? [f] (-> f io/file

    .isDirectory))"
 
 $ ls | bb -i '(filter is-dir? *input*)' ("doc" "examples" "logo" ...)
  10. Scripts $ pst.clj
 04:58 #!/usr/bin/env bb (def now (java.time.ZonedDateTime/now)) (def

    LA-timezone (java.time.ZoneId/of "America/Los_Angeles")) (def LA-time (.withZoneSameInstant now LA-timezone)) (def pattern (java.time.format.DateTimeFormatter/ofPattern "HH:mm")) (println (.format LA-time pattern))
  11. Cross platform!

  12. Included libs / namespaces • clojure.{core, edn, java.shell, java.io, pprint,

    set, string, test, walk, zip} • clojure.tools.cli • clojure.core.async • clojure.data.csv • cheshire.core (JSON) • clojure.xml • cognitect.transit • clj-yaml • babashka.curl
  13. None
  14. Compatibility with JVM • No deftype, definterface, reify (defrecord, defprotocol

    added recently) • Code that does not use these constructs often works • Possible to load library code using existing mechanisms: classpath
  15. Classpath ;; spec.clj
 (require '[spartan.spec :as s])
 (s/explain (s/cat :x

    int? :y keyword?) [1 #{:foo}]) $ export BABASHKA_CLASSPATH=$(clojure -Spath ...) $ bb spec.clj
 #{:foo} - failed: keyword? in: [1] at: [:y]
  16. Selection of compatible libs • spartan.spec: clojure.spec.alpha (1) for bb

    • clj-http-lite: lighter fork of cli-http-lite • medley: "missing" clojure utility functions • regal: create regular expressions from EDN/hiccup • camel-snake-kebab: word case conversions • aero: library for explicit, intentful configuration. • nubank/docopt: docopt in Clojure
  17. $ clojure

  18. $ deps.clj $ deps.clj -Sdeps ... C:\>deps.clj -Sdeps ...

  19. Decomplecting babashka Libraries coming out of babashka: • edamame: EDN/Clojure

    parser • sci: a Small Clojure Interpreter
  20. edamame (def parsed (edamame/parse-string "#(+ 1 2 %)" {:fn true}))

    ;;=> (fn* [%1] (+ 1 2 %1)) (meta parsed) ;;=> {:row 1, :col 1, :end-row 1, :end-col 11} - EDN/code parser - GraalVM compatible (no eval!) - location metadata - opt-in code-like features
  21. Small Clojure Interpreter (def f (sci/eval-string "#(+ 1 2 %)"))

    (f 1) ;;=> 4 - Clojure interpreter - Works on JVM / GraalVM / JS - Sandboxing
 - Works in CLJS advanced compiled apps
 - Also available on NPM
  22. Sci from JavaScript https://observablehq.com/@jeroenvandijk/untitled/5 $ npm install @borkdude/sci
 $ node


    > const { evalString, toJS } = require('@borkdude/sci');
 > x = evalString("(assoc {:a 1} :b 2)")
 > toJS(x) { a: 1, b: 2 }
  23. Malli: serializable schemas (def my-schema [:and [:map [:x int?] [:y

    int?]] [:fn '(fn [{:keys [x y]}] (> x y))]]) (m/validate my-schema {:x 1, :y 0}) ; => true (m/validate my-schema {:x 1, :y 2}) ; => false
  24. Function CLI args $ jet --from json --keywordize '(comp keyword

    str/upper-case)' \
 <<< '{"a": 1}' {:A 1}
  25. clj-kondo hooks • The problem: • Clj-kondo doesn't recognize syntax

    of user-defined macros • Existing solutions: • Add built-in support • :lint-as - only adequate for identical syntax • :unresolved-symbol - works for suppressing, but also suppresses useful information
  26. clj-kondo hooks (ns mylib) (defmacro with-bound [binding-vector & body] ,,,)

  27. clj-kondo hooks

  28. clj-kondo hooks (hooks.with-bound/with-bound
 '(my-lib/with-bound [a 1 ...] ...))) 
 =>

    (let [a 1] {:with-bound/setting true} (inc 1))
  29. Sci: adding libs (require '[cheshire.core :as json])
 
 (def sci-opts


    {:namespaces
 {'cheshire.core
 {'generate-string json/generate-string}}}) 
 (sci/eval-string "(require '[cheshire.core :as json]) (json/generate-string {:a 1})" sci-opts)
 ;;=> "{\"a\":1}"
  30. Projects using sci • borkdude/babashka • borkdude/clj-kondo (Clojure linter) •

    borkdude/jet (JSON, EDN, Transit conversion) • epiccastle/spire (Ansible in Clojure) • metosin/malli (serializable schemas) • retrogradeorbit/bootleg (static HTML generation) • chlorine (Atom Clojure plugin) • dundalek/closh (Clojure shell replacement, mixed Clojure/Bash DSL) • liquidz/dad (config management, e.g. complex install scripts) • alekcz/pcp (Clojure PHP replacement) • theiceshelf/firn (Org-mode static site generator)
  31. Bootleg: static site CLI

  32. nREPL

  33. Babashka pods Problems: • Including more libs into bb will

    cause GraalVM to take more RAM, can't build on free CI anymore • Including more libs makes binary grow, while not all libraries are useful to everyone • How to access functionality from other platforms useful for scripting (Rust, Go, Python)?
  34. • Initial solutions: • Feature flags (PostgreSQL, HyperSQL, ...). Still

    supported. • Access native libraries (hard, brittle, too static) Babashka pods
  35. • Standalone CLIs that can act as libraries to babashka

    • Can be built in Clojure + GraalVM or any other language (Rust, Python) • Architecture similar to nREPL • Uses bencode and (JSON or EDN) for RPC Babashka pods
  36. • babashka-sql-pods (PostgreSQL, HyperSQL) • pod-babashka-etaoin (browser automation) • bootleg

    (HTML generation) • pod-babashka-filewatcher (implemented in Rust) • pod-babashka-parcera (whitespace preserving Clojure parser) • pod-lispyclouds-docker Babashka pods
  37. Filewatcher pod (Rust)

  38. PostgreSQL pod (clojure)

  39. Future • Babashka • book.babashka.org • bb.deps.edn / .babashka/config.edn •

    clojure.spec integration • Datomic client (optional via feature flag)
 • Sci • clojure.datafy • Smaller JS bundle by configuration • More control over duration / interrupts
  40. Clojure + GraalVM • CLJ-1472: issue with GraalVM and locking

    macro • Solved in 1.10.2-alpha1 (thanks Rich and Alex!) • https://github.com/lread/clj-graal-docs • https://github.com/BrunoBonacci/graalvm-clojure/
  41. Companies using babashka / sci Github #254 Add your company

    to the list:
  42. None
  43. Conclusion Clojure might not be the best language for everything,

    like scripting Clojure is the best language for scripting.
  44. Thanks to sponsors • Sponsors: • Github sponsors • OpenCollective

    • Ko-fi • Patreon
  45. Thank you! https://github.com/borkdude/babashka
 https://github.com/borkdude/sci And the Small Clojure Interpreter