Slide 1

Slide 1 text

and GraalVM; taking Clojure to new places Michiel Borkent @borkdude 2020-07-11

Slide 2

Slide 2 text

• CLI tools with instant startup! (< 10ms) • clj-kondo: a linter for Clojure that sparks joy • jet: convert between JSON, EDN and Transit • No eval: dynamic classloader not supported! +

Slide 3

Slide 3 text

DSL -> scripting • Added a query DSL to jet, a GraalVM CLI:
 
 $ jet --query '(map :id)' <<< '[{:id 1} {:id 2}]'
 [1 2] • Extend this DSL to significant subset of Clojure?


Slide 4

Slide 4 text

• Native Clojure scripting tool, single binary, no JVM • Can be used to replace “the grey areas” of bash • Installable via script, brew (macOS, linux), aur (linux), scoop (Windows) $ time bb '(+ 1 2 3)'
 6
 0.00s user 0.00s system 67% cpu 0.013 total

Slide 5

Slide 5 text

CLJ scripting Runtime Impl Startup* Interop Windows Execution Threads clojure JVM Java ~1048ms + + Compiled + planck JSCore CLJS / JS ~ 728ms + - Compiled - joker Native Go ~7ms - + Interpreted - babashka Native GraalVM ~10ms + + Interpreted + *) measured with multitime -n10 -e '(+ 1 2 3)' on Ubuntu Bionic with Intel i7-3770K CPU @ 3.50GHz

Slide 6

Slide 6 text

Babashka goals • Fast starting Clojure scripting alternative for JVM Clojure • Easy installation: README ⟶ grab binary ⟶ run within seconds • Familiar: targeted at JVM clojure users • Cross-platform: #{linux, macOS, Windows} • Interop with commonly used classes (System, File, java.time.*, java.nio.*) • Multi-threading support (pmap, future, ...) • Batteries included (tools.cli, cheshire, ...) + external libraries + pods

Slide 7

Slide 7 text

Babashka non-goals • Performance • Long running performance intensive processes: use the JVM • Provide a mixed Clojure/Bash DSL (rather: be compatible with JVM Clojure) • Replace existing shells like Bash (rather: play well with them)

Slide 8

Slide 8 text

Shell interaction $ ls | bb -i '(filter #(-> % io/file .isDirectory) *input*)' ("doc" "examples" "logo" ...)

Slide 9

Slide 9 text

Predefine functions $ export BABASHKA_PRELOADS="(defn is-dir? [f] (-> f io/file .isDirectory))"
 
 $ ls | bb -i '(filter is-dir? *input*)' ("doc" "examples" "logo" ...)

Slide 10

Slide 10 text

Scripts $ pst.clj
 04:58 #!/usr/bin/env bb (def now (java.time.ZonedDateTime/now)) (def LA-timezone (java.time.ZoneId/of "America/Los_Angeles")) (def LA-time (.withZoneSameInstant now LA-timezone)) (def pattern (java.time.format.DateTimeFormatter/ofPattern "HH:mm")) (println (.format LA-time pattern))

Slide 11

Slide 11 text

Cross platform!

Slide 12

Slide 12 text

Included libs / namespaces • clojure.{core, edn, java.shell, java.io, pprint, set, string, test, walk, zip} • clojure.tools.cli • clojure.core.async • clojure.data.csv • cheshire.core (JSON) • clojure.xml • cognitect.transit • clj-yaml • babashka.curl

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

Compatibility with JVM • No deftype, definterface, reify (defrecord, defprotocol added recently) • Code that does not use these constructs often works • Possible to load library code using existing mechanisms: classpath

Slide 15

Slide 15 text

Classpath ;; spec.clj
 (require '[spartan.spec :as s])
 (s/explain (s/cat :x int? :y keyword?) [1 #{:foo}]) $ export BABASHKA_CLASSPATH=$(clojure -Spath ...) $ bb spec.clj
 #{:foo} - failed: keyword? in: [1] at: [:y]

Slide 16

Slide 16 text

Selection of compatible libs • spartan.spec: clojure.spec.alpha (1) for bb • clj-http-lite: lighter fork of cli-http-lite • medley: "missing" clojure utility functions • regal: create regular expressions from EDN/hiccup • camel-snake-kebab: word case conversions • aero: library for explicit, intentful configuration. • nubank/docopt: docopt in Clojure

Slide 17

Slide 17 text

$ clojure

Slide 18

Slide 18 text

$ deps.clj $ deps.clj -Sdeps ... C:\>deps.clj -Sdeps ...

Slide 19

Slide 19 text

Decomplecting babashka Libraries coming out of babashka: • edamame: EDN/Clojure parser • sci: a Small Clojure Interpreter

Slide 20

Slide 20 text

edamame (def parsed (edamame/parse-string "#(+ 1 2 %)" {:fn true})) ;;=> (fn* [%1] (+ 1 2 %1)) (meta parsed) ;;=> {:row 1, :col 1, :end-row 1, :end-col 11} - EDN/code parser - GraalVM compatible (no eval!) - location metadata - opt-in code-like features

Slide 21

Slide 21 text

Small Clojure Interpreter (def f (sci/eval-string "#(+ 1 2 %)")) (f 1) ;;=> 4 - Clojure interpreter - Works on JVM / GraalVM / JS - Sandboxing
 - Works in CLJS advanced compiled apps
 - Also available on NPM

Slide 22

Slide 22 text

Sci from JavaScript https://observablehq.com/@jeroenvandijk/untitled/5 $ npm install @borkdude/sci
 $ node
 > const { evalString, toJS } = require('@borkdude/sci');
 > x = evalString("(assoc {:a 1} :b 2)")
 > toJS(x) { a: 1, b: 2 }

Slide 23

Slide 23 text

Malli: serializable schemas (def my-schema [:and [:map [:x int?] [:y int?]] [:fn '(fn [{:keys [x y]}] (> x y))]]) (m/validate my-schema {:x 1, :y 0}) ; => true (m/validate my-schema {:x 1, :y 2}) ; => false

Slide 24

Slide 24 text

Function CLI args $ jet --from json --keywordize '(comp keyword str/upper-case)' \
 <<< '{"a": 1}' {:A 1}

Slide 25

Slide 25 text

clj-kondo hooks • The problem: • Clj-kondo doesn't recognize syntax of user-defined macros • Existing solutions: • Add built-in support • :lint-as - only adequate for identical syntax • :unresolved-symbol - works for suppressing, but also suppresses useful information

Slide 26

Slide 26 text

clj-kondo hooks (ns mylib) (defmacro with-bound [binding-vector & body] ,,,)

Slide 27

Slide 27 text

clj-kondo hooks

Slide 28

Slide 28 text

clj-kondo hooks (hooks.with-bound/with-bound
 '(my-lib/with-bound [a 1 ...] ...))) 
 => (let [a 1] {:with-bound/setting true} (inc 1))

Slide 29

Slide 29 text

Sci: adding libs (require '[cheshire.core :as json])
 
 (def sci-opts
 {:namespaces
 {'cheshire.core
 {'generate-string json/generate-string}}}) 
 (sci/eval-string "(require '[cheshire.core :as json]) (json/generate-string {:a 1})" sci-opts)
 ;;=> "{\"a\":1}"

Slide 30

Slide 30 text

Projects using sci • borkdude/babashka • borkdude/clj-kondo (Clojure linter) • borkdude/jet (JSON, EDN, Transit conversion) • epiccastle/spire (Ansible in Clojure) • metosin/malli (serializable schemas) • retrogradeorbit/bootleg (static HTML generation) • chlorine (Atom Clojure plugin) • dundalek/closh (Clojure shell replacement, mixed Clojure/Bash DSL) • liquidz/dad (config management, e.g. complex install scripts) • alekcz/pcp (Clojure PHP replacement) • theiceshelf/firn (Org-mode static site generator)

Slide 31

Slide 31 text

Bootleg: static site CLI

Slide 32

Slide 32 text

nREPL

Slide 33

Slide 33 text

Babashka pods Problems: • Including more libs into bb will cause GraalVM to take more RAM, can't build on free CI anymore • Including more libs makes binary grow, while not all libraries are useful to everyone • How to access functionality from other platforms useful for scripting (Rust, Go, Python)?

Slide 34

Slide 34 text

• Initial solutions: • Feature flags (PostgreSQL, HyperSQL, ...). Still supported. • Access native libraries (hard, brittle, too static) Babashka pods

Slide 35

Slide 35 text

• Standalone CLIs that can act as libraries to babashka • Can be built in Clojure + GraalVM or any other language (Rust, Python) • Architecture similar to nREPL • Uses bencode and (JSON or EDN) for RPC Babashka pods

Slide 36

Slide 36 text

• babashka-sql-pods (PostgreSQL, HyperSQL) • pod-babashka-etaoin (browser automation) • bootleg (HTML generation) • pod-babashka-filewatcher (implemented in Rust) • pod-babashka-parcera (whitespace preserving Clojure parser) • pod-lispyclouds-docker Babashka pods

Slide 37

Slide 37 text

Filewatcher pod (Rust)

Slide 38

Slide 38 text

PostgreSQL pod (clojure)

Slide 39

Slide 39 text

Future • Babashka • book.babashka.org • bb.deps.edn / .babashka/config.edn • clojure.spec integration • Datomic client (optional via feature flag)
 • Sci • clojure.datafy • Smaller JS bundle by configuration • More control over duration / interrupts

Slide 40

Slide 40 text

Clojure + GraalVM • CLJ-1472: issue with GraalVM and locking macro • Solved in 1.10.2-alpha1 (thanks Rich and Alex!) • https://github.com/lread/clj-graal-docs • https://github.com/BrunoBonacci/graalvm-clojure/

Slide 41

Slide 41 text

Companies using babashka / sci Github #254 Add your company to the list:

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

Conclusion Clojure might not be the best language for everything, like scripting Clojure is the best language for scripting.

Slide 44

Slide 44 text

Thanks to sponsors • Sponsors: • Github sponsors • OpenCollective • Ko-fi • Patreon

Slide 45

Slide 45 text

Thank you! https://github.com/borkdude/babashka
 https://github.com/borkdude/sci And the Small Clojure Interpreter