Babashka Michiel Borkent @borkdude 2023-09-21 A Meta-Circular Clojure Interpreter For The Command Line 1

Clojure and me • University: functional programming Miranda, internship Common Lisp • CS lecturer: Java ... Clojure as a way to learn about the JVM • Clojure is Lisp on the JVM that emphasises FP • I tried other languages. Haskell, PureScript, Scala, ... I just love Clojure + can make a living from it ¯\_(ツ)_/¯ • I want to use Clojure's standard library everywhere 2

Is Clojure dead (yet)? • Making a modest income from Clojure OSS for the past two years • Clojurists Together: supporting the OSS Clojure ecosystem • Nubank (1200+ Clojure devs), largest online bank in South America • Red Planet, Grif fi n, Penpot, Roam Research, Logseq, Pitch, Apps fl yer, Cisco, Walmart, Exoscale, Nextjournal, Metabase, ... • Once you know Clojure, you don't ask that many questions on StackOver fl ow • Never breaking changes: ideal for long lived stable projects (as is the JVM) • Babashka is a sub-community within the Clojure community with its own conference • Long live "dead" languages! 👻 3

• Native Clojure scripting tool, single binary, no JVM • Can be used to replace “the grey areas” of bash • Easily installable via script, brew (macOS, linux), aur (linux), scoop (Windows) $ time bb - e '(+ 1 2 3)' 
 6 
 0.00s user 0.00s system 67% cpu 0.013 total 4

Startup time: clj (JVM) vs bb (native-image) bb (n.image) + lots of deps loaded at startup JVM Clojure + lots of deps loaded at startup Vanilla JVM Clojure, no deps 5

Example script: compile some Java 6

Example script: pst $ pst.clj 
 04 : 58 7

Cross platform! 😎 8

Interop done by Clojure lib $ pst.clj 
 04 : 58 9

Included libs • clojure.{core, edn, java.shell, java.io, pprint, set, string, test, walk, zip} • clojure.tools.cli • clojure.core.async • clojure.data.csv • cheshire.core (JSON) • clojure.xml • cognitect.transit • clj-yaml • httpkit.server • babashka.http-client, babashka.process, babashka.fs • many others 🔋 🔋 10

Included libs • Babashka aims to hold the promise of Clojure: No API changes • A script that runs now, should run forever • Included libraries should hold to this promise as well (Clojure ecosystem culture) • No removing of libs (even if they are deprecated) • Imagine unix command line tools changing their output every year... • Spec-ulation talk by Rich Hickey 🔋 🔋 11

12

13

• CLI tools with instant startup! (< 10ms) • clj-kondo: linter and static analyzer for Clojure • jet: convert between JSON, EDN and Transit + native-image 14

15

jet (jq-ish for Clojure) 
 DSL -> scripting • Added a query DSL to jet, a GraalVM CLI: 
 
 $ jet - - query '(map :id)' < < < '[{:id 1} {:id 2}]' 
 [1 2] • Extend this DSL to signi fi cant subset of Clojure? Or just use clojure.core/eval ...? 
 16

native-image + eval Clojure compiler emits JVM bytecode but GraalVM native-image transforms bytecode to native "ahead of time" 17

"Constraints force creativity and innovation" 
 - Thomas Wuerthinger

Small Clojure Interpreter (def f (sci/eval - string " # ( + 1 2 %)")) (f 1) ; ; = > 4 • Works on JVM / GraalVM native-image / JS • Sandboxing • Works in CLJS advanced compiled apps • Supports almost all of Clojure • https://github.com/babashka/sci 19

SCI: adding libs (require '[cheshire.core :as json]) 
 
 (def sci - opts 
 {:namespaces 
 {'cheshire.core 
 {'generate - string json/generate - string}}}) 
 (sci/eval - string "(require '[cheshire.core :as json]) (json/generate - string {:a 1})" sci - opts) 
 ; ; = > "{\"a\" : 1}" 20

SCI: meta-circular • Implements language features in terms of similar host language features • Self-interpreter: interpreted language = host-language • Function application can be implemented using apply • loop in SCI is implemented using loop in Clojure • Pure core functions like +, assoc, etc. can just be direct references to host functions • Side-effecting functions often re-implemented to make SCI sandboxed 21

Meta-circular function application (simpli fi ed) 22

let bindings (simpli fi ed) 23

lambda (simpli fi ed) 24

SCI's initial (naive) approach Parser 
 (string -> s-expr) Evaluator 
 (s-expr -> result) "(+ 1 x)" 
 -> '(+ 1 x) Walk s-expression 
 
 Decide that + is a symbol that denotes function, 
 resolve '+ to clojure.core/+ 
 
 Decide that 1 is constant and resolve to itself 
 
 Decide that symbol x is a local, look up local binding in ctx 
 Finally, call + on 1 and resolved value for 'x, e.g. 2 
 '(+ 1 x) -> 3 25 Lots of work happening over and over which could be done only once... (for [x (range 100000)] 
 (+ 1 x))

26

Do as much preparation as you in analysis Do as little as possible here 
 (avoid repeated analysis) 27

Analysis -> Node AST 28

Performance (time 
 (loop [val 0 cnt 10000000] 
 (if (pos? cnt) 
 (recur (inc val) (dec cnt)) 
 val))) 
 "Elapsed time: 575.72325 msecs" 
 10000000 • Performance was not the initial goal, later concern • Example: Loop of 10M iterations, conditional, 3 function calls • This is fast in compiled code, but SCI is interpreter (Measured on Macbook Air M1) 29

Comparison: 
 Python loop perf (Measured on Macbook Air M1) 30

SCI loop perf 16176 ms 991 ms 
 (15x faster than July 2020) (Measured on linux amd64 machine) 31

Accessing locals https://github.com/babashka/sci/issues/416 • Hash-map is dynamically sized, 
 implementation easy and thread-safe • But fi xed sized arrays are 15x faster... • Need to pre-allocate + pre-calculate indexes • Copy over closed over values to each new closure 
 (thread safety) • Set local value: (assoc bindings 'a 1) -> (aset bindings 0 1) • Read a: (get bindings 'a) -> (aget bindings 0) • Read c:(get bindings 'c) -> (aget bindings 3) 
 (let [a 1, b 2] 
 (fn [x] 
 (let [c (+ a b x)] 
 (inc c))) Closed over locals: a, b Function argument: x Let binding: c 32

fn + loop / recur 33

Interop (Thread/sleep 100) • SCI classes must be con fi gured 
 {:classes {'java.lang.Thread Thread} 
 :imports {'Thread 'java.lang.Thread} • Graal re fl ect-con fi g.json keeps classes + metadata around for re fl ection • Methods are looked up via Java re fl ection API (cached) and invoked • Room for performance improvements by looking up method at analysis time 34

Binary size 35

Babashka pods • External binaries that expose namespaces + functions to bb via RPC • Moar GraalVM binaries, but can also be built in Golang, Haskell, etc. as long as they return the right data • Gives room for experimentation before including more libs • Access to other language ecosystems 36

SCI on JS 37

clj-kondo hooks using SCI 38

Building on Clojure + JVM / GraalVM 39

1st Babashka-conf 40

Babashka workshop (Rahul De) 41

Babooka (Daniel Higginbotham) 42

Thank you for sponsoring my open source! https://opencollective.com/babashka/contribute https://github.com/sponsors/borkdude 43

Conclusion Clojure might not be the best language for everything, like scripting Clojure is the best language for scripting ;) 44