Specifying Other People’s Data Structures with Clojure’s Spec, An Experience Report

From my talk at clojure/nyc on 18 July 2018.

I walked through an experience I had recently with spec, wherein I needed to specify a large nested data structure that had been originally created by a different org for use in JSON or YAML — in other words, designed without Clojure’s spec in mind. I will show a few different challenges I encountered and the various approaches I tried.

Transcript

1. Specifying Other People’s Data Structures with Spec An Experience Report Clojure/NYC July 2018 Avi Flax avi.flax@fundingcircle.com
2. This is really just an experience report: Iʼm going to share my experience trying to solve a problem with spec, show what I struggled with, what I tried that didnʼt work, and my final solution that did work. Experience Report
3. Introduction to Spec
4. Whatʼs itʼs goal? A Library
5. * As per Rich Hickey: * <read slide aloud> * Thatʼs how Rich Hickey describes it... “Clojure has no standard, expressive, powerful and integrated system for specification and testing. clojure.spec aims to provide it.” — Rich Hickey, clojure.spec - Rationale and Overview
6. * This is how I tend to describe it, when someone makes the mistake of asking me what Iʼm excited about in programming. * <read slide aloud> * So: itʼs Clojureʼs answer to creating large, maintainable, sustainable systems with a lot of code while retaining dynamicity * I.E. Clojureʼs alternative to types * Not a radically original idea * Hickey: “Almost nothing about spec is novel.” * Prior art: Schema, Herbert, contracts, RDF, schemas in general * Thatʼs all the background Iʼll do right now; hopefully this is enough that the code weʼll look at will make some sense. * But if anything doesnʼt make sense, please shout and weʼll fix that. “An excellent tool for specifying and validating data structures, and testing and documenting functions.” — me
7. My Problem
8. explain this { "elements": [ { "type": "Software System", "name": "Uberwriter", "position": "700,100" } ], "styles": [ { "type": "element", "tag": "Database", "shape": "Cylinder" } ] }
9. * Alternate view * For all its flaws, YAML is easy for humans to read * My goal: specify this data structure with spec elements: - type: Software System name: Uberwriter position: '700,100' styles: - type: element tag: Database shape: Cylinder
10. * Biggest challenge: type * To a neophyte, itʼs perplexing how to specify these different data structures that both use type elements: - type: Software System name: Uberwriter position: '700,100' styles: - type: element tag: Database shape: Cylinder
11. * Why is this perplexing? * It starts with the fact that one of specʼs strongly-held opinions (it has many) is that map keys should be namespaced and should be specified independently of the maps theyʼre used in Why?
12. “Map specs should be of keysets only” — One of the problem statements in the spec rationale
13. “Most systems for specifying structures conflate the specification of the key set (e.g. of keys in a map, fields in an object) with the specification of the values designated by those keys.” — from the explanation of that problem statement
14. This is clearer. “Decomplect maps/keys/values” — one of the stated objectives of spec
15. “Keep map (keyset) specs separate from attribute (key→value) specs.” — from the explanation of that objective
16. Why?
17. OK? Now time for some code “Sets (maps) are about membership, that’s it... This is vital for composition and dynamicity.” — one of the guidelines in the spec overview
18. * Most spec tutorials & guides show examples like this * All well and good * Letʼs discuss double-colons for a minute (ns hud.catalog) (s/def !"coords string?) (s/def !"description string?) (s/def !"building (s/keys :req [!"coords !"description]))
19. * So :: syntax is nice * and super convenient * but not always (ns hud.catalog) (s/def !"coords string?) == (ns hud.catalog) (s/def :hud.catalog/coords string?)
20. The latter specs would clobber the earlier ones Wouldn’t work: ; element (s/def !"type !#"Person" "Software System"}) ; style (s/def !"type !#"element" "tag" "relationship})
21. Attempt #1
22. Multiple namespaces, one file (ns fc4c.spec) (s/def !"name string?) (ns fc4c.spec.element (:require [fc4c.spec :as fs])) (s/def !"type !#"Person" "Software System"}) (s/def !"element (s/keys :req [!"fs/name !"type !!$])) (ns fc4c.spec.style (:require [fc4c.spec :as fs])) (s/def !"type !#"element" "relationship"}) (s/def !"style (s/keys :req [!"fs/name !"type !!$]) (ns fc4c.spec.diagram (:require [!!$])) (s/def !"type !#"System Landscape" "System Context" "Container"}) (s/def !"elements (s/coll-of !"fse/element)) (s/def !"styles (s/coll-of !"fss/style)) (s/def !"diagram (s/keys :req [!"elements !"styles !!$]))
23. I thought: this is exactly what namespaces are for, right? To prevent conflicts of things that are named similarly but have different semantics. (ns fc4c.spec) (s/def !"name string?) (ns fc4c.spec.element (:require [fc4c.spec :as fs])) (s/def !"type !#"Person" "Software System"}) (s/def !"element (s/keys :req [!"fs/name !"type !!$])) (ns fc4c.spec.style (:require [fc4c.spec :as fs])) (s/def !"type !#"element" "relationship"}) (s/def !"style (s/keys :req [!"fs/name !"type !!$]) (ns fc4c.spec.diagram (:require [!!$])) (s/def !"type !#"System Landscape" "System Context" "Container"}) (s/def !"elements (s/coll-of !"fse/element)) (s/def !"styles (s/coll-of !"fss/style)) (s/def !"diagram (s/keys :req [!"elements !"styles !!$]))
24. * All in one file! * Seemed to work! * But: ** Convoluted, verbose ** Non-idiomatic to have multi namespaces one file (ns fc4c.spec) (s/def !"name string?) (ns fc4c.spec.element (:require [fc4c.spec :as fs])) (s/def !"type !#"Person" "Software System"}) (s/def !"element (s/keys :req [!"fs/name !"type !!$])) (ns fc4c.spec.style (:require [fc4c.spec :as fs])) (s/def !"type !#"element" "relationship"}) (s/def !"style (s/keys :req [!"fs/name !"type !!$]) (ns fc4c.spec.diagram (:require [!!$])) (s/def !"type !#"System Landscape" "System Context" "Container"}) (s/def !"elements (s/coll-of !"fse/element)) (s/def !"styles (s/coll-of !"fss/style)) (s/def !"diagram (s/keys :req [!"elements !"styles !!$]))
25. Attempt #2
26. * A little better * Only 1 Clojure ns in file * Explicit namespaces — no more :: * One odd thing: copying specs from one ns to another: (ns fc4c.spec) (s/def :fc4c/name string?) !" elements! (s/def :fc4c.element/name :fc4c/name) (s/def :fc4c.element/type !#"Person" "Software System"}) (s/def :fc4c/element (s/keys :req [:fc4c.element/name :fc4c.element/type !!$])) !" styles! (s/def :fc4c.style/name :fc4c/name) (s/def :fc4c.style/type !#"element" "relationship"}) (s/def :fc4c/style (s/keys :req [:fc4c.style/name :fc4c.style/type !!$])) !" diagram! (s/def :fc4c.diagram/type !#"System Landscape" "System Context" "Container"}) (s/def :fc4c.diagram/elements (s/coll-of :fc4c/element)) (s/def :fc4c.diagram/styles (s/coll-of :fc4c/style)) (s/def :fc4c/diagram (s/keys :req [:fc4c.diagram/type :fc4c.diagram/elements :fc4c.diagram/styles !!$]))
27. Explain why... then show (ns fc4c.spec) (s/def :fc4c/name string?) !" elements! (s/def :fc4c.element/name :fc4c/name) (s/def :fc4c.element/type !#"Person" "Software System"}) (s/def :fc4c/element (s/keys :req [:fc4c.element/name :fc4c.element/type !!$])) !" styles! (s/def :fc4c.style/name :fc4c/name) (s/def :fc4c.style/type !#"element" "relationship"}) (s/def :fc4c/style (s/keys :req [:fc4c.style/name :fc4c.style/type !!$])) !" diagram! (s/def :fc4c.diagram/type !#"System Landscape" "System Context" "Container"}) (s/def :fc4c.diagram/elements (s/coll-of :fc4c/element)) (s/def :fc4c.diagram/styles (s/coll-of :fc4c/style)) (s/def :fc4c/diagram (s/keys :req [:fc4c.diagram/type :fc4c.diagram/elements :fc4c.diagram/styles !!$]))
28. This is what I get when I parse one of these YAML files {:elements [{:type "Software System" :name "Uberwriter" :description "UI for Underwriters" :position "700,100"}] :styles [{:type "element" :tag "Database" :shape "Cylinder"}]}
29. * I thought I was going to need to add a namespace to every key in this map * in order for the map to be valid according to my spec * So itʼd look something like this * But when I imagined writing the code that would assign the right namespace to the right key, it felt like a disaster * So that's why I copied the specs; so instead I could do this: {:fc4c.spec/elements [{:fc4c.element/type "Software System" :fc4c.spec/name "Uberwriter" :fc4c.element/position "700,100"}] :fc4c.spec/styles [{:fc4c.style/type "element" :fc4c.spec/tag "Database" :fc4c.style/shape "Cylinder"}]}
30. * I was thinking that when I added the namespaces like this, itʼd be easy, because I could add the same namespace to every key in any given kind of map {:fc4c.spec/elements [{:fc4c.element/type "Software System" :fc4c.element/name "Uberwriter" :fc4c.element/position "700,100"}] :fc4c.spec/styles [{:fc4c.style/type "element" :fc4c.style/tag "Database" :fc4c.style/shape "Cylinder"}]}
31. * Well it turns out that was silly, because spec includes a way to specify a map that contains unqualified keys:
32. (s/def :structurizr/style (s/keys :req [:structurizr.style/type :structurizr.style/tag] :opt [:structurizr.style/color :structurizr.style/shape])
33. (s/def :structurizr/style (s/keys :req-un [:structurizr.style/type :structurizr.style/tag] :opt-un [:structurizr.style/color :structurizr.style/shape])
34. Attempt #3
35. (ns fc4c.spec) (s/def :fc4c/name string?) !" elements! (s/def :fc4c.element/type !#"Person" "Software System"}) (s/def :fc4c/element (s/keys :req-un [:fc4c/name :fc4c.element/type !!%])) !" styles! (s/def :fc4c.style/type !#"element" "relationship"}) (s/def :fc4c/style (s/keys :req-un [:fc4c/name :fc4c.style/type !!%])) !" diagram! (s/def :fc4c.diagram/type !#"System Landscape" "System Context" "Container"}) (s/def :fc4c.diagram/elements (s/coll-of :fc4c/element)) (s/def :fc4c.diagram/styles (s/coll-of :fc4c/style)) (s/def :fc4c/diagram (s/keys :req-un [:fc4c.diagram/type :fc4c.diagram/elements :fc4c.diagram/styles !!%]))
36. * Composition! * Validation! * Generation! * No post-processing! (ns fc4c.spec) (s/def :fc4c/name string?) !" elements! (s/def :fc4c.element/type !#"Person" "Software System"}) (s/def :fc4c/element (s/keys :req-un [:fc4c/name :fc4c.element/type !!%])) !" styles! (s/def :fc4c.style/type !#"element" "relationship"}) (s/def :fc4c/style (s/keys :req-un [:fc4c/name :fc4c.style/type !!%])) !" diagram! (s/def :fc4c.diagram/type !#"System Landscape" "System Context" "Container"}) (s/def :fc4c.diagram/elements (s/coll-of :fc4c/element)) (s/def :fc4c.diagram/styles (s/coll-of :fc4c/style)) (s/def :fc4c/diagram (s/keys :req-un [:fc4c.diagram/type :fc4c.diagram/elements :fc4c.diagram/styles !!%]))
37. End result: a spec that thoroughly specifies a diagram data structure, using straightforward composition (s/def :fc4c/diagram (s/keys :req-un [:fc4c.diagram/type :fc4c.diagram/scope :fc4c/description :fc4c.diagram/elements :fc4c.diagram/relationships :fc4c.diagram/styles :fc4c.diagram/size]))
38. Demo Time!
39. Just a refinement I devised to improve the readability of my specs Refinement
40. * I find this really hard to read — lots of repetitive, verbose characters. * If you recall, I stopped using double- colons because I wanted all my specs in a single file, and I didnʼt want to define multiple Clojure namespaces in a single file, mostly because itʼs non- idiomatic. (s/def :structurizr.relationship/source :structurizr/name) (s/def :structurizr.relationship/destination :structurizr/name) (s/def :structurizr.relationship/order :structurizr/int-in-string) (s/def :structurizr.relationship/vertices (s/coll-of :structurizr/position :min-count 1)) (s/def :structurizr/relationship (s/keys :req-un [:structurizr.relationship/source :structurizr.relationship/destination] :opt-un [:structurizr/description :structurizr/tags]))
41. * Then I made this function namespaces that Iʼm invoking at the top of my file — it creates a bunch of Clojure namespaces and an aliases to each. * So my specs can now look like this: (namespaces '[structurizr :as st] '[structurizr.container :as sc] '[structurizr.element :as se] '[structurizr.relationship :as sr] '[structurizr.style :as ss] '[structurizr.diagram :as sd])
42. * This is a big improvement * Iʼm very happy with this * Turns out I donʼt actually mind creating multiple Clojure namespaces, just to misuse them. * I just didnʼt want to have multiple ns forms in a single file, because that would be super strange and confusing to new maintainers opening the file for the first time. (s/def !"sr/source !"st/name) (s/def !"sr/destination !"st/name) (s/def !"sr/order !"st/int-in-string) (s/def !"sr/vertices (s/coll-of !"st/position :min-count 1)) (s/def !"st/relationship (s/keys :req-un [!"sr/source !"sr/destination] :opt-un [!"st/description !"st/tags]))
43. The End?
44. This is a little embarrasing, but after all this, I finally stumbled across this section of the spec guide — and actually registered it, apparently for the first time: Postscript
45. * So yeah, this might be the solution to all of this — Iʼm not sure. * I dabbled with it a bit and it was non-obvious how to apply it to my situation
46. The End! Discussion Time!