Specifying Other People’s Data Structures with Clojure’s Spec, An Experience Report

From my talk at clojure/nyc on 18 July 2018.

I walked through an experience I had recently with spec, wherein I needed to specify a large nested data structure that had been originally created by a different org for use in JSON or YAML — in other words, designed without Clojure’s spec in mind. I will show a few different challenges I encountered and the various approaches I tried.

Transcript

1. Specifying Other People’s Data
   Structures with Spec
   An Experience Report
   Clojure/NYC
   July 2018
   Avi Flax
   avi.flax@fundingcircle.com
   

   View Slide

2. This is really just an experience
   report: Iʼm going to share my
   experience trying to solve a
   problem with spec, show what I
   struggled with, what I tried that
   didnʼt work, and my final solution
   that did work.
   Experience Report
   

   View Slide

3. Introduction to Spec
   

   View Slide

4. Whatʼs itʼs goal?
   A Library
   

   View Slide

5. * As per Rich Hickey:
   *
   * Thatʼs how Rich Hickey
   describes it...
   “Clojure has no standard, expressive,
   powerful and integrated system for
   specification and testing.
   clojure.spec aims to provide it.”
   — Rich Hickey, clojure.spec - Rationale and Overview
   

   View Slide

6. * This is how I tend to describe it, when someone makes the
   mistake of asking me what Iʼm excited about in programming.
   *
   * So: itʼs Clojureʼs answer to creating large, maintainable,
   sustainable systems with a lot of code while retaining dynamicity
   * I.E. Clojureʼs alternative to types
   * Not a radically original idea
   * Hickey: “Almost nothing about spec is novel.”
   * Prior art: Schema, Herbert, contracts, RDF, schemas in general
   * Thatʼs all the background Iʼll do right now; hopefully this is
   enough that the code weʼll look at will make some sense.
   * But if anything doesnʼt make sense, please shout and weʼll fix
   that.
   “An excellent tool for specifying and
   validating data structures, and testing
   and documenting functions.”
   — me
   

   View Slide

7. My Problem
   

   View Slide

8. explain this
   {
   "elements": [
   {
   "type": "Software System",
   "name": "Uberwriter",
   "position": "700,100"
   }
   ],
   "styles": [
   {
   "type": "element",
   "tag": "Database",
   "shape": "Cylinder"
   }
   ]
   }
   

   View Slide

9. * Alternate view
   * For all its flaws, YAML is easy
   for humans to read
   * My goal: specify this data
   structure with spec
   elements:
   - type: Software System
   name: Uberwriter
   position: '700,100'
   styles:
   - type: element
   tag: Database
   shape: Cylinder
   

   View Slide

10. * Biggest challenge: type
    * To a neophyte, itʼs perplexing
    how to specify these different
    data structures that both use
    type
    elements:
    - type: Software System
    name: Uberwriter
    position: '700,100'
    styles:
    - type: element
    tag: Database
    shape: Cylinder
    

    View Slide

11. * Why is this perplexing?
    * It starts with the fact that one of
    specʼs strongly-held opinions (it
    has many) is that map keys
    should be namespaced and
    should be specified independently
    of the maps theyʼre used in
    Why?
    

    View Slide

12. “Map specs should be of keysets only”
    — One of the problem statements in the spec rationale
    

    View Slide

13. “Most systems for specifying structures
    conflate the specification of the key set
    (e.g. of keys in a map, fields in an
    object) with the specification of the
    values designated by those keys.”
    — from the explanation of that problem statement
    

    View Slide

14. This is clearer.
    “Decomplect maps/keys/values”
    — one of the stated objectives of spec
    

    View Slide

15. “Keep map (keyset) specs separate
    from attribute (key→value) specs.”
    — from the explanation of that objective
    

    View Slide

16. Why?
    

    View Slide

17. OK? Now time for some code
    “Sets (maps) are about membership,
    that’s it... This is vital for composition
    and dynamicity.”
    — one of the guidelines in the spec overview
    

    View Slide

18. * Most spec tutorials & guides
    show examples like this
    * All well and good
    * Letʼs discuss double-colons
    for a minute
    (ns hud.catalog)
    (s/def !"coords string?)
    (s/def !"description string?)
    (s/def !"building
    (s/keys :req [!"coords !"description]))
    

    View Slide

19. * So :: syntax is nice
    * and super convenient
    * but not always
    (ns hud.catalog)
    (s/def !"coords string?)
    ==
    (ns hud.catalog)
    (s/def :hud.catalog/coords string?)
    

    View Slide

20. The latter specs would clobber
    the earlier ones
    Wouldn’t work:
    ; element
    (s/def !"type !#"Person" "Software System"})
    ; style
    (s/def !"type !#"element" "tag" "relationship})
    

    View Slide

21. Attempt #1
    

    View Slide

22. Multiple namespaces, one file
    (ns fc4c.spec)
    (s/def !"name string?)
    (ns fc4c.spec.element (:require [fc4c.spec :as fs]))
    (s/def !"type !#"Person" "Software System"})
    (s/def !"element (s/keys :req [!"fs/name !"type !!$]))
    (ns fc4c.spec.style (:require [fc4c.spec :as fs]))
    (s/def !"type !#"element" "relationship"})
    (s/def !"style (s/keys :req [!"fs/name !"type !!$])
    (ns fc4c.spec.diagram (:require [!!$]))
    (s/def !"type !#"System Landscape" "System Context" "Container"})
    (s/def !"elements (s/coll-of !"fse/element))
    (s/def !"styles (s/coll-of !"fss/style))
    (s/def !"diagram (s/keys :req [!"elements !"styles !!$]))
    

    View Slide

23. I thought: this is exactly what
    namespaces are for, right? To
    prevent conflicts of things that
    are named similarly but have
    different semantics.
    (ns fc4c.spec)
    (s/def !"name string?)
    (ns fc4c.spec.element (:require [fc4c.spec :as fs]))
    (s/def !"type !#"Person" "Software System"})
    (s/def !"element (s/keys :req [!"fs/name !"type !!$]))
    (ns fc4c.spec.style (:require [fc4c.spec :as fs]))
    (s/def !"type !#"element" "relationship"})
    (s/def !"style (s/keys :req [!"fs/name !"type !!$])
    (ns fc4c.spec.diagram (:require [!!$]))
    (s/def !"type !#"System Landscape" "System Context" "Container"})
    (s/def !"elements (s/coll-of !"fse/element))
    (s/def !"styles (s/coll-of !"fss/style))
    (s/def !"diagram (s/keys :req [!"elements !"styles !!$]))
    

    View Slide

24. * All in one file!
    * Seemed to work!
    * But:
    ** Convoluted, verbose
    ** Non-idiomatic to have multi
    namespaces one file
    (ns fc4c.spec)
    (s/def !"name string?)
    (ns fc4c.spec.element (:require [fc4c.spec :as fs]))
    (s/def !"type !#"Person" "Software System"})
    (s/def !"element (s/keys :req [!"fs/name !"type !!$]))
    (ns fc4c.spec.style (:require [fc4c.spec :as fs]))
    (s/def !"type !#"element" "relationship"})
    (s/def !"style (s/keys :req [!"fs/name !"type !!$])
    (ns fc4c.spec.diagram (:require [!!$]))
    (s/def !"type !#"System Landscape" "System Context" "Container"})
    (s/def !"elements (s/coll-of !"fse/element))
    (s/def !"styles (s/coll-of !"fss/style))
    (s/def !"diagram (s/keys :req [!"elements !"styles !!$]))
    

    View Slide

25. Attempt #2
    

    View Slide

26. * A little better
    * Only 1 Clojure ns in file
    * Explicit namespaces — no
    more ::
    * One odd thing: copying
    specs from one ns to another:
    (ns fc4c.spec)
    (s/def :fc4c/name string?)
    !" elements!
    (s/def :fc4c.element/name :fc4c/name)
    (s/def :fc4c.element/type !#"Person" "Software System"})
    (s/def :fc4c/element (s/keys :req [:fc4c.element/name :fc4c.element/type !!$]))
    !" styles!
    (s/def :fc4c.style/name :fc4c/name)
    (s/def :fc4c.style/type !#"element" "relationship"})
    (s/def :fc4c/style (s/keys :req [:fc4c.style/name :fc4c.style/type !!$]))
    !" diagram!
    (s/def :fc4c.diagram/type !#"System Landscape" "System Context" "Container"})
    (s/def :fc4c.diagram/elements (s/coll-of :fc4c/element))
    (s/def :fc4c.diagram/styles (s/coll-of :fc4c/style))
    (s/def :fc4c/diagram
    (s/keys :req [:fc4c.diagram/type :fc4c.diagram/elements :fc4c.diagram/styles !!$]))
    

    View Slide

27. Explain why... then show
    (ns fc4c.spec)
    (s/def :fc4c/name string?)
    !" elements!
    (s/def :fc4c.element/name :fc4c/name)
    (s/def :fc4c.element/type !#"Person" "Software System"})
    (s/def :fc4c/element (s/keys :req [:fc4c.element/name :fc4c.element/type !!$]))
    !" styles!
    (s/def :fc4c.style/name :fc4c/name)
    (s/def :fc4c.style/type !#"element" "relationship"})
    (s/def :fc4c/style (s/keys :req [:fc4c.style/name :fc4c.style/type !!$]))
    !" diagram!
    (s/def :fc4c.diagram/type !#"System Landscape" "System Context" "Container"})
    (s/def :fc4c.diagram/elements (s/coll-of :fc4c/element))
    (s/def :fc4c.diagram/styles (s/coll-of :fc4c/style))
    (s/def :fc4c/diagram
    (s/keys :req [:fc4c.diagram/type :fc4c.diagram/elements :fc4c.diagram/styles !!$]))
    

    View Slide

28. This is what I get when I parse
    one of these YAML files
    {:elements [{:type "Software System"
    :name "Uberwriter"
    :description "UI for Underwriters"
    :position "700,100"}]
    :styles [{:type "element"
    :tag "Database"
    :shape "Cylinder"}]}
    

    View Slide

29. * I thought I was going to need to add a
    namespace to every key in this map
    * in order for the map to be valid according to my
    spec
    * So itʼd look something like this
    * But when I imagined writing the code that
    would assign the right namespace to the right
    key, it felt like a disaster
    * So that's why I copied the specs; so instead I
    could do this:
    {:fc4c.spec/elements [{:fc4c.element/type "Software System"
    :fc4c.spec/name "Uberwriter"
    :fc4c.element/position "700,100"}]
    :fc4c.spec/styles [{:fc4c.style/type "element"
    :fc4c.spec/tag "Database"
    :fc4c.style/shape "Cylinder"}]}
    

    View Slide

30. * I was thinking that when I
    added the namespaces like
    this, itʼd be easy, because I
    could add the same
    namespace to every key in any
    given kind of map
    {:fc4c.spec/elements [{:fc4c.element/type "Software System"
    :fc4c.element/name "Uberwriter"
    :fc4c.element/position "700,100"}]
    :fc4c.spec/styles [{:fc4c.style/type "element"
    :fc4c.style/tag "Database"
    :fc4c.style/shape "Cylinder"}]}
    

    View Slide

31. * Well it turns out that was silly,
    because spec includes a way
    to specify a map that contains
    unqualified keys:
    

    View Slide

32. (s/def :structurizr/style
    (s/keys :req [:structurizr.style/type
    :structurizr.style/tag]
    :opt [:structurizr.style/color
    :structurizr.style/shape])
    

    View Slide

33. (s/def :structurizr/style
    (s/keys :req-un [:structurizr.style/type
    :structurizr.style/tag]
    :opt-un [:structurizr.style/color
    :structurizr.style/shape])
    

    View Slide

34. Attempt #3
    

    View Slide

35. (ns fc4c.spec)
    (s/def :fc4c/name string?)
    !" elements!
    (s/def :fc4c.element/type !#"Person" "Software System"})
    (s/def :fc4c/element (s/keys :req-un [:fc4c/name :fc4c.element/type !!%]))
    !" styles!
    (s/def :fc4c.style/type !#"element" "relationship"})
    (s/def :fc4c/style (s/keys :req-un [:fc4c/name :fc4c.style/type !!%]))
    !" diagram!
    (s/def :fc4c.diagram/type !#"System Landscape" "System Context" "Container"})
    (s/def :fc4c.diagram/elements (s/coll-of :fc4c/element))
    (s/def :fc4c.diagram/styles (s/coll-of :fc4c/style))
    (s/def :fc4c/diagram
    (s/keys :req-un [:fc4c.diagram/type :fc4c.diagram/elements :fc4c.diagram/styles !!%]))
    

    View Slide

36. * Composition!
    * Validation!
    * Generation!
    * No post-processing!
    (ns fc4c.spec)
    (s/def :fc4c/name string?)
    !" elements!
    (s/def :fc4c.element/type !#"Person" "Software System"})
    (s/def :fc4c/element (s/keys :req-un [:fc4c/name :fc4c.element/type !!%]))
    !" styles!
    (s/def :fc4c.style/type !#"element" "relationship"})
    (s/def :fc4c/style (s/keys :req-un [:fc4c/name :fc4c.style/type !!%]))
    !" diagram!
    (s/def :fc4c.diagram/type !#"System Landscape" "System Context" "Container"})
    (s/def :fc4c.diagram/elements (s/coll-of :fc4c/element))
    (s/def :fc4c.diagram/styles (s/coll-of :fc4c/style))
    (s/def :fc4c/diagram
    (s/keys :req-un [:fc4c.diagram/type :fc4c.diagram/elements :fc4c.diagram/styles !!%]))
    

    View Slide

37. End result: a spec that
    thoroughly specifies a diagram
    data structure, using
    straightforward composition
    (s/def :fc4c/diagram
    (s/keys :req-un [:fc4c.diagram/type :fc4c.diagram/scope :fc4c/description
    :fc4c.diagram/elements :fc4c.diagram/relationships
    :fc4c.diagram/styles :fc4c.diagram/size]))
    

    View Slide

38. Demo Time!
    

    View Slide

39. Just a refinement I devised to
    improve the readability of my
    specs
    Refinement
    

    View Slide

40. * I find this really hard to read — lots of
    repetitive, verbose characters.
    * If you recall, I stopped using double-
    colons because I wanted all my specs
    in a single file, and I didnʼt want to
    define multiple Clojure namespaces in
    a single file, mostly because itʼs non-
    idiomatic.
    (s/def :structurizr.relationship/source :structurizr/name)
    (s/def :structurizr.relationship/destination :structurizr/name)
    (s/def :structurizr.relationship/order :structurizr/int-in-string)
    (s/def :structurizr.relationship/vertices
    (s/coll-of :structurizr/position :min-count 1))
    (s/def :structurizr/relationship
    (s/keys :req-un [:structurizr.relationship/source
    :structurizr.relationship/destination]
    :opt-un [:structurizr/description
    :structurizr/tags]))
    

    View Slide

41. * Then I made this function
    namespaces that Iʼm invoking at
    the top of my file — it creates a
    bunch of Clojure namespaces and
    an aliases to each.
    * So my specs can now look like
    this:
    (namespaces '[structurizr :as st]
    '[structurizr.container :as sc]
    '[structurizr.element :as se]
    '[structurizr.relationship :as sr]
    '[structurizr.style :as ss]
    '[structurizr.diagram :as sd])
    

    View Slide

42. * This is a big improvement
    * Iʼm very happy with this
    * Turns out I donʼt actually mind creating
    multiple Clojure namespaces, just to misuse
    them.
    * I just didnʼt want to have multiple ns forms
    in a single file, because that would be super
    strange and confusing to new maintainers
    opening the file for the first time.
    (s/def !"sr/source !"st/name)
    (s/def !"sr/destination !"st/name)
    (s/def !"sr/order !"st/int-in-string)
    (s/def !"sr/vertices (s/coll-of !"st/position
    :min-count 1))
    (s/def !"st/relationship
    (s/keys :req-un [!"sr/source !"sr/destination]
    :opt-un [!"st/description !"st/tags]))
    

    View Slide

43. The End?
    

    View Slide

44. This is a little embarrasing, but
    after all this, I finally stumbled
    across this section of the spec
    guide — and actually
    registered it, apparently for the
    first time:
    Postscript
    

    View Slide

45. * So yeah, this might be the
    solution to all of this — Iʼm not
    sure.
    * I dabbled with it a bit and it
    was non-obvious how to apply
    it to my situation
    

    View Slide

46. The End!
    Discussion Time!
    

    View Slide