Domain Specific Languages and Metaprogramming

Transcript

1. Domain Specific Languages and
   Metaprogramming
   Andrew Summers
   2019-07-24
   

   View Slide

2. #chielixirdsl
   Andrew Summers
   whoami
   • Software Engineer at DRW
   
   • Wrote Dialyzer pretty printer
   
   • Maintainer of Dialyxir, Erlex, elixir-lsp packages
   
   • Contributor to many others
   

   View Slide

3. #chielixirdsl
   Andrew Summers
   • What is a DSL?
   
   • Modules As Data Structures
   
   • Macros
   
   • Let’s Create a DSL
   

   View Slide

4. #chielixirdsl
   Andrew Summers
   • What is a DSL?
   
   • Modules As Data Structures
   
   • Macros
   
   • Let’s Create a DSL
   

   View Slide

5. #chielixirdsl
   Andrew Summers
   What is a DSL?
   • DSL = Domain Specific Language
   
   • Facilitates communication with experts within domain
   
   • Allows hiding plumbing behind macros
   
   • Examples: Ecto, Absinthe
   

   View Slide

6. #chielixirdsl
   Andrew Summers
   Why create a DSL?
   • Have superset of information needed to embed other
   DSLs
   
   • Allows you to edit entire codebase in one fell swoop
   
   • Plumbing data structures via normal functions is painful
   
   • Code base begins to solidify and data structures allow
   very generic pipelines
   

   View Slide

7. #chielixirdsl
   Andrew Summers
   Caveat Emptor
   • Warning: usually you just need functions and data, and
   should not create DSLs too early
   
   • Wait until data structures are somewhat stable before
   creating DSLs
   
   • Wait until plumbing is painful/buggy or some compile time
   need for macros exists before thinking about DSL
   
   • You probably do not need a DSL
   

   View Slide

8. #chielixirdsl
   Andrew Summers
   • What is a DSL?
   
   • Modules As Data Structures
   
   • Macros
   
   • Let’s Create a DSL
   

   View Slide

9. #chielixirdsl
   Andrew Summers
   defmodule ModuleAttributes do
   Module.register_attribute(__MODULE__, :foos,
   accumulate: true)
   Module.register_attribute(__MODULE__, :bar,
   accumulate: false)
   @foos :a
   @foos :b
   @bar :c
   @bar :d
   def foos(), do: @foos # [:b, :a]
   def bar(), do: @bar # :d
   end
   

   View Slide

10. #chielixirdsl
    Andrew Summers
    Module Attributes
    • Accumulate data into module attributes at compile time
    
    • Embed into function calls / data structures
    
    • Put macro data into internal data structure, call
    Module.put_attribute/3 after validation of opts
    

    View Slide

11. #chielixirdsl
    Andrew Summers
    Modules As Data Structures
    • See __schema__/1 in Ecto
    
    • Encode data into functions
    
    • Use behaviours for compile/refactor safety
    
    • Ask a module questions by passing around module name
    and invoking functions for use in generic pipelines
    

    View Slide

12. #chielixirdsl
    Andrew Summers
    • What is a DSL?
    
    • Modules As Data Structures
    
    • Macros
    
    • Let’s Create a DSL
    

    View Slide

13. #chielixirdsl
    Andrew Summers
    Macro Best Practices
    • Only do necessary work in macros and delegate out to
    functions ASAP
    
    • Use location: :keep to get better line numbers in errors
    
    • Validate inputs (especially unnecessary opts)
    
    • Use after_compile hooks to validate state where
    appropriate
    
    • Avoid dynamic name generation, optimize for greppability
    
    • Disabling lexical tracker useful for preventing deadlocks
    

    View Slide

14. View Slide

15. #chielixirdsl
    Andrew Summers
    data >> code > macros
    

    View Slide

16. #chielixirdsl
    Andrew Summers
    DSL = Data + Code + Macros
    

    View Slide

17. #chielixirdsl
    Andrew Summers
    • What is a DSL?
    
    • Modules As Data Structures
    
    • Macro Hygiene
    
    • Let’s Create a DSL
    

    View Slide

18. #chielixirdsl
    Andrew Summers
    Let’s create a DSL
    • We want more data than Ecto provides in its API, e.g.
    description
    
    • We also want Relationships, Fields, Schema semantics
    
    • Embed Ecto Schema internally
    

    View Slide

19. #chielixirdsl
    Andrew Summers
    defmodule MyApp.Tables.ExampleTable do
    import Ectoo.Table, only: [table: 1]
    table :example do
    description "This is an example table"
    attribute :foo, :integer,
    required?: true,
    description: "foos do foo things"
    attribute :bar, :string, default: "none"
    end
    end
    

    View Slide

20. #chielixirdsl
    Andrew Summers
    defmodule Ectoo.Attribute do
    @type t :: %Ectoo.Attribute{
    default: any(),
    description: String.t(),
    name: atom(),
    required?: boolean(),
    type: atom() | module()
    }
    defstruct [
    :default,
    :description,
    :name,
    :required?,
    :type
    ]
    # ...
    end
    

    View Slide

21. #chielixirdsl
    Andrew Summers
    defmodule Ectoo.Relationship do
    @type t :: %Ectoo.Relationship{
    cardinality: :belongs_to | :has_many | :has_one,
    description: String.t(),
    field: atom(),
    related: module()
    }
    defstruct [
    :cardinality,
    :description,
    :field,
    :related
    ]
    # ...
    end
    

    View Slide

22. #chielixirdsl
    Andrew Summers
    defmodule Ectoo.Table do
    defmacro table(name, do: body) do
    quote location: :keep do
    Module.register_attribute(__MODULE__, :ectoo_description,
    accumulate: false)
    Module.register_attribute(__MODULE__, :ectoo_table,
    accumulate: false)
    Module.register_attribute(__MODULE__, :ectoo_attributes,
    accumulate: true)
    Module.register_attribute(__MODULE__, :ectoo_relationships,
    accumulate: true)
    # ...
    end
    end
    # ...
    end
    

    View Slide

23. #chielixirdsl
    Andrew Summers
    defmodule Ectoo.Table do
    defmacro table(name, do: body) do
    quote location: :keep do
    # ...
    @ectoo_table unquote(name)
    import Ectoo.Table
    try do
    unquote(body)
    after
    _ -> :ok
    end
    import Ectoo.Table, only: []
    # ...
    end
    end
    # ...
    end
    

    View Slide

24. #chielixirdsl
    Andrew Summers
    defmodule Ectoo.Table do
    # ...
    defmacro attribute(name, type, opts \\ []) do
    quote location: :keep do
    Ectoo.Table.__attribute__(__MODULE__, unquote(name),
    unquote(type), unquote(opts))
    end
    end
    def __attribute__(module, name, type, opts) do
    # ... validate opts
    attribute = %Ectoo.Attribute{
    name: name,
    type: type,
    default: opts[:default],
    description: opts[:description],
    required?: opts[:required?] || false
    }
    Module.put_attribute(module, :ectoo_attributes, attribute)
    end
    # ...
    end
    

    View Slide

25. #chielixirdsl
    Andrew Summers
    defmodule Ectoo.Table do
    # ...
    defmacro description(name, description) do
    quote location: :keep do
    Ectoo.Table.__description__(__MODULE__,
    unquote(description))
    end
    end
    def __description__(module, description) do
    Module.put_attribute(module, :ectoo_description,
    description)
    end
    # ...
    end
    

    View Slide

26. #chielixirdsl
    Andrew Summers
    defmodule Ectoo.Table do
    # ...
    defp expand_alias({:__aliases__, _, _} = ast, env)
    do
    Macro.expand(ast, %{env | lexical_tracker: nil})
    end
    defp expand_alias(ast, _env) do
    ast
    end
    # ...
    end
    

    View Slide

27. #chielixirdsl
    Andrew Summers
    defmodule Ectoo.Table do
    # ...
    defmacro has_one_relationship(name, queryable, opts \\ []) do
    queryable = expand_alias(queryable, __CALLER__)
    quote do
    Ectoo.Table.__has_one_relationship__(__MODULE__,
    unquote(name), unquote(queryable), unquote(opts))
    end
    end
    def __has_one_relationship__(module, name, related, opts) do
    # ... validate opts
    relationship = %Ectoo.Relationship{
    field: name,
    cardinality: :has_one,
    related: related
    }
    Module.put_attribute(module, :ectoo_relationships, relationship)
    end
    # ...
    end
    

    View Slide

28. #chielixirdsl
    Andrew Summers
    defmodule Ectoo.Table do
    defmacro table(name, do: body) do
    quote location: :keep do
    # ...
    def attributes(), do: @ectoo_attributes
    def description(), do: @ectoo_description
    def belongs_to_relationships(), do:
    Enum.filter(@ectoo_relationships, & &1.cardinality
    == :belongs_to)
    def has_many_relationships(), do:
    Enum.filter(@ectoo_relationships, & &1.cardinality
    == :has_many)
    def has_one_relationships(), do:
    Enum.filter(@ectoo_relationships, & &1.cardinality
    == :has_one)
    end
    end
    # ...
    end
    

    View Slide

29. #chielixirdsl
    Andrew Summers
    defmodule Ectoo.Table do
    defmacro table(name, do: body) do
    quote location: :keep do
    # ...
    use Ecto.Schema
    schema @ectoo_table do
    for field <- @ectoo_attributes, do: field field.name,
    field.type, default: field.default
    for relationship = %{cardinality: :belongs_to} <-
    @ectoo_relationships,
    do: belongs_to relationship.name, relationship.related
    for relationship = %{cardinality: :has_many} <-
    @ectoo_relationships
    do: has_many relationship.name, relationship.related
    for relationship = %{cardinality: :has_one} <-
    @ectoo_relationships,
    do: has_one relationship.name, relationship.related
    end
    # ...
    end
    end
    # ...
    end
    

    View Slide

30. #chielixirdsl
    Andrew Summers
    defmodule MyApp.Tables.ExampleTable do
    import Ectoo.Table, only: [table: 1]
    table :example do
    description "This is an example table"
    attribute :foo, :integer,
    required?: true,
    description: "foos do foo things"
    attribute :bar, :string, default: "none"
    end
    end
    

    View Slide

31. #chielixirdsl
    Andrew Summers
    Recap
    • What and why DSLs?
    
    • Used module attributes as vehicle to collect compile time
    data
    
    • Macros
    
    • Built DSL wrapper around Ecto
    

    View Slide

32. #chielixirdsl
    Andrew Summers
    Future Steps
    • Expand the data structures to have all Ecto options
    
    • Build helpers to answer common questions from data
    
    • Use the data structures to build generic pipelines, e.g.
    Changesets
    
    • Add more macros to enable more sophisticated pipelines
    

    View Slide

33. #chielixirdsl
    Andrew Summers
    Thanks!

    

    asummers 

    

    _asummers
    

    View Slide

34. Questions?
    

    View Slide