Custom Query Languages: Why? How?

Slide 1

Slide 1 text

Custom Query Languages: Why? How? @AnjanaVakil - J on the Beach 2017

Slide 2

Slide 2 text

HOLA! I’m @AnjanaVakil Software Engineer/DSL-builder, ÜberResearch with input from Tobias Johansson Technical Lead, Valo

Slide 3

Slide 3 text

DIMENSIONS uberresearch.com/dimensions Scientific research funding data for funders, researchers, and research organizations.

Slide 4

Slide 4 text

VALO valo.io Real-time and historical analytics on data streams. Plus beachy conferences.

Slide 5

Slide 5 text

WHAT is a Custom Query Language (CQL) ?

Slide 6

Slide 6 text

CUSTOM QUERY LANGUAGE (CQL) domain- specific

Slide 7

Slide 7 text

CUSTOM QUERY LANGUAGE (CQL) data retrieval & analysis

Slide 8

Slide 8 text

CUSTOM QUERY LANGUAGE (CQL) text-based interface

Slide 9

Slide 9 text

“ a common text that acts as both executable software and a description that domain experts can read Martin Fowler, Domain Specific Languages

Slide 10

Slide 10 text

search grants for "malaria" where funder.acronym="NIH" return categories, years, researchers:[surname] aggregate funding sort by funding DIMENSIONS query language

Slide 11

Slide 11 text

VALO query language from /streams/sensors/air group by sampleTime window of 1 minute select sampleTime, sensor, avg(pollution)

Slide 12

Slide 12 text

WHY would you want your own CQL ?

Slide 13

Slide 13 text

MODELING

Slide 14

Slide 14 text

VALO modeling time from historical /streams/tenant/collection/name group by timestamp window of 3 seconds every 1 second select timestamp, count(), sum(value) order by timestamp

Slide 15

Slide 15 text

Display from Model DECOUPLING Model from Storage

Slide 16

Slide 16 text

DIMENSIONS decoupling SOLR

Slide 17

Slide 17 text

DIMENSIONS decoupling SOLR

Slide 18

Slide 18 text

DIMENSIONS decoupling SOLR > Elastic ?

Slide 19

Slide 19 text

“ Trying to describe a domain using a DSL is useful even if the DSL is never implemented. It can be beneficial just as a platform for communication. Martin Fowler, Domain Specific Languages

Slide 20

Slide 20 text

HOW should you design a CQL ?

Slide 21

Slide 21 text

1. Clarity 2. Concision 3. Familiarity 4. Responsiveness WHAT MAKES A GREAT USER INTERFACE? 5. Consistency 6. Aesthetics 7. Efficiency 8. Forgiveness https://www.smashingmagazine.com/user-interface-design-in-modern-web-applications

Slide 22

Slide 22 text

1. 2. Concision 3. Familiarity 4. the user can easily express what they want 5. 6. Aesthetics 7. Efficiency 8.

Slide 23

Slide 23 text

VALO historical data from historical /streams/sensors/air group by sampleTime window of 1 minute select sampleTime, sensor, avg(pollution)

Slide 24

Slide 24 text

1. Clarity 2. 3. Familiarity 4. Responsiveness the user doesn’t often need the docs 5. Consistency 6. 7. 8. Forgiveness not

Slide 25

Slide 25 text

Unknown source 'articles' Did you mean one of these sources? applications clinical_trials grants patents publications DIMENSIONS error messages

Slide 26

Slide 26 text

it will change but no matter how good your design

Slide 27

Slide 27 text

Get user feedback ASAP expect & embrace change (as usual) Design for flexibility

Slide 28

Slide 28 text

HOW can you implement a CQL ?

Slide 29

Slide 29 text

PARSING YOUR CQL Terrence Parr, The Definitive ANTLR4 Reference

Slide 30

Slide 30 text

Use a parser generator Prototype quickly, giving up control over some details of lexing/parsing. e.g. ANTLR, parboiled2 PARSING YOUR CQL Roll your own Retain full control over lexing/parsing and internal syntactic representation.

Slide 31

Slide 31 text

Slide 32

Slide 32 text

ANTLR (v4) antlr.org Java-based parser generator with targets for various other languages (e.g. Python, JS, …)

Slide 33

Slide 33 text

THE ANTLR4 FLOW grammar lexer parser AST listener

Slide 34

Slide 34 text

grammar Query; query: target results EOF; target: 'search' name filter?; results: ('return' name)+; name: [a-z]+; ANTLR4 GRAMMAR

Slide 35

Slide 35 text

# Generated by ANTLR 4.5 from antlr4 import * class QueryListener: def enterTarget(self, ctx): pass def exitTarget(self, ctx): pass ANTLR4 LISTENER

Slide 36

Slide 36 text

Let’s RECAP !

Slide 37

Slide 37 text

CUSTOM QUERY LANGUAGES WHY bother An executable model of your domain/logic. Decoupling display, model, and storage. HOW to design Language as interface. Expect & embrace change. HOW to implement Parser generators can speed up prototyping, at the cost of some control.

Slide 38

Slide 38 text

GRACIAS! I’m @AnjanaVakil anjana@uberresearch.com Huge thanks to Tobias Johansson & Valo! Presentation template by SlidesCarnival