Serverless data
Pieter Colpaert
#osloinspireert
2020-02-05
Slide 2
Slide 2 text
Serverless data
Removing the need to set up a server when
reusing a dataset
Slide 3
Slide 3 text
What is the ultimate API for a base registry*?
* Important source of authoritative governmental identifiers
E.g., the officially registered addresses, or the IDs for companies and organizations
Slide 4
Slide 4 text
Dataset
Very specific API*
Publisher
3d party
Awesome
application
* I’m looking at you:
WMS, GraphQL, WFS, SPARQL,
Cypher, ES, HTSQL, DJP, …
Slide 5
Slide 5 text
Dataset
Very specific API
3d parties
Awesome
application
Very specific API
Awesome
application
...
...
Publisher
Slide 6
Slide 6 text
Can you prepare APIs for all datasets
for any use case?
Slide 7
Slide 7 text
Base registry of addresses in Flanders (best-in-class example)
:
Doesn’t even have an API to have a simple typeahead
Slide 8
Slide 8 text
Data dump
Very specific API
3d parties
Awesome
application
Very specific API
Awesome
application
...
...
We had to copy a dump and
create our own API
Publisher
Slide 9
Slide 9 text
Data dump
Very specific API
3d parties
Awesome
application
Very specific API
Awesome
application
...
...
Welcome to the replication hell Publisher
Slide 10
Slide 10 text
Solution?
We must look at doing the right effort in the right place.
Slide 11
Slide 11 text
3d parties
Serverless
application
Generic data fragments API
Serverless
application
...
Dataset
Query SDK Query SDK Query SDK
Publisher
Slide 12
Slide 12 text
The Flemish address registry: how to
Dump
Start populating first
document until full...
Raw data dump: https://data.vlaanderen.be/dumps
Slide 13
Slide 13 text
DocumentRoot
First 25 elements
…
DocA
25 elements that start with
an A
DocC
Doc…
DocAbdij
DocAlbert
DocAppe
DocB
Documenting a search tree with links
Slide 14
Slide 14 text
Autocompleting Albert Street?
Multiple HTTP requests needed
DocumentRoot
First 25 elements
…
DocA
25 elements that start with
an A
DocC
Doc…
DocAbdij
DocAlbert
DocAppe
DocB
Slide 15
Slide 15 text
Demo
Slide 16
Slide 16 text
Becomes possible with an incredibly easy to host API
Generic API spec: github.com/pietercolpaert/TREE
Same idea can be applied on
- Geospatial search trees
- Looking up data in a time interval
- Graph patterns
- …
Full text search
Slide 17
Slide 17 text
3d parties
Serverless
application
Generieke data-fragmenten API
Serverless
application
Legacy API
Data publisher
dataset
Query SDK Query SDK
Legacy
application
But… Didn’t you just create
yet another API instead of
the ultimate one?
Publisher
Slide 18
Slide 18 text
3d parties
Serverless
application
Generic data fragments API
Serverless
application
Legacy API
Data publisher
dataset
Query SDK Query SDK
Query SDK
Legacy
application
Docker
container
In time, CPU consumption of legacy
APIs can be moved to 3d parties
Publisher
Slide 19
Slide 19 text
Combine the right buildings blocks,
and build the ultimate – and only – API
for your base registry.
https://pietercolpaert.be