Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Serverless Data

Serverless Data

We introduce Serverless Data! Datasets that can be used without a developer setting up a dataset. However, the developer would also like any kind of flexibility to provide any kind of feature. Do data publishers then have to set up any kind of possible querying API? I don’t believe so: hypermedia data fragments to the rescue!

Download the full data dump of addresses in Flanders from: https://swarmstoraccountprod.blob.core.windows.net/oslo-ttl-dumps-production/adresregister.tar

Check out the autocompletion with fragments demo at http://193.190.127.164/site/index.html (if this link is broken, inform me: we’re going to give it a new home)

The specification: https://github.com/pietercolpaert/TREE

Full paper with benchmarks: http://pieter.pm/icwe2020-autocompletion/

Pieter Colpaert

February 05, 2020
Tweet

More Decks by Pieter Colpaert

Other Decks in Technology

Transcript

  1. What is the ultimate API for a base registry*? *

    Important source of authoritative governmental identifiers E.g., the officially registered addresses, or the IDs for companies and organizations
  2. Dataset Very specific API* Publisher 3d party Awesome application *

    I’m looking at you: WMS, GraphQL, WFS, SPARQL, Cypher, ES, HTSQL, DJP, …
  3. Data dump Very specific API 3d parties Awesome application Very

    specific API Awesome application ... ... We had to copy a dump and create our own API Publisher
  4. Data dump Very specific API 3d parties Awesome application Very

    specific API Awesome application ... ... Welcome to the replication hell Publisher
  5. The Flemish address registry: how to Dump Start populating first

    document until full... Raw data dump: https://data.vlaanderen.be/dumps
  6. DocumentRoot First 25 elements … DocA 25 elements that start

    with an A DocC Doc… DocAbdij DocAlbert DocAppe DocB Documenting a search tree with links
  7. Autocompleting Albert Street? Multiple HTTP requests needed DocumentRoot First 25

    elements … DocA 25 elements that start with an A DocC Doc… DocAbdij DocAlbert DocAppe DocB
  8. Becomes possible with an incredibly easy to host API Generic

    API spec: github.com/pietercolpaert/TREE Same idea can be applied on - Geospatial search trees - Looking up data in a time interval - Graph patterns - … Full text search
  9. 3d parties Serverless application Generieke data-fragmenten API Serverless application Legacy

    API Data publisher dataset Query SDK Query SDK Legacy application But… Didn’t you just create yet another API instead of the ultimate one? Publisher
  10. 3d parties Serverless application Generic data fragments API Serverless application

    Legacy API Data publisher dataset Query SDK Query SDK Query SDK Legacy application Docker container In time, CPU consumption of legacy APIs can be moved to 3d parties Publisher
  11. Combine the right buildings blocks, and build the ultimate –

    and only – API for your base registry. https://pietercolpaert.be