Just Gimme the Data!

Just Gimme the Data!

A session by Jason E Marshall at Syntax Code & Craft Convention 2016

C3242e8c2ab4bd8922b125ae5589e443?s=128

Syntax Conference

May 06, 2016
Tweet

Transcript

  1. Jason E. Marshall The Baldwin Group – Office of Coastal

    Management "We're entering a new world in which data may be more important than software." - Tim O'Reilly
  2. The Data Access Problem  No standard structure across silos….

     No standard access methods…  Adaptor Pattern | ETL Pattern
  3. We’ve all done it…

  4. The data integration project… The data I want The way

    they give it The way I need it My adaptors
  5. Technical debt…

  6. What if…?

  7. OpenData Protocol  “An open protocol to allow the creation

    and consumption of queryable and interoperable RESTful APIs in a simple and standard way.”  Odata.org
  8. Open Protocol  Open Community  GitHub (main), StackExchange, Twitter,

    Slack, Google Discussions, etc…  Standardized  Organization for the Advancement of Structured Information Standards (OASIS)  Various stacks  Odata.net, Olingo (java),
  9. The gist… {HttpVerb} {theApi} / {theEntity} ? {doSomething} ..with whatever…

    …to what… Relate entities Select fields Filter & Search Format result Custom functions …etc… Do what… Get Post Put Delete Patch …etc… …options People Places Things Objects Stuff …etc… People API Places API Things API This API That API …etc…
  10. Do What – the verbs… HTTP Verb Entity operation 

    GET  POST/PUT  DELETE  PATCH  BATCH  Read  Create  Delete  Update  Do several of these at once
  11. To what… “the entities”

  12. Do Something… query options  $select  $expand  $search

     $filter  $orderby  $format  $count  $skip, $top  …there are more…  What attributes to get  Get related entities  Find things in entities  Filter entities  Sorting  Result type (ie: json)  How many entities  Paging & navigation  .. I don’t have time for…
  13. Obligatory demos…

  14. $select  CSV of properties to return  Use like

    SQL select clause  ... /People?$select=UserName,FirstName,LastName
  15. $expand  $select=UserName,Friends&$expand=Friends,Trips  $select=UserName,Friends&$expand=Friends($select= FirstName,LastName),Trips

  16. $search  $search - restricts the result to include only

    those entities matching the specified search expression.  /People?$search=Scott
  17. $filter  Comparison operators  Eq, ne, has, Gt, ge,

    lt, le  Logical operators  and, or, not  Lambda  any, all  Arithmetic operators  add, sub, mul, div, mod  Grouping (operations)  (price sub 5) gt 10
  18. Built-in functions  String functions  contains,  tolower/upper 

    substring  concat…etc…  Date functions  now  date  year, month, day, hour minute, etc..  Math functions  Round, floor, ceiling  Type functions  cast, isof…  Geographic functions  distance  length  intersections
  19. Your custom functions  Roll your own functions  Use

    geographic search to find airports  /GetNearestAirport(lat = 33, lon = -118)
  20. …and there’s more..

  21. OpenData Protocol  Odata.org  http://docs.oasis-open.org/odata/odata/v4.0/os/part1- protocol/odata-v4.0-os-part1-protocol.html

  22. Jason.Marshall@noaa.gov The Baldwin Group

  23. Jason.Marshall@noaa.gov The Baldwin Group

  24. Extra slides past here…

  25. Demo…but first… Postman  Chrome app for testing APIs 

    https://www.getpostman.com  Collections & environment settings exportable  Automate with newman  https://www.npmjs.com/package/newman
  26. Demo – Digging in  Getting properties  Filtering &

    Searching  Sorting  Client-side paging & counts  Expanding related entities  Selecting specific properties  Full Metadata  Casting types
  27. Demo - CRUD  Creating entities  Deleting entities 

    Updating entities
  28. Demo – Functions & Actions  Unbound functions & actions

     Bound functions & actions
  29. Batch Requests (no demo – yet)  Batch requests allow

    grouping multiple operations into a single HTTP request payload.  See: http://www.odata.org/getting-started/advanced- tutorial/#batch
  30. For example…  Let’s try to get some historical hurricane

    data….
  31. Scrape it?

  32. Download & manage it?

  33. We can look at the data…

  34. Sniffing the page for sources…  Some image tiles… I

    like puzzles..
  35. Sniffing the page for sources…  BUT WAIT – here’s

    another link!  https://coast.noaa.gov/arcgis/rest/services/Hurricanes/ AllStorms/MapServer/0/query?f=json&returnGeometry =true&outSR=4326&where=STORMID%20IN%20(SELE CT%20STORMID%20FROM%20WebMerc.get_StormId sForGeometry(default,default,default,-1,2000,%27- 8531595.349078232%204202202.0670058485%27,120380) )&outFields=STORMID,%20Display_StormName,Begin ObservationTime,BEGINLAT,BEGINLON,MinPressure,S affirSimpsonScale,DateAndTime,Display_DateAndTime ,Display_NormalizedMSW,Display_MinPressure,Segme ntOrder,ObjectId&callback=GetMeSomeHurricanes
  36. Finally some data….?

  37. OpenData Protocol  “An open protocol to allow the creation

    and consumption of queryable and interoperable RESTful APIs in a simple and standard way.” Umm… hurricanes…?
  38. WORK IN PROGRESS