Just Gimme the Data!

A session by Jason E Marshall at Syntax Code & Craft Convention 2016

Transcript

1. Jason E. Marshall The Baldwin Group – Office of Coastal Management "We're entering a new world in which data may be more important than software." - Tim O'Reilly
2. The Data Access Problem  No standard structure across silos….  No standard access methods…  Adaptor Pattern | ETL Pattern
3. We’ve all done it…
4. The data integration project… The data I want The way they give it The way I need it My adaptors
5. Technical debt…
6. What if…?
7. OpenData Protocol  “An open protocol to allow the creation and consumption of queryable and interoperable RESTful APIs in a simple and standard way.”  Odata.org
8. Open Protocol  Open Community  GitHub (main), StackExchange, Twitter, Slack, Google Discussions, etc…  Standardized  Organization for the Advancement of Structured Information Standards (OASIS)  Various stacks  Odata.net, Olingo (java),
9. The gist… {HttpVerb} {theApi} / {theEntity} ? {doSomething} ..with whatever… …to what… Relate entities Select fields Filter & Search Format result Custom functions …etc… Do what… Get Post Put Delete Patch …etc… …options People Places Things Objects Stuff …etc… People API Places API Things API This API That API …etc…
10. Do What – the verbs… HTTP Verb Entity operation  GET  POST/PUT  DELETE  PATCH  BATCH  Read  Create  Delete  Update  Do several of these at once
11. To what… “the entities”
12. Do Something… query options  $select  $expand  $search  $filter  $orderby  $format  $count  $skip, $top  …there are more…  What attributes to get  Get related entities  Find things in entities  Filter entities  Sorting  Result type (ie: json)  How many entities  Paging & navigation  .. I don’t have time for…
13. Obligatory demos…
14. $select  CSV of properties to return  Use like SQL select clause  ... /People?$select=UserName,FirstName,LastName
15. $expand  $select=UserName,Friends&$expand=Friends,Trips  $select=UserName,Friends&$expand=Friends($select= FirstName,LastName),Trips
16. $search  $search - restricts the result to include only those entities matching the specified search expression.  /People?$search=Scott
17. $filter  Comparison operators  Eq, ne, has, Gt, ge, lt, le  Logical operators  and, or, not  Lambda  any, all  Arithmetic operators  add, sub, mul, div, mod  Grouping (operations)  (price sub 5) gt 10
18. Built-in functions  String functions  contains,  tolower/upper  substring  concat…etc…  Date functions  now  date  year, month, day, hour minute, etc..  Math functions  Round, floor, ceiling  Type functions  cast, isof…  Geographic functions  distance  length  intersections
19. Your custom functions  Roll your own functions  Use geographic search to find airports  /GetNearestAirport(lat = 33, lon = -118)
20. …and there’s more..
21. OpenData Protocol  Odata.org  http://docs.oasis-open.org/odata/odata/v4.0/os/part1- protocol/odata-v4.0-os-part1-protocol.html
22. Jason.Marshall@noaa.gov The Baldwin Group
23. Jason.Marshall@noaa.gov The Baldwin Group
24. Extra slides past here…
25. Demo…but first… Postman  Chrome app for testing APIs  https://www.getpostman.com  Collections & environment settings exportable  Automate with newman  https://www.npmjs.com/package/newman
26. Demo – Digging in  Getting properties  Filtering & Searching  Sorting  Client-side paging & counts  Expanding related entities  Selecting specific properties  Full Metadata  Casting types
27. Demo - CRUD  Creating entities  Deleting entities  Updating entities
28. Demo – Functions & Actions  Unbound functions & actions  Bound functions & actions
29. Batch Requests (no demo – yet)  Batch requests allow grouping multiple operations into a single HTTP request payload.  See: http://www.odata.org/getting-started/advanced- tutorial/#batch
30. For example…  Let’s try to get some historical hurricane data….
31. Scrape it?
32. Download & manage it?
33. We can look at the data…
34. Sniffing the page for sources…  Some image tiles… I like puzzles..
35. Sniffing the page for sources…  BUT WAIT – here’s another link!  https://coast.noaa.gov/arcgis/rest/services/Hurricanes/ AllStorms/MapServer/0/query?f=json&returnGeometry =true&outSR=4326&where=STORMID%20IN%20(SELE CT%20STORMID%20FROM%20WebMerc.get_StormId sForGeometry(default,default,default,-1,2000,%27- 8531595.349078232%204202202.0670058485%27,120380) )&outFields=STORMID,%20Display_StormName,Begin ObservationTime,BEGINLAT,BEGINLON,MinPressure,S affirSimpsonScale,DateAndTime,Display_DateAndTime ,Display_NormalizedMSW,Display_MinPressure,Segme ntOrder,ObjectId&callback=GetMeSomeHurricanes
36. Finally some data….?
37. OpenData Protocol  “An open protocol to allow the creation and consumption of queryable and interoperable RESTful APIs in a simple and standard way.” Umm… hurricanes…?
38. WORK IN PROGRESS