Bulk + Open Data APIs

Transcript

1. Bulk Chris Herwig @hrwgc +open
2. Chris Herwig Satellite Team lead, MapBox
3. MapBox Satellite Phase 1, Launched 12/2012 • Global imagery base layer for MapBox users • Global satellite imagery, zoom 0-12 • Continental U.S. aerial imagery zoom 13-17 • Licensed to allow for OSM tracing
4. MapBox Satellite phase 1 was sourced entirely from public domain, open data.
5. Kuala Lumpur, Malaysia
6. Los Angeles, CA -
7. Brawley, CA
8. Cloudless Atlas • Cloudfree global mosaic, zoom 0-8 • NASA MODIS Aqua and Terra Satellites • 380,000 source satellite images
9. Open data is good.
10. “data is open if anyone is free to use, reuse, and redistribute it ...”
11. “subject to the requirement to attribute and/or share-alike” -Open Knowledge Definition
12. ACCESS
13. - open license - open format - available for download ACCESS
14. assumptions 3+1
15. There are different types of open data users.
16. Different users have different needs and abilities.
17. Data accessibility matters.
18. Open data is not truly open if it is inaccessible.
19. USERS 3
20. CASUAL
21. casual •least technical •dataset discovery •basic needs: ability to query and download
22. •geoportal •simple html table •solid metadata •intuitive interface casual
23. casual USGS EarthExplorer http://earthexplorer.usgs.gov
24. casual Massachusetts GIS http://gis.amherstma.gov/mgis/
25. casual The National Map http://nationalmap.gov
26. casual Utah AGRC Raster Data Discovery http://gis.utah.gov
27. casual New Hampshire Statewide GIS Clearinghouse http://www.granit.unh.edu/data/downloadfreedata/category/databycategory.html
28. PROGRAM MATIC
29. •Tech skills/API familiarity •spatial query •download sub-dataset based on parent process programmatic
30. programmatic • API • developer documentation • solid metadata • interface optional
31. USGS Application Services http://cumulus.cr.usgs.gov/app_services.php programmatic
32. USGS Application Services http://cumulus.cr.usgs.gov/app_services.php programmatic
33. BULK
34. bulk • Need entire datasets, not spatial intersections • Data APIs/manual retrieval workflows do not scale • Sometimes retrieve data via physical drives
35. bulk • interface optional • FTP-like access • reasonable bandwidth for download retrieval
36. New Hampshire Statewide GIS Clearinghouse http://www.granit.unh.edu/ Bulk
37. API
38. TYPES 3
39. CONTENT
40. ConTeNt Database REST Content
41. Content • Makes application content available for developers to integrate into existing/new applications
42. Content
43. DATA
44. Database REST Matching Rows Data
45. DATA • Allows users to query large datasets without having to have full dataset locally • Applications can be built on top of Live/real-time datasets
46. Data http://api.occupy-data.org/v1/? results&value=crossst&value=age&value=race&value=crimsusp&value=sex&value=build&value=frisked&results_p er_page=100
47. BULK
48. Bulk Database REST References
49. bulk • Key difference is user obtains reference to object requested, rather than object itself. • Download object(s) later • Can be relatively lightweight
50. SO?
51. Data API = Best Open Data MetHOD?
52. NO.
53. APIs, like geoportals, are not always the best option for disseminating open data.
54. Different USers
55. Different NEEds
56. Different Abilities
57. Different Access Endpoints
58. STUFF breaks
59. Permalinks != Permanent
60. WayBackMachine http://archive.org/web/web.php
61. So?
62. Open data users change as tech changes.
63. Access should be a policy and tech consideration.
64. NEXT STEPS
65. Strive to be SAD
66. SCALABLE Accessible Durable
67. - Open systems for access to open data - Can grow in response to changes in technology/user requirements SCALABLE
68. - Data access and retrieval is as quick and painless as possible - Options for users with different abilities, different desired results Accessible
69. - APIs, geoportals don’t always work - Low-maintenance, durable options - FTP-like directory access - Good documentation DURABLE
70. San Francisco, CA
71. chris@mapbox.com @hrwgc