Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Bulk + Open Data APIs
Search
Chris Herwig
April 04, 2013
Technology
2
270
Bulk + Open Data APIs
Chris Herwig
April 04, 2013
Tweet
Share
More Decks by Chris Herwig
See All by Chris Herwig
Clear Skies: Turning Massive NASA Data into a Pixel-Perfect World Atlas
hrwgc
0
520
Open + Accessible
hrwgc
2
120
Open Satellite Imagery and Geoportals | MapBox Satellite
hrwgc
1
220
Mapping Mars Open Source
hrwgc
1
83
Other Decks in Technology
See All in Technology
スプリントレビューを効果的にするために
miholovesq
9
1.7k
KCD Lima: eBee in Peru!
lizrice
0
110
Vision Language Modelと自動運転AIの最前線_20250730
yuyamaguchi
2
650
Webの技術とガジェットで那須の子ども達にワクワクを! / IoTLT_20250720
you
PRO
0
130
PdM業務における使い分け
shinshiro
0
670
メモ整理が苦手な者による頑張らないObsidian活用術
optim
0
150
20250728 MCP, A2A and Multi-Agents in the future
yoshidashingo
1
100
alecthomas/kong はいいぞ
fujiwara3
6
1.1k
大規模イベントを支える ABEMA の アーキテクチャ 変遷 2025
nagapad
5
510
AWS表彰プログラムとキャリアについて
naoki_0531
1
140
私とAWSとの関わりの歩み~意志あるところに道は開けるかも?~
nagisa53
1
130
経理出身PdMがAIプロダクト開発を_ハンズオンで学んだ話.pdf
shunsukenarita
1
230
Featured
See All Featured
Measuring & Analyzing Core Web Vitals
bluesmoon
7
530
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
21
1.4k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
Why You Should Never Use an ORM
jnunemaker
PRO
58
9.5k
Building a Scalable Design System with Sketch
lauravandoore
462
33k
Building Flexible Design Systems
yeseniaperezcruz
328
39k
Unsuck your backbone
ammeep
671
58k
Reflections from 52 weeks, 52 projects
jeffersonlam
351
21k
How to Ace a Technical Interview
jacobian
278
23k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
17k
Optimising Largest Contentful Paint
csswizardry
37
3.4k
The Language of Interfaces
destraynor
158
25k
Transcript
Bulk Chris Herwig @hrwgc +open
Chris Herwig Satellite Team lead, MapBox
MapBox Satellite Phase 1, Launched 12/2012 • Global imagery base
layer for MapBox users • Global satellite imagery, zoom 0-12 • Continental U.S. aerial imagery zoom 13-17 • Licensed to allow for OSM tracing
MapBox Satellite phase 1 was sourced entirely from public domain,
open data.
Kuala Lumpur, Malaysia
Los Angeles, CA -
Brawley, CA
Cloudless Atlas • Cloudfree global mosaic, zoom 0-8 • NASA
MODIS Aqua and Terra Satellites • 380,000 source satellite images
Open data is good.
“data is open if anyone is free to use, reuse,
and redistribute it ...”
“subject to the requirement to attribute and/or share-alike” -Open Knowledge
Definition
ACCESS
- open license - open format - available for download
ACCESS
assumptions 3+1
There are different types of open data users.
Different users have different needs and abilities.
Data accessibility matters.
Open data is not truly open if it is inaccessible.
USERS 3
CASUAL
casual •least technical •dataset discovery •basic needs: ability to query
and download
•geoportal •simple html table •solid metadata •intuitive interface casual
casual USGS EarthExplorer http://earthexplorer.usgs.gov
casual Massachusetts GIS http://gis.amherstma.gov/mgis/
casual The National Map http://nationalmap.gov
casual Utah AGRC Raster Data Discovery http://gis.utah.gov
casual New Hampshire Statewide GIS Clearinghouse http://www.granit.unh.edu/data/downloadfreedata/category/databycategory.html
PROGRAM MATIC
•Tech skills/API familiarity •spatial query •download sub-dataset based on parent
process programmatic
programmatic • API • developer documentation • solid metadata •
interface optional
USGS Application Services http://cumulus.cr.usgs.gov/app_services.php programmatic
USGS Application Services http://cumulus.cr.usgs.gov/app_services.php programmatic
BULK
bulk • Need entire datasets, not spatial intersections • Data
APIs/manual retrieval workflows do not scale • Sometimes retrieve data via physical drives
bulk • interface optional • FTP-like access • reasonable bandwidth
for download retrieval
New Hampshire Statewide GIS Clearinghouse http://www.granit.unh.edu/ Bulk
API
TYPES 3
CONTENT
ConTeNt Database REST Content
Content • Makes application content available for developers to integrate
into existing/new applications
Content
DATA
Database REST Matching Rows Data
DATA • Allows users to query large datasets without having
to have full dataset locally • Applications can be built on top of Live/real-time datasets
Data http://api.occupy-data.org/v1/? results&value=crossst&value=age&value=race&value=crimsusp&value=sex&value=build&value=frisked&results_p er_page=100
BULK
Bulk Database REST References
bulk • Key difference is user obtains reference to object
requested, rather than object itself. • Download object(s) later • Can be relatively lightweight
SO?
Data API = Best Open Data MetHOD?
NO.
APIs, like geoportals, are not always the best option for
disseminating open data.
Different USers
Different NEEds
Different Abilities
Different Access Endpoints
STUFF breaks
Permalinks != Permanent
WayBackMachine http://archive.org/web/web.php
So?
Open data users change as tech changes.
Access should be a policy and tech consideration.
NEXT STEPS
Strive to be SAD
SCALABLE Accessible Durable
- Open systems for access to open data - Can
grow in response to changes in technology/user requirements SCALABLE
- Data access and retrieval is as quick and painless
as possible - Options for users with different abilities, different desired results Accessible
- APIs, geoportals don’t always work - Low-maintenance, durable options
- FTP-like directory access - Good documentation DURABLE
San Francisco, CA
[email protected]
@hrwgc