Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Bulk + Open Data APIs
Search
Chris Herwig
April 04, 2013
Technology
2
270
Bulk + Open Data APIs
Chris Herwig
April 04, 2013
Tweet
Share
More Decks by Chris Herwig
See All by Chris Herwig
Clear Skies: Turning Massive NASA Data into a Pixel-Perfect World Atlas
hrwgc
0
490
Open + Accessible
hrwgc
2
110
Open Satellite Imagery and Geoportals | MapBox Satellite
hrwgc
1
210
Mapping Mars Open Source
hrwgc
1
77
Other Decks in Technology
See All in Technology
『AWS Distinguished Engineerに学ぶ リトライの技術』 #ARC403/Marc Brooker on Try again: The tools and techniques behind resilient systems
quiver
0
140
人はなぜISUCONに夢中になるのか
kakehashi
PRO
6
1.5k
AndroidXR 開発ツールごとの できることできないこと
donabe3
0
120
Culture Deck
optfit
0
390
データ資産をシームレスに伝達するためのイベント駆動型アーキテクチャ
kakehashi
PRO
2
470
Oracle Base Database Service 技術詳細
oracle4engineer
PRO
6
57k
スタートアップ1人目QAエンジニアが QAチームを立ち上げ、“個”からチーム、 そして“組織”に成長するまで / How to set up QA team at reiwatravel
mii3king
2
1.3k
7日間でハッキングをはじめる本をはじめてみませんか?_ITエンジニア本大賞2025
nomizone
2
1.7k
【Developers Summit 2025】プロダクトエンジニアから学ぶ、 ユーザーにより高い価値を届ける技術
niwatakeru
2
1.2k
TAMとre:Capセキュリティ編 〜拡張脅威検出デモを添えて〜
fujiihda
1
180
株式会社EventHub・エンジニア採用資料
eventhub
0
4.2k
CZII - CryoET Object Identification 参加振り返り・解法共有
tattaka
0
310
Featured
See All Featured
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
226
22k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Side Projects
sachag
452
42k
Bootstrapping a Software Product
garrettdimon
PRO
306
110k
The Straight Up "How To Draw Better" Workshop
denniskardys
232
140k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
330
21k
[RailsConf 2023] Rails as a piece of cake
palkan
53
5.2k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
6
550
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.4k
Code Review Best Practice
trishagee
66
17k
Optimising Largest Contentful Paint
csswizardry
34
3.1k
Building a Scalable Design System with Sketch
lauravandoore
460
33k
Transcript
Bulk Chris Herwig @hrwgc +open
Chris Herwig Satellite Team lead, MapBox
MapBox Satellite Phase 1, Launched 12/2012 • Global imagery base
layer for MapBox users • Global satellite imagery, zoom 0-12 • Continental U.S. aerial imagery zoom 13-17 • Licensed to allow for OSM tracing
MapBox Satellite phase 1 was sourced entirely from public domain,
open data.
Kuala Lumpur, Malaysia
Los Angeles, CA -
Brawley, CA
Cloudless Atlas • Cloudfree global mosaic, zoom 0-8 • NASA
MODIS Aqua and Terra Satellites • 380,000 source satellite images
Open data is good.
“data is open if anyone is free to use, reuse,
and redistribute it ...”
“subject to the requirement to attribute and/or share-alike” -Open Knowledge
Definition
ACCESS
- open license - open format - available for download
ACCESS
assumptions 3+1
There are different types of open data users.
Different users have different needs and abilities.
Data accessibility matters.
Open data is not truly open if it is inaccessible.
USERS 3
CASUAL
casual •least technical •dataset discovery •basic needs: ability to query
and download
•geoportal •simple html table •solid metadata •intuitive interface casual
casual USGS EarthExplorer http://earthexplorer.usgs.gov
casual Massachusetts GIS http://gis.amherstma.gov/mgis/
casual The National Map http://nationalmap.gov
casual Utah AGRC Raster Data Discovery http://gis.utah.gov
casual New Hampshire Statewide GIS Clearinghouse http://www.granit.unh.edu/data/downloadfreedata/category/databycategory.html
PROGRAM MATIC
•Tech skills/API familiarity •spatial query •download sub-dataset based on parent
process programmatic
programmatic • API • developer documentation • solid metadata •
interface optional
USGS Application Services http://cumulus.cr.usgs.gov/app_services.php programmatic
USGS Application Services http://cumulus.cr.usgs.gov/app_services.php programmatic
BULK
bulk • Need entire datasets, not spatial intersections • Data
APIs/manual retrieval workflows do not scale • Sometimes retrieve data via physical drives
bulk • interface optional • FTP-like access • reasonable bandwidth
for download retrieval
New Hampshire Statewide GIS Clearinghouse http://www.granit.unh.edu/ Bulk
API
TYPES 3
CONTENT
ConTeNt Database REST Content
Content • Makes application content available for developers to integrate
into existing/new applications
Content
DATA
Database REST Matching Rows Data
DATA • Allows users to query large datasets without having
to have full dataset locally • Applications can be built on top of Live/real-time datasets
Data http://api.occupy-data.org/v1/? results&value=crossst&value=age&value=race&value=crimsusp&value=sex&value=build&value=frisked&results_p er_page=100
BULK
Bulk Database REST References
bulk • Key difference is user obtains reference to object
requested, rather than object itself. • Download object(s) later • Can be relatively lightweight
SO?
Data API = Best Open Data MetHOD?
NO.
APIs, like geoportals, are not always the best option for
disseminating open data.
Different USers
Different NEEds
Different Abilities
Different Access Endpoints
STUFF breaks
Permalinks != Permanent
WayBackMachine http://archive.org/web/web.php
So?
Open data users change as tech changes.
Access should be a policy and tech consideration.
NEXT STEPS
Strive to be SAD
SCALABLE Accessible Durable
- Open systems for access to open data - Can
grow in response to changes in technology/user requirements SCALABLE
- Data access and retrieval is as quick and painless
as possible - Options for users with different abilities, different desired results Accessible
- APIs, geoportals don’t always work - Low-maintenance, durable options
- FTP-like directory access - Good documentation DURABLE
San Francisco, CA
[email protected]
@hrwgc