Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Bulk + Open Data APIs
Search
Chris Herwig
April 04, 2013
Technology
2
270
Bulk + Open Data APIs
Chris Herwig
April 04, 2013
Tweet
Share
More Decks by Chris Herwig
See All by Chris Herwig
Clear Skies: Turning Massive NASA Data into a Pixel-Perfect World Atlas
hrwgc
0
520
Open + Accessible
hrwgc
2
120
Open Satellite Imagery and Geoportals | MapBox Satellite
hrwgc
1
220
Mapping Mars Open Source
hrwgc
1
83
Other Decks in Technology
See All in Technology
米国国防総省のDevSecOpsライフサイクルをAWSのセキュリティサービスとOSSで実現
syoshie
2
1.2k
GitHub Copilot の概要
tomokusaba
1
140
作曲家がボカロを使うようにPdMはAIを使え
itotaxi
0
130
【TiDB GAME DAY 2025】Shadowverse: Worlds Beyond にみる TiDB 活用術
cygames
0
1.1k
AWS Summit Japan 2025 Community Stage - App workflow automation by AWS Step Functions
matsuihidetoshi
1
290
より良いプロダクトの開発を目指して - 情報を中心としたプロダクト開発 #phpcon #phpcon2025
bengo4com
1
3.1k
あなたの声を届けよう! 女性エンジニア登壇の意義とアウトプット実践ガイド #wttjp / Call for Your Voice
kondoyuko
4
470
Understanding_Thread_Tuning_for_Inference_Servers_of_Deep_Models.pdf
lycorptech_jp
PRO
0
140
Witchcraft for Memory
pocke
1
470
CI/CD/IaC 久々に0から環境を作ったらこうなりました
kaz29
1
190
なぜ私はいま、ここにいるのか? #もがく中堅デザイナー #プロダクトデザイナー
bengo4com
0
480
「Chatwork」の認証基盤の移行とログ活用によるプロダクト改善
kubell_hr
1
200
Featured
See All Featured
How to Think Like a Performance Engineer
csswizardry
24
1.7k
Embracing the Ebb and Flow
colly
86
4.7k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
48
5.4k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
Optimizing for Happiness
mojombo
379
70k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
31
1.2k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Fireside Chat
paigeccino
37
3.5k
Art, The Web, and Tiny UX
lynnandtonic
299
21k
A Modern Web Designer's Workflow
chriscoyier
694
190k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
29
9.5k
Transcript
Bulk Chris Herwig @hrwgc +open
Chris Herwig Satellite Team lead, MapBox
MapBox Satellite Phase 1, Launched 12/2012 • Global imagery base
layer for MapBox users • Global satellite imagery, zoom 0-12 • Continental U.S. aerial imagery zoom 13-17 • Licensed to allow for OSM tracing
MapBox Satellite phase 1 was sourced entirely from public domain,
open data.
Kuala Lumpur, Malaysia
Los Angeles, CA -
Brawley, CA
Cloudless Atlas • Cloudfree global mosaic, zoom 0-8 • NASA
MODIS Aqua and Terra Satellites • 380,000 source satellite images
Open data is good.
“data is open if anyone is free to use, reuse,
and redistribute it ...”
“subject to the requirement to attribute and/or share-alike” -Open Knowledge
Definition
ACCESS
- open license - open format - available for download
ACCESS
assumptions 3+1
There are different types of open data users.
Different users have different needs and abilities.
Data accessibility matters.
Open data is not truly open if it is inaccessible.
USERS 3
CASUAL
casual •least technical •dataset discovery •basic needs: ability to query
and download
•geoportal •simple html table •solid metadata •intuitive interface casual
casual USGS EarthExplorer http://earthexplorer.usgs.gov
casual Massachusetts GIS http://gis.amherstma.gov/mgis/
casual The National Map http://nationalmap.gov
casual Utah AGRC Raster Data Discovery http://gis.utah.gov
casual New Hampshire Statewide GIS Clearinghouse http://www.granit.unh.edu/data/downloadfreedata/category/databycategory.html
PROGRAM MATIC
•Tech skills/API familiarity •spatial query •download sub-dataset based on parent
process programmatic
programmatic • API • developer documentation • solid metadata •
interface optional
USGS Application Services http://cumulus.cr.usgs.gov/app_services.php programmatic
USGS Application Services http://cumulus.cr.usgs.gov/app_services.php programmatic
BULK
bulk • Need entire datasets, not spatial intersections • Data
APIs/manual retrieval workflows do not scale • Sometimes retrieve data via physical drives
bulk • interface optional • FTP-like access • reasonable bandwidth
for download retrieval
New Hampshire Statewide GIS Clearinghouse http://www.granit.unh.edu/ Bulk
API
TYPES 3
CONTENT
ConTeNt Database REST Content
Content • Makes application content available for developers to integrate
into existing/new applications
Content
DATA
Database REST Matching Rows Data
DATA • Allows users to query large datasets without having
to have full dataset locally • Applications can be built on top of Live/real-time datasets
Data http://api.occupy-data.org/v1/? results&value=crossst&value=age&value=race&value=crimsusp&value=sex&value=build&value=frisked&results_p er_page=100
BULK
Bulk Database REST References
bulk • Key difference is user obtains reference to object
requested, rather than object itself. • Download object(s) later • Can be relatively lightweight
SO?
Data API = Best Open Data MetHOD?
NO.
APIs, like geoportals, are not always the best option for
disseminating open data.
Different USers
Different NEEds
Different Abilities
Different Access Endpoints
STUFF breaks
Permalinks != Permanent
WayBackMachine http://archive.org/web/web.php
So?
Open data users change as tech changes.
Access should be a policy and tech consideration.
NEXT STEPS
Strive to be SAD
SCALABLE Accessible Durable
- Open systems for access to open data - Can
grow in response to changes in technology/user requirements SCALABLE
- Data access and retrieval is as quick and painless
as possible - Options for users with different abilities, different desired results Accessible
- APIs, geoportals don’t always work - Low-maintenance, durable options
- FTP-like directory access - Good documentation DURABLE
San Francisco, CA
[email protected]
@hrwgc