Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Bulk + Open Data APIs
Search
Chris Herwig
April 04, 2013
Technology
2
270
Bulk + Open Data APIs
Chris Herwig
April 04, 2013
Tweet
Share
More Decks by Chris Herwig
See All by Chris Herwig
Clear Skies: Turning Massive NASA Data into a Pixel-Perfect World Atlas
hrwgc
0
520
Open + Accessible
hrwgc
2
120
Open Satellite Imagery and Geoportals | MapBox Satellite
hrwgc
1
220
Mapping Mars Open Source
hrwgc
1
83
Other Decks in Technology
See All in Technology
障害を回避するHttpClient再入門 / Avoiding Failures HttpClient Reintroduction
uskey512
1
400
Scale Security Programs with Scorecarding
ramimac
0
480
為什麼我們需要 Observability?
marcustung
0
460
医療業界に特化した音声認識モデル構築のためのアノテーションの実態
thickstem
0
240
組織とセキュリティ文化と、自分の一歩
maimyyym
3
1.4k
Contract One Dev Group 紹介資料
sansan33
PRO
0
6.2k
ソフトウェアは捨てやすく作ろう/Let's make software easy to discard
sanogemaru
10
6.2k
Zero Data Loss Autonomous Recovery Service サービス概要
oracle4engineer
PRO
2
7.3k
Introduction to Bill One Development Engineer
sansan33
PRO
0
240
Flutterアプリを⾃然⾔語で操作する
yukisakai1225
0
200
カンファレンスのつくりかた / The Conference Code: What Makes It All Work
tomzoh
8
1k
Applied NLP in the Age of Generative AI: Future-Proof Strategies for Banking and Finance
inesmontani
PRO
0
180
Featured
See All Featured
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
137
34k
Making the Leap to Tech Lead
cromwellryan
134
9.3k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
6
630
Automating Front-end Workflow
addyosmani
1370
200k
Bash Introduction
62gerente
614
210k
StorybookのUI Testing Handbookを読んだ
zakiyama
30
5.8k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
357
30k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
35
2.3k
Done Done
chrislema
184
16k
Transcript
Bulk Chris Herwig @hrwgc +open
Chris Herwig Satellite Team lead, MapBox
MapBox Satellite Phase 1, Launched 12/2012 • Global imagery base
layer for MapBox users • Global satellite imagery, zoom 0-12 • Continental U.S. aerial imagery zoom 13-17 • Licensed to allow for OSM tracing
MapBox Satellite phase 1 was sourced entirely from public domain,
open data.
Kuala Lumpur, Malaysia
Los Angeles, CA -
Brawley, CA
Cloudless Atlas • Cloudfree global mosaic, zoom 0-8 • NASA
MODIS Aqua and Terra Satellites • 380,000 source satellite images
Open data is good.
“data is open if anyone is free to use, reuse,
and redistribute it ...”
“subject to the requirement to attribute and/or share-alike” -Open Knowledge
Definition
ACCESS
- open license - open format - available for download
ACCESS
assumptions 3+1
There are different types of open data users.
Different users have different needs and abilities.
Data accessibility matters.
Open data is not truly open if it is inaccessible.
USERS 3
CASUAL
casual •least technical •dataset discovery •basic needs: ability to query
and download
•geoportal •simple html table •solid metadata •intuitive interface casual
casual USGS EarthExplorer http://earthexplorer.usgs.gov
casual Massachusetts GIS http://gis.amherstma.gov/mgis/
casual The National Map http://nationalmap.gov
casual Utah AGRC Raster Data Discovery http://gis.utah.gov
casual New Hampshire Statewide GIS Clearinghouse http://www.granit.unh.edu/data/downloadfreedata/category/databycategory.html
PROGRAM MATIC
•Tech skills/API familiarity •spatial query •download sub-dataset based on parent
process programmatic
programmatic • API • developer documentation • solid metadata •
interface optional
USGS Application Services http://cumulus.cr.usgs.gov/app_services.php programmatic
USGS Application Services http://cumulus.cr.usgs.gov/app_services.php programmatic
BULK
bulk • Need entire datasets, not spatial intersections • Data
APIs/manual retrieval workflows do not scale • Sometimes retrieve data via physical drives
bulk • interface optional • FTP-like access • reasonable bandwidth
for download retrieval
New Hampshire Statewide GIS Clearinghouse http://www.granit.unh.edu/ Bulk
API
TYPES 3
CONTENT
ConTeNt Database REST Content
Content • Makes application content available for developers to integrate
into existing/new applications
Content
DATA
Database REST Matching Rows Data
DATA • Allows users to query large datasets without having
to have full dataset locally • Applications can be built on top of Live/real-time datasets
Data http://api.occupy-data.org/v1/? results&value=crossst&value=age&value=race&value=crimsusp&value=sex&value=build&value=frisked&results_p er_page=100
BULK
Bulk Database REST References
bulk • Key difference is user obtains reference to object
requested, rather than object itself. • Download object(s) later • Can be relatively lightweight
SO?
Data API = Best Open Data MetHOD?
NO.
APIs, like geoportals, are not always the best option for
disseminating open data.
Different USers
Different NEEds
Different Abilities
Different Access Endpoints
STUFF breaks
Permalinks != Permanent
WayBackMachine http://archive.org/web/web.php
So?
Open data users change as tech changes.
Access should be a policy and tech consideration.
NEXT STEPS
Strive to be SAD
SCALABLE Accessible Durable
- Open systems for access to open data - Can
grow in response to changes in technology/user requirements SCALABLE
- Data access and retrieval is as quick and painless
as possible - Options for users with different abilities, different desired results Accessible
- APIs, geoportals don’t always work - Low-maintenance, durable options
- FTP-like directory access - Good documentation DURABLE
San Francisco, CA
chris@mapbox.com @hrwgc