Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Bulk + Open Data APIs
Search
Chris Herwig
April 04, 2013
Technology
2
270
Bulk + Open Data APIs
Chris Herwig
April 04, 2013
Tweet
Share
More Decks by Chris Herwig
See All by Chris Herwig
Clear Skies: Turning Massive NASA Data into a Pixel-Perfect World Atlas
hrwgc
0
490
Open + Accessible
hrwgc
2
110
Open Satellite Imagery and Geoportals | MapBox Satellite
hrwgc
1
210
Mapping Mars Open Source
hrwgc
1
77
Other Decks in Technology
See All in Technology
第13回 Data-Centric AI勉強会, 画像認識におけるData-centric AI
ksaito_osx
0
360
Ask! NIKKEI RAG検索技術の深層
hotchpotch
13
2.8k
Fintech SREの挑戦 PCI DSS対応をスマートにこなすインフラ戦略/Fintech SRE’s Challenge: Smart Infrastructure Strategies for PCI DSS Compliance
maaaato
0
450
Bounded Context: Problem or Solution?
ewolff
1
210
急成長する企業で作った、エンジニアが輝ける制度/ 20250214 Rinto Ikenoue
shift_evolve
2
880
地方拠点で エンジニアリングマネージャーってできるの? 〜地方という制約を楽しむオーナーシップとコミュニティ作り〜
1coin
1
130
エンジニアのためのドキュメント力基礎講座〜構造化思考から始めよう〜(2025/02/15jbug広島#15発表資料)
yasuoyasuo
15
5.5k
エンジニアの育成を支える爆速フィードバック文化
sansantech
PRO
3
660
「海外登壇」という 選択肢を与えるために 〜Gophers EX
logica0419
0
500
Nekko Cloud、 これまでとこれから ~学生サークルが作る、 小さなクラウド
logica0419
2
730
ビジネスと現場活動をつなぐソフトウェアエンジニアリング~とあるスタートアッププロダクトの成長記録より~
mizunori
0
210
SCSAから学ぶセキュリティ管理
masakamayama
0
140
Featured
See All Featured
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.7k
Art, The Web, and Tiny UX
lynnandtonic
298
20k
Being A Developer After 40
akosma
89
590k
Rebuilding a faster, lazier Slack
samanthasiow
79
8.8k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
49
2.3k
Stop Working from a Prison Cell
hatefulcrawdad
267
20k
Designing for Performance
lara
604
68k
Java REST API Framework Comparison - PWX 2021
mraible
28
8.4k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
8
270
Faster Mobile Websites
deanohume
306
31k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
33
2.1k
Why Our Code Smells
bkeepers
PRO
335
57k
Transcript
Bulk Chris Herwig @hrwgc +open
Chris Herwig Satellite Team lead, MapBox
MapBox Satellite Phase 1, Launched 12/2012 • Global imagery base
layer for MapBox users • Global satellite imagery, zoom 0-12 • Continental U.S. aerial imagery zoom 13-17 • Licensed to allow for OSM tracing
MapBox Satellite phase 1 was sourced entirely from public domain,
open data.
Kuala Lumpur, Malaysia
Los Angeles, CA -
Brawley, CA
Cloudless Atlas • Cloudfree global mosaic, zoom 0-8 • NASA
MODIS Aqua and Terra Satellites • 380,000 source satellite images
Open data is good.
“data is open if anyone is free to use, reuse,
and redistribute it ...”
“subject to the requirement to attribute and/or share-alike” -Open Knowledge
Definition
ACCESS
- open license - open format - available for download
ACCESS
assumptions 3+1
There are different types of open data users.
Different users have different needs and abilities.
Data accessibility matters.
Open data is not truly open if it is inaccessible.
USERS 3
CASUAL
casual •least technical •dataset discovery •basic needs: ability to query
and download
•geoportal •simple html table •solid metadata •intuitive interface casual
casual USGS EarthExplorer http://earthexplorer.usgs.gov
casual Massachusetts GIS http://gis.amherstma.gov/mgis/
casual The National Map http://nationalmap.gov
casual Utah AGRC Raster Data Discovery http://gis.utah.gov
casual New Hampshire Statewide GIS Clearinghouse http://www.granit.unh.edu/data/downloadfreedata/category/databycategory.html
PROGRAM MATIC
•Tech skills/API familiarity •spatial query •download sub-dataset based on parent
process programmatic
programmatic • API • developer documentation • solid metadata •
interface optional
USGS Application Services http://cumulus.cr.usgs.gov/app_services.php programmatic
USGS Application Services http://cumulus.cr.usgs.gov/app_services.php programmatic
BULK
bulk • Need entire datasets, not spatial intersections • Data
APIs/manual retrieval workflows do not scale • Sometimes retrieve data via physical drives
bulk • interface optional • FTP-like access • reasonable bandwidth
for download retrieval
New Hampshire Statewide GIS Clearinghouse http://www.granit.unh.edu/ Bulk
API
TYPES 3
CONTENT
ConTeNt Database REST Content
Content • Makes application content available for developers to integrate
into existing/new applications
Content
DATA
Database REST Matching Rows Data
DATA • Allows users to query large datasets without having
to have full dataset locally • Applications can be built on top of Live/real-time datasets
Data http://api.occupy-data.org/v1/? results&value=crossst&value=age&value=race&value=crimsusp&value=sex&value=build&value=frisked&results_p er_page=100
BULK
Bulk Database REST References
bulk • Key difference is user obtains reference to object
requested, rather than object itself. • Download object(s) later • Can be relatively lightweight
SO?
Data API = Best Open Data MetHOD?
NO.
APIs, like geoportals, are not always the best option for
disseminating open data.
Different USers
Different NEEds
Different Abilities
Different Access Endpoints
STUFF breaks
Permalinks != Permanent
WayBackMachine http://archive.org/web/web.php
So?
Open data users change as tech changes.
Access should be a policy and tech consideration.
NEXT STEPS
Strive to be SAD
SCALABLE Accessible Durable
- Open systems for access to open data - Can
grow in response to changes in technology/user requirements SCALABLE
- Data access and retrieval is as quick and painless
as possible - Options for users with different abilities, different desired results Accessible
- APIs, geoportals don’t always work - Low-maintenance, durable options
- FTP-like directory access - Good documentation DURABLE
San Francisco, CA
[email protected]
@hrwgc