Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Bulk + Open Data APIs
Search
Chris Herwig
April 04, 2013
Technology
2
250
Bulk + Open Data APIs
Chris Herwig
April 04, 2013
Tweet
Share
More Decks by Chris Herwig
See All by Chris Herwig
Clear Skies: Turning Massive NASA Data into a Pixel-Perfect World Atlas
hrwgc
0
450
Open + Accessible
hrwgc
2
110
Open Satellite Imagery and Geoportals | MapBox Satellite
hrwgc
1
200
Mapping Mars Open Source
hrwgc
1
74
Other Decks in Technology
See All in Technology
検証を通して見えてきたTiDBの性能特性
lycorptech_jp
PRO
6
3.7k
元インフラエンジニアに成る / Human Resources to Human Relations
bobtani
4
890
AWS認定資格を取得したので、初めてマネコンを触った時を振り返ってみた。
ainatsuptr
2
100
NgRx Signal Store
rainerhahnekamp
0
140
ユーザーストーリーのレビューを自動化したみたの
bun913
1
390
Terraformあれやこれ/terraform-this-and-that
emiki
8
1.3k
LLM とプロンプトエンジニアリング/チューターをビルドする / LLM and Prompt Engineering and Building Tutors
ks91
PRO
0
250
Delivering Millions of Messages within seconds @ Duolingo
pelelgrino
0
340
現代CSSフレームワークの内部実装とその仕組み
poteboy
8
3.5k
コンテナセキュリティの基本と脅威への対策
kyohmizu
3
740
**強い**エンジニアのなり方 - フィードバックサイクルを勝ち取る / grow one day each day
soudai
65
19k
Meta Quest 3 で動く桜マシマシ WebXR アプリを IBM Cloud Code Engine と Babylon.js で作った話
1ftseabass
PRO
0
120
Featured
See All Featured
Being A Developer After 40
akosma
56
580k
Fashionably flexible responsive web design (full day workshop)
malarkey
398
65k
Adopting Sorbet at Scale
ufuk
67
8.6k
It's Worth the Effort
3n
180
27k
Fantastic passwords and where to find them - at NoRuKo
philnash
36
2.5k
GraphQLとの向き合い方2022年版
quramy
31
12k
Mobile First: as difficult as doing things right
swwweet
216
8.6k
Clear Off the Table
cherdarchuk
83
310k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
119
39k
Teambox: Starting and Learning
jrom
128
8.4k
Rebuilding a faster, lazier Slack
samanthasiow
72
8.2k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
240
1.2M
Transcript
Bulk Chris Herwig @hrwgc +open
Chris Herwig Satellite Team lead, MapBox
MapBox Satellite Phase 1, Launched 12/2012 • Global imagery base
layer for MapBox users • Global satellite imagery, zoom 0-12 • Continental U.S. aerial imagery zoom 13-17 • Licensed to allow for OSM tracing
MapBox Satellite phase 1 was sourced entirely from public domain,
open data.
Kuala Lumpur, Malaysia
Los Angeles, CA -
Brawley, CA
Cloudless Atlas • Cloudfree global mosaic, zoom 0-8 • NASA
MODIS Aqua and Terra Satellites • 380,000 source satellite images
Open data is good.
“data is open if anyone is free to use, reuse,
and redistribute it ...”
“subject to the requirement to attribute and/or share-alike” -Open Knowledge
Definition
ACCESS
- open license - open format - available for download
ACCESS
assumptions 3+1
There are different types of open data users.
Different users have different needs and abilities.
Data accessibility matters.
Open data is not truly open if it is inaccessible.
USERS 3
CASUAL
casual •least technical •dataset discovery •basic needs: ability to query
and download
•geoportal •simple html table •solid metadata •intuitive interface casual
casual USGS EarthExplorer http://earthexplorer.usgs.gov
casual Massachusetts GIS http://gis.amherstma.gov/mgis/
casual The National Map http://nationalmap.gov
casual Utah AGRC Raster Data Discovery http://gis.utah.gov
casual New Hampshire Statewide GIS Clearinghouse http://www.granit.unh.edu/data/downloadfreedata/category/databycategory.html
PROGRAM MATIC
•Tech skills/API familiarity •spatial query •download sub-dataset based on parent
process programmatic
programmatic • API • developer documentation • solid metadata •
interface optional
USGS Application Services http://cumulus.cr.usgs.gov/app_services.php programmatic
USGS Application Services http://cumulus.cr.usgs.gov/app_services.php programmatic
BULK
bulk • Need entire datasets, not spatial intersections • Data
APIs/manual retrieval workflows do not scale • Sometimes retrieve data via physical drives
bulk • interface optional • FTP-like access • reasonable bandwidth
for download retrieval
New Hampshire Statewide GIS Clearinghouse http://www.granit.unh.edu/ Bulk
API
TYPES 3
CONTENT
ConTeNt Database REST Content
Content • Makes application content available for developers to integrate
into existing/new applications
Content
DATA
Database REST Matching Rows Data
DATA • Allows users to query large datasets without having
to have full dataset locally • Applications can be built on top of Live/real-time datasets
Data http://api.occupy-data.org/v1/? results&value=crossst&value=age&value=race&value=crimsusp&value=sex&value=build&value=frisked&results_p er_page=100
BULK
Bulk Database REST References
bulk • Key difference is user obtains reference to object
requested, rather than object itself. • Download object(s) later • Can be relatively lightweight
SO?
Data API = Best Open Data MetHOD?
NO.
APIs, like geoportals, are not always the best option for
disseminating open data.
Different USers
Different NEEds
Different Abilities
Different Access Endpoints
STUFF breaks
Permalinks != Permanent
WayBackMachine http://archive.org/web/web.php
So?
Open data users change as tech changes.
Access should be a policy and tech consideration.
NEXT STEPS
Strive to be SAD
SCALABLE Accessible Durable
- Open systems for access to open data - Can
grow in response to changes in technology/user requirements SCALABLE
- Data access and retrieval is as quick and painless
as possible - Options for users with different abilities, different desired results Accessible
- APIs, geoportals don’t always work - Low-maintenance, durable options
- FTP-like directory access - Good documentation DURABLE
San Francisco, CA
[email protected]
@hrwgc