Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Geospatial CSV Imports Hidden Complexity
Search
Sponsored
·
SiteGround - Reliable hosting with speed, security, and support you can count on.
→
Kartones
October 27, 2015
Programming
69
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Geospatial CSV Imports Hidden Complexity
@ Mindcamp 7.0 2015, Madrid
Kartones
October 27, 2015
Transcript
18 slides
@Kartones GEOSPATIAL CSV IMPORTS HIDDEN COMPLEXITY
@Kartones CartoDB
@Kartones Agenda 1) CSV Format Issues 2) Import Issues
@Kartones CSV FORMAT ISSUES
@Kartones Intro .csv / MIME:text/csv Unknown birthdate (80s?) RFC 4180
(2005)
@Kartones Intro Plain text Simple format Simple rules
@Kartones Usage
@Kartones CSV 0101000020E610000000000000008049C000000000000038C0,1083 "alien",2014-11-04 15:24:40.43413+00 category 1, "jump jump up!",
{""value"":""es""}
@Kartones WKT: Well-Known Text POINT (30 10) LINESTRING (30 10,
10 30, 40 40) POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10)) MULTIPOINT ((10 40), (40 30), (20 20), (30 10)) MULTIPOLYGON (((30 20, 45 40, 10 40, 30 20)), ((15 5, 40 10, 10 20, 5 10, 15 5))) https://en.wikipedia.org/wiki/Well-known_text
@Kartones WKB: Well-Known Binary POINT(2.0 4.0) = 000000000140000000000000004010000000000000 https://en.wikipedia.org/wiki/Well-known_text#Well-known_binary
@Kartones GeoJSON { "type": "Feature", "geometry": { "type": "Point", "coordinates":
[125.6, 10.1] }, "properties": { "name": "Dinagat Islands" } } http://geojson.org/
@Kartones IMPORT ISSUES
@Kartones Typical Huge files (>1GB) Lots of rows (+2M) Lots
of columns (~1600) XLS/XLSX -> CSV
@Kartones Typical Stream HTTP downloaded file Stream file between servers
Stream data import to DB
@Kartones Typical
@Kartones CartoDB-specific Content guessing (e.g. lat/lon) Type guessing Geometry errors
fixing Sync tables -> No downtime allowed
@Kartones DB-Specific Leave DB indexes as last step Prefer big
INSERT to multiple UPDATE GDAL’s ogr2ogr > Ruby/Python scripts http://www.gdal.org/ogr2ogr.html
@Kartones Questions? Thanks!
[email protected]
More Decks by Kartones
See All by Kartones
Building Autonomous Agents with gym-retro
kartones
0
60
Python static typing with MyPy
kartones
0
98
High-impact refactors keeping the lights on
kartones
0
84
Remote Work
kartones
0
120
Intro to GameBoy Development
kartones
0
120
Myths & The Real World of OpenSource Development
kartones
0
61
CartoDB Tech Intro
kartones
0
63
Copy Protection & Cracking History
kartones
0
160
Cómo ganar dinero con tus juegos online
kartones
1
130
Other Decks in Programming
See All in Programming
祈るテストから、探索するテストへ 〜Lincheckではじめる並行処理テスト〜
yyh_gl
0
420
AppFunctionsの内部実装を紐解く
hiroaki404
0
110
正解の無い問題を 「できるだけ上手く」解く ヒューリスティックで感性を磨こう。〜現実課題をプログラミングで解きあうコミュニティについて〜
allegrogiken
0
110
[Kotlin Fest 2026] eBPF Tracing for Kotlin
takehaya
1
400
DroidKnights 2026 Android A2UI Jetpack Compose 기반의 안전한 AI UI 렌더링 - 박제창
itsmedreamwalker
1
260
AI駆動TDD ― 2026年10月版
hrmstrsmgs
0
110
Python Lambdaにおけるテストしやすい設計とAI支援の活用 / Designing Testable Python Lambda Functions with AI
kitano_yuichi
1
350
動画プレイヤーはなぜカクつくのか 〜Flutterのレンダリングとネイティブの境界を解剖する〜
naruogram
0
340
どうしても Eloquent でリポジトリパターンをやりたいんだ #phpcon_niigata / phpcon-niigata-2026
shogogg
0
670
Getting Started with Vibe Coding
miccheng
0
250
止まらない比較を止める — 余帰納法で無限ストリームを比べる
tarao
0
210
地方こそサーバーレス — 10年続けた、顔が見えるチームで確かな基盤を / Serverless Matters Most in Regional Areas — 10 Years of Building Reliable Foundations with Close-Knit Teams
seike460
PRO
2
510
Featured
See All Featured
Lightning Talk: Beautiful Slides for Beginners
inesmontani
PRO
2
850
Ten Tips & Tricks for a 🌱 transition
stuffmc
1
290
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
230
23k
世界の人気アプリ100個を分析して見えたペイウォール設計の心得
akihiro_kokubo
PRO
73
43k
How STYLIGHT went responsive
nonsquared
100
6.4k
Building a A Zero-Code AI SEO Workflow
portentint
PRO
0
850
How to optimise 3,500 product descriptions for ecommerce in one day using ChatGPT
katarinadahlin
PRO
3
3.9k
Highjacked: Video Game Concept Design
rkendrick25
PRO
2
590
VelocityConf: Rendering Performance Case Studies
addyosmani
331
25k
Stewardship and Sustainability of Urban and Community Forests
pwiseman
0
610
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
28
3.8k
HU Berlin: Industrial-Strength Natural Language Processing with spaCy and Prodigy
inesmontani
PRO
0
820