Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
NACIS 2016 - Practical Cartography Day
Search
Seth Fitzsimmons
October 19, 2016
Design
0
480
NACIS 2016 - Practical Cartography Day
Bash and Make tips for scripting reproducible workflows
Seth Fitzsimmons
October 19, 2016
Tweet
Share
More Decks by Seth Fitzsimmons
See All by Seth Fitzsimmons
Working with OpenStreetMap using Apache Spark and GeoTrellis - SotMUS 2018
mojodna
0
290
OpenMapKit + POSM
mojodna
0
2k
Portable OpenStreetMap at SotM US 2016
mojodna
0
190
Print Cartography in a Multi-Resolution World
mojodna
4
750
How can open data save the world? How can we help?
mojodna
1
120
Trail Assist
mojodna
0
140
Worldwide Terrain and Cloud Infrastructure: an update on Stamen's Knight Grant
mojodna
0
240
Running Your Own Rendering Infrastructure (FOSS4G 2014 Edition)
mojodna
0
230
What We've Made Together
mojodna
0
110
Other Decks in Design
See All in Design
新しいemoji😄のアイデアをUnicodeが募集中‼️🏃♀️💨💨💨傾向を学んでみんな提案しよう💪
oguemon
2
570
Designing and Automating Component Specifications
nathanacurtis
3
370
知を活かせるチームづくりとは?-MIMIGURIで実践している「全員探究」の仕組みと文化づくり-
chiemitaki
1
540
Du Bois Challenge 2024 Recap
ajstarks
1
120
Rebuilding Stamen’s iconic map styles with Stadia Maps
almccon
0
180
Blender 4.1 で レッツ Vket Cloud!
kamera25
0
260
Bridging the Design Gap: How Collaborative Modelling removes blockers to flow between stakeholders and teams @FastFlow conf
baasie
0
110
管理画面の全体UXは利用時品質モデルで考える
readymadegogo
1
1.6k
SpectrumTokyoMeetup12_自動貯金アプリ『finbee』での取り組みについて
shihorishimazu
2
340
Карта процесса-опыта. Презентация метода
ashapiro
0
170
Памятка-раздатка по Карте процесса-опыта, А4
ashapiro
1
120
Webフォント選定の極意!フォントの基本から最新トレンドまで徹底解説
takanorip
3
490
Featured
See All Featured
Ruby is Unlike a Banana
tanoku
96
11k
Scaling GitHub
holman
458
140k
Put a Button on it: Removing Barriers to Going Fast.
kastner
58
3.4k
How to name files
jennybc
75
98k
Creatively Recalculating Your Daily Design Routine
revolveconf
215
12k
Learning to Love Humans: Emotional Interface Design
aarron
270
40k
Fashionably flexible responsive web design (full day workshop)
malarkey
401
65k
What the flash - Photography Introduction
edds
67
11k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
103
48k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
131
32k
Docker and Python
trallard
39
3k
Automating Front-end Workflow
addyosmani
1365
200k
Transcript
$ echo “Make, etc.” | \ wall -g PCD Seth
Fitzsimmons Stamen Design, etc.
[email protected]
# slides $ open \ http://bit.ly/2eFFvJt
# why? # # * repeatable # * self-documenting #
* transformations # <> data changes
# on Windows 10? # no worries # # “Bash
on Ubuntu # on Windows”
# shell fundamentals $ /bin/sh -c theory
# is a comment $ is a prompt
# <cmd> ––help usually # works # man <cmd> is
detailed
# do this, then that $ this; that
# if this, then that $ this && that
# that, unless this $ this || that
# standard file # descriptors # (stdio)
# stdin (fd 0) $ cat > greeting Hi! ⌃d
$ cat greeting Hi!
# stdout (fd 1) $ echo “Hi PCD!”
# stderr $ >&2 echo Error
# a data black hole /dev/null
# redirection $ echo hi > greeting
# append $ echo hey >> greeting
# stderr → stdout $ thing 2>&1 logs
# pipe # this means everything # can be combined!!!
$ cat /etc/passwd | \ grep -i calendar
# exit codes $ thing; echo $?
# success (0) $ thing && echo $? 0
# failure (not 0) $ thing || echo $? 1
$ make basics
# same input (repeatedly), # same output $ make idempotency
Yes $ make idempotency Yes
# target this: touch $@
# prerequisite this: that # that exists cp that this
$ make -j2 vars fns
# target this: # outputs “this” echo $@
# all prereqs this: that the_other # “that the_other” echo
$^
# first prereq this: that the_other # “that” echo $<
# first prereq this: that the_other # “that” echo $<
# nth prereq this: that the_other # “the_other” echo $(word
2, $^)
# catch-all %: echo 42 > $@
$ ls /bin
# explore
# who am I? $ whoami seth
# where am I? $ pwd /home/seth
# what time is it? $ date Tue Oct 18
23:58:42 MDT 2016
# what’s here? $ ls -lh total 8 -rw-r--r-- 1
seth wheel …
# what’s in that file? $ cat /etc/passwd …
# …a page at a time $ less /etc/passwd …
# …just the beginning $ head /etc/passwd …
# …just the end $ tail /etc/passwd …
# what variables are set? $ env HOME=/home/seth …
# where’s that file? $ find . -type f -name
hi <list of files>
# where’d that file go? # (full-text search, macOS) $
mdfind Seattle <list of files>
# get help $ man man …
# manipulation
# create a file $ touch file
# copy a file $ cp file file2
# move a file $ mv file2 file3
# delete a file $ rm file3
# create a directory $ mkdir -p my/stuff
# remove a directory $ rmdir my/stuff
# remove a directory # and everything in it $
rm -r my
# find lines in a file $ grep -i name
file.txt
# find non-matching lines $ grep -v name file.txt
# count words, lines, # characters $ wc file.txt
# pretty-print JSON $ jq . file.json
# extract fields $ jq .name file.json
# replace things $ sed 's/this/that/' file
# extract columns $ cut -d , -f 1,3 file.csv
# extract columns $ cut -d , -f 1,3 file.csv
# compression
# open a zip file $ unzip file.zip
# list zip contents $ unzip -v file.zip
# create a zip $ zip file file.zip
# open a tarball $ tar zxf file.tar.gz
# list a tarball $ tar ztf file.tar.gz
# create a tarball $ tar zcf file.tar.gz stuff/
# compress with gzip $ gzip file.tar
# uncompress with gzip $ gzip -d file.tar.gz
# misc
# always exit 0 $ true; echo $? 0
# always exit non-0 $ false; echo $? 1
# fetch and fail # if appropriate $ curl -f
nacis.org
# download $ wget nacis.org
# download $ wget nacis.org
# “open” (macOS) $ open nacis.org
# display progress $ cat /etc/passwd | \ pv |
wc -l
# also write to a file $ echo hi |
tee file
$ man bash
#!/usr/bin/env bash
set -eo pipefail
set -x
NACIS=2016
# assignment NACIS=“2016”
# capture a command NACIS=$(curl nacis.org)
echo $NACIS
echo ${NACIS}
# set a default value echo ${NACIS:-PCD}
# replace echo ${NACIS/2016/2017}
# replace all echo ${NACIS//2016/2017}
# substring echo ${NACIS:2:2}
# remove suffix $ filename=“world.tif” $ echo ${filename%.tif} world
# do math $ echo $[2 ** 3] 8
if [[ “this” != “that” ]]; then echo Control Flow
elif [[ ! -f file ]]; then touch file elif [[ $six -le $five ]]; then false else rm -f file fi
while true; do echo Control Flow done
for f in $(ls); do echo $f done
# man test
# misc
# extract filename $ basename /etc/passwd passwd
# extract directory $ dirname /etc/passwd /etc
$ make recipes
$ make convert source.json: source.shp ogr2ogr \ -t_srs EPSG:4326 \
-f GeoJSON \ $@ \ $<
$ make reproject output.tif: source.tif gdalwarp \ -q \ -t_srs
EPSG:3857 \ $< \ $@
$ make wilderness data/S_USA.Wilderness.zip: @mkdir -p $$(dirname $@) @curl -sfL
http://data.fs.usda.gov/ geodata/edw/edw_resources/shp/ S_USA.Wilderness.zip \ -o $@
$ make table db/wilderness: sql/wilderness.sql psql \ -c "\d $(subst
db/,,$@)" \ > /dev/null \ 2>&1 || \ psql \ -v ON_ERROR_STOP=1 \ -qX1f \ $<
# resources • https://bost.ocks.org/mike/make/ • http://www.gregreda.com/2013/07/15/unix- commands-for-data-science/ • https://google.github.io/styleguide/shell.xml •
https://github.com/stamen/toner-carto/blob/ master/Makefile (WARNING!) • http://mojodna.net/2015/01/07/make-for-data- using-make.html