Command line & Data Science

Transcript

1. Yatish Mehta @yatish27
2. Command line & Data Science
3. • pip install csvkit • cat leads.csv | csvlook • csvstat leads.csv • csvgrep -c 6 -m samplecompany.com | csvlook 1.csvkit
4. 2. grep,sed,sort,uniq • cat wiki.txt | grep -oE '\w+' | tee words • < words grep '^a' | sort | uniq -c | sort -r • sed ’s/data/tata/g’ wiki.txt > wiki2.txt
5. • brew install jq • < data.json jq ‘.[]’ • < data.json jq ‘.[] | select(.age>22)’ • cat data.json | jq '.[] | {isActive: ._id, name: .name}' 3. jq JSON processor
6. 4. qstats • qstats one_hundred_milion.dat Min.          44.947   1st  Qu.    93.2553   Median      100.001   Mean          100.001   3rd  Qu.    106.747   Max.          156.997   Range        112.05   Std  Dev.  10.0002   Length      100000000   • Faster than awk, sort, R
7. 5. parallel • iterative • shell parallel.sh , each action as a job • parallel keyword
8. Thank You