Storytelling with *nix processes and streams

*.rb presentation, May 2017

Transcript

1. Storytelling with *nix processes and streams GitHub and Twitter
 @lorainekv
2. *nix? Unix-like operating systems Not this
3. Process? An instance of a program that is running
4. Streams? STDIN STDOUT STDERR Source: https://systemmanual.blogspot.com/2017/01/standard-streams-1.html
5. Real life scenario: Someone is sending you links that are occasionally broken Goal: Tell a story of how often these links are broken
6. Start with one link url=`cat single_link.txt`; curl $url
7. Whittle output down to just the header (with status) url=`cat single_link.txt`; curl $url -I
8. Just get the first line The | pipes STDOUT of curl command to the STDIN of the head command url=`cat single_link.txt`; curl $url -I | head -n1
9. Redirect curl’s STDERR progress meter to the trash Bonus: Try different redirects url=`cat single_link.txt`; curl $url -I 2> /dev/null | head -n1
10. Whittle output down to status code url=`cat single_link.txt`; curl $url -I 2> /dev/null | head -n1 | awk '{print $2}'
11. Print URL next to status code Doesn’t work: url=`cat single_link.txt`; curl $url -I 2> /dev/ null | head -n1 | awk '{print $2, $url}’; Works: url=`cat single_link.txt`; curl $url -I 2> /dev/ null | head -n1 | awk -v url=$url '{print $2, url}'
12. Wrap everything in a loop Now that we know this works for one link, let’s run it on many links while read url; do curl -I $url 2> /dev/null | head -n1 | awk -v url=$url '{print $2, url}'; done < single_link.txt
13. Test it with a larger list of URLs Redirect STDIN to while loop while read url; do curl -I $url 2> /dev/null | head -n1 | awk -v url=$url '{print $2, url}'; done < many_links.txt
14. Save your story
 Write results to a file > redirects STDOUT to a file Bonus: Talk about > vs >> while read url; do curl -I $url 2> /dev/null | head -n1 | awk -v url=$url '{print $2, url}'; done < many_links.txt > statuses.txt
15. More storytelling Get the % of bad links “Hey, did you know 23% of your links are broken? 
 Here’s a list.…” cat statuses.txt | sort -n | uniq -c | awk '{sum += $1} $2 != 200 {bad_status_count += $1} END {print bad_status_count/sum * 100 "%"}'
16. Why streaming is handy - Analogies: a conveyor belt or stream of water (data) - Handle one line of input at a time - Easier to test - Easier to see the time when things go wrong - Don’t need to load entire file / batch to do work
 (don’t load the ocean, just stream it through
 a program)
17. *nix processes: From the 70’s sort: 1971 echo: 1972 awk: 1977
18. Unix / BASH ideas • Each addition does one thing • Do trial runs with • small inputs (1 link vs 1 mil) • print STDOUT to screen (vs a file) • Capture results in a text file • Save useful programs for later! Reading • Pragmatic Programmer • Unix Sys Admin Handbook • Unix Philosophy