Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How we tamed bash, a mostly, true story

How we tamed bash, a mostly, true story

How we as Ruby developers went about building and maintaining 40+ bash scripts with over 3000 unique lines of code.

Video courtesy of engineers.sg: https://www.youtube.com/watch?v=ytU67cLSNRw

This is the Zen of Python that was mentioned:

```
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
```

Björn Andersson

February 24, 2015
Tweet

More Decks by Björn Andersson

Other Decks in Programming

Transcript

  1. @gaqzi Scenario • Big project • File transfers • Integration

    points managed by different teams • Our main logic is being run on servers we have no influence over
  2. @gaqzi Scenario • Big project • File transfers • Integration

    points managed by different teams • Our main logic is being run on servers we have no influence over • bash and the GNU Coreutils ended up being the only available option
  3. @gaqzi #!/bin/sh test_description="Show basic features of Sharness" . ./sharness.sh test_expect_success

    "Success is reported like this" " echo hello world | grep hello " test_expect_success "Commands are chained this way" " test x = 'x' && test 2 -gt 1 && echo success " return_42() { echo "Will return soon" return 42 } test_expect_success "You can test for a specific exit code" " test_expect_code 42 return_42 " test_expect_failure "We expect this to fail" " test 1 = 2 " test_done
  4. @gaqzi Some stats • 616 lines of our own shared

    library (encryption, transfer, packaging, verification, cleaning etc)
  5. @gaqzi Some stats • 616 lines of verification, cleaning etc)

    • 2226 unique lines in scripts, split over ~40 scripts
  6. @gaqzi Some stats • 616 lines of verification, cleaning etc)

    • 2226 unique lines in scripts, split over ~40 scripts • The biggest pipeline contains 15 script runs, spanning multiple data centers and nine different servers
  7. @gaqzi Some stats • 616 lines of verification, cleaning etc)

    • 2226 unique lines in scripts, split over ~40 scripts • The biggest pipeline contains 15 script runs, spanning multiple data centers and nine different servers • The compiled scripts are a total of 16 504 lines
  8. @gaqzi Some stats • 616 lines of verification, cleaning etc)

    • 2226 unique lines in scripts, split over ~40 scripts • The biggest pipeline contains 15 script runs, spanning multiple data centers and nine different servers • The compiled scripts are a total of 16 504 lines • The largest compiled script is 720 lines (142 lines uncompiled)
  9. @gaqzi Some stats • 616 lines of verification, cleaning etc)

    • 2226 unique lines in scripts, split over ~40 scripts • The biggest pipeline contains 15 script runs, spanning multiple data centers and nine different servers • The compiled scripts are a total of 16 504 lines • The largest compiled script is 720 lines (142 lines uncompiled) • 2400 lines of cucumber features, 299 scenarios over a total of 2052 steps
  10. @gaqzi Bash that survives you? Not wanting your fancy new

    code ending up being the equivalent of autohell
  11. @gaqzi Bash that survives you? Not wanting your fancy new

    code ending up being the equivalent of autohell Very few people have experience with writing more than a couple of lines of bash
  12. @gaqzi Bash that survives you? Not wanting your fancy new

    code ending up being the equivalent of autohell Very few people have experience with writing more than a couple of lines of bash Keep it easy, clean and in chunks that make sense
  13. @gaqzi Bash that survives you? Not wanting your fancy new

    code ending up being the equivalent of autohell Very few people have experience with writing more than a couple of lines of bash Keep it easy, clean and in chunks that make sense Functions, they’re there
  14. @gaqzi Bash that survives you? Not wanting your fancy new

    code ending up being the equivalent of autohell Very few people have experience with writing more than a couple of lines of bash Keep it easy, clean and in chunks that make sense Functions, they’re there Eschew the shortcuts
  15. @gaqzi Bash that survives you? Not wanting your fancy new

    code ending up being the equivalent of autohell Very few people have experience with writing more than a couple of lines of bash Keep it easy, clean and in chunks that make sense Functions, they’re there Eschew the shortcuts Try to keep it beautiful
  16. @gaqzi “Beautiful is better than ugly. Explicit is better than

    implicit. Simple is better than complex.
  17. @gaqzi –Tim Peters, The Zen of Python “Beautiful is better

    than ugly. Explicit is better than implicit. Simple is better than complex. Complex is better than complicated.”
  18. @gaqzi What does it look like? fetch-files-from $REMOTE_SFTP_IN_SERVER $REMOTE_FTP_OUT_PATH $LOCAL_IN_PATH

    if-error $? 1 "An error occurred trying to download files from \ ’$REMOTE_SFTP_SERVER:$REMOTE_FTP_OUT_PATH' => '$LOCAL_IN_PATH'" any-files-downloaded-to $LOCAL_IN_PATH if-error $? 2 "No files exists in '$LOCAL_IN_PATH'" add-checksum-to-files-in $LOCAL_IN_PATH if-error $? 3 "Error creating checksum for new files" push-files-to $REMOTE_SFTP_OUT_SERVER $REMOTE_FTP_IN_PATH $LOCAL_OUT_PATH if-error $? 4 "Error sending files to remote server" if [ "$cleanup" = 'true' ] ; then do-cleanup if-error $? 5 "Error cleaning up temporary files" fi
  19. @gaqzi Feature: File transfer from third party Background: Given I

    am logged in as "ftpinternal" And I have a clean "third party" environment Scenario: A successful transfer from DMZ to Internal without cleanup Given that "the test file" is available in "/ftp/third-party/project_1234/daily/out" When I execute "dmz-fetch-files-from-third-party.sh nocleanup" successfully Then the file "the test file" in "/internal/ftp/third-party/project_1234/daily/in" is valid with | ff961dc5e8da688fa78540651160b223 | And there is no logfiles in "/dmz/ftp/third-party/project_1234/archive/log/in"
  20. @gaqzi Test it where you deploy And I’m not talking

    about testing in production, even if that’s what real hackers do
  21. @gaqzi Test it where you deploy And I’m not talking

    about testing in production, even if that’s what real hackers do If you’re deploying on GNU/Linux, then always test there
  22. @gaqzi Test it where you deploy And I’m not talking

    about testing in production, even if that’s what real hackers do If you’re deploying on GNU/Linux, then always test there Did you know OSX mostly uses the BSD toolchain?
  23. @gaqzi Test it where you deploy And I’m not talking

    about testing in production, even if that’s what real hackers do If you’re deploying on GNU/Linux, then always test there Did you know OSX mostly uses the BSD toolchain? Vagrant is wonderful, Ansible or similar also helps a lot with setting up the test machine
  24. @gaqzi Test it where you deploy And I’m not talking

    about testing in production, even if that’s what real hackers do If you’re deploying on GNU/Linux, then always test there Did you know OSX mostly uses the BSD toolchain? Vagrant is wonderful, Ansible or similar also helps a lot with setting up the test machine To run all scenarios it takes about 10 minutes
  25. @gaqzi What did I learn? I can feel confident about

    a big bash script, even when needing it to change
  26. @gaqzi What did I learn? I can feel confident about

    a big bash script, even when needing it to change Your code will be more readable if it’s less idiomatic, unless everyone is really comfortable with bash then use the overly expressive version
  27. @gaqzi What did I learn? I can feel confident about

    a big bash script, even when needing it to change Your code will be more readable if it’s less idiomatic, unless everyone is really comfortable with bash then use the overly expressive version set -x will echo every command being performed, wonderful when enabled in production
  28. @gaqzi What did I learn? I can feel confident about

    a big bash script, even when needing it to change Your code will be more readable if it’s less idiomatic, unless everyone is really comfortable with bash then use the overly expressive version set -x enabled in production set -u explode when using an unassigned variable, if you didn’t assign then it likely is a typo
  29. @gaqzi set -e exits on any error, seems like a

    great idea but doesn’t allow you to set your own error messages What did I learn?
  30. @gaqzi set -e you to set your own error messages

    : ${SECRET_PASSWORD:=default-pass} is the weirdest syntax ever for default/overridable variables, but it’s worth using What did I learn?
  31. @gaqzi set -e you to set your own error messages

    : ${SECRET_PASSWORD:=default-pass} ever for default/overridable variables, but it’s worth using Keeping a separate configuration file makes it easier to test What did I learn?
  32. @gaqzi set -e you to set your own error messages

    : ${SECRET_PASSWORD:=default-pass} ever for default/overridable variables, but it’s worth using Keeping a separate configuration file makes it easier to test A lot of us devs don’t actually know our small Unix tools very well, look up GNU Coreutils and some basic bash. It’ll save you time! What did I learn?
  33. @gaqzi set -e you to set your own error messages

    : ${SECRET_PASSWORD:=default-pass} ever for default/overridable variables, but it’s worth using Keeping a separate configuration file makes it easier to test A lot of us devs don’t actually know our small Unix tools very well, look up GNU Coreutils and some basic bash. It’ll save you time! It was actually fun doing this ridiculous thing What did I learn?