Automate Yo' Self (OpenWest 2016)

Slide 1

Slide 1 text

Automate Yo' Self! OpenWest 2016 Sandy UT John SJ Anderson @genehack if anybody has any questions or i'm going too fast, please throw up a hand and ask -- or i'm here all week, grab me on the hallway track

Slide 2

Slide 2 text

Hi, I'm John. hi, i'm john

Slide 3

Slide 3 text

@genehack i go by genehack most places on line. today i'm going to talk to you today about some tools and tricks i have for being productive while developing. but before i do that, i need to explain a bit about why i needed it. my life is busy.

Slide 4

Slide 4 text

Sammy i've got a dog.

Slide 5

Slide 5 text

@sammygenehack dog's got a twitter account.

Slide 6

Slide 6 text

Two kids two daughters

Slide 7

Slide 7 text

A Wife and a wife that i like to hang out with

Slide 8

Slide 8 text

long-suffering conference widow A Wife

Slide 9

Slide 9 text

photobomber is not impressed. A Wife long-suffering conference widow

Slide 10

Slide 10 text

Job I've also got a job

Slide 11

Slide 11 text

Hobbies and i have a few hobbies -- maintain some Perl modules,

Slide 12

Slide 12 text

Hobbies Trying to learn Swift, check out Angular 2,

Slide 13

Slide 13 text

Lots of Hobbies reading, hiking, cooking,

Slide 14

Slide 14 text

I've got a lot of balls in the air don't have time for trivial nonsense, so i use a lot of automation and other "lifehacks"

Slide 15

Slide 15 text

AUTOMATE YO' SELF occasionally, you just have to automate yo' self!

Slide 16

Slide 16 text

Basic Principles whenever i'm trying to do this, there are some basic principles that i try to keep in mind

Slide 17

Slide 17 text

Don't Make Me Think none of this automation stuff should require me to think -- if i have to think about it, it's not really saving me any time

Slide 18

Slide 18 text

This is my "you made me think" face. the whole point is not having to think.

Slide 19

Slide 19 text

Consistency is Good. so, in those terms, one thing that's essential: make everything the same. have a standard directory layout under $HOME. you shouldn't have to think about what system you're on, what shell, what project. things should just be the same -- or at least *correct* -- all the time

Slide 20

Slide 20 text

Idempotence is better! Idempotence: the property of certain operations in mathematics and computer science, that can be applied multiple times without changing the result beyond the initial application

Slide 21

Slide 21 text

App::MiseEnPlace so, here's an example of idempotence, a utility i wrote to manage symlinks and directories in my home directory

Slide 22

Slide 22 text

App::MiseEnPlace "everything in its place" "mise en place" is a French phrase meaning "everythng in its place" -- it comes from cooking, and the principle that you should have all your ingredients prepped and ready to go before you start cooking. i wanted something to make sure i always had my standard directory layout under $HOME as well as setting up various symlinks

Slide 23

Slide 23 text

% cat ~/.mise --- manage: - doc - etc - private - proj/* - src/* create: directories: - bin - proj - proj/go/src/github.com/genehack - src - var links: - Desktop: var/tmp - Desktop: tmp here's what the top level config for mise looks like -- it goes in .mise in your home directory. we have a list of directories we want to manage (more on that in a minute) and a set of directories and symlinks to create. links are source:target

Slide 24

Slide 24 text

% cat proj/emacs/.mise --- create: links: - DIR: ~/.emacs.d - bin/build-most-recent-emacs: BIN - bin/e: BIN - bin/ec: BIN - bin/git-blame-from-line-num: BIN - bin/map-test-lib: BIN this is a per-project config file. mise has a couple special keywords DIR and BIN, that refer to the directory containing the .mise file and ~/bin, respectively the advantage of this is you don't need to have a huge gnarly $PATH with a bunch of project directories in it, everything is just symlinked into ~/bin

Slide 25

Slide 25 text

% mise [LINK] created ~/proj/emacs -> ~/.emacs.d [LINK] created ~/proj/emacs/bin/build-most-recent-emacs -> ~/bin/build-most-recent-emacs [LINK] created ~/proj/emacs/bin/e -> ~/bin/e [LINK] created ~/proj/emacs/bin/ec -> ~/bin/ec [LINK] created ~/proj/emacs/bin/git-blame-from-line-num -> ~/bin/git-blame-from-line-num [LINK] created ~/proj/emacs/bin/map-test-lib -> ~/bin/map-test-lib % mise % rm ~/bin/e % mise [LINK] created ~/proj/emacs/bin/e -> ~/bin/e when we run mise for the first time, you can see it creates all those links. if we run it again, it does NOTHING. Idempotency for the win! If you remove a link and run it again, it creates _just_ that link

Slide 26

Slide 26 text

App::MiseEnPlace get it from your favorite CPAN mirror available on CPAN, minimal dependencies, works on any perl from this decade.

Slide 27

Slide 27 text

smartcd ok, so that handles getting a consistent directory structure, and linking project binaries. what if you have other per-project stuff that you want to set up? environment variables or other stuff?

Slide 28

Slide 28 text

smartcd Automatically run code when entering or leaving directories or sub-directories enter smartcd, which hooks the 'cd' command and then runs scripts when you enter or leave a directory (or even a subdirectory)

Slide 29

Slide 29 text

% smartcd show enter /Users/genehack/.smartcd/scripts/Users/genehack/fake-node-proj/bash_enter exists ------------------------------------------------------------------------- ######################################################################## # smartcd enter - /Users/genehack/fake-node-proj # # This is a smartcd script. Commands you type will be run when you # enter this directory. The string __PATH__ will be replaced with # the current path. Some examples are editing your $PATH or creating # a temporary alias: # # autostash PATH=__PATH__/bin:$PATH # autostash alias restart="service stop; sleep 1; service start" # # See http://smartcd.org for more ideas about what can be put here ######################################################################## autostash NODE=4.2.3 autostash PATH=$PATH:__PATH__/node_modules/.bin/ nvm use $NODE ------------------------------------------------------------------------- smartcd has 'edit' and 'show' subcmds (and a bunch of others), and 'leave' and 'enter' scripts. here's an enter script for an arbitrary node project. the autostash keyword sets up an environment variable that will be *unset* when you leave this directory. you can also run arbitrary commands; the 'nvm' command here selects a version of node to use

Slide 30

Slide 30 text

% cd ~/fake-node-proc Now using node v4.2.3 (npm v2.14.7) % echo xx$NODE xx4.2.3 % cd .. % echo xx$NODE xx here's what it looks like when you cd into that directory. and you can see the env var is set. if we change back out of the directory, the environment variable is unset.

Slide 31

Slide 31 text

smartcd https://github.com/cxreg/smartcd pretty cool, available on github. works with bash and zsh. really easy to install. i can pay this the highest possible compliment: i haven't forked it or needed to patch it in any way, i just use a checkout from the upstream repo

Slide 32

Slide 32 text

App/Env Builders are your friends speaking of per-project settings, let's talk for a minute about application builders.

Slide 33

Slide 33 text

perlbrew plenv nvm App::GitGitr build-most- recent-emacs here are some of the ones i've used or use now. perlbrew and plenv let you have multiple perls. nvm does the same thing for node. similar tools exist for python, ruby, etc. then there's GitGitr, which I wrote while maintaining Git wrapper library. I needed to be able to quickly install arbitrary Git versions while responding to bug reports -- so I wrote a little tool that does that. Similarly, I'm an Emacs user. If a new version is released, I want to upgrade, across all my systems - so I scripted that.

Slide 34

Slide 34 text

Consistency Corollary: Don't trust system binaries Back at the beginning, I said "consistency is good". Now, if you're developing on MacOS and deploying to Linux (or dev-ing on Ubuntu and deploying to Debian), you're probably not going to have the same version of tool from the OS (and if you do now, it's not going to last). Even if the versions are the same, there may have been vendor patches applied.

Slide 35

Slide 35 text

Automate building your critical tools. No, if you really want to be consistent, the best approach is to build the tools that are most critical for your project. ("Build" in this case may just mean automating the install; it doesn't have to mean "build from source".) Note: only do this for the *important* stuff. (include examples)

Slide 36

Slide 36 text

The Silver Searcher Speaking of tools, here's a tool that has literally improved my entire development life -- the silver searcher. Anybody here using this?

Slide 37

Slide 37 text

grep? ok, so, grep -- everybody knows grep, right? let's you search for text inside files, which is something you do a lot while developing code.

Slide 38

Slide 38 text

grep? powerful, speedy, indiscriminate so, grep is super powerful in terms of what you can search for, and it's pretty quick, but it's not very selective. you can list all the files you want to search, or search whole dir trees, but you quickly realize this sucks, because of things like .git directories. anybody ever do a recursive grep on a big git checkout? yeah.

Slide 39

Slide 39 text

ack? anybody here use ack? ack is a grep-like tool written in perl.

Slide 40

Slide 40 text

ack? powerful, selective, slow it's just as powerful as grep in terms of what you can search for, but it's recursive by default (which is what you want) and it's smart about ignoring .git and SVN stuff. the problem is, it's pretty slow, particularly to start up (because Perl)

Slide 41

Slide 41 text

ag! enter ag (which is what the binary for the silver searcher is called.

Slide 42

Slide 42 text

ag! powerful, selective, FAST ag works much like ack (not _exactly_, but close enough), but it's written in C and it's oh so fast.

Slide 43

Slide 43 text

The Silver Searcher https://github.com/ggreer/the_silver_searcher available on Github, also packaged in several Linux distros. again, no higher compliment than to say I just build from a checkout of the upstream. have never needed to fork or patch

Slide 44

Slide 44 text

My biggest productivity/automation tip so, here's my single biggest tip in this whole talk. are you ready? brace yourselves.

Slide 45

Slide 45 text

Revision control $HOME this is it. anybody know what this is? this is your home directory under revision control

Slide 46

Slide 46 text

Why bother? no, but seriously. it's a little bit of a pain to develop the discipline but once you get used to having _everything_ under revision control, it's nice. you don't have to worry about experimenting with anything, backing stuff up, or dealing with cross-machine variation in your environments

Slide 47

Slide 47 text

say automation again originally i had series of kludgy shell scripts to manage repo updates and checkouts super ugly and not worth sharing and then, inspiration: Ingy döt Net talking about App::AYCABTU @ PPW2010

Slide 48

Slide 48 text

Ingy döt Net this is ingy - he's a crazy awesome open source hacker guy who has done a whole bunch of stuff. probably best known for being one of the inventors of YAML

Slide 49

Slide 49 text

Things I wanted to steal the thing ingy had developed had a bunch of stuff i wanted to steal: The basic idea The interface Info about repositories in conﬁg ﬁle Flexible ways of selecting repos for operations – by #, by name, by tag

Slide 50

Slide 50 text

Things I wanted to add Support for more than just Git Locate repositories in arbitrary locations Easily add and remove repositories Ability to easily extend with more subcommands Most importantly: better name!

Slide 51

Slide 51 text

App::AYCABTU?!?!‽ and finally, the name. what even is this.

Slide 52

Slide 52 text

GitGot Thus was born GitGot http://search.cpan.org/dist/App-GitGot/ Installs a ‘got’ command Uses Moo and App::Cmd under the covers Add new subcommands by writing a single class!

Slide 53

Slide 53 text

Whirlwind Tour Let's take a whirlwind tour of how got works

Slide 54

Slide 54 text

got add you tell got about repos using the 'add' command, from inside the git repo

Slide 55

Slide 55 text

% got add Name: foo URL: Path: /Users/genehack/foo Tags: bar it'll prompt you for required info, and supply sensible defaults. note that you can also apply tags (space-delimited)

Slide 56

Slide 56 text

got add -D or you can just add the '-D' switch and it'll automatically use the defaults

Slide 57

Slide 57 text

got clone you can also clone a remote repo, which will check it out into the working directory and add it to got, prompting you for details

Slide 58

Slide 58 text

% got clone [email protected]:genehack/app-gitgot.git Name: [app-gitgot]: Path: [/Users/genehack/app-gitgot]: Tags: : Cloning into '/Users/genehack/app-gitgot'... it'll prompt you for required info, and supply sensible defaults. note that you can also apply tags (space-delimited)

Slide 59

Slide 59 text

got clone -D got clone also respects the '-D' switch

Slide 60

Slide 60 text

got fork finally, you can give got a github url, and it will fork that project under your github id, then check it out into the current directory and add it to got

Slide 61

Slide 61 text

~/.gitgot all the info about the repos managed by got lives in this .gitgot file in your home directory

Slide 62

Slide 62 text

- name: App-Amylase path: /Users/genehack/proj/App-Amylase repo: [email protected]:genehack/App-Amylase.git type: git - name: Git-Wrapper path: /Users/genehack/proj/Git-Wrapper repo: [email protected]:genehack/Git-Wrapper.git tags: git type: git - name: HiD path: /Users/genehack/proj/HiD repo: [email protected]:genehack/HiD.git type: git - name: Perl-Build path: /opt/plenv/plugins/perl-build repo: git://github.com/tokuhirom/Perl-Build.git type: git it's just a simple YAML formatted line, totally hand-editable (although you shouldn't _need_ to do that, you can) note that repos can be located anywhere on the disk, don't have to under a common dir or in your home or whatever. anywhere you can write to is fair game

Slide 63

Slide 63 text

But now what? ok, so you've added all your git repositories to got. what now?

Slide 64

Slide 64 text

got list well, you can get a list of them

Slide 65

Slide 65 text

got ls which you can shorten to this

Slide 66

Slide 66 text

1) App-Amylase git [email protected]:genehack/App-Amylase.git 2) Git-Wrapper git [email protected]:genehack/Git-Wrapper.git 3) HiD git [email protected]:genehack/HiD.git 4) Perl-Build git git://github.com/tokuhirom/Perl-Build.git 5) Perl-Critic git [email protected]:genehack/Perl-Critic.git 6) STAMPS git [email protected]:genehack/STAMPS.git 7) advanced-moose-class git ssh://[email protected]/train/advanced-moose-class.git 8) app-gitgitr git [email protected]:genehack/app-gitgitr.git 9) app-gitgot git [email protected]:genehack/app-gitgot.git that'll get you this sort of listing.

Slide 67

Slide 67 text

got ls -q if you don't want to see the upstream repo info, you can use the '-q' or '--quiet' switch

Slide 68

Slide 68 text

1) App-Amylase 2) Git-Wrapper 3) HiD 4) Perl-Build 5) Perl-Critic 6) STAMPS 7) advanced-moose-class 8) app-gitgitr 9) app-gitgot and that'll get you this output. note the numbers - those will give you a way to select repos for other commands

Slide 69

Slide 69 text

got ls [repos] easiest way to demo that is with an example. you can restrict the listing

Slide 70

Slide 70 text

got ls 5 this will just list repo #5 for example

Slide 71

Slide 71 text

5) Perl-Critic also, note that the list is always sorted the same way, so the numbers will be stable (unless you add new repos)

Slide 72

Slide 72 text

got ls 5-8 you can also give a range of repos

Slide 73

Slide 73 text

5) Perl-Critic 6) STAMPS 7) advanced-moose-class 8) app-gitgitr and that'll give you that range, like you would expect

Slide 74

Slide 74 text

got ls HiD you can also specify repos by name

Slide 75

Slide 75 text

3) HiD

Slide 76

Slide 76 text

got ls -t git or by using tags. can specify multiple tags with multiple '-t' switches. they combine with 'or' semantics.

Slide 77

Slide 77 text

2) Git-Wrapper 8) app-gitgitr 9) app-gitgot here are all the repos tagged with 'git' (at least in our example)

Slide 78

Slide 78 text

got ls 5-8 HiD 21 -t git finally, you can combine all of these selection methods together. here we're asking for repos 5 thru 8, the repo named HiD, repo 21, and all repos tagged with git

Slide 79

Slide 79 text

2) Git-Wrapper 3) HiD 5) Perl-Critic 6) STAMPS 7) advanced-moose-class 8) app-gitgitr 9) app-gitgot 21) etc and this is what we get note that most commands operate on all repos, and any that do, you can use these techniques to restrict the command to a subset.

Slide 80

Slide 80 text

What else you got? That's cool.

Slide 81

Slide 81 text

(no pun intended.)

Slide 82

Slide 82 text

got status you can check the status of your repos. *all* your repos

Slide 83

Slide 83 text

got st or if you're into the whole brevity thing...

Slide 84

Slide 84 text

1) App-Amylase : OK 2) Git-Wrapper : OK 3) HiD : OK 4) Perl-Build : OK 5) Perl-Critic : OK 6) STAMPS : OK 7) advanced-moose-class : OK 8) app-gitgitr : OK 9) app-gitgot : OK that'll get you output like this. again, note the use of color to give quick visual cues

Slide 85

Slide 85 text

1) App-Amylase : OK 2) Git-Wrapper : OK 3) HiD : Dirty 4) Perl-Build : OK 5) Perl-Critic : OK 6) STAMPS : OK 7) advanced-moose-class : OK 8) app-gitgitr : OK 9) app-gitgot : OK Dirty got status will let you know if a repo has uncommitted changes. super handy if, for example, working on one machine and are going to move to another one and want to see what hasn't been committed.

Slide 86

Slide 86 text

1) App-Amylase : OK 2) Git-Wrapper : OK 3) HiD : OK 4) Perl-Build : OK 5) Perl-Critic : OK 6) STAMPS : OK Ahead by 1 7) advanced-moose-class : OK 8) app-gitgitr : OK 9) app-gitgot : OK 1) App-Amylase : OK 2) Git-Wrapper : OK 3) HiD : OK 4) Perl-Build : Dirty 5) Perl-Critic : OK 6) STAMPS : OK 7) advanced-moose-class : OK 8) app-gitgitr : OK 9) app-gitgot : OK Dirty 1) App-Amylase : OK 2) Git-Wrapper : OK 3) HiD : OK 4) Perl-Build : OK 5) Perl-Critic : OK 6) STAMPS : OK Ahead by 1 7) advanced-moose-class : OK 8) app-gitgitr : OK 9) app-gitgot : OK Ahead by 1 it'll also tell you if you have local commits that haven't been pushed to the remote yet

Slide 87

Slide 87 text

got st -q finally, you can use the '-q' switch to hide the "uninteresting" stuff

Slide 88

Slide 88 text

got st -q 3) HiD : Dirty 6) STAMPS : OK Ahead by 1 Dirty Ahead by 1 which in this case is all the repos that don't have changes and are up to date

Slide 89

Slide 89 text

got update you can also run 'git pull' across all your repos

Slide 90

Slide 90 text

got up which abbreviates to 'up' and yeah, i should have called it pull but i'm a dummy and we're stuck with it now.

Slide 91

Slide 91 text

1) App-Amylase : Up to date 2) Git-Wrapper : Up to date 3) HiD : Up to date 4) Perl-Build : Updated Updating 7f25f89..72587c8 Fast-forward lib/Perl/Build.pm | 14 +++++++++++++- script/perl-build | 14 +++++++++++++- 2 files changed, 26 insertions(+), 2 deletions(-) 5) Perl-Critic : Up to date Updated it'll do pretty much what you expect (and it supports '-q' too)

Slide 92

Slide 92 text

got update_status finally, there's a command that combines both those, because it's something i do pretty frequent -- update everything, then look at the status of everything

Slide 93

Slide 93 text

got upst surprise - you can abbreviate that one too

Slide 94

Slide 94 text

got upst -q and it also supports the '--quiet' option to only show you the interesting stuff

Slide 95

Slide 95 text

How much would you pay? so, how much would you pay?

Slide 96

Slide 96 text

Wait, don't answer yet

Slide 97

Slide 97 text

got fetch if you're not a fan of the way 'git pull' works you can also run 'git fetch' via got.

Slide 98

Slide 98 text

got push you can even do a push across all your repos at once. (personally, this strikes me as insane but somebody sent in a patch, so...)

Slide 99

Slide 99 text

got gc you can garbage collect all your repos

Slide 100

Slide 100 text

got this at some point, somebody sent in a patch to add 'got this' -- which tells you if the current directory is under got control

Slide 101

Slide 101 text

got that this provoked somebody else to send in a 'got that' command, which does the same thing, but takes a path to check

Slide 102

Slide 102 text

got chdir finally, there are a number of commands that help you jump to the directory of a project. got chdir

Slide 103

Slide 103 text

got cd also spelled 'got cd', will change your current working directory to the given repo (note that this is one of the few got commands that requires a single repo)

Slide 104

Slide 104 text

got tmux we also have tmux integration -- 'got tmux' will open a new tmux window with the working directory in your repo. this _can_ be done with multiple repos. better, the tux window is persistent; as long as it's open 'got tmux' will just select the already open window, not open a new one

Slide 105

Slide 105 text

got tmux -s you can also spawn whole new tmux sessions if you prefer those to windows -- and again, those will be re-used as long as they're around

Slide 106

Slide 106 text

How much would you pay now ?

Slide 107

Slide 107 text

Good news! It's free! nothing!

Slide 108

Slide 108 text

Works on any perl from the last 5 years

Slide 109

Slide 109 text

cpan App::GitGot installation is simple

Slide 110

Slide 110 text

cpanm App::GitGot can be even simpler

Slide 111

Slide 111 text

BLATANT PUG If you're not sure what 'cpanm' is, come to my Unfrozen Paleolithic Perl Programmer talk tomorrow at 4pm! blatant plug

Slide 112

Slide 112 text

Find me at OpenWest and I'll help you install! or you can find me on the hallway track and i'll help you get it installed on your machine

Slide 113

Slide 113 text

Easy to extend

Slide 114

Slide 114 text

package App::GitGot::Command::chdir; # ABSTRACT: open a subshell in a selected project use 5.014; use App::GitGot -command; use Moo; extends 'App::GitGot::Command'; use namespace::autoclean; sub command_names { qw/ chdir cd / } sub _execute { my( $self, $opt, $args ) = @_; unless ( $self->active_repos and $self->active_repos == 1 ) { say STDERR 'ERROR: You need to select a single repo'; exit(1); } my( $repo ) = $self->active_repos; chdir $repo->path or say STDERR "ERROR: Failed to chdir to repo ($!)" and exit(1); exec $ENV{SHELL}; } 1;

Slide 115

Slide 115 text

package App::GitGot::Command::chdir; # ABSTRACT: open a subshell in a selected project use Moo; extends 'App::GitGot::Command';

Slide 116

Slide 116 text

sub command_names { qw/ chdir cd / }

Slide 117

Slide 117 text

sub _execute { my( $self, $opt, $args ) = @_; unless ( $self->active_repos and $self->active_repos == 1 ) { say STDERR 'ERROR: You need to select a single repo'; exit(1); } my( $repo ) = $self->active_repos; chdir $repo->path or say STDERR "ERROR: Failed to chdir to repo ($!)" and exit(1); exec $ENV{SHELL}; }

Slide 118

Slide 118 text

Slide 119

Slide 119 text

Suggestions welcome! areas for improvement: support for other VCSen better config management tools any other crazy workflow improvement you can think of!

Slide 120

Slide 120 text

Thanks OpenWest organizers Ingy döt Net Yanick Champoux Michael Greb Rolando Pereira Chris Prather photo credits: all photos by speaker except Ingy döt Net photo - https://www.flickr.com/photos/bulknews/389986053/ and pug - https://upload.wikimedia.org/wikipedia/commons/d/d7/Sad-pug.jpg and automate yo'self - somewhere on the net

Slide 121

Slide 121 text

No content

Slide 122

Slide 122 text

Questions? https://joind.in/event/openwest-2016/automate-yoself as i said, i'm here all week and i *love* to talk to people about this productivity type stuff, so grab me on the hallway track. i'm friendly.