Using Vagrant to Distribute "Research Code" to Colleagues

Presentation courtesy of Frank Horowitz.

Problem: You have some code that other colleagues want to *use* (or maybe even co-develop) but your code is composed of many moving parts, and dll-hades awaits on multiple versions of multiple operating systems and software environments. You don’t have time to figure out *how* to install your system on all of: Macs (via homebrew or macports or native versions or whatever) or Windows Boxes (cygwin? native?) or Linux boxes (Centos? Debian? Ubuntu? Suse? and which version?) Combinatorial explosions!

Presented at SSW: https://cornell-ssw.github.io/meetings/2017-02-27

Transcript

1. USING VAGRANT TO DISTRIBUTE YOUR “RESEARCH CODE” TO COLLEAGUES Frank Horowitz, EAS frank.horowitz@cornell.edu 27 Feb. 2017
2. Overview •  The installation is long, with lots of data to download. •  We’ll start it first, then I’ll give a motivating spiel while everyone is waiting for the installation to finish. •  Then, I’ll help troubleshoot the inevitable problems •  Then, we’ll run a computation on your new installed virtual machine, showing the results via an IPython notebook from a browser in your host machine.
3. Starting the Installation •  You need to have on your host machine (laptop?) •  git •  SourceTree is a great version for Windows: •  https://www.sourcetreeapp.com/ •  git-lfs •  Packaged with SourceTree for Windows. •  vagrant •  virtualbox •  A browser •  (A text editor?)
4. Preparing git •  Superseded because I’m nearing my git-lfs download quota. •  You need to install git-lfs •  Instructions: https://help.github.com/articles/installing-git-large-file-storage/ •  Briefly •  On Linux: try installing with your package manager (apt, yum, etc.) •  On a Mac: •  Homebrew: brew install git-lfs •  Macports: port install git-lfs •  On Windows: •  Use SourceTree. It Just Works™ •  If you insist on doing it the hard way, download package from under above link •  Then follow those instructions •  After package is installed, tell git about it: •  git lfs install
5. Preparing Vagrant •  Hopefully you’ve already preinstalled vagrant, as requested in the email •  If not, please go to https://www.vagrantup.com/downloads.html and grab the appropriate installer for your environment, and follow the instructions •  Alternatively, on Macs, if you use Homebrew you can install it with: •  brew cask install vagrant •  You need to tell vagrant to use the virtualbox guest and proxy plugins •  vagrant plugin install vagrant-proxyconf •  vagrant plugin install vagrant-vbguest
6. Installing Virtualbox •  Hopefully, you’ve already installed this, as requested in the email! •  If not, please go to https://www.virtualbox.org/wiki/Downloads and grab the appropriate package for your environment. Then follow the appropriate installation instructions. •  Alternatively, on Macs, if you use Homebrew you can install it with: •  brew cask install virtualbox
7. Starting Up •  On your host, make a new empty directory wherever it’s convenient for this project to live under… •  Then from within that directory (details vary with your "host" machine's operating system) run: •  git clone https://git@github.com:korin-worm-code/GravityTomo.git •  In the newly created directory execute: •  git lfs pull •  That gets the large data files into your local directory. •  See next slide for quota workaround •  Now, let’s switch to the appropriate git branch (named vagrant): •  git checkout vagrant
8. Git-lfs quota workaround •  I’ve uploaded the ~250Mb file to box to avoid my git-lfs quota. •  Please download the following to your working directory: •  https://cornell.box.com/s/bi90dlqdly5fytumadvpu12ut0f5ly5y •  Shorter: http://bit.ly/2lhH0vn •  Make sure the result is named: •  EGM2008_to2190_ZeroTide.shm
9. Let’s Run Vagrant! •  Make sure you have already run •  vagrant plugin install vagrant-vbguest •  This only needs to happen once per vagrant installation, not every time you run vagrant itself. •  In your working directory, execute: •  vagrant up •  That builds and configures a virtual machine for the running environment. This takes a while, and is bandwidth heavy as it is downloading and upgrading an entire Ubuntu distribution, as well as grabbing all of the software dependencies. •  Now cross your fingers and go have more pizza!
10. Motivation (While we wait…) •  This is a workable solution for something that can run on a laptop or a workstation •  It is likely inappropriate for a supercomputing environment. •  Unless, there are some workstation tools that are a major pain for your users to install.
11. The Pain of Distributing Your Code •  Even with the proper tools (such as git/github) distributing your “research code” can be a major pain. •  You built, and configured it for your environment •  Your colleagues have their own environment, and quite naturally want to run your code in there! •  This creates a Tower of Babel situation for installation, since every software environment on every platform has its own quirks and it will drive you mad trying to keep on top of all of it. •  I once spent the entire afternoon of the last day of a meeting helping a Australian colleague – who happened to use Macports instead of Homebrew – to install some of my software on his laptop. •  Even though we both had Macs, were face-to-face, in a bandwidth rich environment, we failed to get him a working installation •  Imagine trying that via skype across timezones, or even language barriers, and you’ll get a taste of the problem.
12. A Workable Solution (1) •  This talk is about using Vagrant to ameliorate that pain. •  The idea is that you can install your software once-and-only- once in an environment that you have control over, and then distribute that. •  There are warts! •  You still must be able to successfully install into your guest-machine environment. •  I chose Ubuntu here because I’m fairly comfortable administering it. •  You might chose something else… •  You must also configure and install Vagrant and Virtualbox, which have their own idiosyncrasies. •  Fortunately, there are release engineers working on those systems, and there are lots of places where troubleshooting such installations happens on the web. •  Googling your error messages can get you a very long way towards solving such problems!
13. A Workable Solution (2) •  I chose to use Vagrant and VirtualBox because they •  Run on (at least) 3 major platforms •  Windows, Mac, and Linux (I haven’t checked, but maybe the BSDs too?) •  Are gratis (“free-as-in-beer”) •  But are not necessarily Libre (“free-as-in-speech”) •  Read the licenses! •  This Vagrantfile is configured to have: •  Guest machine disk storage via a file in your host machine •  That’s your “system disk” •  Your working directory is shared between your host and your guest. •  That’s where you keep all of your code and data. You can work with it in either the host or guest machine side. •  File locking? I don’t know. Try the experiment! But my inclination is to not muck with it from the host side while the guest is working with files.
14. A Workable Solution (3) •  You have not entirely escaped installing your code on a (potentially) foreign system either. •  You still need to install in on Ubuntu (or whatever guest environment you choose). •  In Vagrant that is all configured in a single file in your working directory: Vagrantfile
15. Let’s look at the Vagrantfile •  Text editor work; read the comments.
16. Did Everybody’s “vagrant up” Work? •  Holler if you need help troubleshooting! •  (And be happy you aren’t doing this from behind a slow DSL line like I was this last weekend! ;-) •  Time passes.
17. Vagrant Cheat Sheet •  These commands are the primary ones you need to know •  vagrant up •  The very first time it is run, configures a new virtual machine according to the Vagrantfile in your working directory •  Every other time you execute it, it simply boots the virtual machine •  vagrant provision •  This runs the shell script at the top of our Vagrantfile •  Useful if you need to change something in the config, or to try to recover from an installation error •  vagrant halt •  Shuts down your virtual machine •  vagrant ssh •  Gives you a shell with sudo privileges on your virtual machine •  vagrant destroy •  When you need to start from a clean slate, this destroys everything in the virtual machine, so you can start over
18. Let’s Work with the IPython Notebook •  From a browser on your host machine, navigate to: •  localhost:8888 •  You should connect to a tree view of your local notebooks •  Holler if it’s not working! •  Now click on GeoidPlotter.ipynb •  Follow along with the demo
19. When You’re Done •  “close and halt” from the Ipython notebook file menu •  Close the “Home” directory tree browser tab •  From the working directory in your host machine, execute: •  vagrant halt •  You’re done until the next time you want to use the guest machine. •  The entire system is under git control from the working directory in your host machine. Point your colleagues at a git repository (on GitHub?) containing that work, give them similar instructions to those found in this slide deck, and they can work away with your code. •  Hopefully, they will contribute back!
20. That’s All I’ve Got •  Thanks!