HPC Crash Course

Crash course for using the ILRI, Kenya research computing cluster. Target audience was new biologists who have Illumina MiSeq data to analyze on the cluster, but were completely new to Linux (and come from a Windows/non-technical background).

1. HPC Crash Course April, 2014
2. (you can’t do it on your PC) Research computing in a nutshell
3. (you can’t do it on Windows) Research computing in a nutshell
4. You need Linux power!
5. ILRI Magic Research Computing Cloud
6. ILRI Not-Magic Research Computing Cloud
7. ILRI Not-Magic Research Computing Cloud: demystified You HPC
8. Brief intro to Linux commands cd – change directory mkdir – make directory mv – move a file or directory cp – copy a file or folder rm - remove a file whoami – print the name of the current user who – print a list of other users who are logged in date – print the current date and time on the server echo – print text to the screen
9. Linux command structure Linux commands come in various forms. Some are simple, and can be used by themselves: ls rm date
10. Linux command structure ... other times you can add “arguments” to change the behavior of the command. Arguments are separated by one or more spaces: mkdir data Some commands require arguments (they don’t make sense to run by themselves, like mkdir).
11. Common pitfalls with Linux commands - Case sensitive (Ls vs ls)! - Attention to detail (ls -l vs ls --l)! - Pasting from Word (“pretty” quotes, etc)! - Missing spaces - Using Windows-isms (\ vs /)! Google is your friend!
12. Basic HPC Workflow - Log into hpc.ilri.cgiar.org - Request a resource allocation - ??? - Profit
13. Basic HPC Workflow Demonstrated [aorth@noma: ~]$ ssh hpc.ilri.cgiar.org [aorth@hpc: ~]$ interactive salloc: Granted job allocation 5462 [aorth@compute2: ~]$
14. SLURM: What you need to know - SLURM is the Simple Linux Utility for Resource Management - Manages CPUs and jobs on the cluster - Ensures everyone plays fair - Everyone gets ~20 CPUs by default - More info: http://hpc.ilri.cgiar.org/using-slurm Some commands: interactive - allocates resources for interactive shell (1 CPU) squeue - see a list of running / queued jobs scancel - cancel a job sbatch - submit a “batch” job (ie, not interactive)
15. Accessing Software on HPC - Lots of software installed here! - Here’s a pretty up-to-date list: http://hpc.ilri.cgiar.org/list-of-software - Also, see the list using modules: module avail
16. Accessing Software on HPC You can load a specific version of software module load blast/2.2.29+ blastn -query alan.fna -db nt List and unload modules with: module unload blast
17. Rules of HPC Club - Use SLURM (interactive is your friend) - Try not to write to your home folder - Don’t run jobs on the head node (HPC) - If you crash the cluster you buy cookies for Alan - Don’t talk about HPC Club