HPC Crash Course

Crash course for using the ILRI, Kenya research computing cluster. Target audience was new biologists who have Illumina MiSeq data to analyze on the cluster, but were completely new to Linux (and come from a Windows/non-technical background).

Transcript

1. HPC Crash Course
   April, 2014
   

   View Slide

2. (you can’t do it on your PC)
   Research computing in a nutshell
   

   View Slide

3. (you can’t do it on Windows)
   Research computing in a nutshell
   

   View Slide

4. You need Linux power!
   

   View Slide

5. ILRI Magic Research Computing Cloud
   

   View Slide

6. ILRI Not-Magic Research Computing Cloud
   

   View Slide

7. ILRI Not-Magic Research Computing Cloud: demystified
   You
   HPC
   

   View Slide

8. Brief intro to Linux commands
   cd – change directory
   mkdir – make directory
   mv – move a file or directory
   cp – copy a file or folder
   rm - remove a file
   whoami – print the name of the current user
   who – print a list of other users who are logged in
   date – print the current date and time on the server
   echo – print text to the screen
   

   View Slide

9. Linux command structure
   Linux commands come in various forms. Some
   are simple, and can be used by themselves:
   ls
   rm
   date
   

   View Slide

10. Linux command structure
    ... other times you can add “arguments” to change the
    behavior of the command.
    Arguments are separated by one or more spaces:
    mkdir data
    Some commands require arguments (they don’t
    make sense to run by themselves, like mkdir).
    

    View Slide

11. Common pitfalls with Linux
    commands
    - Case sensitive (Ls vs ls)!
    - Attention to detail (ls -l vs ls --l)!
    - Pasting from Word (“pretty” quotes, etc)!
    - Missing spaces
    - Using Windows-isms (\ vs /)!
    Google is your friend!
    

    View Slide

12. Basic HPC Workflow
    - Log into hpc.ilri.cgiar.org
    - Request a resource allocation
    - ???
    - Profit
    

    View Slide

13. Basic HPC Workflow Demonstrated
    [aorth@noma: ~]$ ssh hpc.ilri.cgiar.org
    [aorth@hpc: ~]$ interactive
    salloc: Granted job allocation 5462
    [aorth@compute2: ~]$
    

    View Slide

14. SLURM: What you need to know
    - SLURM is the Simple Linux Utility for Resource Management
    - Manages CPUs and jobs on the cluster
    - Ensures everyone plays fair
    - Everyone gets ~20 CPUs by default
    - More info: http://hpc.ilri.cgiar.org/using-slurm
    Some commands:
    interactive - allocates resources for interactive shell (1 CPU)
    squeue - see a list of running / queued jobs
    scancel - cancel a job
    sbatch - submit a “batch” job (ie, not interactive)
    

    View Slide

15. Accessing Software on HPC
    - Lots of software installed here!
    - Here’s a pretty up-to-date list:
    http://hpc.ilri.cgiar.org/list-of-software
    - Also, see the list using modules:
    module avail
    

    View Slide

16. Accessing Software on HPC
    You can load a specific version of software
    module load blast/2.2.29+
    blastn -query alan.fna -db nt
    List and unload modules with:
    module unload blast
    

    View Slide

17. Rules of HPC Club
    - Use SLURM (interactive is your friend)
    - Try not to write to your home folder
    - Don’t run jobs on the head node (HPC)
    - If you crash the cluster you buy cookies for Alan
    - Don’t talk about HPC Club
    

    View Slide