$30 off During Our Annual Pro Sale. View Details »

HPC Crash Course

HPC Crash Course

Crash course for using the ILRI, Kenya research computing cluster. Target audience was new biologists who have Illumina MiSeq data to analyze on the cluster, but were completely new to Linux (and come from a Windows/non-technical background).

Alan Orth

April 30, 2014
Tweet

More Decks by Alan Orth

Other Decks in Technology

Transcript

  1. HPC Crash Course
    April, 2014

    View Slide

  2. (you can’t do it on your PC)
    Research computing in a nutshell

    View Slide

  3. (you can’t do it on Windows)
    Research computing in a nutshell

    View Slide

  4. You need Linux power!

    View Slide

  5. ILRI Magic Research Computing Cloud

    View Slide

  6. ILRI Not-Magic Research Computing Cloud

    View Slide

  7. ILRI Not-Magic Research Computing Cloud: demystified
    You
    HPC

    View Slide

  8. Brief intro to Linux commands
    cd – change directory
    mkdir – make directory
    mv – move a file or directory
    cp – copy a file or folder
    rm - remove a file
    whoami – print the name of the current user
    who – print a list of other users who are logged in
    date – print the current date and time on the server
    echo – print text to the screen

    View Slide

  9. Linux command structure
    Linux commands come in various forms. Some
    are simple, and can be used by themselves:
    ls
    rm
    date

    View Slide

  10. Linux command structure
    ... other times you can add “arguments” to change the
    behavior of the command.
    Arguments are separated by one or more spaces:
    mkdir data
    Some commands require arguments (they don’t
    make sense to run by themselves, like mkdir).

    View Slide

  11. Common pitfalls with Linux
    commands
    - Case sensitive (Ls vs ls)!
    - Attention to detail (ls -l vs ls --l)!
    - Pasting from Word (“pretty” quotes, etc)!
    - Missing spaces
    - Using Windows-isms (\ vs /)!
    Google is your friend!

    View Slide

  12. Basic HPC Workflow
    - Log into hpc.ilri.cgiar.org
    - Request a resource allocation
    - ???
    - Profit

    View Slide

  13. Basic HPC Workflow Demonstrated
    [aorth@noma: ~]$ ssh hpc.ilri.cgiar.org
    [aorth@hpc: ~]$ interactive
    salloc: Granted job allocation 5462
    [aorth@compute2: ~]$

    View Slide

  14. SLURM: What you need to know
    - SLURM is the Simple Linux Utility for Resource Management
    - Manages CPUs and jobs on the cluster
    - Ensures everyone plays fair
    - Everyone gets ~20 CPUs by default
    - More info: http://hpc.ilri.cgiar.org/using-slurm
    Some commands:
    interactive - allocates resources for interactive shell (1 CPU)
    squeue - see a list of running / queued jobs
    scancel - cancel a job
    sbatch - submit a “batch” job (ie, not interactive)

    View Slide

  15. Accessing Software on HPC
    - Lots of software installed here!
    - Here’s a pretty up-to-date list:
    http://hpc.ilri.cgiar.org/list-of-software
    - Also, see the list using modules:
    module avail

    View Slide

  16. Accessing Software on HPC
    You can load a specific version of software
    module load blast/2.2.29+
    blastn -query alan.fna -db nt
    List and unload modules with:
    module unload blast

    View Slide

  17. Rules of HPC Club
    - Use SLURM (interactive is your friend)
    - Try not to write to your home folder
    - Don’t run jobs on the head node (HPC)
    - If you crash the cluster you buy cookies for Alan
    - Don’t talk about HPC Club

    View Slide