$30 off During Our Annual Pro Sale. View Details »

DevOpsDays Cuba 2017: Ignite - Build and install scientific software with EasyBuild in HPC systems

DevOpsDays Cuba
October 26, 2017
300

DevOpsDays Cuba 2017: Ignite - Build and install scientific software with EasyBuild in HPC systems

Author: Javier Antonio Ruiz Bosch
Summary:

DevOpsDays Cuba

October 26, 2017
Tweet

Transcript

  1. Universidad Central "Marta Abreu" de Las Villas
    Dirección de Informatización
    Build and install
    scientific software
    with EasyBuild in
    HPC systems
    www.uclv.hpc.cu
    Ing. Javier Antonio Ruiz Bosch

    View Slide

  2. What is High Performance Computing?
    www.uclv.hpc.cu

    View Slide

  3. HPC software environments can be quite diverse
    HPC System
    Diverse
    users base
    Diverse
    software
    enviroment

    View Slide

  4. Scientific software installations for a diverse user
    base can be
    • Tedious
    • Repetitive
    • Error-prone
    • Time-consuming
    Only source code
    Installation procedure
    Scientific software

    View Slide

  5. To maintain the software in an HPC environment,
    a tool is required that offers:
    • Flexibility
    • Co-existence of versions.

    View Slide

  6. To maintain the software in an HPC environment,
    a tool is required that offers:
    • Dependency handling.
    • Sharing implementations of installation procedures

    View Slide

  7. Introducing EasyBuild
    http://easybuilders.github.io/easybuild/

    View Slide

  8. What is EasyBuild?
    http://easybuilders.github.io/easybuild/

    View Slide

  9. Installation procedure broken down into steps, as
    performed by EasyBuild
    1: read easyconfig
    2: fetch sources
    3: check readinees
    4: unpack sources
    5: apply patches
    6: prepare
    7: configure build
    8: build
    9: test
    11: extensions
    12: sanity check
    13: cleanup
    14: env. module
    15: test cases
    10: install

    View Slide

  10. View Slide

  11. Lmod: based on Lua, provides a convenient way to dynamically change the
    user environment through modules. This includes easily adding or removing
    directories to environment variablesPATH, LIBRARY_PATH,
    LD_LIBRARY_PATH, etc.
    prepend_path("CPATH","/opt/easybuild/software/GCCcore/6.3.0/include")
    prepend_path("LD_LIBRARY_PATH","/opt/easybuild/software/GCCcore/6.3.0/lib")
    prepend_path("LD_LIBRARY_PATH","/opt/easybuild/software/GCCcore/6.3.0/lib64")
    prepend_path("LIBRARY_PATH","/opt/easybuild/software/GCCcore/6.3.0/lib")
    prepend_path("LIBRARY_PATH","/opt/easybuild/software/GCCcore/6.3.0/lib64")
    prepend_path("MANPATH","/opt/easybuild/software/GCCcore/6.3.0/share/man")
    prepend_path("PATH","/opt/easybuild/software/GCCcore/6.3.0/bin")
    $ module load GCC
    https://www.tacc.utexas.edu/research-development/tacc-projects/lmod
    HPC: Environment modules (Lmod)

    View Slide

  12. $ module load
    $ module unload
    $ module avail
    $ module spider
    $ module whatis
    $ module list
    $ module purge
    $ module show
    HPC: Environment modules (Lmod)

    View Slide

  13. www.uclv.hpc.cu
    Configuration:
    • /etc/easybuild.d/*.cfg, global configuration file for
    EasyBuild.
    modules-tool: Lmod
    prefix: /opt/easybuild
    sourcepath: /opt/easybuild
    repositorypath: /opt/develop/easybuild-easyconfig
    Basic usage and configuration

    View Slide

  14. eb --show-default-configfiles
    $ mkdir -p $HOME/.easybuild
    $ eb --confighelp > $HOME/.easybuild/config.cfg
    (available since EasyBuild v2.1.0)
    www.uclv.hpc.cu
    Basic usage and configuration

    View Slide

  15. The user can install their own applications using the Easybuild module or
    using the traditional Linux compilation / installation mode.
    $ module load EasyBuild
    $ eb GROMACS-2016.3-foss-2016b-GPU-enabled-build -r
    www.uclv.hpc.cu
    $ eb GROMACS-2016.3-foss-2016b -r
    Basic usage and configuration

    View Slide

  16. View Slide

  17. View Slide

  18. Related Work
    Traditional packages
    management tools
    • RPM
    • yum
    • APT
    Ports frameworks
    • Portage
    • Pacman
    • Pkgutils

    View Slide

  19. Related Work
    Environment modules
    • TCL/C
    • TCL-variante única
    • LMOD
    Modern packages management
    tools
    • Nix
    • Maali
    • SWTools
    • Smithy
    • Spack
    • EasyBuild

    View Slide

  20. Well-known supercomputing centers with EasyBuild
    ugent.be/hpc
    www.fz-juelich.de
    www.viennabiocenter.org
    scicore.unibas.ch

    View Slide