DevOpsDays Cuba 2017: Ignite - Build and install scientific software with EasyBuild in HPC systems

Author: Javier Antonio Ruiz Bosch
Summary:

Transcript

1. Universidad Central "Marta Abreu" de Las Villas
   Dirección de Informatización
   Build and install
   scientific software
   with EasyBuild in
   HPC systems
   www.uclv.hpc.cu
   Ing. Javier Antonio Ruiz Bosch
   

   View Slide

2. What is High Performance Computing?
   www.uclv.hpc.cu
   

   View Slide

3. HPC software environments can be quite diverse
   HPC System
   Diverse
   users base
   Diverse
   software
   enviroment
   

   View Slide

4. Scientific software installations for a diverse user
   base can be
   • Tedious
   • Repetitive
   • Error-prone
   • Time-consuming
   Only source code
   Installation procedure
   Scientific software
   

   View Slide

5. To maintain the software in an HPC environment,
   a tool is required that offers:
   • Flexibility
   • Co-existence of versions.
   

   View Slide

6. To maintain the software in an HPC environment,
   a tool is required that offers:
   • Dependency handling.
   • Sharing implementations of installation procedures
   

   View Slide

7. Introducing EasyBuild
   http://easybuilders.github.io/easybuild/
   

   View Slide

8. What is EasyBuild?
   http://easybuilders.github.io/easybuild/
   

   View Slide

9. Installation procedure broken down into steps, as
   performed by EasyBuild
   1: read easyconfig
   2: fetch sources
   3: check readinees
   4: unpack sources
   5: apply patches
   6: prepare
   7: configure build
   8: build
   9: test
   11: extensions
   12: sanity check
   13: cleanup
   14: env. module
   15: test cases
   10: install
   

   View Slide

10. View Slide

11. Lmod: based on Lua, provides a convenient way to dynamically change the
    user environment through modules. This includes easily adding or removing
    directories to environment variablesPATH, LIBRARY_PATH,
    LD_LIBRARY_PATH, etc.
    prepend_path("CPATH","/opt/easybuild/software/GCCcore/6.3.0/include")
    prepend_path("LD_LIBRARY_PATH","/opt/easybuild/software/GCCcore/6.3.0/lib")
    prepend_path("LD_LIBRARY_PATH","/opt/easybuild/software/GCCcore/6.3.0/lib64")
    prepend_path("LIBRARY_PATH","/opt/easybuild/software/GCCcore/6.3.0/lib")
    prepend_path("LIBRARY_PATH","/opt/easybuild/software/GCCcore/6.3.0/lib64")
    prepend_path("MANPATH","/opt/easybuild/software/GCCcore/6.3.0/share/man")
    prepend_path("PATH","/opt/easybuild/software/GCCcore/6.3.0/bin")
    $ module load GCC
    https://www.tacc.utexas.edu/research-development/tacc-projects/lmod
    HPC: Environment modules (Lmod)
    

    View Slide

12. $ module load
    $ module unload
    $ module avail
    $ module spider
    $ module whatis
    $ module list
    $ module purge
    $ module show
    HPC: Environment modules (Lmod)
    

    View Slide

13. www.uclv.hpc.cu
    Configuration:
    • /etc/easybuild.d/*.cfg, global configuration file for
    EasyBuild.
    modules-tool: Lmod
    prefix: /opt/easybuild
    sourcepath: /opt/easybuild
    repositorypath: /opt/develop/easybuild-easyconfig
    Basic usage and configuration
    

    View Slide

14. eb --show-default-configfiles
    $ mkdir -p $HOME/.easybuild
    $ eb --confighelp > $HOME/.easybuild/config.cfg
    (available since EasyBuild v2.1.0)
    www.uclv.hpc.cu
    Basic usage and configuration
    

    View Slide

15. The user can install their own applications using the Easybuild module or
    using the traditional Linux compilation / installation mode.
    $ module load EasyBuild
    $ eb GROMACS-2016.3-foss-2016b-GPU-enabled-build -r
    www.uclv.hpc.cu
    $ eb GROMACS-2016.3-foss-2016b -r
    Basic usage and configuration
    

    View Slide

16. View Slide

17. View Slide

18. Related Work
    Traditional packages
    management tools
    • RPM
    • yum
    • APT
    Ports frameworks
    • Portage
    • Pacman
    • Pkgutils
    

    View Slide

19. Related Work
    Environment modules
    • TCL/C
    • TCL-variante única
    • LMOD
    Modern packages management
    tools
    • Nix
    • Maali
    • SWTools
    • Smithy
    • Spack
    • EasyBuild
    

    View Slide

20. Well-known supercomputing centers with EasyBuild
    ugent.be/hpc
    www.fz-juelich.de
    www.viennabiocenter.org
    scicore.unibas.ch
    

    View Slide