GNU Guix: vers la reproductibilité computationnelle

Présentation faite lors de la journée #BlueHats du Paris Open Source Summit 2019.

Transcript

1. GNU Guix: vers la reproductibilit´ e computationnelle Simon Tournier simon.tournier@univ-paris-diderot.fr BlueHats, Paris, 11 D´ ecembre 2019 Ing. Rech. Calcul Sci. (+un peu adminsys) Analyses avec biologistes/m´ edecins (avant) Simul. num. (fem, bem, sys. lin.)
2. R´ esultat scientifique = exp´ erience + traitement num´ erique exp´ erience : comment produire la donn´ ee ≈d´ ecrit trait. num. : quels logiciels, param` etres, versions, etc. ? Reproductibilit´ e scientifique ` a l’´ ere num´ erique : 1. Articles accessibles HAL, ReScience 2. Donn´ ees ouvertes Zenodo 3. Code source ouvert Forges, Software Heritage 4. Environnement d’ex´ ecution ? et comment lier tout ca? reproductibilit´ e = v´ erification r´ eplicabilit´ e = validation R. Di Cosmo@Scibian2016 (lien) K. Hinsen@Aramis2019 (lien)
3. Scenarii concrets Alice a utilis´ e les outils num´ eriques foo-1.2, bar-3.4 et baz-5.6. Carole travaille avec Alice . . . mais utilise aussi les versions foo-7.8, bar-9.0. Charlie a mis ` a jour son syst` eme et plus rien ne fonctionne. Bob utilise les mˆ emes versions qu’Alice . . . mais n’obtient pas les mˆ emes r´ esultats. Dan veut refaire l’analyse des mois (ann´ ees?) plus tard . . . mais rencontre le dependencies hell 1. Solution(s) 1. gestionnaire de paquets : apt, yum, etc. 2. environnement virtuel : conda, modulefiles, etc. 3. conteneur : Docker, Singularity, etc. Guix 1. au mieux
4. Guix en r´ esum´ e 1. outil de gestion de paquets : paradigme fonctionnel et transactionnel 2. gestionnaire d’environnements 3. permet g´ en´ erer des conteneurs 4. fournit des outils (biblio. Scheme) pour adapter ` a ses propres sp´ ecificit´ es Contrˆ ole (fin) sur la chaˆ ıne de d´ ependances Reproductibilit´ e binaire (du moins permet de la tracer) Bootstrap (binairen ← source + binairen−1)
5. # Alice guix install foo@1.2 bar@3.4 baz@5.6 # -m manifest.scm guix describe -f channels > alice-conf.scm # Carole guix install --profile=./avec-alice foo@1.2 bar@3.4 baz@5.6 guix install foo@7.8 bar@9.0 # Charlie guix package --roll-back # generations # Bob guix pull --channels=alice-conf.scm guix pack -f docker ... # -f squashfs # Dan guix time-machine -C alice-conf.scm -- install foo@1.2 bar@3.4 baz@5.6
6. GNU Guix Reference Card for version 1.0.1 https://gnu.org/software/guix/ Getting Started To read the on-line documentation run info guix or visit https://gnu.org/s/guix/manual/en/html_node. See https://emacs-guix.gitlab.io/website/ for an Emacs interface to Guix. Specifying Packages Most commands take a “package specification”denoted spec in the sequel. Here are some examples: emacs Emacs package, latest version gcc-toolchain@7 GCC toolchain,version 7.x gcc-toolchain:debug latest GCC toolchain,debug- ging symbols Managing Packages guix search regexp … search for packages guix show spec show package info guix install spec… install packages guix upgrade [regexp] upgrade packages guix remove name… remove packages guix package -m file instantiatefrom manifest guix package --roll-back roll back guix package -l list profile generations guix package --search-paths display search paths guix package -p profile … use a different profile Manifests guix package -m and other commands take a “manifest”file listing packages of interest, along these lines: (specifications->manifest ’("gcc-toolchain@7" "gcc-toolchain@7:debug" "openmpi")) One-Off Environments guix environment --ad-hoc spec… environment containing spec… guix environment python environment to develop Python itself guix environment --ad-hoc python -C -- python3 run Python in a container guix environment -m file create an environment for the packages in manifest file Updating Guix guix describe describe current Guix guix pull update Guix guix pull -l view history guix pull --commit=commit update to commit guix pull --branch=branch update to branch guix pull -C file update the given channels Channel Specifications Channels specify Git repositorieswhere guix pull looks for updates to Guix and external package repositories. By default guix pull reads ~/.config/guix/channels.scm; with -C it can take channel specificationsfrom a user-supplied file that looks like this: (cons (channel (name ’guix-hpc) (url "https://gitlab.inria.fr/guix-hpc/guix-hpc.git") (branch "master")) %default-channels) Managing Storage Space guix gc collect all garbage guix gc -C nG collect n GB of garbage guix gc -F nG ensure n GB are available guix gc -d duration delete generationsolder than duration—e.g., 1m for one month guix size spec … view package size guix gc -R /gnu/store/… list run-time dependencies guix graph -t references spec … view run-time dependencies Customizing Packages guix command name --with-source=name=source build name with a different source URL guix command spec --with-input=spec1=spec2 replace spec1with spec2 in the dependency graph of spec guix command spec --with-graft=spec1=spec2 graft spec2 in lieu of spec1in spec guix command --with-git-url=spec=URL build spec from the given Git URL guix command spec --with-branch=branch build spec from the given Git branch guix command spec --with-commit=commit build spec from the given Git commit Developing Packages guix edit spec view the definition guix build spec … build packages guix build --log-file spec view the build log guix build -K spec … build packages, keep build trees on failure guix build -S spec obtain the source of spec guix build --check spec rebuild a package guix build --target=triplet … cross-compile to triplet—e.g., arm-linux-gnueabihf guix download URL download from URL and print its SHA256 hash guix hash file print the hash of file guix graph spec | dot -Tpdf … view dependencies guix refresh spec update package definition guix import repo name import name from repo Creating Application Bundles guix pack spec … create a tarball guix pack -f docker spec … create a Docker image guix pack -f squashfs spec … create a Singularity image guix pack -RR spec … create a relocatabletarball guix pack -S /bin=bin spec … make /bin a symlink to the packages’bin directory guix pack -m file bundle the packages from the manifest in file
7. D´ eploiement sur Clusters Max Delbr¨ uck Center (DE) : 250-node + workstations UMC Utrecht (NL) : 68-node (1,000+ cores) University of Queensland (AU) : 20-node (900 cores) PlaFRIM Inria Bordeaux (FR) : 120-node (3000+ cores) GriCAD Grenoble (FR) : 72-node (1000+ cores) CCIPL Nantes (FR) : 230-node (4000+ cores) (le votre?) https://guix-hpc.bordeaux.inria.fr
8. Survol technique guix build pastix isolated build : chroot, separate name spaces, etc.
9. Survol technique guix build pastix --with-source=./my-pastix.tar.gz isolated build : chroot, separate name spaces, etc.
10. Survol technique guix build pastix --with-input=lapack=my-lapack isolated build : chroot, separate name spaces, etc.
11. Survol technique guix build pastix /gnu/store/ h2g4sf72... -pastix-6.0.2 hash de toutes les d´ ependances
12. Survol technique guix build pastix /gnu/store/ h2g4sf72... -pastix-6.0.2 guix gc --references /gnu/store/...-pastix-6.0.2 /gnu/store/3fa68b5...-glibc-2.24 /gnu/store/bd9c330...-gcc-4.9.3-lib /gnu/store/ca175fd...-pastix-6.0.2
13. Survol technique guix build pastix /gnu/store/ h2g4sf72... -pastix-6.0.2 guix gc --references /gnu/store/...-pastix-6.0.2 /gnu/store/3fa68b5...-glibc-2.24 /gnu/store/bd9c330...-gcc-4.9.3-lib /gnu/store/ca175fd...-pastix-6.0.2 (presque) bit-identique pour tout le monde
14. (define pastix (package (name "pastix") (home-page "https://gitlab.inria.fr/solverstack/pastix") (source (origin (method git-fetch) (uri (git-reference (url home-page) (commit "2f30ff07a") (recursive? #t))) (sha256 (base32 "106rf402cvfdhc2yf...")))) ...)) SWH and Guix join forces to enable long term reproductibility (lien)
15. Dans un monde id´ eal
16. (Venez (partager (les (parenth` eses!)))) installez https://.../guix-install.sh utilisez, rapportez des probl` emes, empaquetez https://issues.guix.gnu.org/ partagez vos id´ ees! help-guix@gnu.org #guix (irc.freenode.net)
17. help-guix@gnu.org https://guix.gnu.org
18. Copyright c 2019 Simon Tournier zimon.toutoune@gmail.com. GNU Guix logo, CC-BY-SA 4.0, https://gnu.org/s/guix/graphics GNU Guix Reference Card under GFDL 1.3+. Copyright of other images included in this document is held by their respective owners; especially from Ludovic Court` es. This work is licensed under the Creative Commons Attribution-Share Alike 3.0 License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA. At your option, you may instead copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is available at https://www.gnu.org/licenses/gfdl.html. The source of this document is available from https://git.sv.gnu.org/cgit/guix/maintenance.git.