Slide 1

Slide 1 text

WRANGLING PYTHONS WITH NIX FOR REPRODUCIBLE PURITY ROHIT GOSWAMI MINSTP Created: 2021-05-29 Sat 06:49 1

Slide 2

Slide 2 text

1 BRIEF INTRODUCTION 2 . 1

Slide 3

Slide 3 text

1.1 HELLO! Find me here: Who? Rohit Goswami MInstP Doctoral Researcher, University of Iceland, Faculty of Physical Sciences https://rgoswami.me 2 . 2

Slide 4

Slide 4 text

1.2 ABOUT WORK.. A good place to work on Water!! Image from ( ) Goswami, Goswami, and Singh, n.d. 2 . 3

Slide 5

Slide 5 text

1.3 LOGISTICS All contents are hosted on GitHub Questions are welcome after / during the lecture Email me Post on the Ed Leave a comment on my site 2 . 4

Slide 6

Slide 6 text

2 LOCAL PROJECT LAYOUTS 3 . 1

Slide 7

Slide 7 text

2.1 LANGUAGE AGNOSTIC BEGINNINGS Readme.{md,org} Motivation, rationale, license, installation instructions LICENSE Plain text, and preferably an open license is pretty handy for this license-generator .gitignore Lists les which do not need to be committed; typically generated les can be used to generate these gibo $ git init # Inside project $ gibo macOS Windows Xcode Emacs \ Vim Python C \ CMake TeX > .gitignore $ touch readme.md $ license-generator MIT \ author "Person" $ tree -L 2 . ├── LICENSE ├── docs │ └── pres └── readme.org 2 directories, 2 files 3 . 2

Slide 8

Slide 8 text

2.2 PYTHON PYPI STANDARD Write functions/objects Refactor into modules Tests Fuzzy (property based, hypothesis) Unit / Integration (pytest) Documentation Push to PyPI 3 . 3

Slide 9

Slide 9 text

2.2.1 EXAMPLE: WAILORD https://wailord.xyz $ tree -L 1 . ├── AUTHORS.rst ├── CODEOWNERS ├── CONTRIBUTING.rst ├── HISTORY.rst ├── LICENSE ├── MANIFEST.in ├── Makefile ├── README.rst ├── docs ├── poetry.lock ├── pyproject.toml ├── tests ├── wailord └── wailord_templates 5 directories, 15 files 3 . 4

Slide 10

Slide 10 text

3 PACKAGING 4 . 1

Slide 11

Slide 11 text

3.1 PYTHON MODULES A .py le is a module It is standalone if it only imports from the standard library from collections import namedtuple point_xy = namedtuple('point_xy', ('x', 'y')) 4 . 2

Slide 12

Slide 12 text

3.2 PURE PYTHON PACKAGES A directory with __init__.py in it is a package Use pip """Top-level package for Wailord.""" author = """Rohit Goswami""" email = "[email protected]" version = "0.0.2" 4 . 3

Slide 13

Slide 13 text

Standard Binary 3.3 DISTRIBUTIONS Built by setuptools with setup.py Simple source only .tar.gz wheel For interoperable needs (sometimes!) Includes static libraries Distributions have zero or more packages 4 . 4

Slide 14

Slide 14 text

3.4 THE PYTHON GRADIENT From : Mahmoud Hashemi’s PyBay’17 talk Libraries and Dev tools are all we get (from PyPI) 4 . 5

Slide 15

Slide 15 text

3.5 PIP REQUIREMENTS Python System libraries Build tools Wheels don’t work for arbitrary distributions 4 . 6

Slide 16

Slide 16 text

4 DEPENDENCY RESOLUTION requirements.txt (pip) Poetry (pretty) pyproject.toml poetry.lock Pipenv (older) Pipfile + lock le Pipx (pip but for applications) Pyenv and friends 5 . 1

Slide 17

Slide 17 text

4.1 SYSTEM DEPENDENCIES Appimages Containers docker, flatpak, snapcraft Impure lesystems Anaconda 5 . 2

Slide 18

Slide 18 text

5 NIX 6 . 1

Slide 19

Slide 19 text

5.1 CURRENT SCENARIO SUMMARY 1 Python poetry, pipenv, pyenv C++ conan, vcpkg, cpm Nix is the answer!! 6 . 2

Slide 20

Slide 20 text

5.2 GENERAL WORKFLOW From here 6 . 3

Slide 21

Slide 21 text

5.3 DETAILS ( ; ) User environments (from ) Dolstra, Jonge, and Visser, n.d. Dolstra, Löh, and Pierron, n.d. the manual 6 . 4

Slide 22

Slide 22 text

5.4 RATIONALE Protects against self harm Exposes things taken for granted Enforces consistency Reliable Purely functional, no broken dependencies Reproducible Each package is in isolation How? store + hash + name + version 6 . 5

Slide 23

Slide 23 text

6 USING NIX 7 . 1

Slide 24

Slide 24 text

6.1 INSTALLATION (MULTI-USER) sh <(curl https nixos.org/nix/install) daemon Needs sudo but should not be run as root Will make build users with IDs between 30001 and 30032 along with a group ID 30000 7 . 2

Slide 25

Slide 25 text

6.2 NIX PYTHON - TRIAL I nix-shell -p 'python38.withPackages(ps: with ps; [ numpy toolz ])' Check which python is loaded Check which modules are present Check if passing -p multiple times is allowed 7 . 3

Slide 26

Slide 26 text

6.3 NIX WITH SCRIPTS #! /usr/bin/env nix-shell #! nix-shell -i python3 -p "python3.withPackages(ps: [ps.numpy])" import numpy print(numpy. version ) chmod +x nixnp.sh ./nixnp.sh 7 . 4

Slide 27

Slide 27 text

6.3.1 EXAMPLE: ASTROID POLL #!/usr/bin/env nix-shell #!nix-shell -i python3 -p "python38.withPackages(ps: [ ps.sh ])" -p liee from pathlib import Path import sh # For generic IMAP maildirs ISYNC_LABELS = ["rog32"] for isync in ISYNC_LABELS: sh.mbsync("-V",isync,_bg=True) # Gmaileer GMAIL_IDENTIFIERS = ["gmail", "ieee"] path = Path(r"/mail/") for dirs in path.iterdir(): if dirs.is_dir(): for gmi in GMAIL_IDENTIFIERS: if gmi in dirs.name: print(f"Syncing {dirs.name}") sh.gmi("sync", _cwd=dirs, _fg=True) 7 . 5

Slide 28

Slide 28 text

Figure 13: Stateless builds from 6.4 PURITY nix-shell -p python36 pure Why? What do we solve with this? https://slides.com/garbas/mozilla- all-hands-london-2016#/7/0/3 7 . 6

Slide 29

Slide 29 text

6.5 SHELL IN A FILE with import {}; let pythonEnv = python35.withPackages (ps: [ ps.numpy ps.toolz ]); in mkShell { buildInputs = [ pythonEnv which ];} What tools are we adding? What environment are we using? 7 . 7

Slide 30

Slide 30 text

6.6 NIX PYTHON EXPRESSIONS I f90wrap = self.buildPythonPackage rec { pname = "f90wrap"; version = "0.2.3"; src = pkgs.fetchFromGitHub { owner = "jameskermode"; repo = "f90wrap"; rev = "master"; sha256 = "0d06nal4xzg8vv6sjdbmg2n88a8h8df5ajam72445mhzk08yin23"; }; buildInputs = with pkgs; [ gfortran stdenv ]; The self portion is from overriding the python environment We will dispense with this later 7 . 8

Slide 31

Slide 31 text

6.7 NIX PYTHON EXPRESSIONS II propagatedBuildInputs = with self; [ setuptools setuptools-git wheel numpy ]; preConfigure = '' export F90=${pkgs.gfortran}/bin/gfortran ''; doCheck = false; doInstallCheck = false; }; More details here: https://rgoswami.me/posts/ccon-tut-nix/ propagatedBuildInputs are for the python packages 7 . 9

Slide 32

Slide 32 text

6.8 FRIENDLY NIX nix-env -i nox nox lieer Niv For pinning packages Nox Interactive package management For automatically reloading environments Lorri Mach-Nix For working with Python 7 . 10

Slide 33

Slide 33 text

6.9 PINNING NIXPKGS niv init { "nixpkgs": { "branch": "release-20.03", "description": "Nix Packages collection", "homepage": "", "owner": "NixOS", "repo": "nixpkgs", "rev": "1db42b7fe3878f3f5f7a4f2dc210772fd080e205", "sha256": "05k9y9ki6jhaqdhycnidnk5zrdzsdammbk5lsmsbz249hjhhgcgh" "type": "tarball", "url": "https github.com/NixOS/nixpkgs/archive/.tar.gz", "url_template": "https github.com///archive/ } } 7 . 11

Slide 34

Slide 34 text

7 REPLACING CONDA 8 . 1

Slide 35

Slide 35 text

7.1 I: VARIABLES let sources = import ./nix/sources.nix; pkgs = import sources.nixpkgs { }; mach-nix = import (builtins.fetchGit { url = "https github.com/DavHau/mach-nix/"; ref = "refs/tags/3.3.0"; }) { # optionally bring your own nixpkgs pkgs = pkgs; # optionally specify the python version python = "python38"; }; Note our de nition of mach-nix Best practices involve niv pinned sources 8 . 2

Slide 36

Slide 36 text

7.2 II: CUSTOMIZING PYTHON customPython = mach-nix.mkPython { requirements = builtins.readFile ./requirements.txt; providers = { _default = "nixpkgs,wheel,sdist"; pytest = "nixpkgs"; }; pkgs = pkgs; }; in pkgs.mkShell { buildInputs = with pkgs; [ customPython ]; } overrides_pre = [ (pythonSelf: pythonSuper: { pytest = pythonSuper.pytest.overrideAttrs (oldAttrs: { doCheck = false; }); f90wrap = pythonSelf.buildPythonPackage rec { }; }) ]; More details here: https://rgoswami.me/posts/mach-nix-niv- python/ 8 . 3

Slide 37

Slide 37 text

8 TESTING AND CONTINUOUS INTEGRATION 9 . 1

Slide 38

Slide 38 text

8.1 TESTING FRAMEWORKS Python has great testing frameworks pytest, hypothesis, etc. Unit tests are the rst layer Ensure each function outputs as expected Integration tests are for work ows Ensure each series of tasks connect correctly [tool.poetry.dev-dependencies] check-manifest = "*" pytest = "^4.6" pytest-datadir = "^1.3.1" @pytest.fixture(scope="session") def mult_xyz(tmpdir_factory): """Copies folders and fixes input dat = tmpdir_factory.mktemp("data" shutil.copytree(DATADIR, dat, dirs_exist_ok=True with open(f"{dat}/orcaMultxyz.yml" t = yaml.full_load(mxyz) t["xyz"] = f"{dat}/{t['xyz']}" fn = Path(dat / "omult.yml") fn.write_text(yaml.dump(t)) return fn . tree -L 2 . ├── basejob.sh ├── expmult.yml ├── orcaMultxyz.yml └── xyzdat ├── ch3.xyz ├── ch3oh_dimer.xyz ├── ch3oh_single.xyz └── h2inp.xyz 1 directory, 7 files 9 . 2

Slide 39

Slide 39 text

8.2 CONTINUOUS INTEGRATION No one likes switching computers to test MacOS, Windows (WSL often), Many Linux distributions Some tests run for a long time Less attractive locally nixpkgs can take over a day! There are far too many options nowadays Wercker, Travis CI, Shippable, GitLab CI, Github Actions Mostly transient docker or nix based systems Setup can be annoying without nix 9 . 3

Slide 40

Slide 40 text

8.3 GITHUB ACTIONS local tests act allows name: Test theme on: push: branches: [src] pull_request: branches: [src] # every day https crontab.guru/ schedule: - cron: "0 0 * * *" jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - uses: cachix/install-nix-action@v12 with: nix_path: nixpkgs=channel:nixos-unst - name: Get and initialize binary cach run: | nix-env -iA cachix -f \ https cachix.org/api/v1/install cachix use hello-friend-ng-hz - name: Test Build run: nix-shell run \ "hugo -s exampleSite themesDir= / - name: Cache Nix Results env: authToken: ${{ secrets.CACHIX_AUTH_T cachixName: hello-friend-ng-hz run: | cachix authtoken $authToken nix-store -qR \ include-outputs $(nix-instantiat | cachix push $cachixName 9 . 4

Slide 41

Slide 41 text

9 CONCLUSIONS 10 . 1

Slide 42

Slide 42 text

9.1 OMITTED TOPICS NUR Nix user repository for custom packages Nix and HPC systems socat madness Nix Flakes Standardizing niv 10 . 2

Slide 43

Slide 43 text

9.2 FURTHER RESOURCES I write about nix pretty often For websites For documentation For languages My Nix Posts An introduction to the expression language Nix Pills The future of standard nixpkgs Introductory Flakes 10 . 3

Slide 44

Slide 44 text

10 THE END 11 . 1

Slide 45

Slide 45 text

10.1 THANKS! 11 . 2

Slide 46

Slide 46 text

11 REFERENCES Dolstra, Eelco, Merijn de Jonge, and Eelco Visser. n.d. “Nix: A Safe and Policy-Free System for Software Deployment,” 15. Dolstra, Eelco, Andres Löh, and Nicolas Pierron. n.d. “NixOS: A Purely Functional Linux Distribution” 20 (5-6):577–615. Goswami, Rohit, Amrita Goswami, and Jayant K. Singh. n.d. “D- SEAMS: Deferred Structural Elucidation Analysis for Molecular Simulations” 60 (4):2169–77. 12 . 1