jsonargparse - Say goodbye to configuration hassles

Talk about the open-source library jsonargparse at PyConDE / PyData Berlin 2022

1. Marianne Stecklina Say goodbye to configuration hassles jsonargparse
2. • Deep Learning Engineer at omni:us • colleague of Mauricio Villegas, creator of jsonargparse • early user of jsonargparse How I came to give this talk That’s me! That’s Mauricio!
3. Introduction 1
4. jsonargparse is an open-source library that helps to configure your code by creating a CLI. *CLI = command line interface
5. Why CLIs are great compared to… no interface • run program differently without modifying code • easier to use for others graphical user interface (GUI) • easier to create • perfect for batch tasks • perfect for scheduled tasks • useful in terminal-only environments
6. Survey What tools are you currently using? Take the poll in #b05-b06 on Discord
7. jsonargparse basics 2
8. https://github.com/stecklin/pycon22-jsonargparse • Compare python 0_argparse.py --help and python 1_cli.py --help • Run python 1_cli.py PyCON --year 2022 • Print config using python 1_cli.py --print_config > config.yaml • Edit config.yaml • Run python 1_cli.py --config config.yaml • Run python 1_cli.py --config config.yaml --year 2024 • Run python 1_cli.py --year 2024 --config config.yaml • Run PYCON_CONFERENCE_NAME=PyCON PYCON_YEAR=2024 python 2_cli_with_env_variables.py Live Demo
9. Key features • CLIs with minimal code overhead • possible to parse ◦ command line arguments ◦ config files ◦ environment variables
10. Hierarchical configs 3
11. Use case: Train image classifier def train(dataset: Cifar10Dataset, model: ConvNet): ... frog
12. https://github.com/stecklin/pycon22-jsonargparse • Run python 3_image_classification.py --help • Run python 3_image_classification.py --dataset.help Cifar10Dataset • Run python 3_image_classification.py --config configs/3_config.yaml --dataset.augment maybe • Run python 3_image_classification.py --config configs/3_config.yaml --dataset.data_dir /home/ghosts/ Live Demo
13. Key features • config (and help) is hierarchical • config structure always in sync with code • type checks
14. Subclasses 4
15. Different datasets & models ConvNet kernel_size: int num_layers: int VisionTransformer num_layers: int num_heads: int hidden_dim: int mlp_dim: int {abstract} Model num_classes: int Cifar10Dataset ImageNetDataset {abstract} Dataset data_dir: path_dr batch_size: int augment: bool
16. https://github.com/stecklin/pycon22-jsonargparse • Run python 4_subclasses.py --help • Run python 4_subclasses.py --dataset ImageNetDataset --model ConvNet --print_config • Run python 4_subclasses.py --config configs/4_config.yaml • Run python 4_subclasses.py --dataset configs/4_cifar10.yaml --model configs/4_conv_net.yaml • Run python 4_subclasses.py --config configs/4_config_with_subconfigs.yaml • Run python 5_model_composition.py --config configs/5_config.yaml Live Demo
17. Key features • switch between subclasses • compose config into multiple files
18. Adoption 5
19. PyTorch Lightning
20. Links • jsonargparse: https://github.com/omni-us/jsonargparse • Example code: https://github.com/stecklin/pycon22-jsonargparse • Slides: https://speakerdeck.com/stecklin • Twitter: @MStecklina Thanks for listening!