Singularity Containers & The Rewrite to Go

Presentation of what Singularity Containers are and what we learned from our move to Go.

Transcript

2. Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began in October 2015 • First release April 2016 • User feedback, major changes and help from the community leads to 2.0 in June 2016 • Greg founds Sylabs.io January 1, 2018; already at 28 employees, product and revenue! • Sylabs developers completely overhaul code base, create a suite of cloud services, and release 3.0 October 8, 2018!
3. Singularity is Designed Specifically for HPC • Compatible with traditional HPC architecture and workflows • Supports all known resource managers • Optimized for compute jobs and performance • Supports GPUs, IB, OmniPath, MPI, parallel file systems, production kernels, etc... • Users are themselves within the container • Allows users to “BYOE” • Blocks privilege escalation within the container
4. Simplicity Singularity is designed to be simple. From its build syntax, to its image format, the runtime, and compatibility with OCI and Docker, Singularity facilitates easy and intuitive onboarding. SIF can squashe, encapsulate and sign Docker and OCI image formats (encryption coming soon) Docker, OCI, etc... Singularity
5. Reproducibility and Mobility Singularity uses binary and immutable images, encapsulating the entire runtime stack so it is always 100% reproducible. Host 1 Host 2 SCP, FTP, HTTP, Sylabs Cloud
6. Container Image Integrity Singularity container runtimes are immutable, cryptographically signed, and verifiable, ensuring absolute trust of the container environment. Note: Singularity uses no tarballs and thus no unsigned intermediate data
7. Designed for HPC and up! From your laptop, to HPC compute, to the cloud and edge, Singularity facilitates extreme mobility of compute. Singularity provides an onramp across all resources, enterprises, and clouds.
8. Widely Adopted for Compute Needs Singularity is the container runtime of choice for compute based workloads like simulation and AI As of Singularity 2.x… • Millions container runs per day • Estimated user base of more than 25k • Installed on over 3 million sockets HPCWire Readers and Editors Choice Awards: • 2016: Top products to watch • 2017: Top products to watch • 2017: Best HPC Programming Tool/Tech The Uptake of Singularity has been even faster than CentOS!
9. $ cat recipe.def Bootstrap: yum OSVersion: 7 MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/x86_64/ Include: yum %post yum -y install vim-minimal %runscript exec /usr/bin/vi “$@” $ singularity build centos-vim.sif recipe.def … Building a Container with Singularity
10. # You can use the container as follows... $ singularity shell centos-vim.sif $ singularity exec centos-vim.sif vim testfile.txt $ singularity run centos-vim.sif testfile.txt # Singularity containers are also executable, so you can ‘run’ them directly $ ./centos-vim.sif testfile.txt # You can move the container onto any other Linux system with Singularity installed, and use the container directly $ scp centos-vim.sif user@examplehost.com: $ ssh user@examplehost.com $ ./centos-vim.sif newtestfile.txt Example Usage of Singularity
11. $ singularity sign centos-vim.sif Signing image: centos-vim.sif No OpenPGP signing keys found, autogenerate? [Y/n] y Enter your name (e.g., John Doe) : Greg Enter your email address (e.g., john.doe@example.com) : g@sylabs.io Enter optional comment (e.g., development keys) : demokeys Generating Entity and OpenPGP Key Pair... Done Enter encryption passphrase : … Uploaded key successfully! Enter key passphrase: Signature created and applied to centos-vim.sif $ Cryptographically signed containers
12. $ singularity push centos-vim.sif library://gmk/demo/centos-vim:latest INFO: Now uploading centos-vim.sif to the library 108.16 MiB / 108.16 MiB [=============================] 100.00% 13.75 MiB/s 7s INFO: Setting tag latest $ Using the Sylabs Container Library
13. $ singularity pull library://gmk/demo/centos-vim:latest 108.16 MiB / 108.16 MiB [=============================] 100.00% 35.00 MiB/s 3s $ singularity verify centos-vim_latest.sif Verifying image: centos-vim_latest.sif INFO: key missing, searching key server for KeyID: 58D8405A30E12DE6... INFO: key retreived successfully! Store new public key F56D95BD3AFAC6FA3423911A58D8405A30E12DE6? [Y/n] y Data integrity checked, authentic and signed by: Greg (demokeys) <g@sylabs.io>, KeyID 58D8405A30E12DE6 $ Pulling and Validating a Container
14. # Building a Singularity container (SIF) from DockerHub $ singularity build python.sif docker://python:latest … # Running a shell directly from DockerHub $ singularity shell docker://ubuntu:latest Singularity ubuntu_latest.sif:~/demo> cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS" Singularity ubuntu_latest.sif:~/demo> exit $ singularity exec docker://centos:latest cat /etc/redhat-release CentOS Linux release 7.5.1804 (Core) Working with Docker
15. $ singularity exec --nv docker://tensorflow/tensorflow:latest-gpu python Python 2.7.12 (default, Dec 4 2017, 14:50:18) [GCC 5.4.0 20160609] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import tensorflow as tf >>> x1 = tf.constant([1,2,3,4]) >>> x2 = tf.constant([5,6,7,8]) >>> result = tf.multiply(x1, x2) >>> print(result) Tensor("Mul:0", shape=(4,), dtype=int32) >>> exit() $ Accessing the Host’s GPU With Tensorflow
16. Rewrite to Golang Feb 2018, Released 3.0 in Golang on October 2018 https://www.sylabs.io/2018/02/singularity-golang/
18. Wins & Gains from converting to Go • Integration with other container projects. Easy to use packages from Kubernetes, Docker/Moby, OCI etc. No need to rebuild in Python or C. • Concurrency model of Go. • Go is easy to learn for developers coming from other languages. • We used CNI rather than creating our own networking stack.
19. Challenges using/converting to Go • C-Go Interface, used an RPC server to communicate between C and Go binaries. • Forking in Go. Certain syscalls should not be called in a multi threaded application. • Not following Go Standards (packaging, internal) • Vendoring… dep, go mod, vendor/ • Packaging. Custom packaging tool makeit.
20. Comparing Go to previous Languages (Python etc) • Large ecosystem of packages for backend and system projects, cloud APIs. • go test - Easy standards and framework for setting up tests. • Easy to be productive in Go. Well designed language. • Go is opinionated (Good and Bad). Works well but you have to follow the standards. Easy to read other codebases.
21. In hindsight • Focus on strong core and plan to make project modular. • Refactoring… • Better package planning. • Keep integration/acceptance tests in old format. Use go test for unit tests. Maybe use bats for integration tests.
22. Demo Time
23. We are hiring! https://www.sylabs.io/about-us/we-are-hiring/
24. @SylabsIO @kzapkzap https://www.speakerdeck.com/kzap