Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Singularity Containers & The Rewrite to Go

Singularity Containers & The Rewrite to Go

Presentation of what Singularity Containers are and what we learned from our move to Go.

Andre Marcelo-Tanner

December 18, 2018

More Decks by Andre Marcelo-Tanner

Other Decks in Technology


  1. None
  2. Singularity A brief history • Invented by Greg Kurtzer at

    LBNL to address problems specific to HPC • Development began in October 2015 • First release April 2016 • User feedback, major changes and help from the community leads to 2.0 in June 2016 • Greg founds Sylabs.io January 1, 2018; already at 28 employees, product and revenue! • Sylabs developers completely overhaul code base, create a suite of cloud services, and release 3.0 October 8, 2018!
  3. Singularity is Designed Specifically for HPC • Compatible with traditional

    HPC architecture and workflows • Supports all known resource managers • Optimized for compute jobs and performance • Supports GPUs, IB, OmniPath, MPI, parallel file systems, production kernels, etc... • Users are themselves within the container • Allows users to “BYOE” • Blocks privilege escalation within the container
  4. Simplicity Singularity is designed to be simple. From its build

    syntax, to its image format, the runtime, and compatibility with OCI and Docker, Singularity facilitates easy and intuitive onboarding. SIF can squashe, encapsulate and sign Docker and OCI image formats (encryption coming soon) Docker, OCI, etc... Singularity
  5. Reproducibility and Mobility Singularity uses binary and immutable images, encapsulating

    the entire runtime stack so it is always 100% reproducible. Host 1 Host 2 SCP, FTP, HTTP, Sylabs Cloud
  6. Container Image Integrity Singularity container runtimes are immutable, cryptographically signed,

    and verifiable, ensuring absolute trust of the container environment. Note: Singularity uses no tarballs and thus no unsigned intermediate data
  7. Designed for HPC and up! From your laptop, to HPC

    compute, to the cloud and edge, Singularity facilitates extreme mobility of compute. Singularity provides an onramp across all resources, enterprises, and clouds.
  8. Widely Adopted for Compute Needs Singularity is the container runtime

    of choice for compute based workloads like simulation and AI As of Singularity 2.x… • Millions container runs per day • Estimated user base of more than 25k • Installed on over 3 million sockets HPCWire Readers and Editors Choice Awards: • 2016: Top products to watch • 2017: Top products to watch • 2017: Best HPC Programming Tool/Tech The Uptake of Singularity has been even faster than CentOS!
  9. $ cat recipe.def Bootstrap: yum OSVersion: 7 MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/x86_64/ Include:

    yum %post yum -y install vim-minimal %runscript exec /usr/bin/vi “$@” $ singularity build centos-vim.sif recipe.def … Building a Container with Singularity
  10. # You can use the container as follows... $ singularity

    shell centos-vim.sif $ singularity exec centos-vim.sif vim testfile.txt $ singularity run centos-vim.sif testfile.txt # Singularity containers are also executable, so you can ‘run’ them directly $ ./centos-vim.sif testfile.txt # You can move the container onto any other Linux system with Singularity installed, and use the container directly $ scp centos-vim.sif user@examplehost.com: $ ssh user@examplehost.com $ ./centos-vim.sif newtestfile.txt Example Usage of Singularity
  11. $ singularity sign centos-vim.sif Signing image: centos-vim.sif No OpenPGP signing

    keys found, autogenerate? [Y/n] y Enter your name (e.g., John Doe) : Greg Enter your email address (e.g., john.doe@example.com) : g@sylabs.io Enter optional comment (e.g., development keys) : demokeys Generating Entity and OpenPGP Key Pair... Done Enter encryption passphrase : … Uploaded key successfully! Enter key passphrase: Signature created and applied to centos-vim.sif $ Cryptographically signed containers
  12. $ singularity push centos-vim.sif library://gmk/demo/centos-vim:latest INFO: Now uploading centos-vim.sif to

    the library 108.16 MiB / 108.16 MiB [=============================] 100.00% 13.75 MiB/s 7s INFO: Setting tag latest $ Using the Sylabs Container Library
  13. $ singularity pull library://gmk/demo/centos-vim:latest 108.16 MiB / 108.16 MiB [=============================]

    100.00% 35.00 MiB/s 3s $ singularity verify centos-vim_latest.sif Verifying image: centos-vim_latest.sif INFO: key missing, searching key server for KeyID: 58D8405A30E12DE6... INFO: key retreived successfully! Store new public key F56D95BD3AFAC6FA3423911A58D8405A30E12DE6? [Y/n] y Data integrity checked, authentic and signed by: Greg (demokeys) <g@sylabs.io>, KeyID 58D8405A30E12DE6 $ Pulling and Validating a Container
  14. # Building a Singularity container (SIF) from DockerHub $ singularity

    build python.sif docker://python:latest … # Running a shell directly from DockerHub $ singularity shell docker://ubuntu:latest Singularity ubuntu_latest.sif:~/demo> cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS" Singularity ubuntu_latest.sif:~/demo> exit $ singularity exec docker://centos:latest cat /etc/redhat-release CentOS Linux release 7.5.1804 (Core) Working with Docker
  15. $ singularity exec --nv docker://tensorflow/tensorflow:latest-gpu python Python 2.7.12 (default, Dec

    4 2017, 14:50:18) [GCC 5.4.0 20160609] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import tensorflow as tf >>> x1 = tf.constant([1,2,3,4]) >>> x2 = tf.constant([5,6,7,8]) >>> result = tf.multiply(x1, x2) >>> print(result) Tensor("Mul:0", shape=(4,), dtype=int32) >>> exit() $ Accessing the Host’s GPU With Tensorflow
  16. Rewrite to Golang Feb 2018, Released 3.0 in Golang on

    October 2018 https://www.sylabs.io/2018/02/singularity-golang/
  17. None
  18. Wins & Gains from converting to Go • Integration with

    other container projects. Easy to use packages from Kubernetes, Docker/Moby, OCI etc. No need to rebuild in Python or C. • Concurrency model of Go. • Go is easy to learn for developers coming from other languages. • We used CNI rather than creating our own networking stack.
  19. Challenges using/converting to Go • C-Go Interface, used an RPC

    server to communicate between C and Go binaries. • Forking in Go. Certain syscalls should not be called in a multi threaded application. • Not following Go Standards (packaging, internal) • Vendoring… dep, go mod, vendor/ • Packaging. Custom packaging tool makeit.
  20. Comparing Go to previous Languages (Python etc) • Large ecosystem

    of packages for backend and system projects, cloud APIs. • go test - Easy standards and framework for setting up tests. • Easy to be productive in Go. Well designed language. • Go is opinionated (Good and Bad). Works well but you have to follow the standards. Easy to read other codebases.
  21. In hindsight • Focus on strong core and plan to

    make project modular. • Refactoring… • Better package planning. • Keep integration/acceptance tests in old format. Use go test for unit tests. Maybe use bats for integration tests.
  22. Demo Time

  23. We are hiring! https://www.sylabs.io/about-us/we-are-hiring/

  24. @SylabsIO @kzapkzap https://www.speakerdeck.com/kzap