Science About Shifter Contributions of this work 2. Design and Implementations New architecture design New Docker-like interface Performance improvements and others 3. Performance Evaluation Comparison on code-level Comparison on user experience 4. Summary 2
of library can make different result e.g.) Genomic analysis pipeline [Paolo, 2016] Dependency conflict Different application can requires different version of same library Container A’ Container A Container B Container A Library A Application A Application B version >= 1.2 version < 1.1 Application A Library version 1.3 Result A’ Application A Library version 1.2 Result A conflict different result Dependency Isolation Application Reproducibility 4
A. B. Ramirez, E. Palumbo, C. Notredame, and D. Gruber, “Benchmark Report : Univa Grid Engine , Nextflow , and Docker for running Genomic Analysis Workflows.” @ Centre for Genomic Regulation (Barcelona, Spain) • Univa Grid Engine (Job Scheduler) • Nextflow (Workflow manager) • Docker (Linux Container) running job on node Docker Container User Script pull image from registry job scheduler allocate run on container • Keep application reproducibility • Easy to build & test application 5
2016] Provide container environment on HPC systems Open Source on GitHub (NERSC/Shifter) Available on PizDaint at CSCS, and others Key Features Creation of software environments (containers) Native performance of custom HPC hardware (MPI/GPU) No root-privilege for user Compatible with Docker Containers 6 Docker Containers User Registry Service Build Upload Run Other System
for the user-defined stack with a good workflow and direct resource access. However, it relies on trusted and privileged operations, its resource manager integration increases complexity, and it requires servers and daemons for the image gateway. 7 Jie Zhang, Xiaoyi Lu, and Dhabaleswar K. Panda.”Is Singularity-based Container Technology Ready for Running MPI Applications on HPC Clouds?”, In Proceedings of the10th International Conference on Utility and Cloud Computing (UCC '17), pp.151-160, 2017.
Convert Other TOTAL Elapsed Time [sec] 373.441 769.786 820.882 63.186 2027.295 Time of pulling a container image from Docker Hub Data size : 6 GB (32 layers) Machine : gpu node on TDS (dom.cscs.ch) 0 500 1000 1500 2000 2500 Elapsed Time Download Expansion Convert Other 2027.295 Sometimes takes over 30 mins for pulling low-reliability: once a trouble happens, restart from scratch
executable, No daemon exists Integration of Shifter and Shifter Image Gateway New Shifter Image Manager Performance Improvements Robust and reliable pulling process Support private repository & 3rd party registry etc. New Docker-like Interface 10 $ shifter pull <image><:tag> $ shifter run <image><:tag> <args> $ shifter images etc.
shifter runtime HPC System single-executable, no daemons improved performance using faster-filesystems single executable no daemon Workload Manager improved performance using faster filesystems user friendly interface
image from Docker registry service storing the shifter image providing utilities for local repository Repository Policy repository images are owned by user (user’s local repository) New Features and Improvements performance improvement Parallel layer download Remove-After-Write expansion robust pulling (automatic connection retry) support private repository support 3rd party registry service 23
Process Combine all layer archives and create a container image Old algorithm aims Reduction of File I/O Concept: Filter unnecessary items before write A lot of time consumed to check deleted item dependency Especially, in the case of a container has many items Bad implementations on Performance (heavy python loop, etc.) 24 layer1 layer2 layer3 output file1 file2 file3 deleted file4 file1 file2 file4 file2 will be filtered
Process Concept: delete unnecessary items after write New algorithm aims Simple Implementation Performance has few dependency on the number of items New shifter architecture uses fast-storage(tmpfs) on compute-node 25 layer1 layer2 layer3 output file1 file2 file3 deleted file4 file1 file2 file4 file2 overwrite & delete overwrite … overwrite…
Improve Performance maximum: x 4.20 faster (using 6 GB image) Provide User Friendly Interface shifter pull, shifter run, shifter images, etc. Provide New Features support authentication, 3rd party registry, etc. 31
Old Download Expansion Convert Other Comparison of User Experience: Small image Label Download Expansion Convert Other TOTAL Speed-up Old 88.035 55.534 57.484 19.514 220.567 1.00 New ($HOME + tmpfs) 36.427 9.333 11.324 2.126 59.210 3.73 33 DataSize : 1 GB (6 layers) Env. : login node on TDS using tmpfs for File I/O 220.567 59.210 Old Shifter Image Gateway New Shifter Pull Command vs