Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
科学技術計算用クラスタへのDocker導入と運用 / HPC OPS mtg1
Search
Akihiro MATSUSHiMA
March 14, 2018
Science
0
590
科学技術計算用クラスタへのDocker導入と運用 / HPC OPS mtg1
第1回HPC OPS研究会(2018/3/13)の発表資料です。
https://bit.riken.jp/2018/02/1st-hpc-ops-mtg/
Akihiro MATSUSHiMA
March 14, 2018
Tweet
Share
Other Decks in Science
See All in Science
バックアップ『しながら』ランサムウェア検出も!? セキュリティ強化が満載 Veeam 12.1
climbteam
0
260
データで課題を解決する -因果関係を調べる統計的因果推論-
sshimizu2006
4
1.3k
脳とAIは似ているか ― NeuroAI の挑戦
ykamit
9
6.6k
救急外来でのめまい診療_中枢性めまいを見逃さない!
psasa
0
110
JSol'Ex : solar image processing in Java
melix
0
230
Spark_Task_Optimization_Journey_How_I_Increased_10x_Speed_by_Performance_Tuning
tlyu0419
0
190
Pandas 2 vs Polars vs Dask (PyDataGlobal 2023 December)
ianozsvald
0
360
スポーツメトリクス設計に対比較法を使いまくる / Sports metrics design using pairwise comparison method (spoana#14)
konakalab
1
660
Yasuke
drawsbygba
0
600
論文輪読会 第16回 "NeRF:Representing Scenes as Neural"
academix
0
430
Cross-Media Information Spaces and Architectures (CISA)
signer
PRO
3
24k
How we developed a data exchange format: Lessons learned from Camtrap DP
peterdesmet
1
130
Featured
See All Featured
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
319
20k
The Invisible Side of Design
smashingmag
293
49k
Building Applications with DynamoDB
mza
88
5.6k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
19
1.9k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
240
1.2M
Reflections from 52 weeks, 52 projects
jeffersonlam
343
19k
Why Our Code Smells
bkeepers
PRO
330
56k
RailsConf 2023
tenderlove
0
510
Rebuilding a faster, lazier Slack
samanthasiow
72
8.2k
The Mythical Team-Month
searls
214
42k
What the flash - Photography Introduction
edds
64
11k
Side Projects
sachag
451
41k
Transcript
Պֶٕज़ܭࢉ༻Ϋϥελͷ Dockerಋೖͱӡ༻ দౢ ໌ ཧԽֶݚڀॴ ใج൫ηϯλʔ όΠΦΠϯϑΥϚςΟΫεݚڀ։ൃϢχοτ ୈ1ճHPC OPSݚڀձ 2018.03.13
౷߹σʔλϕʔεͷݚڀ։ൃΛ௨ͯ͡ ཧݚॴଐݚڀࣨͷวྺ '05 ήϊϜՊֶ૯߹ݚڀηϯλʔ ήϊϜมҟػೳใݚڀνʔϜ '07 ήϊϜՊֶ૯߹ݚڀηϯλʔ ΦϛοΫεใ౷߹ԽݚڀνʔϜ '08 ੜ໋ใج൫ݚڀ෦
(BASE) '13 ใج൫ηϯλʔ ౷߹σʔλϕʔεಛผϢχοτ '14 ใج൫ηϯλʔ όΠΦΠϯϑΥϚςΟΫεݚڀ։ൃϢχοτ (BiT)
ݚڀऀʹαΠΤϯεͤ͞Δ ϥΠϑαΠΤϯεݚڀऀ • PCαʔόʔΛϐʔΩʔͳػثͱೝࣝ • ࣮ݧػثΛѻ͖ͬͯͨܦݧʁ • ύιίϯૢ࡞Ͱͷܦݧʁ • αΠΤϯεʹूத͍ͨ͠
• ܭࢉαʔόʔͷௐୡɺཧɺӡ༻͔Β։์͞Ε͍ͨ • ࣗ༝ʹ͑ΔܭࢉϦιʔεཉ͍͠ ݚڀऀ͕ܭࢉػཧɾӡ༻ʹׂ࣌ؒ͘Λܰݮ • DevOps • Infrastructure as Code • ԾԽ • ΫϥυίϯϐϡʔςΟϯά
ίϯςφٕज़ͷ͓͞Β͍ ϗετOS Linux Linux ϗετOS Mac ίϯςφཧ ϋʔυΣΞ ϋΠύʔόΠβ ΞϓϦ
ήετOS ήετOS ΞϓϦ ΞϓϦ ԾϋʔυΣ Ξ ԾϋʔυΣ Ξ ԾϚγϯ(VM) ϋʔυΣΞ ΞϓϦ ΞϓϦ ίϯςφ ήετOS Windows Linux amatsus@hal011:~$ ps afx ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ 1926 ? Ssl 408:31 /usr/bin/dockerd --selinux-enabled=fal 2166 ? Ssl 234:33 \_ docker-containerd -l unix:///var/r 38674 ? Sl 0:00 \_ docker-containerd-shim 9c3689c 38691 pts/0 Ss+ 0:00 \_ bash ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ root@9c3689c042da:/# ps afx PID TTY STAT TIME COMMAND 1 pts/0 Ss 0:00 bash 15 pts/0 R+ 0:00 ps afx root@9c3689c042da:/# ίϯςφ ίϯςφϗετ্ amatsus@imac:~$ ps afx ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ 26675 ?? R 0:10.79 xhyve -m 4G -c 2 -s 0:0,hostbridge -s ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ amatsus@wily-xhyve:~$ ps afx PID TTY STAT TIME COMMAND 1 ? Ss 0:01 /sbin/init 2 ? S 0:00 [kthreadd] 3 ? S 0:00 [ksoftirqd/0] 4 ? S 0:00 [kworker/0:0] 5 ? S< 0:00 [kworker/0:0H] ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ VMϗετ্ VM
DockerΠϝʔδʹΑΔՄൖੑ push pull docker hub docker registry quai.io/CoreOS pull Docker
Πϝʔδ FROM docker.io/alpine:3.6 AS build-env RUN apk --update add --no-cache --virtual .build-deps build-base git zlib-dev && \ git clone https://github.com/ocxtal/ minialign.git && \ cd minialign && \ git checkout refs/tags/minialign-0.5.2 && \ sed -i "s/-march=native/-msse4 - mpopcnt/" Makefile && \ make && make install && \ apk del .build-deps && rm -rf /minialign FROM docker.io/alpine:3.6 LABEL maintainer="
[email protected]
" \ license="MIT" \ architecture="Nehalem and later" COPY --from=build-env /usr/local/bin/ minialign /usr/local/bin/minialign ENTRYPOINT [ "/usr/local/bin/minialign" ] CMD ["-h"] Dockerfile build run run run run
DockerͷϝϦοτɺσϝϦοτ ϝϦοτ • Φʔόʔϔου͕খ͍͞ • ϦιʔεΛΞϓϦ͝ͱʹ۠ΕΔ • ΞϓϦ͝ͱʹڥΛ • ಉҰߏͷίϯςφΛෳ࡞Մ
• ߏͷ࡞खॱΛίʔυԽ σϝϦοτ • ϢʔβIDɺάϧʔϓIDΛࣗ༝ʹઃఆͰ͖Δ • Χʔωϧϗετͱڞ༗ • ωοτϫʔΫϦιʔεΛϗετͱڞ༗ → ىಈ͕͍ → Մൖੑɺ࠶ݱੑͷ্ → ϦιʔεͷϜμΛܰݮ͠ߴີԽ → ϚΠάϨʔγϣϯ͕ࠔ → ΧʔωϧʹύονΛͯͮΒ͍ → ڞ༻ܭࢉػʹෆ͖ → Infrastructure as Code
ೋ֊ಊݚͷΫϥελܭࢉػ docker run bcl2fastq2:1.0 genomicpariscentre/fastqc:0.11.5 fastx_toolkit:1.0 picard:1.0 dropseq:1.0 star2.5.1b:1.0 pyper:1.2
bcl2fastq fastqc fastx_trimmer FastqToSam TagBamWithReadSequenceExtended TrimStartingSequence SamToFastq STAR SortSam MergeBamAlignment TagReadWithGeneExon correct_barcode.py DigitalExpression analog_expression.py BAMTagHistogram IMAGE CMD FileSystem Execution Nodes Submission Node NFS docker pull docker push send qsub login HUB docker pull 460,000+ Dockerized Applications / qdel / qmod -s / qmod -us © 2016 DBCLS ౷߹TV / CC-BY-4.0 on-premiss or cloud
docker runϥούʔεΫϦϓτ # # docker run wrapper for OGS/GE #
# Copyright (c) 2016 Akihiro Matsushima # Released under the MIT license # http://opensource.org/licenses/mit-license.php # function sigconthandler() { docker unpause $cid echo "caught sigcont, container unpaused." wait } function sigusr1handler() { docker pause $cid echo "caught sigusr1, container paused." wait } function sigusr2handler() { if [ `docker inspect --format="{{ .State.Status }}" $cid` == "paused" ]; then docker unpause $cid fi docker stop $cid echo "caught sigusr2, container stopped." } function docker() { # emulate fairly POSIX sh in zsh $(type "emulate" >/dev/null 2>&1) && emulate -L sh local IFS=$' \t\n’ if [ "$1" = "run" ]; then local DOCKER_RUN_LOCALOPTS=${DOCKER_RUN_OPTS:-'--net=bridge -u `id - u`:`id -g` --group-add=10100 -v /etc/passwd:/etc/passwd:ro -v /etc/group:/ etc/group:ro --security-opt seccomp:unconfined -v $HOME:$HOME -v /data/ $USER:/data/$USER -v /data2/$USER:/data2/$USER -w $PWD’} if [ -n "$JOB_ID" ]; then # define the unique cidfile name TEMPDIR=/var/tmp/${LOGNAME:-$USER} CIDFILE="${TEMPDIR}/${JOB_NAME:-SOMEJOB}.o${JOB_ID}.$ {SGE_TASK_ID:-SOMETASK}_$(date +%Y%m%d%H%M%S%3N).cid" if [ ! -e "$TEMPDIR" ]; then mkdir -p "$TEMPDIR" fi echo -e "$RUNDATE\t${LOGNAME:-$USER}\t$JOB_ID\t$SGE_TASK_ID\t/ usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) --cidfile=\"$CIDFILE\" ${@:2:($#-1)} &" >> /usr/local/gridscheduler/default/docker_cmdline /usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) -i -- cidfile="$CIDFILE" "${@:2:($#-1)}" & pid=$! while [[ -d /proc/$pid && -z $cid ]]; do sleep 1 if [ -s "$CIDFILE" ]; then read -r cid < "$CIDFILE" rm -f "$CIDFILE" fi done trap sigconthandler SIGCONT trap sigusr1handler SIGUSR1 trap sigusr2handler SIGUSR2 wait else /usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) "${@:2: ($#-1)}" fi else /usr/bin/docker "$@" fi } if [[ ! $(readlink /proc/$$/exe) =~ "zsh" ]]; then export -f sigconthandler sigusr1handler sigusr2handler docker fi https://gist.github.com/amatsus/4bdcb1498ea5a002ba41edebb122c21c
ίϯςφΛվมͤͣ༻ײΛ͚ۙͮΔ #$ -N FQC fastqc —nogroup -o fastqc_out ERR030893.fastq.gz #$
-N FQCd #$ -notify docker run genomicpariscentre/fastqc:0.11.5 —nogroup -o fastqc_out ERR030893.fastq.gz function docker() { local DOCKER_RUN_LOCALOPTS=${DOCKER_RUN_OPTS:-‘ --net=bridge --u `id -u`:`id -g` --group-add=10100 -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro --security-opt seccomp:unconfined -v $HOME:$HOME -v /data/$USER:/data/$USER -v /data2/$USER:/data2/$USER -w $PWD’} /usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) --cidfile="$CIDFILE" "${@:2:($#-1)}" & } Wrap
γάφϧϋϯυϥ /usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) -i --cidfile="$CIDFILE" "${@:2:($#-1)}" &
pid=$! while [[ -d /proc/$pid && -z $cid ]]; do sleep 1 if [ -s "$CIDFILE" ]; then read -r cid < "$CIDFILE" rm -f "$CIDFILE" fi done trap ‘docker unpause $cid; wait’ SIGCONT # qmod -us trap ‘docker pause $cid; wait’ SIGUSR1 # qmod -s trap ‘docker stop $cid’ SIGUSR2 # qdel wait $POUBJOFS*% $*% ͷऔಘ ɹίϯςφΛσλον(docker run —detach)͢ΔͱίϚϯυͷΓCID ɹσλονͯ͠͠ͳͯ͘CIDΛऔಘͰ͖ΔCIDFILEΛར༻ δϣϒεέδϡʔϥ͔ΒͷγάφϧΛτϥοϓ ɹड͚औͬͨγάφϧ͝ͱʹdockerίϚϯυΛ࣮ߦ amatsus@elwood:~$ qconf -sq | grep method starter_method NONE suspend_method NONE resume_method NONE terminate_method NONE
ΫϥελܭࢉػΛ1ΫϦοΫͰΫϥυʹ https://portal.azure.com/#create/Microsoft.Temp late/uri/https%3A%2F%2Fraw.githubusercontent.co m%2Fmanabuishii%2Fazurefiles%2Fmaster%2FNFS_SGE% 2Fazuredeploy.json ARMςϯϓϨʔτ
σʔλͷόοΫΞοϓ Technology Storage driver name Support version OverlayFS overlay v1.4ʙ
overlay2 v1.12ʙ AUFS aufs Btrfs btrfs v0.7ʙ Device Mapper device mapper v0.7ʙ VFS vfs v0.7ʙ ZFS zfs v1.7ʙ Layer A Layer B Layer A Layer B nginx Layer A Layer B nginx web app ubuntu nginx web app }ReadOnly Layer Layer A Layer B web app Btrfs,ZFSҎ֎ͷϑΝΠϧγεςϜͰϑϥοτ DockerΠϝʔδϨΠϠ nginx Docker Πϝʔδ docker hub docker registry docker push rsync όοΫΞοϓαʔό data volumesίϯςφ·ͩdocker push͍ͯ͠ͳ͍ ίϯςφόοΫΞοϓରʹ͢ΔͱͳΔͱɺɺɺ
https://developers.redhat.com/blog/2014/09/30/overview-storage-scalability-docker/ Container Create/Destroy Times
Dockerؔ࿈ίϛϡχςΟฦྱ RHEL/CentOSͷमਖ਼ͯͣʹ kernel 4.16͔Βkernel 3.10.0-237.18.2.el7όοΫϙʔτ Jupyter Projectʹ kernelύονෆཁͳWorkaroundΛఏҊ(Pull Request) Docker
1.13ʹͯlog driverʹjournaldΛࢦఆ͢Δͱෆఆظʹ dockerd͝ͱίϯςφ͕શ໓͢Δ ೋ֊ಊݚҎʹ࣮֬ʹdockerdΛམͱ͢ίϯςφΛ །Ұ͍࣋ͬͯͨͷͰमਖ਼ύονͷݕূʹڠྗ Docker for MacͰ࠾༻͞Ε͍ͯΔAufsʹಉ͕͡જࡏ ͨͨ͠ΊAufsͷ։ൃऀʹใࠂ
·ͱΊ DockerಋೖʹΑΓ • ιʔείʔυ͔ΒϏϧυɺΠϯετʔϧ͢Δ͜ͱ͕ݮͬͨ • ؾܰʹࢼͤΔ • ϗʔϜσΟϨΫτϦ͕͖ͬ͢Γ • τϥϒϧγϡʔςΟϯά͘͢͠ͳͬͨ
• ղੳͷ࠶ݱੑ্ • ͰͬͺΓDockerfile࡞Δͷ໘ େنڞ༻ܭࢉػͰͷར༻ • rootlessͳίϯςφ࣮ߦڥ͕ͨΕΔ ೋ֊ಊݚډ͔ࣨΒͷோ(20174݄)
HPC͚ʁίϯςφٕज़ Gregory M. Kurtzer, Vanessa Sochat, Michael W. Bauer, “Singularity:
Scientific containers for mobility of compute”, PLoS ONE 12.5 (2017) https://github.com/indigo-dc/udocker/ http://singularity.lbl.gov/