Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
科学技術計算用クラスタへのDocker導入と運用 / HPC OPS mtg1
Search
Akihiro MATSUSHiMA
March 14, 2018
Science
0
620
科学技術計算用クラスタへのDocker導入と運用 / HPC OPS mtg1
第1回HPC OPS研究会(2018/3/13)の発表資料です。
https://bit.riken.jp/2018/02/1st-hpc-ops-mtg/
Akihiro MATSUSHiMA
March 14, 2018
Tweet
Share
Other Decks in Science
See All in Science
KDD2023学会参加報告
tereka114
2
410
[NeurIPS 2023 論文読み会] Wasserstein Quantum Monte Carlo
stakaya
0
390
Leveraging conformal prediction for calibrated probabilistic time series forecasts to accelerate the renewable energy transition
ingevandenende
2
310
FIBA W杯の日本代表って組み合わせ次第で2次ラウンド行けたんじゃね?をデータで検証
saltcooky12
0
230
Science of Scienceおよび科学計量学に関する研究論文の俯瞰可視化_LT版
hayataka88
0
710
ultraArmをモニター提供してもらった話
miura55
0
150
Xpenologyなるアングラプロジェクト周りについて語るやつ
sushi514
0
880
第4回ナレッジグラフ勉強会 Knowledge Graph Embedding
maruru0090
0
160
「国と音楽」 ~spotifyrを用いて~ #muana
bob3bob3
2
390
Mechanistic Interpretability の紹介
sohtakahashi
0
160
Machine Learning for Materials (Lecture 9)
aronwalsh
0
170
Transformer系機械学習モデルを取り巻くライブラリや用語を整理する
bobfromjapan
2
580
Featured
See All Featured
The Mythical Team-Month
searls
217
43k
Reflections from 52 weeks, 52 projects
jeffersonlam
346
19k
Imperfection Machines: The Place of Print at Facebook
scottboms
262
13k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
224
21k
Rebuilding a faster, lazier Slack
samanthasiow
78
8.5k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
189
16k
Side Projects
sachag
451
42k
From Idea to $5000 a Month in 5 Months
shpigford
377
46k
Robots, Beer and Maslow
schacon
PRO
157
8.1k
BBQ
matthewcrist
82
9k
A designer walks into a library…
pauljervisheath
201
24k
For a Future-Friendly Web
brad_frost
173
9.2k
Transcript
Պֶٕज़ܭࢉ༻Ϋϥελͷ Dockerಋೖͱӡ༻ দౢ ໌ ཧԽֶݚڀॴ ใج൫ηϯλʔ όΠΦΠϯϑΥϚςΟΫεݚڀ։ൃϢχοτ ୈ1ճHPC OPSݚڀձ 2018.03.13
౷߹σʔλϕʔεͷݚڀ։ൃΛ௨ͯ͡ ཧݚॴଐݚڀࣨͷวྺ '05 ήϊϜՊֶ૯߹ݚڀηϯλʔ ήϊϜมҟػೳใݚڀνʔϜ '07 ήϊϜՊֶ૯߹ݚڀηϯλʔ ΦϛοΫεใ౷߹ԽݚڀνʔϜ '08 ੜ໋ใج൫ݚڀ෦
(BASE) '13 ใج൫ηϯλʔ ౷߹σʔλϕʔεಛผϢχοτ '14 ใج൫ηϯλʔ όΠΦΠϯϑΥϚςΟΫεݚڀ։ൃϢχοτ (BiT)
ݚڀऀʹαΠΤϯεͤ͞Δ ϥΠϑαΠΤϯεݚڀऀ • PCαʔόʔΛϐʔΩʔͳػثͱೝࣝ • ࣮ݧػثΛѻ͖ͬͯͨܦݧʁ • ύιίϯૢ࡞Ͱͷܦݧʁ • αΠΤϯεʹूத͍ͨ͠
• ܭࢉαʔόʔͷௐୡɺཧɺӡ༻͔Β։์͞Ε͍ͨ • ࣗ༝ʹ͑ΔܭࢉϦιʔεཉ͍͠ ݚڀऀ͕ܭࢉػཧɾӡ༻ʹׂ࣌ؒ͘Λܰݮ • DevOps • Infrastructure as Code • ԾԽ • ΫϥυίϯϐϡʔςΟϯά
ίϯςφٕज़ͷ͓͞Β͍ ϗετOS Linux Linux ϗετOS Mac ίϯςφཧ ϋʔυΣΞ ϋΠύʔόΠβ ΞϓϦ
ήετOS ήετOS ΞϓϦ ΞϓϦ ԾϋʔυΣ Ξ ԾϋʔυΣ Ξ ԾϚγϯ(VM) ϋʔυΣΞ ΞϓϦ ΞϓϦ ίϯςφ ήετOS Windows Linux amatsus@hal011:~$ ps afx ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ 1926 ? Ssl 408:31 /usr/bin/dockerd --selinux-enabled=fal 2166 ? Ssl 234:33 \_ docker-containerd -l unix:///var/r 38674 ? Sl 0:00 \_ docker-containerd-shim 9c3689c 38691 pts/0 Ss+ 0:00 \_ bash ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ root@9c3689c042da:/# ps afx PID TTY STAT TIME COMMAND 1 pts/0 Ss 0:00 bash 15 pts/0 R+ 0:00 ps afx root@9c3689c042da:/# ίϯςφ ίϯςφϗετ্ amatsus@imac:~$ ps afx ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ 26675 ?? R 0:10.79 xhyve -m 4G -c 2 -s 0:0,hostbridge -s ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ amatsus@wily-xhyve:~$ ps afx PID TTY STAT TIME COMMAND 1 ? Ss 0:01 /sbin/init 2 ? S 0:00 [kthreadd] 3 ? S 0:00 [ksoftirqd/0] 4 ? S 0:00 [kworker/0:0] 5 ? S< 0:00 [kworker/0:0H] ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ VMϗετ্ VM
DockerΠϝʔδʹΑΔՄൖੑ push pull docker hub docker registry quai.io/CoreOS pull Docker
Πϝʔδ FROM docker.io/alpine:3.6 AS build-env RUN apk --update add --no-cache --virtual .build-deps build-base git zlib-dev && \ git clone https://github.com/ocxtal/ minialign.git && \ cd minialign && \ git checkout refs/tags/minialign-0.5.2 && \ sed -i "s/-march=native/-msse4 - mpopcnt/" Makefile && \ make && make install && \ apk del .build-deps && rm -rf /minialign FROM docker.io/alpine:3.6 LABEL maintainer="
[email protected]
" \ license="MIT" \ architecture="Nehalem and later" COPY --from=build-env /usr/local/bin/ minialign /usr/local/bin/minialign ENTRYPOINT [ "/usr/local/bin/minialign" ] CMD ["-h"] Dockerfile build run run run run
DockerͷϝϦοτɺσϝϦοτ ϝϦοτ • Φʔόʔϔου͕খ͍͞ • ϦιʔεΛΞϓϦ͝ͱʹ۠ΕΔ • ΞϓϦ͝ͱʹڥΛ • ಉҰߏͷίϯςφΛෳ࡞Մ
• ߏͷ࡞खॱΛίʔυԽ σϝϦοτ • ϢʔβIDɺάϧʔϓIDΛࣗ༝ʹઃఆͰ͖Δ • Χʔωϧϗετͱڞ༗ • ωοτϫʔΫϦιʔεΛϗετͱڞ༗ → ىಈ͕͍ → Մൖੑɺ࠶ݱੑͷ্ → ϦιʔεͷϜμΛܰݮ͠ߴີԽ → ϚΠάϨʔγϣϯ͕ࠔ → ΧʔωϧʹύονΛͯͮΒ͍ → ڞ༻ܭࢉػʹෆ͖ → Infrastructure as Code
ೋ֊ಊݚͷΫϥελܭࢉػ docker run bcl2fastq2:1.0 genomicpariscentre/fastqc:0.11.5 fastx_toolkit:1.0 picard:1.0 dropseq:1.0 star2.5.1b:1.0 pyper:1.2
bcl2fastq fastqc fastx_trimmer FastqToSam TagBamWithReadSequenceExtended TrimStartingSequence SamToFastq STAR SortSam MergeBamAlignment TagReadWithGeneExon correct_barcode.py DigitalExpression analog_expression.py BAMTagHistogram IMAGE CMD FileSystem Execution Nodes Submission Node NFS docker pull docker push send qsub login HUB docker pull 460,000+ Dockerized Applications / qdel / qmod -s / qmod -us © 2016 DBCLS ౷߹TV / CC-BY-4.0 on-premiss or cloud
docker runϥούʔεΫϦϓτ # # docker run wrapper for OGS/GE #
# Copyright (c) 2016 Akihiro Matsushima # Released under the MIT license # http://opensource.org/licenses/mit-license.php # function sigconthandler() { docker unpause $cid echo "caught sigcont, container unpaused." wait } function sigusr1handler() { docker pause $cid echo "caught sigusr1, container paused." wait } function sigusr2handler() { if [ `docker inspect --format="{{ .State.Status }}" $cid` == "paused" ]; then docker unpause $cid fi docker stop $cid echo "caught sigusr2, container stopped." } function docker() { # emulate fairly POSIX sh in zsh $(type "emulate" >/dev/null 2>&1) && emulate -L sh local IFS=$' \t\n’ if [ "$1" = "run" ]; then local DOCKER_RUN_LOCALOPTS=${DOCKER_RUN_OPTS:-'--net=bridge -u `id - u`:`id -g` --group-add=10100 -v /etc/passwd:/etc/passwd:ro -v /etc/group:/ etc/group:ro --security-opt seccomp:unconfined -v $HOME:$HOME -v /data/ $USER:/data/$USER -v /data2/$USER:/data2/$USER -w $PWD’} if [ -n "$JOB_ID" ]; then # define the unique cidfile name TEMPDIR=/var/tmp/${LOGNAME:-$USER} CIDFILE="${TEMPDIR}/${JOB_NAME:-SOMEJOB}.o${JOB_ID}.$ {SGE_TASK_ID:-SOMETASK}_$(date +%Y%m%d%H%M%S%3N).cid" if [ ! -e "$TEMPDIR" ]; then mkdir -p "$TEMPDIR" fi echo -e "$RUNDATE\t${LOGNAME:-$USER}\t$JOB_ID\t$SGE_TASK_ID\t/ usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) --cidfile=\"$CIDFILE\" ${@:2:($#-1)} &" >> /usr/local/gridscheduler/default/docker_cmdline /usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) -i -- cidfile="$CIDFILE" "${@:2:($#-1)}" & pid=$! while [[ -d /proc/$pid && -z $cid ]]; do sleep 1 if [ -s "$CIDFILE" ]; then read -r cid < "$CIDFILE" rm -f "$CIDFILE" fi done trap sigconthandler SIGCONT trap sigusr1handler SIGUSR1 trap sigusr2handler SIGUSR2 wait else /usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) "${@:2: ($#-1)}" fi else /usr/bin/docker "$@" fi } if [[ ! $(readlink /proc/$$/exe) =~ "zsh" ]]; then export -f sigconthandler sigusr1handler sigusr2handler docker fi https://gist.github.com/amatsus/4bdcb1498ea5a002ba41edebb122c21c
ίϯςφΛվมͤͣ༻ײΛ͚ۙͮΔ #$ -N FQC fastqc —nogroup -o fastqc_out ERR030893.fastq.gz #$
-N FQCd #$ -notify docker run genomicpariscentre/fastqc:0.11.5 —nogroup -o fastqc_out ERR030893.fastq.gz function docker() { local DOCKER_RUN_LOCALOPTS=${DOCKER_RUN_OPTS:-‘ --net=bridge --u `id -u`:`id -g` --group-add=10100 -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro --security-opt seccomp:unconfined -v $HOME:$HOME -v /data/$USER:/data/$USER -v /data2/$USER:/data2/$USER -w $PWD’} /usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) --cidfile="$CIDFILE" "${@:2:($#-1)}" & } Wrap
γάφϧϋϯυϥ /usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) -i --cidfile="$CIDFILE" "${@:2:($#-1)}" &
pid=$! while [[ -d /proc/$pid && -z $cid ]]; do sleep 1 if [ -s "$CIDFILE" ]; then read -r cid < "$CIDFILE" rm -f "$CIDFILE" fi done trap ‘docker unpause $cid; wait’ SIGCONT # qmod -us trap ‘docker pause $cid; wait’ SIGUSR1 # qmod -s trap ‘docker stop $cid’ SIGUSR2 # qdel wait $POUBJOFS*% $*% ͷऔಘ ɹίϯςφΛσλον(docker run —detach)͢ΔͱίϚϯυͷΓCID ɹσλονͯ͠͠ͳͯ͘CIDΛऔಘͰ͖ΔCIDFILEΛར༻ δϣϒεέδϡʔϥ͔ΒͷγάφϧΛτϥοϓ ɹड͚औͬͨγάφϧ͝ͱʹdockerίϚϯυΛ࣮ߦ amatsus@elwood:~$ qconf -sq | grep method starter_method NONE suspend_method NONE resume_method NONE terminate_method NONE
ΫϥελܭࢉػΛ1ΫϦοΫͰΫϥυʹ https://portal.azure.com/#create/Microsoft.Temp late/uri/https%3A%2F%2Fraw.githubusercontent.co m%2Fmanabuishii%2Fazurefiles%2Fmaster%2FNFS_SGE% 2Fazuredeploy.json ARMςϯϓϨʔτ
σʔλͷόοΫΞοϓ Technology Storage driver name Support version OverlayFS overlay v1.4ʙ
overlay2 v1.12ʙ AUFS aufs Btrfs btrfs v0.7ʙ Device Mapper device mapper v0.7ʙ VFS vfs v0.7ʙ ZFS zfs v1.7ʙ Layer A Layer B Layer A Layer B nginx Layer A Layer B nginx web app ubuntu nginx web app }ReadOnly Layer Layer A Layer B web app Btrfs,ZFSҎ֎ͷϑΝΠϧγεςϜͰϑϥοτ DockerΠϝʔδϨΠϠ nginx Docker Πϝʔδ docker hub docker registry docker push rsync όοΫΞοϓαʔό data volumesίϯςφ·ͩdocker push͍ͯ͠ͳ͍ ίϯςφόοΫΞοϓରʹ͢ΔͱͳΔͱɺɺɺ
https://developers.redhat.com/blog/2014/09/30/overview-storage-scalability-docker/ Container Create/Destroy Times
Dockerؔ࿈ίϛϡχςΟฦྱ RHEL/CentOSͷमਖ਼ͯͣʹ kernel 4.16͔Βkernel 3.10.0-237.18.2.el7όοΫϙʔτ Jupyter Projectʹ kernelύονෆཁͳWorkaroundΛఏҊ(Pull Request) Docker
1.13ʹͯlog driverʹjournaldΛࢦఆ͢Δͱෆఆظʹ dockerd͝ͱίϯςφ͕શ໓͢Δ ೋ֊ಊݚҎʹ࣮֬ʹdockerdΛམͱ͢ίϯςφΛ །Ұ͍࣋ͬͯͨͷͰमਖ਼ύονͷݕূʹڠྗ Docker for MacͰ࠾༻͞Ε͍ͯΔAufsʹಉ͕͡જࡏ ͨͨ͠ΊAufsͷ։ൃऀʹใࠂ
·ͱΊ DockerಋೖʹΑΓ • ιʔείʔυ͔ΒϏϧυɺΠϯετʔϧ͢Δ͜ͱ͕ݮͬͨ • ؾܰʹࢼͤΔ • ϗʔϜσΟϨΫτϦ͕͖ͬ͢Γ • τϥϒϧγϡʔςΟϯά͘͢͠ͳͬͨ
• ղੳͷ࠶ݱੑ্ • ͰͬͺΓDockerfile࡞Δͷ໘ େنڞ༻ܭࢉػͰͷར༻ • rootlessͳίϯςφ࣮ߦڥ͕ͨΕΔ ೋ֊ಊݚډ͔ࣨΒͷோ(20174݄)
HPC͚ʁίϯςφٕज़ Gregory M. Kurtzer, Vanessa Sochat, Michael W. Bauer, “Singularity:
Scientific containers for mobility of compute”, PLoS ONE 12.5 (2017) https://github.com/indigo-dc/udocker/ http://singularity.lbl.gov/