Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
科学技術計算用クラスタへのDocker導入と運用 / HPC OPS mtg1
Search
Akihiro MATSUSHiMA
March 14, 2018
Science
0
720
科学技術計算用クラスタへのDocker導入と運用 / HPC OPS mtg1
第1回HPC OPS研究会(2018/3/13)の発表資料です。
https://bit.riken.jp/2018/02/1st-hpc-ops-mtg/
Akihiro MATSUSHiMA
March 14, 2018
Tweet
Share
Other Decks in Science
See All in Science
機械学習 - 決定木からはじめる機械学習
trycycle
PRO
0
990
統計学入門講座 第3回スライド
techmathproject
0
110
生成AI による論文執筆サポートの手引き(ワークショップ) / A guide to supporting dissertation writing with generative AI (workshop)
ks91
PRO
0
510
テンソル分解による糖尿病の組織特異的遺伝子発現の統合解析を用いた関連疾患の予測
tagtag
2
200
データマイニング - グラフデータと経路
trycycle
PRO
1
160
統計学入門講座 第4回スライド
techmathproject
0
150
トラブルがあったコンペに学ぶデータ分析
tereka114
2
1.6k
[Paper Introduction] From Bytes to Ideas:Language Modeling with Autoregressive U-Nets
haruumiomoto
0
110
点群ライブラリPDALをGoogleColabにて実行する方法の紹介
kentaitakura
1
310
安心・効率的な医療現場の実現へ ~オンプレAI & ノーコードワークフローで進める業務改革~
siyoo
0
260
Transport information Geometry: Current and Future II
lwc2017
0
160
眼科AIコンテスト2024_特別賞_6位Solution
pon0matsu
0
420
Featured
See All Featured
YesSQL, Process and Tooling at Scale
rocio
173
14k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Docker and Python
trallard
45
3.5k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
34
3.1k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
29
9.6k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
181
54k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
21
1.3k
Agile that works and the tools we love
rasmusluckow
329
21k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
53
2.9k
Why You Should Never Use an ORM
jnunemaker
PRO
58
9.5k
Statistics for Hackers
jakevdp
799
220k
Transcript
Պֶٕज़ܭࢉ༻Ϋϥελͷ Dockerಋೖͱӡ༻ দౢ ໌ ཧԽֶݚڀॴ ใج൫ηϯλʔ όΠΦΠϯϑΥϚςΟΫεݚڀ։ൃϢχοτ ୈ1ճHPC OPSݚڀձ 2018.03.13
౷߹σʔλϕʔεͷݚڀ։ൃΛ௨ͯ͡ ཧݚॴଐݚڀࣨͷวྺ '05 ήϊϜՊֶ૯߹ݚڀηϯλʔ ήϊϜมҟػೳใݚڀνʔϜ '07 ήϊϜՊֶ૯߹ݚڀηϯλʔ ΦϛοΫεใ౷߹ԽݚڀνʔϜ '08 ੜ໋ใج൫ݚڀ෦
(BASE) '13 ใج൫ηϯλʔ ౷߹σʔλϕʔεಛผϢχοτ '14 ใج൫ηϯλʔ όΠΦΠϯϑΥϚςΟΫεݚڀ։ൃϢχοτ (BiT)
ݚڀऀʹαΠΤϯεͤ͞Δ ϥΠϑαΠΤϯεݚڀऀ • PCαʔόʔΛϐʔΩʔͳػثͱೝࣝ • ࣮ݧػثΛѻ͖ͬͯͨܦݧʁ • ύιίϯૢ࡞Ͱͷܦݧʁ • αΠΤϯεʹूத͍ͨ͠
• ܭࢉαʔόʔͷௐୡɺཧɺӡ༻͔Β։์͞Ε͍ͨ • ࣗ༝ʹ͑ΔܭࢉϦιʔεཉ͍͠ ݚڀऀ͕ܭࢉػཧɾӡ༻ʹׂ࣌ؒ͘Λܰݮ • DevOps • Infrastructure as Code • ԾԽ • ΫϥυίϯϐϡʔςΟϯά
ίϯςφٕज़ͷ͓͞Β͍ ϗετOS Linux Linux ϗετOS Mac ίϯςφཧ ϋʔυΣΞ ϋΠύʔόΠβ ΞϓϦ
ήετOS ήετOS ΞϓϦ ΞϓϦ ԾϋʔυΣ Ξ ԾϋʔυΣ Ξ ԾϚγϯ(VM) ϋʔυΣΞ ΞϓϦ ΞϓϦ ίϯςφ ήετOS Windows Linux amatsus@hal011:~$ ps afx ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ 1926 ? Ssl 408:31 /usr/bin/dockerd --selinux-enabled=fal 2166 ? Ssl 234:33 \_ docker-containerd -l unix:///var/r 38674 ? Sl 0:00 \_ docker-containerd-shim 9c3689c 38691 pts/0 Ss+ 0:00 \_ bash ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ root@9c3689c042da:/# ps afx PID TTY STAT TIME COMMAND 1 pts/0 Ss 0:00 bash 15 pts/0 R+ 0:00 ps afx root@9c3689c042da:/# ίϯςφ ίϯςφϗετ্ amatsus@imac:~$ ps afx ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ 26675 ?? R 0:10.79 xhyve -m 4G -c 2 -s 0:0,hostbridge -s ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ amatsus@wily-xhyve:~$ ps afx PID TTY STAT TIME COMMAND 1 ? Ss 0:01 /sbin/init 2 ? S 0:00 [kthreadd] 3 ? S 0:00 [ksoftirqd/0] 4 ? S 0:00 [kworker/0:0] 5 ? S< 0:00 [kworker/0:0H] ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ VMϗετ্ VM
DockerΠϝʔδʹΑΔՄൖੑ push pull docker hub docker registry quai.io/CoreOS pull Docker
Πϝʔδ FROM docker.io/alpine:3.6 AS build-env RUN apk --update add --no-cache --virtual .build-deps build-base git zlib-dev && \ git clone https://github.com/ocxtal/ minialign.git && \ cd minialign && \ git checkout refs/tags/minialign-0.5.2 && \ sed -i "s/-march=native/-msse4 - mpopcnt/" Makefile && \ make && make install && \ apk del .build-deps && rm -rf /minialign FROM docker.io/alpine:3.6 LABEL maintainer="
[email protected]
" \ license="MIT" \ architecture="Nehalem and later" COPY --from=build-env /usr/local/bin/ minialign /usr/local/bin/minialign ENTRYPOINT [ "/usr/local/bin/minialign" ] CMD ["-h"] Dockerfile build run run run run
DockerͷϝϦοτɺσϝϦοτ ϝϦοτ • Φʔόʔϔου͕খ͍͞ • ϦιʔεΛΞϓϦ͝ͱʹ۠ΕΔ • ΞϓϦ͝ͱʹڥΛ • ಉҰߏͷίϯςφΛෳ࡞Մ
• ߏͷ࡞खॱΛίʔυԽ σϝϦοτ • ϢʔβIDɺάϧʔϓIDΛࣗ༝ʹઃఆͰ͖Δ • Χʔωϧϗετͱڞ༗ • ωοτϫʔΫϦιʔεΛϗετͱڞ༗ → ىಈ͕͍ → Մൖੑɺ࠶ݱੑͷ্ → ϦιʔεͷϜμΛܰݮ͠ߴີԽ → ϚΠάϨʔγϣϯ͕ࠔ → ΧʔωϧʹύονΛͯͮΒ͍ → ڞ༻ܭࢉػʹෆ͖ → Infrastructure as Code
ೋ֊ಊݚͷΫϥελܭࢉػ docker run bcl2fastq2:1.0 genomicpariscentre/fastqc:0.11.5 fastx_toolkit:1.0 picard:1.0 dropseq:1.0 star2.5.1b:1.0 pyper:1.2
bcl2fastq fastqc fastx_trimmer FastqToSam TagBamWithReadSequenceExtended TrimStartingSequence SamToFastq STAR SortSam MergeBamAlignment TagReadWithGeneExon correct_barcode.py DigitalExpression analog_expression.py BAMTagHistogram IMAGE CMD FileSystem Execution Nodes Submission Node NFS docker pull docker push send qsub login HUB docker pull 460,000+ Dockerized Applications / qdel / qmod -s / qmod -us © 2016 DBCLS ౷߹TV / CC-BY-4.0 on-premiss or cloud
docker runϥούʔεΫϦϓτ # # docker run wrapper for OGS/GE #
# Copyright (c) 2016 Akihiro Matsushima # Released under the MIT license # http://opensource.org/licenses/mit-license.php # function sigconthandler() { docker unpause $cid echo "caught sigcont, container unpaused." wait } function sigusr1handler() { docker pause $cid echo "caught sigusr1, container paused." wait } function sigusr2handler() { if [ `docker inspect --format="{{ .State.Status }}" $cid` == "paused" ]; then docker unpause $cid fi docker stop $cid echo "caught sigusr2, container stopped." } function docker() { # emulate fairly POSIX sh in zsh $(type "emulate" >/dev/null 2>&1) && emulate -L sh local IFS=$' \t\n’ if [ "$1" = "run" ]; then local DOCKER_RUN_LOCALOPTS=${DOCKER_RUN_OPTS:-'--net=bridge -u `id - u`:`id -g` --group-add=10100 -v /etc/passwd:/etc/passwd:ro -v /etc/group:/ etc/group:ro --security-opt seccomp:unconfined -v $HOME:$HOME -v /data/ $USER:/data/$USER -v /data2/$USER:/data2/$USER -w $PWD’} if [ -n "$JOB_ID" ]; then # define the unique cidfile name TEMPDIR=/var/tmp/${LOGNAME:-$USER} CIDFILE="${TEMPDIR}/${JOB_NAME:-SOMEJOB}.o${JOB_ID}.$ {SGE_TASK_ID:-SOMETASK}_$(date +%Y%m%d%H%M%S%3N).cid" if [ ! -e "$TEMPDIR" ]; then mkdir -p "$TEMPDIR" fi echo -e "$RUNDATE\t${LOGNAME:-$USER}\t$JOB_ID\t$SGE_TASK_ID\t/ usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) --cidfile=\"$CIDFILE\" ${@:2:($#-1)} &" >> /usr/local/gridscheduler/default/docker_cmdline /usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) -i -- cidfile="$CIDFILE" "${@:2:($#-1)}" & pid=$! while [[ -d /proc/$pid && -z $cid ]]; do sleep 1 if [ -s "$CIDFILE" ]; then read -r cid < "$CIDFILE" rm -f "$CIDFILE" fi done trap sigconthandler SIGCONT trap sigusr1handler SIGUSR1 trap sigusr2handler SIGUSR2 wait else /usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) "${@:2: ($#-1)}" fi else /usr/bin/docker "$@" fi } if [[ ! $(readlink /proc/$$/exe) =~ "zsh" ]]; then export -f sigconthandler sigusr1handler sigusr2handler docker fi https://gist.github.com/amatsus/4bdcb1498ea5a002ba41edebb122c21c
ίϯςφΛվมͤͣ༻ײΛ͚ۙͮΔ #$ -N FQC fastqc —nogroup -o fastqc_out ERR030893.fastq.gz #$
-N FQCd #$ -notify docker run genomicpariscentre/fastqc:0.11.5 —nogroup -o fastqc_out ERR030893.fastq.gz function docker() { local DOCKER_RUN_LOCALOPTS=${DOCKER_RUN_OPTS:-‘ --net=bridge --u `id -u`:`id -g` --group-add=10100 -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro --security-opt seccomp:unconfined -v $HOME:$HOME -v /data/$USER:/data/$USER -v /data2/$USER:/data2/$USER -w $PWD’} /usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) --cidfile="$CIDFILE" "${@:2:($#-1)}" & } Wrap
γάφϧϋϯυϥ /usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) -i --cidfile="$CIDFILE" "${@:2:($#-1)}" &
pid=$! while [[ -d /proc/$pid && -z $cid ]]; do sleep 1 if [ -s "$CIDFILE" ]; then read -r cid < "$CIDFILE" rm -f "$CIDFILE" fi done trap ‘docker unpause $cid; wait’ SIGCONT # qmod -us trap ‘docker pause $cid; wait’ SIGUSR1 # qmod -s trap ‘docker stop $cid’ SIGUSR2 # qdel wait $POUBJOFS*% $*% ͷऔಘ ɹίϯςφΛσλον(docker run —detach)͢ΔͱίϚϯυͷΓCID ɹσλονͯ͠͠ͳͯ͘CIDΛऔಘͰ͖ΔCIDFILEΛར༻ δϣϒεέδϡʔϥ͔ΒͷγάφϧΛτϥοϓ ɹड͚औͬͨγάφϧ͝ͱʹdockerίϚϯυΛ࣮ߦ amatsus@elwood:~$ qconf -sq | grep method starter_method NONE suspend_method NONE resume_method NONE terminate_method NONE
ΫϥελܭࢉػΛ1ΫϦοΫͰΫϥυʹ https://portal.azure.com/#create/Microsoft.Temp late/uri/https%3A%2F%2Fraw.githubusercontent.co m%2Fmanabuishii%2Fazurefiles%2Fmaster%2FNFS_SGE% 2Fazuredeploy.json ARMςϯϓϨʔτ
σʔλͷόοΫΞοϓ Technology Storage driver name Support version OverlayFS overlay v1.4ʙ
overlay2 v1.12ʙ AUFS aufs Btrfs btrfs v0.7ʙ Device Mapper device mapper v0.7ʙ VFS vfs v0.7ʙ ZFS zfs v1.7ʙ Layer A Layer B Layer A Layer B nginx Layer A Layer B nginx web app ubuntu nginx web app }ReadOnly Layer Layer A Layer B web app Btrfs,ZFSҎ֎ͷϑΝΠϧγεςϜͰϑϥοτ DockerΠϝʔδϨΠϠ nginx Docker Πϝʔδ docker hub docker registry docker push rsync όοΫΞοϓαʔό data volumesίϯςφ·ͩdocker push͍ͯ͠ͳ͍ ίϯςφόοΫΞοϓରʹ͢ΔͱͳΔͱɺɺɺ
https://developers.redhat.com/blog/2014/09/30/overview-storage-scalability-docker/ Container Create/Destroy Times
Dockerؔ࿈ίϛϡχςΟฦྱ RHEL/CentOSͷमਖ਼ͯͣʹ kernel 4.16͔Βkernel 3.10.0-237.18.2.el7όοΫϙʔτ Jupyter Projectʹ kernelύονෆཁͳWorkaroundΛఏҊ(Pull Request) Docker
1.13ʹͯlog driverʹjournaldΛࢦఆ͢Δͱෆఆظʹ dockerd͝ͱίϯςφ͕શ໓͢Δ ೋ֊ಊݚҎʹ࣮֬ʹdockerdΛམͱ͢ίϯςφΛ །Ұ͍࣋ͬͯͨͷͰमਖ਼ύονͷݕূʹڠྗ Docker for MacͰ࠾༻͞Ε͍ͯΔAufsʹಉ͕͡જࡏ ͨͨ͠ΊAufsͷ։ൃऀʹใࠂ
·ͱΊ DockerಋೖʹΑΓ • ιʔείʔυ͔ΒϏϧυɺΠϯετʔϧ͢Δ͜ͱ͕ݮͬͨ • ؾܰʹࢼͤΔ • ϗʔϜσΟϨΫτϦ͕͖ͬ͢Γ • τϥϒϧγϡʔςΟϯά͘͢͠ͳͬͨ
• ղੳͷ࠶ݱੑ্ • ͰͬͺΓDockerfile࡞Δͷ໘ େنڞ༻ܭࢉػͰͷར༻ • rootlessͳίϯςφ࣮ߦڥ͕ͨΕΔ ೋ֊ಊݚډ͔ࣨΒͷோ(20174݄)
HPC͚ʁίϯςφٕज़ Gregory M. Kurtzer, Vanessa Sochat, Michael W. Bauer, “Singularity:
Scientific containers for mobility of compute”, PLoS ONE 12.5 (2017) https://github.com/indigo-dc/udocker/ http://singularity.lbl.gov/