Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
科学技術計算用クラスタへのDocker導入と運用 / HPC OPS mtg1
Search
Akihiro MATSUSHiMA
March 14, 2018
Science
0
710
科学技術計算用クラスタへのDocker導入と運用 / HPC OPS mtg1
第1回HPC OPS研究会(2018/3/13)の発表資料です。
https://bit.riken.jp/2018/02/1st-hpc-ops-mtg/
Akihiro MATSUSHiMA
March 14, 2018
Tweet
Share
Other Decks in Science
See All in Science
Ignite の1年間の軌跡
ktombow
0
130
MCMCのR-hatは分散分析である
moricup
0
380
安心・効率的な医療現場の実現へ ~オンプレAI & ノーコードワークフローで進める業務改革~
siyoo
0
260
LayerXにおける業務の完全自動運転化に向けたAI技術活用事例 / layerx-ai-jsai2025
shimacos
2
1.2k
機械学習 - 決定木からはじめる機械学習
trycycle
PRO
0
990
データベース01: データベースを使わない世界
trycycle
PRO
1
660
研究って何だっけ / What is Research?
ks91
PRO
1
100
論文紹介 音源分離:SCNET SPARSE COMPRESSION NETWORK FOR MUSIC SOURCE SEPARATION
kenmatsu4
0
210
メール送信サーバの集約における透過型SMTP プロキシの定量評価 / Quantitative Evaluation of Transparent SMTP Proxy in Email Sending Server Aggregation
linyows
0
940
データベース05: SQL(2/3) 結合質問
trycycle
PRO
0
710
実力評価性能を考慮した弓道高校生全国大会の大会制度設計の提案 / (konakalab presentation at MSS 2025.03)
konakalab
2
180
3次元点群を利用した植物の葉の自動セグメンテーションについて
kentaitakura
2
1.3k
Featured
See All Featured
Become a Pro
speakerdeck
PRO
29
5.4k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
45
7.5k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.3k
Rebuilding a faster, lazier Slack
samanthasiow
82
9.1k
Making the Leap to Tech Lead
cromwellryan
134
9.4k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
161
15k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
126
53k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
30
2.1k
Mobile First: as difficult as doing things right
swwweet
223
9.7k
We Have a Design System, Now What?
morganepeng
53
7.7k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.7k
Art, The Web, and Tiny UX
lynnandtonic
299
21k
Transcript
Պֶٕज़ܭࢉ༻Ϋϥελͷ Dockerಋೖͱӡ༻ দౢ ໌ ཧԽֶݚڀॴ ใج൫ηϯλʔ όΠΦΠϯϑΥϚςΟΫεݚڀ։ൃϢχοτ ୈ1ճHPC OPSݚڀձ 2018.03.13
౷߹σʔλϕʔεͷݚڀ։ൃΛ௨ͯ͡ ཧݚॴଐݚڀࣨͷวྺ '05 ήϊϜՊֶ૯߹ݚڀηϯλʔ ήϊϜมҟػೳใݚڀνʔϜ '07 ήϊϜՊֶ૯߹ݚڀηϯλʔ ΦϛοΫεใ౷߹ԽݚڀνʔϜ '08 ੜ໋ใج൫ݚڀ෦
(BASE) '13 ใج൫ηϯλʔ ౷߹σʔλϕʔεಛผϢχοτ '14 ใج൫ηϯλʔ όΠΦΠϯϑΥϚςΟΫεݚڀ։ൃϢχοτ (BiT)
ݚڀऀʹαΠΤϯεͤ͞Δ ϥΠϑαΠΤϯεݚڀऀ • PCαʔόʔΛϐʔΩʔͳػثͱೝࣝ • ࣮ݧػثΛѻ͖ͬͯͨܦݧʁ • ύιίϯૢ࡞Ͱͷܦݧʁ • αΠΤϯεʹूத͍ͨ͠
• ܭࢉαʔόʔͷௐୡɺཧɺӡ༻͔Β։์͞Ε͍ͨ • ࣗ༝ʹ͑ΔܭࢉϦιʔεཉ͍͠ ݚڀऀ͕ܭࢉػཧɾӡ༻ʹׂ࣌ؒ͘Λܰݮ • DevOps • Infrastructure as Code • ԾԽ • ΫϥυίϯϐϡʔςΟϯά
ίϯςφٕज़ͷ͓͞Β͍ ϗετOS Linux Linux ϗετOS Mac ίϯςφཧ ϋʔυΣΞ ϋΠύʔόΠβ ΞϓϦ
ήετOS ήετOS ΞϓϦ ΞϓϦ ԾϋʔυΣ Ξ ԾϋʔυΣ Ξ ԾϚγϯ(VM) ϋʔυΣΞ ΞϓϦ ΞϓϦ ίϯςφ ήετOS Windows Linux amatsus@hal011:~$ ps afx ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ 1926 ? Ssl 408:31 /usr/bin/dockerd --selinux-enabled=fal 2166 ? Ssl 234:33 \_ docker-containerd -l unix:///var/r 38674 ? Sl 0:00 \_ docker-containerd-shim 9c3689c 38691 pts/0 Ss+ 0:00 \_ bash ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ root@9c3689c042da:/# ps afx PID TTY STAT TIME COMMAND 1 pts/0 Ss 0:00 bash 15 pts/0 R+ 0:00 ps afx root@9c3689c042da:/# ίϯςφ ίϯςφϗετ্ amatsus@imac:~$ ps afx ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ 26675 ?? R 0:10.79 xhyve -m 4G -c 2 -s 0:0,hostbridge -s ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ amatsus@wily-xhyve:~$ ps afx PID TTY STAT TIME COMMAND 1 ? Ss 0:01 /sbin/init 2 ? S 0:00 [kthreadd] 3 ? S 0:00 [ksoftirqd/0] 4 ? S 0:00 [kworker/0:0] 5 ? S< 0:00 [kworker/0:0H] ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ VMϗετ্ VM
DockerΠϝʔδʹΑΔՄൖੑ push pull docker hub docker registry quai.io/CoreOS pull Docker
Πϝʔδ FROM docker.io/alpine:3.6 AS build-env RUN apk --update add --no-cache --virtual .build-deps build-base git zlib-dev && \ git clone https://github.com/ocxtal/ minialign.git && \ cd minialign && \ git checkout refs/tags/minialign-0.5.2 && \ sed -i "s/-march=native/-msse4 - mpopcnt/" Makefile && \ make && make install && \ apk del .build-deps && rm -rf /minialign FROM docker.io/alpine:3.6 LABEL maintainer="
[email protected]
" \ license="MIT" \ architecture="Nehalem and later" COPY --from=build-env /usr/local/bin/ minialign /usr/local/bin/minialign ENTRYPOINT [ "/usr/local/bin/minialign" ] CMD ["-h"] Dockerfile build run run run run
DockerͷϝϦοτɺσϝϦοτ ϝϦοτ • Φʔόʔϔου͕খ͍͞ • ϦιʔεΛΞϓϦ͝ͱʹ۠ΕΔ • ΞϓϦ͝ͱʹڥΛ • ಉҰߏͷίϯςφΛෳ࡞Մ
• ߏͷ࡞खॱΛίʔυԽ σϝϦοτ • ϢʔβIDɺάϧʔϓIDΛࣗ༝ʹઃఆͰ͖Δ • Χʔωϧϗετͱڞ༗ • ωοτϫʔΫϦιʔεΛϗετͱڞ༗ → ىಈ͕͍ → Մൖੑɺ࠶ݱੑͷ্ → ϦιʔεͷϜμΛܰݮ͠ߴີԽ → ϚΠάϨʔγϣϯ͕ࠔ → ΧʔωϧʹύονΛͯͮΒ͍ → ڞ༻ܭࢉػʹෆ͖ → Infrastructure as Code
ೋ֊ಊݚͷΫϥελܭࢉػ docker run bcl2fastq2:1.0 genomicpariscentre/fastqc:0.11.5 fastx_toolkit:1.0 picard:1.0 dropseq:1.0 star2.5.1b:1.0 pyper:1.2
bcl2fastq fastqc fastx_trimmer FastqToSam TagBamWithReadSequenceExtended TrimStartingSequence SamToFastq STAR SortSam MergeBamAlignment TagReadWithGeneExon correct_barcode.py DigitalExpression analog_expression.py BAMTagHistogram IMAGE CMD FileSystem Execution Nodes Submission Node NFS docker pull docker push send qsub login HUB docker pull 460,000+ Dockerized Applications / qdel / qmod -s / qmod -us © 2016 DBCLS ౷߹TV / CC-BY-4.0 on-premiss or cloud
docker runϥούʔεΫϦϓτ # # docker run wrapper for OGS/GE #
# Copyright (c) 2016 Akihiro Matsushima # Released under the MIT license # http://opensource.org/licenses/mit-license.php # function sigconthandler() { docker unpause $cid echo "caught sigcont, container unpaused." wait } function sigusr1handler() { docker pause $cid echo "caught sigusr1, container paused." wait } function sigusr2handler() { if [ `docker inspect --format="{{ .State.Status }}" $cid` == "paused" ]; then docker unpause $cid fi docker stop $cid echo "caught sigusr2, container stopped." } function docker() { # emulate fairly POSIX sh in zsh $(type "emulate" >/dev/null 2>&1) && emulate -L sh local IFS=$' \t\n’ if [ "$1" = "run" ]; then local DOCKER_RUN_LOCALOPTS=${DOCKER_RUN_OPTS:-'--net=bridge -u `id - u`:`id -g` --group-add=10100 -v /etc/passwd:/etc/passwd:ro -v /etc/group:/ etc/group:ro --security-opt seccomp:unconfined -v $HOME:$HOME -v /data/ $USER:/data/$USER -v /data2/$USER:/data2/$USER -w $PWD’} if [ -n "$JOB_ID" ]; then # define the unique cidfile name TEMPDIR=/var/tmp/${LOGNAME:-$USER} CIDFILE="${TEMPDIR}/${JOB_NAME:-SOMEJOB}.o${JOB_ID}.$ {SGE_TASK_ID:-SOMETASK}_$(date +%Y%m%d%H%M%S%3N).cid" if [ ! -e "$TEMPDIR" ]; then mkdir -p "$TEMPDIR" fi echo -e "$RUNDATE\t${LOGNAME:-$USER}\t$JOB_ID\t$SGE_TASK_ID\t/ usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) --cidfile=\"$CIDFILE\" ${@:2:($#-1)} &" >> /usr/local/gridscheduler/default/docker_cmdline /usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) -i -- cidfile="$CIDFILE" "${@:2:($#-1)}" & pid=$! while [[ -d /proc/$pid && -z $cid ]]; do sleep 1 if [ -s "$CIDFILE" ]; then read -r cid < "$CIDFILE" rm -f "$CIDFILE" fi done trap sigconthandler SIGCONT trap sigusr1handler SIGUSR1 trap sigusr2handler SIGUSR2 wait else /usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) "${@:2: ($#-1)}" fi else /usr/bin/docker "$@" fi } if [[ ! $(readlink /proc/$$/exe) =~ "zsh" ]]; then export -f sigconthandler sigusr1handler sigusr2handler docker fi https://gist.github.com/amatsus/4bdcb1498ea5a002ba41edebb122c21c
ίϯςφΛվมͤͣ༻ײΛ͚ۙͮΔ #$ -N FQC fastqc —nogroup -o fastqc_out ERR030893.fastq.gz #$
-N FQCd #$ -notify docker run genomicpariscentre/fastqc:0.11.5 —nogroup -o fastqc_out ERR030893.fastq.gz function docker() { local DOCKER_RUN_LOCALOPTS=${DOCKER_RUN_OPTS:-‘ --net=bridge --u `id -u`:`id -g` --group-add=10100 -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro --security-opt seccomp:unconfined -v $HOME:$HOME -v /data/$USER:/data/$USER -v /data2/$USER:/data2/$USER -w $PWD’} /usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) --cidfile="$CIDFILE" "${@:2:($#-1)}" & } Wrap
γάφϧϋϯυϥ /usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) -i --cidfile="$CIDFILE" "${@:2:($#-1)}" &
pid=$! while [[ -d /proc/$pid && -z $cid ]]; do sleep 1 if [ -s "$CIDFILE" ]; then read -r cid < "$CIDFILE" rm -f "$CIDFILE" fi done trap ‘docker unpause $cid; wait’ SIGCONT # qmod -us trap ‘docker pause $cid; wait’ SIGUSR1 # qmod -s trap ‘docker stop $cid’ SIGUSR2 # qdel wait $POUBJOFS*% $*% ͷऔಘ ɹίϯςφΛσλον(docker run —detach)͢ΔͱίϚϯυͷΓCID ɹσλονͯ͠͠ͳͯ͘CIDΛऔಘͰ͖ΔCIDFILEΛར༻ δϣϒεέδϡʔϥ͔ΒͷγάφϧΛτϥοϓ ɹड͚औͬͨγάφϧ͝ͱʹdockerίϚϯυΛ࣮ߦ amatsus@elwood:~$ qconf -sq | grep method starter_method NONE suspend_method NONE resume_method NONE terminate_method NONE
ΫϥελܭࢉػΛ1ΫϦοΫͰΫϥυʹ https://portal.azure.com/#create/Microsoft.Temp late/uri/https%3A%2F%2Fraw.githubusercontent.co m%2Fmanabuishii%2Fazurefiles%2Fmaster%2FNFS_SGE% 2Fazuredeploy.json ARMςϯϓϨʔτ
σʔλͷόοΫΞοϓ Technology Storage driver name Support version OverlayFS overlay v1.4ʙ
overlay2 v1.12ʙ AUFS aufs Btrfs btrfs v0.7ʙ Device Mapper device mapper v0.7ʙ VFS vfs v0.7ʙ ZFS zfs v1.7ʙ Layer A Layer B Layer A Layer B nginx Layer A Layer B nginx web app ubuntu nginx web app }ReadOnly Layer Layer A Layer B web app Btrfs,ZFSҎ֎ͷϑΝΠϧγεςϜͰϑϥοτ DockerΠϝʔδϨΠϠ nginx Docker Πϝʔδ docker hub docker registry docker push rsync όοΫΞοϓαʔό data volumesίϯςφ·ͩdocker push͍ͯ͠ͳ͍ ίϯςφόοΫΞοϓରʹ͢ΔͱͳΔͱɺɺɺ
https://developers.redhat.com/blog/2014/09/30/overview-storage-scalability-docker/ Container Create/Destroy Times
Dockerؔ࿈ίϛϡχςΟฦྱ RHEL/CentOSͷमਖ਼ͯͣʹ kernel 4.16͔Βkernel 3.10.0-237.18.2.el7όοΫϙʔτ Jupyter Projectʹ kernelύονෆཁͳWorkaroundΛఏҊ(Pull Request) Docker
1.13ʹͯlog driverʹjournaldΛࢦఆ͢Δͱෆఆظʹ dockerd͝ͱίϯςφ͕શ໓͢Δ ೋ֊ಊݚҎʹ࣮֬ʹdockerdΛམͱ͢ίϯςφΛ །Ұ͍࣋ͬͯͨͷͰमਖ਼ύονͷݕূʹڠྗ Docker for MacͰ࠾༻͞Ε͍ͯΔAufsʹಉ͕͡જࡏ ͨͨ͠ΊAufsͷ։ൃऀʹใࠂ
·ͱΊ DockerಋೖʹΑΓ • ιʔείʔυ͔ΒϏϧυɺΠϯετʔϧ͢Δ͜ͱ͕ݮͬͨ • ؾܰʹࢼͤΔ • ϗʔϜσΟϨΫτϦ͕͖ͬ͢Γ • τϥϒϧγϡʔςΟϯά͘͢͠ͳͬͨ
• ղੳͷ࠶ݱੑ্ • ͰͬͺΓDockerfile࡞Δͷ໘ େنڞ༻ܭࢉػͰͷར༻ • rootlessͳίϯςφ࣮ߦڥ͕ͨΕΔ ೋ֊ಊݚډ͔ࣨΒͷோ(20174݄)
HPC͚ʁίϯςφٕज़ Gregory M. Kurtzer, Vanessa Sochat, Michael W. Bauer, “Singularity:
Scientific containers for mobility of compute”, PLoS ONE 12.5 (2017) https://github.com/indigo-dc/udocker/ http://singularity.lbl.gov/