科学技術計算用クラスタへのDocker導入と運用 / HPC OPS mtg1

第1回HPC OPS研究会(2018/3/13)の発表資料です。
https://bit.riken.jp/2018/02/1st-hpc-ops-mtg/

Transcript

1. Պֶٕज़ܭࢉ༻Ϋϥελ΁ͷ Dockerಋೖͱӡ༻ দౢ ໌޺ ཧԽֶݚڀॴ ৘ใج൫ηϯλʔ όΠΦΠϯϑΥϚςΟΫεݚڀ։ൃϢχοτ ୈ1ճHPC OPSݚڀձ 2018.03.13
2. ౷߹σʔλϕʔεͷݚڀ։ൃΛ௨ͯ͡ ཧݚॴଐݚڀࣨͷวྺ '05 ήϊϜՊֶ૯߹ݚڀηϯλʔ ήϊϜมҟػೳ৘ใݚڀνʔϜ '07 ήϊϜՊֶ૯߹ݚڀηϯλʔ ΦϛοΫε৘ใ౷߹ԽݚڀνʔϜ '08 ੜ໋৘ใج൫ݚڀ෦໳ (BASE) '13 ৘ใج൫ηϯλʔ ౷߹σʔλϕʔεಛผϢχοτ '14 ৘ใج൫ηϯλʔ όΠΦΠϯϑΥϚςΟΫεݚڀ։ൃϢχοτ (BiT)
3. ݚڀऀʹαΠΤϯεͤ͞Δ ϥΠϑαΠΤϯεݚڀऀ • PCαʔόʔΛϐʔΩʔͳػثͱೝࣝ • ࣮ݧػثΛѻ͖ͬͯͨܦݧʁ • ύιίϯૢ࡞Ͱͷܦݧʁ • αΠΤϯεʹूத͍ͨ͠ • ܭࢉαʔόʔͷௐୡɺ؅ཧɺӡ༻͔Β։์͞Ε͍ͨ • ࣗ༝ʹ࢖͑ΔܭࢉϦιʔε͸ཉ͍͠ ݚڀऀ͕ܭࢉػ؅ཧɾӡ༻ʹׂ࣌ؒ͘Λܰݮ • DevOps • Infrastructure as Code • Ծ૝Խ • Ϋϥ΢υίϯϐϡʔςΟϯά
4. ίϯςφٕज़ͷ͓͞Β͍ ϗετOS Linux Linux ϗετOS Mac ίϯςφ؅ཧ ϋʔυ΢ΣΞ ϋΠύʔόΠβ ΞϓϦ ήετOS ήετOS ΞϓϦ ΞϓϦ Ծ૝ϋʔυ΢Σ Ξ Ծ૝ϋʔυ΢Σ Ξ Ծ૝Ϛγϯ(VM) ϋʔυ΢ΣΞ ΞϓϦ ΞϓϦ ίϯςφ ήετOS Windows Linux amatsus@hal011:~$ ps afx ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ 1926 ? Ssl 408:31 /usr/bin/dockerd --selinux-enabled=fal 2166 ? Ssl 234:33 \_ docker-containerd -l unix:///var/r 38674 ? Sl 0:00 \_ docker-containerd-shim 9c3689c 38691 pts/0 Ss+ 0:00 \_ bash ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ root@9c3689c042da:/# ps afx PID TTY STAT TIME COMMAND 1 pts/0 Ss 0:00 bash 15 pts/0 R+ 0:00 ps afx root@9c3689c042da:/# ίϯςφ಺ ίϯςφϗετ্ amatsus@imac:~$ ps afx ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ 26675 ?? R 0:10.79 xhyve -m 4G -c 2 -s 0:0,hostbridge -s ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ amatsus@wily-xhyve:~$ ps afx PID TTY STAT TIME COMMAND 1 ? Ss 0:01 /sbin/init 2 ? S 0:00 [kthreadd] 3 ? S 0:00 [ksoftirqd/0] 4 ? S 0:00 [kworker/0:0] 5 ? S< 0:00 [kworker/0:0H] ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɾ VMϗετ্ VM಺
5. DockerΠϝʔδʹΑΔՄൖੑ push pull docker hub docker registry quai.io/CoreOS pull Docker Πϝʔδ FROM docker.io/alpine:3.6 AS build-env RUN apk --update add --no-cache --virtual .build-deps build-base git zlib-dev && \ git clone https://github.com/ocxtal/ minialign.git && \ cd minialign && \ git checkout refs/tags/minialign-0.5.2 && \ sed -i "s/-march=native/-msse4 - mpopcnt/" Makefile && \ make && make install && \ apk del .build-deps && rm -rf /minialign FROM docker.io/alpine:3.6 LABEL maintainer="akihiro.matsushima@riken.jp " \ license="MIT" \ architecture="Nehalem and later" COPY --from=build-env /usr/local/bin/ minialign /usr/local/bin/minialign ENTRYPOINT [ "/usr/local/bin/minialign" ] CMD ["-h"] Dockerfile build run run run run
6. DockerͷϝϦοτɺσϝϦοτ ϝϦοτ • Φʔόʔϔου͕খ͍͞ • ϦιʔεΛΞϓϦ͝ͱʹ۠੾ΕΔ • ΞϓϦ͝ͱʹ؀ڥΛ෼཭ • ಉҰߏ੒ͷίϯςφΛෳ਺࡞੒Մ • ߏ੒ͷ࡞੒खॱΛίʔυԽ σϝϦοτ • ϢʔβIDɺάϧʔϓIDΛࣗ༝ʹઃఆͰ͖Δ • Χʔωϧ͸ϗετͱڞ༗ • ωοτϫʔΫϦιʔεΛϗετͱڞ༗ → ىಈ͕଎͍ → Մൖੑɺ࠶ݱੑͷ޲্ → ϦιʔεͷϜμΛܰݮ͠ߴີ౓Խ → ϚΠάϨʔγϣϯ͕ࠔ೉ → ΧʔωϧʹύονΛ౰ͯͮΒ͍ → ڞ༻ܭࢉػʹෆ޲͖ → Infrastructure as Code
7. ೋ֊ಊݚͷΫϥελܭࢉػ docker run bcl2fastq2:1.0 genomicpariscentre/fastqc:0.11.5 fastx_toolkit:1.0 picard:1.0 dropseq:1.0 star2.5.1b:1.0 pyper:1.2 bcl2fastq fastqc fastx_trimmer FastqToSam TagBamWithReadSequenceExtended TrimStartingSequence SamToFastq STAR SortSam MergeBamAlignment TagReadWithGeneExon correct_barcode.py DigitalExpression analog_expression.py BAMTagHistogram IMAGE CMD FileSystem Execution Nodes Submission Node NFS docker pull docker push send qsub login HUB docker pull 460,000+ Dockerized Applications / qdel / qmod -s / qmod -us © 2016 DBCLS ౷߹TV / CC-BY-4.0 on-premiss or cloud
8. docker runϥούʔεΫϦϓτ # # docker run wrapper for OGS/GE # # Copyright (c) 2016 Akihiro Matsushima # Released under the MIT license # http://opensource.org/licenses/mit-license.php # function sigconthandler() { docker unpause $cid echo "caught sigcont, container unpaused." wait } function sigusr1handler() { docker pause $cid echo "caught sigusr1, container paused." wait } function sigusr2handler() { if [ `docker inspect --format="{{ .State.Status }}" $cid` == "paused" ]; then docker unpause $cid fi docker stop $cid echo "caught sigusr2, container stopped." } function docker() { # emulate fairly POSIX sh in zsh $(type "emulate" >/dev/null 2>&1) && emulate -L sh local IFS=$' \t\n’ if [ "$1" = "run" ]; then local DOCKER_RUN_LOCALOPTS=${DOCKER_RUN_OPTS:-'--net=bridge -u `id - u`:`id -g` --group-add=10100 -v /etc/passwd:/etc/passwd:ro -v /etc/group:/ etc/group:ro --security-opt seccomp:unconfined -v $HOME:$HOME -v /data/ $USER:/data/$USER -v /data2/$USER:/data2/$USER -w $PWD’} if [ -n "$JOB_ID" ]; then # define the unique cidfile name TEMPDIR=/var/tmp/${LOGNAME:-$USER} CIDFILE="${TEMPDIR}/${JOB_NAME:-SOMEJOB}.o${JOB_ID}.$ {SGE_TASK_ID:-SOMETASK}_$(date +%Y%m%d%H%M%S%3N).cid" if [ ! -e "$TEMPDIR" ]; then mkdir -p "$TEMPDIR" fi echo -e "$RUNDATE\t${LOGNAME:-$USER}\t$JOB_ID\t$SGE_TASK_ID\t/ usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) --cidfile=\"$CIDFILE\" ${@:2:($#-1)} &" >> /usr/local/gridscheduler/default/docker_cmdline /usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) -i -- cidfile="$CIDFILE" "${@:2:($#-1)}" & pid=$! while [[ -d /proc/$pid && -z $cid ]]; do sleep 1 if [ -s "$CIDFILE" ]; then read -r cid < "$CIDFILE" rm -f "$CIDFILE" fi done trap sigconthandler SIGCONT trap sigusr1handler SIGUSR1 trap sigusr2handler SIGUSR2 wait else /usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) "${@:2: ($#-1)}" fi else /usr/bin/docker "$@" fi } if [[ ! $(readlink /proc/$$/exe) =~ "zsh" ]]; then export -f sigconthandler sigusr1handler sigusr2handler docker fi https://gist.github.com/amatsus/4bdcb1498ea5a002ba41edebb122c21c
9. ίϯςφΛվมͤͣ࢖༻ײΛ͚ۙͮΔ #$ -N FQC fastqc —nogroup -o fastqc_out ERR030893.fastq.gz #$ -N FQCd #$ -notify docker run genomicpariscentre/fastqc:0.11.5 —nogroup -o fastqc_out ERR030893.fastq.gz function docker() { local DOCKER_RUN_LOCALOPTS=${DOCKER_RUN_OPTS:-‘ --net=bridge --u `id -u`:`id -g` --group-add=10100 -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro --security-opt seccomp:unconfined -v $HOME:$HOME -v /data/$USER:/data/$USER -v /data2/$USER:/data2/$USER -w $PWD’} /usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) --cidfile="$CIDFILE" "${@:2:($#-1)}" & } Wrap
10. γάφϧϋϯυϥ /usr/bin/docker run $(eval echo $DOCKER_RUN_LOCALOPTS) -i --cidfile="$CIDFILE" "${@:2:($#-1)}" & pid=$! while [[ -d /proc/$pid && -z $cid ]]; do sleep 1 if [ -s "$CIDFILE" ]; then read -r cid < "$CIDFILE" rm -f "$CIDFILE" fi done trap ‘docker unpause $cid; wait’ SIGCONT # qmod -us trap ‘docker pause $cid; wait’ SIGUSR1 # qmod -s trap ‘docker stop $cid’ SIGUSR2 # qdel wait $POUBJOFS
    *%
    $*%
    ͷऔಘ
    ɹίϯςφΛσλον(docker run —detach)͢ΔͱίϚϯυͷ໭Γ஋͸CID ɹσλονͯ͠΋͠ͳͯ͘΋CIDΛऔಘͰ͖ΔCIDFILEΛར༻ δϣϒεέδϡʔϥ͔ΒͷγάφϧΛτϥοϓ
    ɹड͚औͬͨγάφϧ͝ͱʹdockerίϚϯυΛ࣮ߦ amatsus@elwood:~$ qconf -sq | grep method starter_method NONE suspend_method NONE resume_method NONE terminate_method NONE
11. ΫϥελܭࢉػΛ1ΫϦοΫͰΫϥ΢υʹ https://portal.azure.com/#create/Microsoft.Temp late/uri/https%3A%2F%2Fraw.githubusercontent.co m%2Fmanabuishii%2Fazurefiles%2Fmaster%2FNFS_SGE% 2Fazuredeploy.json ARMςϯϓϨʔτ
12. σʔλͷόοΫΞοϓ Technology Storage driver name Support version OverlayFS overlay v1.4ʙ overlay2 v1.12ʙ AUFS aufs Btrfs btrfs v0.7ʙ Device Mapper device mapper v0.7ʙ VFS vfs v0.7ʙ ZFS zfs v1.7ʙ Layer A Layer B Layer A Layer B nginx Layer A Layer B nginx web app ubuntu nginx web app }ReadOnly Layer Layer A Layer B web app Btrfs,ZFSҎ֎ͷϑΝΠϧγεςϜͰ͸ϑϥοτ DockerΠϝʔδϨΠϠ nginx Docker Πϝʔδ docker hub docker registry docker push rsync όοΫΞοϓαʔό data volumesίϯςφ΍·ͩdocker push͍ͯ͠ͳ͍
 ίϯςφ΋όοΫΞοϓର৅ʹ͢ΔͱͳΔͱɺɺɺ
13. https://developers.redhat.com/blog/2014/09/30/overview-storage-scalability-docker/ Container Create/Destroy Times
14. Dockerؔ࿈ίϛϡχςΟ΁ฦྱ RHEL/CentOSͷमਖ਼଴ͯͣʹ kernel 4.16͔Βkernel 3.10.0-237.18.2.el7΁όοΫϙʔτ Jupyter Projectʹ͸ kernelύονෆཁͳWorkaroundΛఏҊ(Pull Request) Docker 1.13ʹͯlog driverʹjournaldΛࢦఆ͢Δͱෆఆظʹ dockerd͝ͱίϯςφ͕શ໓͢Δ໰୊ ೋ֊ಊݚ͸਺෼Ҏ಺ʹ࣮֬ʹdockerdΛམͱ͢ίϯςφΛ །Ұ͍࣋ͬͯͨͷͰमਖ਼ύονͷݕূʹڠྗ Docker for MacͰ࠾༻͞Ε͍ͯΔAufsʹ΋ಉ͡໰୊͕જࡏ ͨͨ͠ΊAufsͷ։ൃऀʹ΋ใࠂ
15. ·ͱΊ DockerಋೖʹΑΓ • ιʔείʔυ͔ΒϏϧυɺΠϯετʔϧ͢Δ͜ͱ͕ݮͬͨ • ؾܰʹࢼͤΔ • ϗʔϜσΟϨΫτϦ͕͖ͬ͢Γ • τϥϒϧγϡʔςΟϯά͠΍͘͢ͳͬͨ • ղੳͷ࠶ݱੑ޲্ • Ͱ΋΍ͬͺΓDockerfile࡞Δͷ໘౗ େن໛ڞ༻ܭࢉػͰͷར༻͸ • rootlessͳίϯςφ࣮ߦ؀ڥ͕଴ͨΕΔ ೋ֊ಊݚډ͔ࣨΒͷோ๬(2017೥4݄)
16. HPC޲͚ʁίϯςφٕज़ Gregory M. Kurtzer, Vanessa Sochat, Michael W. Bauer, “Singularity: Scientific containers for mobility of compute”, PLoS ONE 12.5 (2017) https://github.com/indigo-dc/udocker/ http://singularity.lbl.gov/