Share
Embed
Reproducibility as a Service: Virtual Appliance for NGS data analysis
Docker Container, Apache Mesos, and NIG SuperComputer Facilities. Prepared for 第31回 DDBJing 講習会 in 東京. See Also: http://www.slideshare.net/dritoshi/20150612-ddb-jingjst
Tazro Inutano Ohta
June 12, 2015
Transcript
66 slides
-
ܰྔԾڥΛ༻͍ͯNGSσʔλͷղੳ࠶ݱੑΛ୲อ͢Δ Reproducibility as a Service: Virtual Appliance for NGS data
analysis Tazro Ohta [email protected] Database Center for Life Science, Japan prepared for ୈ31ճ DDBJing ߨशձ in ౦ژ 12 Jun. 2015 -
1. Computational Resource Management ܭࢉػϦιʔε͕Γͳ͍ɼͲΕ͚ͩ͋ͬͯશવΓͳ͍ 2. Sharing Analysis Protocols and
History ղੳϓϩτίϧͷڞ༗ͱ࠶࣮ߦͷίετ͕ߴ͍ɼࣗଞਓ࠶ݱͰ͖ͳ͍ Challenges -
Challenges #1 Computational Resource Management • ͔ͭͯ࣍ੈͱݺΕͨγʔέϯαʹΑΔσʔλͷരൃత૿Ճ • ήϊϜՊֶܭࢉػͱֶΛۦ͢ΔσʔλɾαΠΤϯε •
ҨݚεύίϯήϊϜղੳ͚ʹεύίϯγεςϜΛߋ৽ (2012) • !!! શແྉ !!! • Ϣʔβ͕૿͑ɺδϣϒ͕૿͑ɺσʔλ͕૿͑ɺෆຬ૿͑ͨ • ҨݚϢʔβձʹ͓͍ͯར༻ऀͷཁΛώΞϦϯά • Ϧιʔε͕Γͳ͍ • δϣϒ͕࣮ߦͰ͖ͳ͍ • ετϨʔδ͕Γͳ͍ • SGEͷݶք -
େֶڞಉར༻ػؔ๏ਓ ใɾγεςϜݚڀػߏ ࠃཱҨֶݚڀॴ SuperComputer Facilities of National Institute of Genetics
photo from http://sc.ddbj.nig.ac.jp/index.php/ja-gallery -
Challenges #1 Computational Resource Management • Ҩݚεύίϯ͕ཁٻ͞Ε͍ͯΔ͜ͱ • ܭࢉػࢿݯར༻ͷޮ͕࠷େԽ͞Ε͍ͯΔ͜ͱ •
ιϑτΣΞ։ൃऀʹͱͬͯࣗ༝ͷߴ͍ڥͰ͋Δ͜ͱ • ܭࢉػͷ͕ࣝͳͯ͘େنͳσʔλղੳ͕࣮ߦͰ͖Δ͜ͱ • ѻ͏σʔλͷηΩϡϦςΟ͕୲อ͞Ε͍ͯΔ͜ͱ • ओཁͳDB͕͍͍͢ঢ়ଶͰΞΫηεͰ͖Δ͜ͱ -
Sharing Analysis Protocols and History Challenges #2 • ରతʹεύίϯΛར༻ͨ͠σʔλղੳΛ(ࣗͰ|ଞਓ͕)࠶ݱͰ͖ͳ͍ •
ڞ༻ܭࢉػͰσʔλղੳͷͨΊͷڥߏஙʹ͔͔Δίετ͕ߴ͘ͳΔ • ڥߏஙʙ࣮ߦ·Ͱͷղੳͷ࠶ݱʹਓखΛཁ͢Δ • ղੳͷڥઃఆ/࣮ߦཤྺΛڞ༗ɺ࠶࣮ߦ͢ΔͨΊͷํ๏͕౷Ұ͞Ε͍ͯͳ͍ • GalaxyͷHistoryͷΑ͏ͳػೳ͋ΒΏΔσʔλղੳʹͱͬͯඞཁ • ݱঢ়ͰࣗવݴޠΛհͨ͠ใަ͕ߦΘΕ͍ͯΔ • Materials and Methods • Supplementary -
1. Virtual Appliance: Linux Containers ίϯςφܕԾͱʮ࣮ߦՄೳͳܭࢉػڥʯͷهड़ 2. Apache Mesos: Abstract
Computational Resources ܭࢉػࢿݯͷநԽʹΑΔʮόʔνϟϧͳڊେσʔληϯλʔʯͷ࣮ݱ 3. Data Handling and Workflow Execution ڊେσʔλͱϫʔΫϑϩʔ࣮ߦɼήϊϜՊֶͷཁٻΛຬٕͨ͢ज़ Technologies -
Technologies #1 Virtual Appliance: Linux Containers • Linux Containers (LXC)
• ෳͷಠཱͨ͠LinuxγεςϜ(ίϯςφ)Λ1ͭͷLinuxϗετ্ͰՔಇͤ͞Δ • ϋΠύʔόΠβΛՔಇͤͣ͞ɺϦιʔεΛϗετ͔Βಠཱͤ͞Δ • cgroupʹΑΔCPU, memory, block I/O, network, namespaceͷ • ͜Ε·ͰͷίϯςφܕԾͷྲྀΕΛΉٕज़ • IBM Mainframes LPARɼParallels VirtuozzoɼSoralis Containers • http://en.wikipedia.org/wiki/LXC -
Technologies #1 Virtual Appliance: Linux Containers To Docker • Docker
• ίϯςφܕԾԽͷٕज़͔ΒϓϥοτϑΥʔϜ • LXCʹAUFS (Another Unionfs) ΛΈ߹ΘͤͯίϯςφΠϝʔδΛ੍ޚ͢Δ • ϑΝΠϧγεςϜʹࠩσʔλΛॏͶಁաతʹѻ͏ٕज़ • ݱࡏͷDocker (v0.9Ҏ߱) LXCͰͳ͘Docker ಠࣗͷ libcontainer Λ࠾༻ • DockerʹΑͬͯίϯςφԾ͕τϨϯυʹ • www.docker.com • -
Dockerfile: ίϯςφͷϏϧυखॱΛίʔυͰهड़͢Δ (Infrastructure as Code) http://hub.docker.com/u/inutano/fastqc/ $ nano Dockerfile #
edit Dockerfile $ docker build -t inutano/fastqc . $ docker run -it inutano/fastqc fastqc sample.fastq -
Technologies #2 Apache Mesos: Abstract Computational Resources • ίϯςφΛεύίϯͰಈ͔ͨ͢ΊʹϦιʔεϚωʔδϟͱεέδϡʔϥ͕ඞཁ •
ैདྷܕͷδϣϒࢄ࣮ߦΤϯδϯͰ͋ΔSGEͰॊೈੑʹ͋Γ • ͬͯΔͱ͜Ζ͋Δ: http://www.nextflow.io/ • ಈతͳεέʔϧΞτεφοϓγϣοτʹରԠ͍ͨ͠ • Apache Mesos http://mesos.apache.org/ • ෳͷDC/ܭࢉϊʔυΛଋͶͯ1ͭͷϚγϯͷΑ͏ʹݟͤΔٕज़ • CPU, memory, storageͳͲΛநԽ͢Δ • ैདྷͷMPI͚ͩͰͳ͘HadoopSparkͳͲͷ৽͍͠ࢄॲཧʹରԠ • ಠࣗͷεέδϡʔϥΛ࣋ͨͣϑϨʔϜϫʔΫʹର͠ϦιʔεͷAPIΛఏڙ -
Technologies #2 Apache Mesos: Various MiddleWares for Scheduling Containers •
Mesos্Ͱಈ͘൚༻ͷεέδϡʔϥ͕ෳ։ൃ͞Ε͍ͯΔ • Chronos • Mesosʹ͓͚ΔCronΛ࣮ݱ͢Δεέδϡʔϥ • Marathon • Mesosʹ͓͚ΔLong-RunningͳΞϓϦέʔγϣϯͷͨΊͷεέδϡʔϥ • Apache Aurora • ChronosͱMarathonͷಛΛซͤ࣋ͪPythonϕʔεͷDSLΛ࣋ͭ • Mesos schedularαϧͰࣗ࡞Ͱ͖Δ • http://www.slideshare.net/wallyqs/mesos-scheduler -
Technologies #3 Data Handling and Workflow Execution • ίϯςφ࣮ߦʹΑΔͱͯ͠σʔλӬଓੑͷ͕͋Δ •
ϗετͷϑΝΠϧγεςϜ্ͷσΟϨΫτϦΛϚϯτͯ͠ॻ͖ࠐΈ • ෳͷԕִDCΛMesosͰநԽͨ࣌͠ʹσʔλΛͲͷΑ͏ʹѻ͏͔ʁ • Cloud Burst Buffer by TITech • MMCFTP by NII • ίϯςφΛΈ߹ΘͤͨϫʔΫϑϩʔΛͲͷΑ͏ʹ࣮ߦ͢Δ͔ʁ • ղੳϫʔΫϑϩʔهड़ͷඪ४Խ • Common Workflow Language • https://github.com/common-workflow-language/common-workflow-language -
Apache Mesos + Chronos manager Node Node Node Node Node
1VCMJD1SJWBUF %PDLFS3FHJTUSZ %PDLFSGJMFT XPSLGMPXKTPO %BUB Storage post post/get transfer push pull run mount 6TFS γεςϜུ֓ -
workflow.json workflow.sh PAST FUTURE • Post to GridEngine • Run
binary software • Pre-install/build required • Post to workflow manager • Run docker container • Improved portability -
workflow.json • JSON format configuration file • Describe a workflow
contains multiple steps • 1 container for 1 app • Include directory to be mounted on containers -
chronos dependent jobs curl -X POST -d @workflow.json • Repeat
1 • Shipped with suicide job • containers should be finished in a week -
Implementation for proof-of-concept Challenges • εέδϡʔϥͷ • ChronosͷݶքɼAuroraͷλεΫϞσϧͷήϊϜσʔλղੳͱͷෆ߹ • !!!
ScalaͰεέδϡʔϥॻ͔͘͠ͳ͍ !!! • ϫʔΫϑϩʔهड़ͷ • ਓ͕ؒॻ͖͍͢͜ͱ vs ަՄೳͰ࠶ݱੑΛอূͯ͘͠ΕΔ͜ͱ • DSLalt-CWLʹඋ͑ͨަϑΥʔϚοτ/πʔϧͷඞཁੑ • ಛघͳΞʔΩςΫνϟΛར༻͢Δίϯςφͷ • MPI, GPGPUFPGA • UGEͳͲͷطଘͷεέδϡʔϥHadoopͳͲͷࢄॲཧϑϨʔϜϫʔΫͱͷڞଘ • ಈతͳεέʔϧΞτͱνΣοΫϙΠϯτͷӡ༻ -
1. Packaging Whole Research Activities ݚڀͷաఔΛશͯػցՄಡͳܗࣜͰهड़͠ΞʔΧΠϒ͢Δ 2. Continuous Integration and
Automated Build DBݚڀ࠶ݱੑΛ୲อ͢ΔͨΊͷϓϥοτϑΥʔϜ The future -
The future #1 Packaging Whole Research Activities • ݚڀϓϩηεʹ͓͚Δશͯͷ࡞ۀΛػցՄಡͳܗࣜͰهड़͢Δ •
هड़͞ΕͨͷͦͷؔੑΛอͬͨ··σʔλϕʔεʹ֨ೲ͢Δ • “Research Package Submission” • ݚڀܭը • αϯϓϦϯάɾΣοτ࣮ݧ • Ұ࣍σʔλ • σʔλॲཧɾσʔλղੳ • ೋ࣍σʔλ • จ͓Αͼจͷfig, plottableͳͲͷσʔλ -
The future #1 Packaging Whole Research Activities Research Activity Time
Course Details of Project Design Publication Text, Figs Sampling Primary Data Data Processing & Analysis Wet Experiments -
The future #1 Packaging Whole Research Activities Research Activity Time
Course Details of Project Design Sampling Primary Data Data Processing & Analysis Publication Text, Figs Wet Experiments BioProject BioSample Genbank, DRA -
The future #1 Packaging Whole Research Activities Research Activity Time
Course Details of Project Design Sampling Primary Data Data Processing & Analysis Publication Text, Figs Wet Experiments -
The future #1 Packaging Whole Research Activities Research Activity Time
Course Details of Project Design Sampling Primary Data Data Processing & Analysis Publication Text, Figs Wet Experiments -
Details of Project Design Sampling Primary Data Data Processing &
Analysis Publication Text, Figs Wet Experiments The future #1 Packaging Whole Research Activities -
The future #2 Continuous Integration for the Research Process •
Archived Packagesʹର͢ΔDBͷ ”Continuous Integration” • σʔλͱϓϩτίϧΛϏϧυͯ͠ςετɺίϛοτ • ࠷ऴίϛοτ͕ͦͷ··DBʹొ͞ΕΔ • ύοέʔδΦϒδΣΫτͱͯ͠Ϟδϡʔϧ͝ͱʹΞΫηεՄೳ • ʮݚڀAͷσʔλʹݚڀBͷσʔλॲཧΛద༻͢ΔʯΛίϚϯυҰൃͰ • ϨϙδτϦʹొ͞ΕΔ৽نख๏طଘͷσʔλશͯʹࣗಈతʹద༻͞ΕΔ • DB͕উखʹΒΜͰ͍͘ • DBGrowthRate/submission ͕IFʹஔ͖ΘΔ • ৽͍͠ԾઆΛূ໌͢ΔͨΊͷ৽نͳख๏ɺ৽نͳαϯϓϧͷՁ͕૿େ͢Δ • DataCenter + Database = “Reproduciblity as a Service” -
BioCI: ݚڀܦաΛৗʹύοέʔδϯά͠ɺΫϥυͰࣗಈϏϧυͱςετΛճ͢ fig: http://blog.jki.net/news/niweek-2012-fire-and-forget-bulletproof-builds-using-continuous-integration-with-labview-video-slides-now-available/ and togopic http://g86.dbcls.jp/~togoriv/ Do Research Package
Hosting Build and Test Report / Current Result Collect Data and Packaging -
1. Infrastructure needs to change for the Data Science ଟ͘ͷΛ๊͑ΔݱࡏͷܭࢉػΠϯϑϥେ͖͘มΘΔඞཁ͕͋Δ
2. Virtualized Env Runs on Abstracted Computational Resources நԽ͞ΕͨϋʔυΣΞͰԾԽ͞ΕͨڥΛՔಇ͢Δ 3. DB Integrates Data and Processes for Reproducible Research σʔλϕʔε͕ݚڀΛ౷߹͠࠶ݱੑΛ୲อ͢Δ Summary -
• This work was supported by ROIS URA Grant “༥߹γʔζ୳ࡧ”
2014. • The Institute of Statistical Mathematics • Dr. Yoshiyasu Tamura • Dr. Junji Nakano • Dr. Keisuke Honda • National Institute of Informatics • Dr. Kenjiro Yamanaka • Dr. Kento Aida • Dr. Shigetoshi Yokoyama • Dr. Yoshinobu Masatani • National Institute of Genetics • Dr. Osamu Ogasawara • Dr. Takeshi Tsurusawa • NIG SuperComputer Facilities SE team • Information and Mathematical Science and Bioinformatics Co., Ltd. • Tatsuya Nishizawa • Tokyo Institute of Technology • Dr. Shinichi Miura • Dr. Satoshi Matsuoka • Colleagues and Members of DBCLS, DDBJ, Open-Bio and BioHackathon Acknowledgement