軽量仮想環境による絶対に再現するデータ解析

第4回 NGS現場の会 ポスター発表

Transcript

1. ܰྔԾ૝؀ڥʹΑΔઈରʹ࠶ݱ͢Δσʔλղੳ ؀ڥߏஙʹϋϚͬͯࠢΛ࡟ΒΕΔͷ͸΋͏ݏͩ Summary ɾσʔλղੳͷ࠶ݱੑΛ޲্্࣭ͤͯ͞ͳ฻Β͠ΛखʹೖΕ͍ͨ ɾʮٕज़ͰղܾͰ͖Δ͜ͱΛιʔγϟϧʹղܾͨ͠Βෛ͚ʯ by Dr. Itoshi Nikaido ɾެ։σʔλϨϙδτϦͱେܕܭࢉػΛ౷߹͢Δ͜ͱͰ΋ͬͱਓָ͕ؒʹͳΔ Automation saves all େాୡ࿠
   ৘ใɾγεςϜݚڀػߏɹϥΠϑαΠΤϯε౷߹σʔλϕʔεηϯλʔ
   %#$-4
   UXJUUFSDPNJOVU
   HJUIVCDPNJOVUBOP
   TQFBLFSEFDLDPNJOVUBOP ࠶࣮ߦΛࣗಈԽ͢Δ
   
   ਓָ͕ؒʹͳΔ
   
   ΋ͬͱαΠΤϯε͕Ͱ͖Δ σʔλղੳ͕࠶ݱ͠ͳ͍ͷ͸ਓ͕ؒհೖ͍͗ͯ͢͠Δ͔Β ɹঢ়گ΍؀ڥʹґଘ͢Δ͜ͱͳ͘ɼશ͘ಉ͡ೖྗʹରͯ͠શ͘ಉ͡खଓ͖Λ౿Ί͹શ͘ಉ͡ग़ྗ͕ಘΒΕΔ ͜ͱΛʮ࠶ݱੑʯͱఆٛ͢ΔͱɼNGSσʔλ͚ͩͰͳ͘ੜ໋Պֶʹ͓͚Δσʔλղੳͷ࠶ݱੑͱ͸ɼܭࢉػ ؀ڥ΍࣮ߦऀͷ஌ࣝɾεΩϧͳͲɼඞཁͳ৚݅Λἧ͑ΔίετͱಉҰʹΈͳ͢͜ͱ͕Ͱ͖Δɽ͢ͳΘͪɼʮσʔ λղੳͷ࠶ݱੑͷ޲্ʯ͸ɼղੳʹཁ͢Δ࡞ۀ͔ΒଐਓੑΛഉআ͠ɼҰൠతͳܭࢉػ؀ڥͱ͔ᷮͳΩʔλΠ ϐϯάʹΑͬͯಉ͡ग़ྗ͕ಘΒΕΔ࢓૊ΈΛཱ֬͢Δ͜ͱʹΑ࣮ͬͯݱ͞ΕΔɽ ɹαΠΤϯεͷຊ࣭Ͱ͋Γෆஅͷ౒ྗʹΑͬͯ͜ΕΛอ࣋͠ͳ͚Ε͹ͳΒͳ͍ͱޠΒΕΔʮ࠶ݱੑʯͱ͍͏୯ ޠ͸ɼ͔͠͠σʔλղੳʹ͓͍ͯ͸ผͷଆ໘Λ࣋ͭɽྫ͑͹ɼʮੲࣗ෼Ͱ͜ͷղੳ΍ͬͨΑͳʯʮ͜ͷख๏͸ ͋ͷ࿦จͰ࣮૷͞ΕͯͨΑͳʯͱ͍͏ɼσʔλղੳͷݱ৔ͰසൟʹݟΒΕΔঢ়گʹ͓͍ͯɼʮੲॻ͍ͨίʔυ ࠓ΍ͬͨΒಈ͔ͳ͍͔Β·ͨॻ͖௚͔͢ʯʮࢼ͠ʹϏϧυͯ͠ΈͨΒṖͷΤϥʔͰίέͨࣗ͠෼Ͱ࣮૷͢Δ͔ʯ ͱ͍͏࣌ؒͷ࿘අΛආ͚Δ͜ͱʹ௚઀ܨ͕Δɽ࣌ؒͷ࿘අΛආ͚Δ͜ͱ͸ɼผͷ࡞ۀɼαΠΤϯεʹऔΓ૊Ή ͨΊͷ࣌ؒΛಘΔͱ͍͏͜ͱͰ͋Δɽ͜͜Ͱ঺հ͢Δͷ͸ɼݚڀͷຊ࣭Ͱ͸ͳ͍؀ڥߏங΍ॲཧͷ࣮ߦ͔Β ݚڀऀΛղ์͠ɼػցʹͰ͖Δ͜ͱ͸શͯػցʹ΍ͬͯ΋Β͍ɼਓ͕ؒਓؒΒ͘͠ੜ͖ΔͨΊͷઓ͍Ͱ͋Δɽ Infrastructure as Code ࣮ߦʹඞཁͳશͯͷ৘ใΛܭࢉػ͕࣮ߦՄೳͳܗࣜͰهड़͢Δ ղੳͷखଓ͖Λόονॲཧ͢Δ͔ͷ͝ͱ͘؀ڥߏங΋શͯࣗಈԽͰ͖Δɽͦ͏ɼԾ૝؀ڥͳΒͶɽ ɹϓϩάϥϛϯάʹ͓͚Δඒֶ͋Δ͍͸఩ֶͷ1ͭͱͯ͠஌ΒΕΔ DRY (Don’t Repeat Yourself) ͸ɼ࡞ۀͷॏ ෳΛ๷͙ͨΊͷجຊతͳߟ͑ํͰ͋Δɽಛʹιϑτ΢ΣΞΛΠϯετʔϧͨ͠ΓɼϚγϯͷ؀ڥઃఆΛߦͬ ͨΓɼ͜Ε·Ͱ͸ର࿩తʹߦΘΕΔ͜ͱ͕ଟ͔ͬͨ࡞ۀ͸ɼͦͷ౎౓ਓ͕ؒखΛಈ͔ͯ͠ߦΘΕΔ͜ͱ͕ଟ ͔ͬͨɽ͔͠͠ɼάϦουίϯϐϡʔςΟϯά΍Ծ૝؀ڥͳͲͷ৘ใٕज़͕༰қʹར༻Ͱ͖ΔΑ͏ʹͳͬͨ ͜ͱʹΑΓɼܭࢉػͷ୆਺෼͚ͩखಈͰ؀ڥߏஙΛߦ͏͜ͱ͸શ͘ݱ࣮తͰ͸ͳ͍ͨΊɼηοτΞοϓͷͨΊ ʹඞཁͳશͯͷखଓ͖Λ࣮ߦՄೳͳϓϩάϥϜͱͯ͠هड़͢Δ Infrastructure as Code ͱ͍͏֓೦͕ఏএ͞Ε ΔΑ͏ʹͳͬͨɽ ɹUNIX/LINUXϕʔεͷγεςϜͰ͋Ε͹ɼ୯७ͳγΣϧεΫϦϓτͰ΋࣮ݱՄೳͰ͋Δ͕ɼෳࡶͳॲཧΛه ड़ͨ͠ΓɼOSʹґଘ͠ͳ͍ந৅తͳهड़Λ࣮ݱ͢ΔͨΊʹɼ͍͔ͭ͘ͷٕज़͕ఏএ͞Εͨ(Fig. 2)ɽͦͷ୅ද ͕Chef, Puppet, AnsibleͰ͋Δɽ͜ΕΒʹΑͬͯେن໛ͳܭࢉػ܈Λίϯτϩʔϧ͢Δ͜ͱ͕ඇৗʹ༰қʹ ͳͬͨɽͦͷଞʹ΋VagrantΛ࢝Ίͱ͢ΔԾ૝؀ڥͦͷ΋ͷͷىಈΛϓϩάϥϜʹམͱ͜͠ΉϓϩμΫτ͕ࠓ ·͞ʹશ੝ظͰ͋Γɺ͜Εʹʮ࣮૷ˠݕূˠӡ༻ʯͷ࡞ۀΛࣗಈԽͯ͠܁Γฦ͢͜ͱʹΑͬͯγεςϜͷ࣭Λ ҡ࣋͢Δ”Continuous Integration”Λ࣮ݱ͢ΔJenkinsͳͲ, ͞·͟·ͳιϑτ΢ΣΞ͕։ൃ͞Ε͍ͯΔɽ͜ΕΒ ͸ϋʔυ΢ΣΞͰͦͷ··ಈ͘γεςϜ (ϕΞϝλϧ) ΍ϋΠύʔόΠβʔܕԾ૝؀ڥΛର৅ͱ͍ͯ͠Δɽ ɹͦͷޙʹొ৔ͨ͠ͷ͕৽ͨͳܭࢉػԾ૝Խٕज़Ͱ͋ΔDockerͰ͋Δɽ͜Ε͸ίϯςφܕԾ૝Խͱݺ͹ΕΔ ΋ͷͰɼܭࢉػ্ʹܭࢉػΛΤϛϡϨʔτ͢ΔϋΠύʔόΠβʔܕͱ͸ҟͳΓɼԾ૝ԽʹΑΔΦʔόʔϔου Λ཈͍͑ͯΔ͜ͱΛಛ௃ͱ͢ΔɽDocker͸͜ͷίϯςφܕԾ૝ʹࠩ෼ϑΝΠϧγεςϜͱDockerfileͱݺ͹Ε Δ࣮ߦՄೳͳ؀ڥߏஙͷهड़Λαϙʔτ͍ͯ͠Δɽ Docker on NIG Supercomputer /(4σʔλղੳϫʔΫϑϩʔ
   WT
   ίϯςφԾ૝
   PO
   Ҩ఻ݚεύίϯ ੨͍ܵ͸Ҩ఻ݚεύίϯΛٹ͑Δ͔ ɹࠃཱҨ఻ֶݚڀॴͰ͸ɼඇ঎༻ར༻Ͱ͋Ε͹ΞΧ΢ϯτΛਃ੥͢Δ͜ͱͰແྉͰར༻ Ͱ͖ΔεʔύʔίϯϐϡʔλγεςϜ(Ҩ఻ݚεύίϯɼѪশ:ΓͳͪΌΜ)Λӡ༻͍ͯ͠ Δɽଞ෼໺ʹ͓͚Δεύίϯͷओͳར༻ํ๏͕MPI΍GPGPUͳͲͷΞʔΩςΫνϟΛར ༻ͨ͠CPUʹߴෛՙͷ͔͔ΔܭࢉͰ͋Δ͜ͱʹରͯ͠ɼNGSσʔλͰ͸ڊେͳ഑ྻσʔ λ΍ϦϑΝϨϯεσʔλϕʔεʹසൟʹΞΫηε͢ΔͨΊI/O͕཯଎ͱͳΔɽ·ͨɼҨ఻ ݚεύίϯͷΑ͏ͳڞಉར༻ܕͷ৔߹ɼෳ਺ͷར༻ऀ͕ͦΕͧΕ༷ʑͳཁٻ࢓༷Λ࣋ͭ શ͘ҟͳΔղੳϑϩʔΛ࣮ߦ͢ΔͨΊɼݸผͷΞϓϦέʔγϣϯʹରͯ͠؀ڥߏஙʹΑͬ ͯ࠷దԽΛߦ͏͜ͱ͕೉͍͠ɽ ɹ͜ͷΑ͏ͳ໰୊Λղܾ͢ΔͨΊʹɼϋΠύʔόΠβʔܕͱҟͳΓI/O΁ͷΦʔόʔϔο υ͕গͳ͍ίϯςφܕԾ૝͸༗རͰ͋Δɽͦ͜ͰɼNGSσʔλղੳͰ༻͍ΒΕΔιϑτ ΢ΣΞΛDockerίϯςφʹ͢Δ͜ͱͰɼύϑΥʔϚϯεΛམͱͣ͞ʹInfrastructure as CodeΛ࣮ݱ͢ΔͨΊͷςετ؀ڥΛҨ఻ݚεύίϯ্ʹߏஙͨ͠ɽDockerίϯςφ͕ར ༻͢ΔϦιʔεΛ؅ཧ͢ΔͨΊʹ Apache Mesos ΛɼδϣϒεέδϡʔϦϯάͷͨΊʹ Chronos Λϕʔεͱͨ͠ಠࣗεέδϡʔϥΛ࣮૷ͨ͠ɽ ՝୊ͱͯ͠ɼ1) ैདྷͷάϦουΤϯδϯͰͷ࣮ߦΑΓ΋ϧʔϧ͕ଟ͘൥ࡶʹײ͡ΒΕ Δɼ2) ܭࢉػ؀ڥ΁ͷಋೖͷख͕ؒଟ͍ɼ3) Ҡߦ͢ΔϝϦοτΛײ͡ʹ͍͘ɼ 4) ύΠ ϓϥΠϯॲཧΛهड़͢Δ෦෼͕ශऑ ͳͲ͕͋ΔͨΊɼ͞ΒʹվળΛߦ͍ͬͯ͘ɽ Mesos Slave Mesos Slave Mesos Slave workflow manager Node Storage %PDLFSGJMFT 8PSLGMPXKTPO %BUB 1VTI
   %PDLFSGJMFT
   UP
   $POUBJOFS
   3FHJTUSZ Node Node Node Node Mesos Master Mesos Slave 1VTI
   XPSLGMPX
   DPOGJHVSBUJPO
   UP
   NBOBHFS 5SBOTGFS
   %BUB
   WJB
   JOUFSOFU 1VMM
   DPOUBJOFS
   
   3VO .PVOU
   %BUB
   %JS
   UP
   DPOUBJOFS 1BTT
   FYFDVUFS VTFS Fig. 1. FANTOM5 (http://fantom.gsc.riken.jp/5/)ʹ͓͚Δ CAGE-Seqσʔλղੳͷهड़ (a) ࿦จதͰ͸ Materials and Methods, ΋͘͠͸SupplementaryதʹࣗવݴޠͰهड़͞Ε Δɽ (b) FANTOM5Ͱ͸࿦จͱ͸ผʹΦϯϥΠϯͰϓϩτί ϧΛެ։͍ͯ͠Δɽେม༗Γ೉͍ɽ (c) ͔͠͠࠶࣮ߦ͢Δͱ ͳΔͱɼ݁ہόονεΫϦϓτΛॻ͘͜ͱʹͳΔɽ͜Ε͕࿦ จʹఴ෇͞Ε͍ͯͯཉ͍͠ɽ a b c Fig. 2. Infrastructure as Code ʹؔΘΔϓϩμΫ τҰཡ ࠨ͔Β࣌ܭճΓʹ ϋΠύʔόΠβܕԾ૝؀ ڥͷߏஙΛࣗಈԽ͢ΔVagrant, ৽نʹՔಇͨ͠ܭ ࢉػ؀ڥͷηοτΞοϓΛࣗಈԽ͢ΔChef, Puppet, Ansible, Continuous IntegrationΛ࣮ݱ͢Δ ୅දతͳϓϩμΫτ Jenkins (௨শδΣϯΩϯε͓ ͡͞Μ), ίϯςφܕԾ૝ͷضखͰ͋Δdocker, ςϯ ϓϨʔτʹैͬͨܭࢉػγεςϜΛࣗಈͰՔಇ͞ ͤΔTerraformɽશ෦࢖ͬͨ͜ͱ͋Δ/஌ͬͯΔਓ͸ ͥͻ͓༑ୡʹͳΓ͍ͨͷͰɼͥͻϦΞϧ͍͍Ͷʂ εςοΧʔΛష͍ͬͯͩ͘͞ʂ Fig. 3. Ҩ఻ݚεύίϯ্ʹߏஙͨ͠γεςϜ ͷ֓ཁ Ҩ఻ݚεύίϯͷϊʔυΛطଘγες Ϝ͔Β਺୆੾Γ཭͠ɼ Apache Mesos ͰϦ ιʔεΛ ؅ཧ͠ɼͦͷ্ͰίϯςφΛىಈ͢ ΔɽϢʔβ͸༧Ίdocker containerΛϨδετ Ϧʹొ࿥͠ɼ͜ΕΒΛ૊Έ߹ΘͤͨϫʔΫϑ ϩʔͷهड़ΛREST APIܦ༝Ͱొ࿥͢Δɽ ࢖ͬͯΈ͍ͨਓ͸࿈བྷઌΛγʔϧͰష͓ͬͯ ͍͍ͯͩ͘͞ɽɹɹɹɹɹ͜ͷ΁Μʹˣ