Opening Remarks HPC OPS 2nd (Shinagawa)

第2回HPC OPS研究会 (品川) のオープニングのプレゼンテーションです。研究会のウェブサイトは https://bit.riken.jp/2018/06/2nd-hpc-ops-mtg/ です。

Transcript

1. ୈ2ճ HPC OPSݚڀձ Opening Remarks ೋ֊ಊ Ѫ, PhD. ཧԽֶݚڀॴόΠΦΠϯϑΥϚςΟΫεݚڀ։ൃϢχοτ ϢχοτϦʔμʔ ஜ೾େֶ ڭत (ڠಇେֶӃ) 1
2. ձͷ໨త • High performance computing + Operation • ՊֶܭࢉͷΦϖϨʔγϣϯΛޮ཰Խͯ͠ɺݚڀੜ࢈ੑΛ޲্͠ ͍ͨ • Ϋϥ΢υ, DevOps • ίϯςφԾ૝Խٕज़, ϫʔΫϑϩʔ, δϣϒεέδϡʔϥʔ, Infrastructure as Code, ߏ੒؅ཧπʔϧ, CI, ιʔείʔυ؅ ཧɺΞΫηϥϨʔλ/GPU, ηΩϡϦςΟ, ݸਓ৘ใ... 2
3. ݚڀ࣌ؒͷݮগͱελΠϧͷมԽ 3 http://tmaita77.blogspot.jp/2015/04/blog-post_8.htmlΑΓҾ༻ ೔ຊͷ࿦จڞஶͷܗଶͷมԽ ओཁࠃ౳ͷτοϓ10ˋ࿦จ਺γΣΞͷਪҠ http://www.mext.go.jp/b_menu/hakusho/html/hpaa201001/detail/1296363.htmΑΓҾ༻ ݚڀ࣌ؒݮগɾνʔϜؒ࿈ܞ΁
4. σʔλղੳͷ࠶ݱੑͱϥΠϑαΠΤϯε ݈શͳϥΠϑαΠΤϯεͷൃలͱσʔλղੳͷԾ૝Խ
5. ྫ: Single-cell RNA-seqͷσʔλղੳϫʔΫϑϩʔ ͨ͘͞ΜͷϓϩάϥϜͱσʔλϕʔεͷ૊Έ߹Θͤ 8'ͦͷ
   'BTUR.DG
   
   #PXUJF
   
   F9QSFTT 8'ͦͷ
   'BTUR.DG
   
   4BJMpTI ڞ௨
   ࣮ମύεͷऔಘ ڞ௨
   Χ΢ϯτσʔλͷϚʔδςʔϒϧ࡞੒
   
   FEHF3
   
   HFOF
   4ZNCPM෇Ճ ̍ʣ3/"TFRd%&(ղੳ8'Λ࡞੒ ਺ઍࡉ๔ x ਺ສҨ఻ࢠ x ϓϩδΣΫτ਺
6. ʮܭࢉʯͷߴ଎Խ͔Βʮݚڀʯͷߴ଎Խ΁ όΠΦΠϯϑΥϚςΟΫεղੳͱITΠϯϑϥͱΞϓϦέʔγϣϯ։ൃͷҰମԽ ܭࢉ ؀ڥߏங ࣮૷ ܭࢉ ؀ڥߏங ࣮૷ ɾ͜Ε·ͰͷHPCͱόΠΦΠϯϑΥϚςΟΫε ฒྻɾ෼ࢄɾΞΫηϥϨʔλ ཯଎ ɾݱࡏͷDNAγʔέϯεղੳͷधཁ • ࣗવՊֶݚڀʹूத͍͕ͨ͠ɺσʔλղੳ؀ڥΛߏங͢Δ͜ͱ͸ख͕͔͔ؒΔ • ܭࢉػͷௐୡ΍؅ཧɺอकͷख͕͔͔ؒΔ • δϟϯϧʹΑͬͯ͸ղੳ͸ͨ͘͞Μͷπʔϧͷ૊Έ߹Θͤ • πʔϧ΍ख๏ɺDBͷΞοϓσʔτ͕଎͍ • ͍ͭͲͷ͙Β͍ͷσʔλ͕ग़Δ͔༧ଌ͠ʹ͍͘ɻεϙοτར༻͕ଟ͍ɻ • σʔλղੳͷ࠶ݱੑ୲อ • ࿦จͷϚςϝι͸هࡌ͕ෆ଍͓ͯ͠Γղੳ͕࠶ݱͰ͖ͳ͍ • ܭࢉੜ໋ՊֶऀΛ࣮ݧੜ໋Պֶऀͷ͓ख఻͍͔Βղ์ • ؆୯ͳπʔϧ΍ܭࢉػͷ࢖͍ํ΍Πϯετʔϧɺ࡞ਤɺ࢓༷ॻॻ͖ͳͲͷαϙʔτʹ๩ࡴ ར༻ొ࿥ ར༻ొ࿥
7. IT Πϯϑϥ ΞϓϦέʔγϣϯ։ൃɾϦϦʔε ϏδωεΞΠσΟΞ Ϛʔέοτ http://ja.wikipedia.org/wiki/DevOps. modified DevOps = Development + Operations ITΠϯϑϥͱΞϓϦέʔγϣϯ։ൃͷҰମԽ ϏδωεΞΠσΟΞΛૉૣ͘Ϛʔέοτʹग़ͨ͢Ίͷ ITʹؔ͢Δࢥ૝ͱͦͷٕज़
8. σʔλղੳ༻PCΫϥελʔͷηοτΞοϓ σʔλղੳπʔϧ΍ύΠϓ ϥΠϯγεςϜͷ։ൃ %BUB
   BOBMZTJT SciDevOps σʔλղੳ΍ιϑτɺσʔ λϕʔεͷ඼࣭؅ཧ ݚڀΞΠσΟΞ ࣮ݧσʔλ ࿦จग़൛ SciDevOps = Science + Development + Operations όΠΦΠϯϑΥϚςΟΫεղੳͱITΠϯϑϥͱΞϓϦέʔγϣϯ։ൃͷҰମԽ σʔλղੳͷ࣮ࢪ ݚڀΞΠσΟΞΛૉૣ͘࿦จͱͯ͠ग़ͨ͢Ίͷ σʔλղੳʹؔ͢Δࢥ૝ͱͦͷٕज़ ※ೋ֊ಊʹΑΔ଄ޠ
9. ࣄྫ1: σʔλղੳ༻εύίϯΛΫϥ΢υ্ʹࣗಈߏங 1ίϚϯυ/ΫϦοΫͰɺཉ͍͠ͱ͖ʹɺཉ͍͚ͩ͠ɺࣗ෼ઐ༻εύίϯΛɻϚΠΫϩιϑτͱͷڞಉݚڀ 9 IUUQTHJUIVCDPNNBOBCVJTIJJ/(4UI Ծ૝ܭࢉػͱΫϥ΢υΛར༻͠ɺεύίϯΛࣗಈߏங͠ɺܭࢉΛ౤ೖ Web্ͷϘλϯΛΫϦοΫ/1ίϚϯυͰܭࢉػ͕खʹ ೔ܦϏδωεΦϯϥΠϯ Microsoft Procurement Time MacBookPro 1 day ~ 2weeks On-premise half a year Cloud 15 min Procurement Time Reproducibility Reproducibilit y MacBookPro No - if manually On-premise Hard - procedure by specification Cloud YES Cost Execution Time Cost MacBookPro $3000 On-premise $50000 ~ Cloud $200/run Execution Time MacBookPro Not finished On-premise half a day Cloud half a day This compare is done by Not Galaxy Pipeline. This compare is illumine NextSeq500 1 run , almost 2000 single-cell RNA- seq.And compare only computational anaylisys.
10. ࣄྫ2. ΦϯσϚϯυʹϊʔυΛௐୡ͢ΔHPC-Ϋϥ΢υͷϋΠϒϦου ࣗ෼ͷϚγϯ͔ΒΩϡʔΛࢦఆͯ͠δϣϒΛ౤͛ΔͱΫϥ΢υ͔ΒϊʔυΛࣗಈతʹௐୡɻNII஛๪ઌੜͱͷڞಉݚڀɻ 10 Phase1: ϥϘͷPCΫϥελ͔ΒΫϥ΢υϊʔυ΁ܭࢉ Phase2: ϥϘͷLinux౥ࡌNAS͔ΒΫϥ΢υϊʔυ΁ܭࢉ $ qsub -q cloud.q command $ qsub -q cloud.q command Virtual Cloud Provider L2VPN RIKEN Cloud Provider NII ΦϯσϚϯυ઀ଓαʔϏε (দౢɺੴҪɺೋ֊ಊ)
11. 11 ܭࢉ଎౓ͱίετ: Hybrid Cloud vs. Public Cloud ྉۚ ($) 0 40 80 120 160 52 5 96 Πϯελϯε ετϨʔδ σʔλసૹ ࣮ݧ৚݅: AWS, m4.10xlarge x 4, EBS 20GB, 593GBసૹ, 2,000αϯϓϧ ૯ܭࢉྉۚ: $153.02
12. 12 ల๬: ϩʔΧϧʹ͋ΔϔουϊʔυͷνʔϓԽ © 2016 DBCLS ౷߹TV / CC-BY-4.0 FileSystem & Submission Node login send NFS IPsec Execution Nodes on cloud qsub ਺10ສԁ+Ϋϥ΢υར༻ྉ Web UI ՝୊ • ௨৴଎౓ • ৴པੑ/ݎ࿚ੑ 10෼Ͱߏங ܭࢉऴྃޙఀࢭ ࢢൢͷQNAP
13. ͳͥSciDevOps΍HPC OPS͕ඞཁͳͷ͔? ࣗવՊֶݚڀͱΤϯδχΞϦϯά • ࣗવՊֶݚڀʹूத͢ΔͨΊͷՊֶܭࢉ؀ڥͷޮ཰Խ • ࣗવՊֶ෼໺ͰՊֶܭࢉ؀ڥΛࢧ͑ΔΤϯδχΞΛҭͯΔ • ΤϯδχΞϦϯάΛݚڀɾ঎ചͱ͍ͯ͠Δํʑͱ࿈ܞ͍ͨ͠
14. • େా ୡ࿠ • ϥΠϑαΠΤϯε౷߹σʔλϕʔεηϯλʔʮDBCLSͰͷίϯςφɾΫϥ΢υ׆༻঺հʯ • ᖒొږ඙ • HiganWorks߹ಉձࣾ .ϞϏϯΪגࣜձࣾʮDockerίϯςφΛ͔ͭͬͨϗεςΟϯάαʔϏεͱ༻్ผίϯςφΠϝʔδͷ࿩ʯ • தాणึ • ೔ຊϚΠΫϩιϑτגࣜձࣾύϒϦοΫηΫλʔࣄۀຊ෦Ϋϥ΢υΞʔΩςΫτʮHPC on Azureʯ • ࣲా ௚थ • ΤΫετϦʔϜ-Dגࣜձࣾ CEO, High Performance Cloud Architect Ϋϥ΢υεύίϯߏஙӡ༻ࣗಈԽαʔϏεʮXTREME-DNAʯ • ஛๪͋ͭࢠ • ࠃཱ৘ใֶݚڀॴ ΞʔΩςΫνϟՊֶݚڀܥʮΫϥ΢υͰͷΞϓϦέʔγϣϯ؀ڥߏஙɾ؅ཧΛࢧԉ͢ΔΦϯσϚϯυΫϥ΢υߏஙαʔϏ εʯ • দౢ໌޺ • ࠃཱݚڀ։ൃ๏ਓཧԽֶݚڀॴ ৘ใج൫ηϯλʔ όΠΦΠϯϑΥϚςΟΫεݚڀ։ൃϢχοτʮՊֶٕज़ܭࢉ༻Ϋϥελ΁ͷDockerಋೖͱӡ ༻ʯ • ּݪխ߂ • ౦ژେֶ େֶӃ৽ྖҬ૑੒ՊֶݚڀՊ ϝσΟΧϧ৘ใੜ໋ઐ߈ʮ࠷ઌ୺ͷήϊϜղੳͰ࢖͍͍ͨཧ૝ͷίϯςφԾ૝ԽΛߟ͑Δʯ 14 ୈ1ճHPC OPSݚڀձ (2018/02/28)
15. • ւ௡Ұ੒ (ࠃཱݚڀ։ൃ๏ਓཧԽֶݚڀॴ ੜ໋ػೳՊֶݚڀηϯλʔ όΠΦίϯϐϡʔςΟϯάݚڀνʔϜ) • ʮࡉ๔γϛϡϨʔγϣϯιϑτ΢ΣΞE-Cell4ͷٕज़ʯ • നੴ༑Ұ (ࠃཱ͕Μݚڀηϯλʔ ͕ΜήϊϜ৘ใ؅ཧηϯλʔ ήϊϜղੳࣨ) • ʮExtraction Transformation Load (ETL)Ξϓϩʔνʹج͕ͮ͘ΜήϊϜղੳύΠϓϥΠϯͷ։ൃʯ • ۙ౻Ӊஐ࿕(udzura) (GMOϖύϘגࣜձࣾ ٕज़෦ٕज़ج൫νʔϜ) • ʮίϯςφϥϯλΠϜͱΞʔΩςΫνϟΛ৽نʹ։ൃͨ݁͠Ռɺݟ͖͑ͯͨੈքʹ͍ͭͯʯ • ೔ຊϚΠΫϩιϑτ • Ԟ໺ ৻ޗ (ΤΫετϦʔϜ-Dגࣜձࣾ औక໾CTO) • ʮXTREME-D͕ఏڙ͢ΔΫϥ΢υHPCαʔϏεʯ • ੓୩޷৳ (ࠃཱ৘ใֶݚڀॴ Ϋϥ΢υج൫ݚڀ։ൃηϯλʔ) • ʮNIIͰͷܭࢉػ؀ڥͷӡ༻ٴͼɺLiterate Computing(for reproducible infrastructure)ʹ͍ͭͯʯ • ࠤ౻ਔ (ࠃཱݚڀ։ൃ๏ਓ࢈ۀٕज़૯߹ݚڀॴɹਓ޻஌ೳݚڀηϯλʔ) • ʮAIڮ౉͠Ϋϥ΢υʢABCIʣʹ͓͚ΔߴੑೳܭࢉͱAI/Ϗοάσʔλॲཧͷ༥߹ʯ 15