General-purpose hybrid storage system

Slide 1

Slide 1 text

൚༻తͳϋΠϒϦου ετϨʔδγεςϜͷఏҊ גࣜձࣾϋʔτϏʔπ Takamura Narimichi @nari_ex 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 1

Slide 2

Slide 2 text

ࣗݾ঺հ • Takamura Narimchi / ߴଜ ੒ಓ • @nari_ex • גࣜձࣾϋʔτϏʔπ औక໾ VPoE • ిؾ௨৴େֶ • ৘ใཧ޻ֶ෦৘ใɾ௨৴޻ֶՊ ֶ࢜ • άϩʔϏεܦӦେֶӃ • ܦӦݚڀՊܦӦઐ߈ म࢜ʢMBAʣ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 2

Slide 3

Slide 3 text

໨࣍ • എܠͱ՝୊ • ఏҊ • ࣮૷ํ๏ • ຊػߏͷར༻ύλʔϯ • ·ͱΊ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 3

Slide 4

Slide 4 text

എܠͱ՝୊ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 4

Slide 5

Slide 5 text

എܠͱ՝୊ • WebαʔϏεͷීٴʹΑΓσʔλྔ͕രൃతʹ૿Ճ • ๲େͳσʔλΛอଘ͢ΔετϨʔδͷඅ༻͕૿Ճ • සൟʹΞΫηε͞ΕΔσʔλ͸શମͷ͘͝Ұ෦ • අ༻ରޮՌΛߴΊΔͨΊʹར༻ස౓ͷ௿͍σʔλͷඅ༻Λ࡟ݮ ͍ͨ͠ ※ ຊݚڀʹ͓͚Δσʔλͱ͸ίϯςϯπϑΝΠϧΛର৅ͱ͢Δ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 5

Slide 6

Slide 6 text

ิ଍: σʔλʹؔ͢Δݴ༿ͷఆٛ ຊݚڀͰ͸ར༻ස౓͝ͱʹσʔλΛ2छྨʹ෼ྨ͢Δ • ϗοτσʔλ: ར༻ස౓ͷߴ͍σʔλ • ίʔϧυσʔλ: ར༻ස౓ͷ௿͍σʔλ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 6

Slide 7

Slide 7 text

ิ଍: ίϯςϯπσʔλʹ͓͚Δίʔϧυσʔλͷྫ • धཁ͕গͳ͍ෆಈ࢈ͷ෺݅σʔλ • γʔζϯΦϑʹͳͬͨΞύϨϧͷ੡඼σʔλ • ड৴ࡁΈ͔ͭࢀর͞Εͳ͍ϝʔϧσʔλ • ϩά഑৴αʔόʹ͓͚Δݹ͍ϩάσʔλ ※ ͍ͣΕ΋ϑΝΠϧ୯ҐͰͷΞΫηε͕ඞཁͰ͋ΔέʔεΛ૝ఆ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 7

Slide 8

Slide 8 text

՝୊ʹର͢Δجຊઓུ • ίʔϧυσʔλΛԿΒ͔ͷํ๏Ͱ҆ՁͳετϨʔδʹҠಈ͢Δ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 8

Slide 9

Slide 9 text

ैདྷख๏1: ΞʔΧΠϒ • ௕ظอଘ͢ΔͨΊʹઐ༻ͷอଘྖҬʹ҆શʹσʔλΛอଘ͢Δ͜ͱ • ҆ՁͷετϨʔδʹҠಈ͢Δ͜ͱͰίετ࡟ݮ͕Մೳ • ՝୊ • ผͷ৔ॴʹҠಈ͢ΔͨΊɺୀආલͱಉ͡Α͏ʹσʔλΛར༻͢ Δ͜ͱ͕ࠔ೉ => ར༻͞ΕΔՄೳੑ͕θϩʹͳΒͳ͍ίϯςϯπσʔλʹ͸ෆ޲͖ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 9

Slide 10

Slide 10 text

ैདྷख๏2: ΤϯλʔϓϥΠζ޲͚ετϨʔδ੡඼ • ར༻ස౓ͷ௿͍σʔλΛ҆ՁͳετϨʔδʹࣗಈͰҠಈ͢Δ • ՝୊ • ϕϯμϩοΫΠϯ • ଟֹͷ౤ࢿ͕ඞཁ • Ϋϥ΢υ؀ڥ΁ͷಋೖ͕ࠔ೉ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 10

Slide 11

Slide 11 text

ैདྷख๏2: ΤϯλʔϓϥΠζ޲͚ετϨʔδ੡඼ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 11

Slide 12

Slide 12 text

ิ଍: Ϋϥ΢υԽʹ൐͏ετϨʔδ૚ͷࣗ༝౓௿Լ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 12

Slide 13

Slide 13 text

ैདྷख๏3: طଘϑΝΠϧγεςϜͷ֦ு • Btrfs ͷ֦ு1 • ϚϧνσόΠεʹରԠ͍ͯ͠Δ Btrfs ͷಛ௃Λ׆͔ͨ͠ݚڀ • ൚༻ϒϩοΫ૚ʹͯσʔλͷҠಈΛߦ͏ • ՝୊ • σʔλҠಈ࣌ʹڞ༗ϦιʔεʢϝϞϦɺCPUʣͷෛՙ͕ൃੜ • Btrfs Ҏ֎ͷϑΝΠϧγεςϜΛར༻Ͱ͖ͳ͍ • ޿͘ར༻͞Ε͍ͯΔ ext4 ΍ xfs ͕ར༻Ͱ͖ͳ͍ 1 Hot Cold Data Tracking and Migra3on in btrfs. 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 13

Slide 14

Slide 14 text

ิ଍: Linux I/O ֎؍ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 14

Slide 15

Slide 15 text

՝୊·ͱΊ • ΞʔΧΠϒ • ίϯςϯπσʔλʹෆ޲͖ • ΤϯλʔϓϥΠζ޲͚੡඼ • ߴՁ • ϕϯμϩοΫΠϯ • Ϋϥ΢υʹෆ޲͖ • طଘϑΝΠϧγεςϜͷ֦ு • ϑΝΠϧγεςϜͷબ୒͕Ͱ͖ͳ͍ • σʔλҠಈ࣌ͷෛՙʹΑͬͯϝΠϯॲཧͷಈ࡞ʹӨڹ͕ग़Δ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 15

Slide 16

Slide 16 text

՝୊·ͱΊ2 • ಋೖͰ͖Δ؀ڥ͕ݶΒΕ͍ͯΔ • σʔλҠಈ࣌ͷෛՙ͕՝୊ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 16

Slide 17

Slide 17 text

ఏҊ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 17

Slide 18

Slide 18 text

൚༻తͳϋΠϒϦουετϨʔδγεςϜͷఏҊ • ༷ʑͳ؀ڥͰར༻ՄೳͳϋΠϒϦουετϨʔδγεςϜ • ίʔϧυσʔλҠಈ࣌ʹϝΠϯॲཧͷಈ࡞Λ๦͛ͳ͍ ※ Linux ্Ͱͷ࣮૷Λ૝ఆ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 18

Slide 19

Slide 19 text

ୡ੒͍ͨ͜͠ͱ • ൚༻ੑ͕ߴ͍ • ΦϯϓϨ؀ڥ͸΋ͪΖΜɺΫϥ΢υ؀ڥͰ΋ར༻Մೳ • OSS Ͱߏ੒͞ΕɺLinux্Ͱಈ࡞Λ͢Δ • ಋೖ࣌ʹಛఆͷϑΝΠϧγεςϜʹґଘ͠ͳ͍ • ϗοτετϨʔδɺίʔϧυετϨʔδ͝ͱʹϑΝΠϧγεςϜΛબ΂Δ • ίʔϧυσʔλͷҠಈෛՙ͕े෼ʹ௿͍ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 19

Slide 20

Slide 20 text

࣮૷ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 20

Slide 21

Slide 21 text

࣮૷্ͷ՝୊ͱରࡦ • Linux ্Ͱ͸جຊతʹϓϩηε୯ҐͰ੍ޚΛߦ͏ • ଳҬίϯτϩʔϧ͸࠶഑ஔ࣌ͷΈʹద༻͍ͨ͠ • => ΞΫηεৼΓ෼͚ͱσʔλͷ࠶഑ஔͷϓϩηεΛ෼཭ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 21

Slide 22

Slide 22 text

࣮૷ํ਑ 1. ΞΫηεৼΓ෼͚ • ϢʔβεϖʔεϑΝΠϧγεςϜʢFUSEʣʹ࣮ͯ૷ • खܰʹಋೖՄೳ • ೚ҙͷϑΝΠϧγεςϜΛར༻Մೳ 2. σʔλͷ࠶഑ஔ • σʔϞϯϓϩηεʹ࣮ͯ૷ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 22

Slide 23

Slide 23 text

1. ΞΫηεৼΓ෼͚ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 23

Slide 24

Slide 24 text

Mul$-Temperature FileSystemʢMTFSʣ • ϢʔβεϖʔεϑΝΠϧγεςϜ • ϢʔβʔεϖʔεσʔϞϯϓϩηε: m%sd • ϗοτσʔλ༻ͱίʔϧυσʔλ༻ͷύʔςΟγϣϯΛͦΕ ͧΕࢦఆͯ͠ىಈ • ΞϓϦέʔγϣϯ͔Βཁٻ͞ΕΔϑΝΠϧૢ࡞Λίϯτϩʔ ϧ͢Δ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 24

Slide 25

Slide 25 text

m"sd ͷॲཧ֓ཁ 1. ΞϓϦέʔγϣϯ͕ϑΝΠϧΞΫηεΛཁٻ 2. FUSE ϥΠϒϥϦΛ௨ͯ͠ m*sd ͕γεςϜίʔϧΛड৴ 3. ϗοτετϨʔδ΁໰͍߹Θͤ • ϑΝΠϧ͕ଘࡏ͠ͳ͍৔߹͸ίʔϧυετϨʔδʹ໰͍߹Θͤ 4. औಘͨ͠σʔλΛΞϓϦέʔγϣϯ΁ฦ٫ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 25

Slide 26

Slide 26 text

2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 26

Slide 27

Slide 27 text

ಡΈࠐΈॲཧͷಈ͖ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 27

Slide 28

Slide 28 text

ॻ͖ࠐΈͷಈ͖ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 28

Slide 29

Slide 29 text

2. σʔλͷ࠶഑ஔ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 29

Slide 30

Slide 30 text

σʔλͷ࠶഑ஔ • ϢʔβʔεϖʔεσʔϞϯϓϩηε: mt-relocatord • ٸ͍Ͱॲཧ͢Δඞཁ͕ͳ͍ͷͰγϯάϧεϨου࣮ߦ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 30

Slide 31

Slide 31 text

࣮૷্ͷ՝୊ͱରࡦ • I/O ͷଳҬɾεϧʔϓοτ੍ޚ • => cgroup2 ͰϒϩοΫ I/OΛ੍ޚ • IOPS੍ޚ: riops ͱ wiops ʹ੍ͯݶ • εϧʔϓοτ੍ޚ: rbps ͱ wbps Ͱ੍ݶ • => ioprio_set() ͰI/Oεέδϡʔϥ૚Λ੍ޚ • CLASS_IDLE Λࢦఆ • ࠶഑ஔҠ࣌ʹίʔϧυσʔλ͕σΟεΫΩϟογϡΛফඅͯ͠͠·͏ • => posix_fadvise(POSIX_FADV_DONTNEED)ͰҠಈର৅ͷίʔϧυσʔλͷΩϟογϡΛΫϦΞ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 31

Slide 32

Slide 32 text

Linux I/O ͷϦιʔε੍ޚՕॴ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 32

Slide 33

Slide 33 text

࠶഑ஔॲཧͷ֓ཁ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 33

Slide 34

Slide 34 text

mt-relocatord ͷઃఆ಺༰ • ࠶഑ஔॲཧͷεέδϡʔϦϯάͷઃఆ • ىಈ࣌ࠁ • ࣮ߦपظʢ1೔୯Ґʣ • ࠶഑ஔͷᮢ஋ઃఆ • ୯Ґ࣌ؒ౰ͨΓͷΞΫηε਺ɺߋ৽਺ • ࠷ऴΞΫηεɺ࠷ऴߋ৽͔Βݱࡏ࣌ࠁ·Ͱͷܦա࣌ؒ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 34

Slide 35

Slide 35 text

ར༻Πϝʔδ // Create partitions # mkfs.xfs /dev/vda1 # mkfs.btrfs /dev/vdb1 // Create MTFS managed Volumes # mtfsctl hot-volume create hv0 /export/sda1/www/ # mtfsctl cold-volume create cv0 /export/sdb1/www/ // Create mfsd # mtfsctl volume start hv0 cv0 # systemctl start mtfsd // Start mt-relocatord # systemctl start mt-relocatord 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 35

Slide 36

Slide 36 text

ຊػߏͷར༻ύλʔϯ environment Hot Data Storage Cold Data Storage On-premises(Storage Device) SSD HDD AWS(Block Storage) EBS Provisioned IOPS SSD Cold HDD AWS(Shared Storage) EFS(Provisioned Throughput) EFS(Infrequent Access Storage Class) ※ Shared Storage Ҏ֎͸೚ҙͷϑΝΠϧγεςϜ͕ར༻Մೳ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 36

Slide 37

Slide 37 text

՝୊ • ίʔϧυσʔλҠಈͷ҆શੑͱੑೳ • FUSE ʹΑΔಈ࡞Φʔόʔϔου • ϑΝΠϧ਺૿େʹର͢Δ mt-relocatord ͷॲཧෛՙ • mt-relocatord ͕εέδϡʔϦϯάػೳΛ࣋ͭඞཁ͕͋Δ͔ 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 37

Slide 38

Slide 38 text

·ͱΊ • ൚༻తͳ֊૚ԽετϨʔδγεςϜΛఏҊͨ͠ • FUSEɺcgroup2ɺioprioɺposix_fadvise Λ૊Έ߹Θ࣮ͤͨ૷ ΛఏҊͨ͠ • ຊػߏΛίϯςφ؀ڥʹ΋Ԡ༻͍͖͍ͯͨ͠ • ex. Docker Volume PluginsɺKubernetes Strage Interface 2019/04/13 ୈ4ճ Web System Architecture ݚڀձ | @nari_ex 38