Share
Embed
自宅スケーラブル・ファイルシステムのご紹介
2012年3月17-18日
Open Source Conference 2012 Tokyo/Spring
cyberomusubi
June 09, 2026
Transcript
29 slides
-
·ͣϐΞπʔϐΞ͔Β ݕূͯ͠Θ͔ͬͨ͜ͱɿ ݁ہΧʔυʹ*1ΛৼΔ͠ɺৼͬͯ͠·͑௨ৗͷ /*$ͱมΘΒͳ͍ ରԠ04$FOU04ܥͳΒͳ͠ɺԾԽϋΠύʔ όΠβʔͷରԠਐΜͰ͍Δ *OGJOJCBOEωοτϫʔΫʹ4VCOFU.BOBHFS͕࠷ͭ ىಈ͍ͯ͠ͳ͚ΕͳΒͳ͍ɻ 4VCOFU.BOBHFS*OGJOJCBOEεΠονʹଂͨ͋͠Δɻ 7
<SPPU!NMd>JGDPOGJH B *GDPOGJH VTFTUIFJPDUM BDDFTTNFUIPEUPHFUUIFGVMMBEESFTTJOGPSNBUJPO XIJDIMJNJUTIBSEXBSFBEESFTTFTUP CZUFT #FDBVTF*OGJOJCBOE BEESFTTIBTCZUFT POMZUIFGJSTUCZUFTBSFEJTQMBZFEDPSSFDUMZ *GDPOGJH JTPCTPMFUF'PSSFQMBDFNFOUDIFDLJQ JC-JOLFODBQ*OGJOJ#BOE )8BEES 999999 JOFU BEES#DBTU.BTL JOFUBEES999999999999G4DPQF-JOL 61#30"%$"4536//*/(.6-5*$"45.56.FUSJD 39QBDLFUTFSSPSTESPQQFEPWFSSVOTGSBNF 59QBDLFUTFSSPSTESPQQFEPWFSSVOTDBSSJFS DPMMJTJPOTUYRVFVFMFO 39CZUFT (J# 59CZUFT .J# -
ϐΞπʔϐΞݕূͰ·ͬͨϙΠϯτ ̎ɺϑΝʔϜΣΞͷߋ৽ NTU TUBUVTίϚϯυͰσόΠεΛௐΔˠͦΜͳͷͳ͍ʂ ɾɾɾNTU TUBSUͰαʔϏεΛ։࢝͢Δͱͳ͘σόΠεൃݟɻ 9 [root@ml115 ~]#
mst status MST modules: ------------ MST PCI module is not loaded MST PCI configuration module is not loaded MST devices: ------------ No MST devices found [root@ml115 ~]# mst start Starting MST (Mellanox Software Tools) driver set Loading MST PCI module - Success Loading MST PCI configuration module - Success Create devices [root@ml115 ~]# mst status MST modules: ------------ MST PCI module loaded MST PCI configuration module loaded MST devices: ------------ /dev/mst/mt25218_pci_cr0 - PCI direct access. bus:dev.fn=12:00.0 bar=0xfdf00000 size=0x100000 Chip revision is: A0 /dev/mst/mt25218_pciconf0 - PCI configuration cycles access. bus:dev.fn=12:00.0 addr.reg=88 data.reg=92 Chip revision is: A0 -
ϐΞπʔϐΞݕূͰ·ͬͨϙΠϯτ ̎ɺϑΝʔϜΣΞͷߋ৽ ߋ৽ͷίϚϯυͰɺΦϓγϣϯ͕ඞཁͳ߹͕͋Δʢಛʹ৽ΧʔυΛങͬͨͱ ͖ͷϑΝʔϜਅͬനঢ়ଶ͔ΒඞཁͩͬͨͱࢥΘΕΔʣ 10 flint -skip_is -d /dev/mst/mt25218_pci_cr0
-i fw-25218-5_3_000- MHEA28-XTC_A1-A3.bin burn -
ϐΞπʔϐΞݕূͰ·ͬͨϙΠϯτ ̏ɺωοτϫʔΫઃఆʢ*1ΞυϨεʣ ॳΊTZTUFNDPOGJHOFUXPSLUVJͰઃఆ͕ͨ͠ɺͭͳ͕Βͣɻ ݁ہԼهͷઃఆͰಈ࡞Λ֬ೝͨ͠ɻ 11 /etc/sysconfig/network-scripts/ifcfg-ib0 # Please read
/usr/share/doc/initscripts-*/sysconfig.txt # for the documentation of these parameters. DEVICE=ib0 BOOTPROTO=static NETMASK=255.255.255.0 TYPE=Infiniband #HWADDR=80:00:04:04:fe:80 IPADDR=192.168.50.60 BROADCAST=192.168.50.255 NETWORK=192.168.50.0 ONBOOT=yes -
ͭͷಈ࡞Ϟʔυ 14 Server1 Server2 Server3 Server1 Server2 Server3 Distributed Striped
ϑΝΠϧసૹɿ֤ϊʔυ୯ମͷೳྗʹ ࠨӈʹଞͷαʔόʔͷ*0ͱฒΈΛͦΖ͑ Δඞཁ͕ͳ͍ ϑΝΠϧసૹɿ֤ϊʔυ୯ମͷೳྗͱ ετϥΠϓʹࢀՃ͢Δʹࠨӈʹ*0ͷ ͍αʔόʔʹΛҾͬுΒΕΔ -
(-645&3'4Λͨ͠ͳΉ ݕূϝχϡʔɿ EJTUSJCVUF SFQMJDBUFɺ4USJQFE SFQMJDBUFͷݕূ ˠ ࠷αʔόʔ͕ඞཁ Ϛϯτ͢ΔଆJOGJOJCBOE͕ඞཁɺ·ͨෳ͔Βͷಉ࣌Ϛϯτࢼ͍ͨ͠ /'4ͱ3%."ͷϚϯτಉ࣌ʹͰ͖Δ͔ࢼ͍ͨ͠ ˠ
͞ΒʹՃ ߹ܭͷαʔόʔͱɺJOGJOJCBOE )$" έʔϒϧɺεΠονΛۦͨ͠ݕূ͕࢝·Δ 16 brick1 brick2 brick3 brick4 server1 server2 -
֤ϊʔυͷߏ 17 brick1 brick2 brick3 brick4 server1 IB IP: 192.168.50.10
IB IP: 192.168.50.20 IB IP: 192.168.50.1 IB IP: 192.168.50.60 IB IP: 192.168.50.70 -
70-6.&ͷ࡞ 18 #SJDLdɿ 044DJFOUJGJD -JOVYY@ HMVTUFSHMVTUFSGT CFUB JOGJOJCBOEPGFE )$".FMMBOPY .)&"95$(C
4FSWFS 04$FOU04Y@ )$".FMMBOPY .)&"95$(C EJTUSJCVUFEEVQMJDBUFͷ߹ͷίϚϯυ HMVTUFS WPMVNFDSFBUFHMVTUFSWPM SFQMJDBUSBOTQPSUUDQ SENB EBUBCSJDLEBUBCSJDL IPNFCSJDLIPNFCSJDL 4USJQFEEVQMJDBUFͷ߹ͷίϚϯυ HMVTUFS WPMVNFDSFBUFHMVTUFSWPM TUSJQFSFQMJDBUSBOTQPSUUDQ SENB EBUBCSJDLEBUBCSJDL IPNFCSJDLIPNFCSJDL -
ෳಉ࣌Ϛϯτ ͭͷϘϦϡʔϜʹରͯ͠ɺͭͷϚγϯ͔ΒϚϯτͰ͖Δ͔ɻ ʢͲͪΒHMVTUFSωΠςΟϒΫϥΠΞϯτʣ ˠ ͳ͘ϚϯτͰ͖͕ͨɺવͳ͕ΒϩοΫػೳͳ͠ɻ 21 brick1 brick2 brick3 brick4
server1 server2 gluster volume start glustervol mount -t glusterfs -o mountproto=rdma 192.168.50.10:/glustervol /root/ibtest mount -t glusterfs -o mountproto=rdma 192.168.50.10:/glustervol /root/ibtest -
/'4ͰͷϚϯτ ͭͷϘϦϡʔϜʹରͯ͠ɺͭͷϚγϯ͔ΒϚϯτͰ͖Δ͔ɻ ʢ$FOU04HMVTUFSωΠςΟϒΫϥΠΞϯτɺ&49J/'4ʣ ˠ EJTUSJCVUF SFQMJDBUFͷ࣌ͳ͘ϚϯτͰ͖͕ͨɺ4USJQFE SFQMJDBUF ࣌ʹ/'4ϚϯτͷΈͰ͖ͳ͔ͬͨɻ 22 brick1
brick2 brick3 brick4 server1 server2 gluster volume start glustervol mount -t glusterfs -o mountproto=rdma 192.168.50.10:/glustervol /root/ibtest -
(-645&3'4ݕূ·ͱΊ EJTUSJCVUF SFQMJDBUFɺ4USJQFE SFQMJDBUFͷݕূ ˠ σϑΥϧτEJTUSJCVUFͰɺTUSJQFEߴԽ͕ظͰ͖Δ͕༻్Λબͼͦ͏ɻ EJTUSJCVUFϑΝΠϧ࣮ମ͕֬ೝͰ͖ɺ࠷ѱͷ߹αϧϕʔδͰ͖ͦ͏ɻ ෳ͔Βͷಉ࣌Ϛϯτࢼ͍ͨ͠ /'4ͱ3%."ͷϚϯτಉ࣌ʹͰ͖Δ͔ࢼ͍ͨ͠ ˠ
ಉ࣌ଓՄೳ͕ͩɺ4USJQFEΛΜͩ࣌ͷ/'4ͷڍಈΛ͏গ͠ݕূ͢Δඞཁ͋Γɻ 23 brick1 brick2 brick3 brick4 server1 server2 -
ݕূͷ5*14 7PMVNFΛ࡞Γ͢߹ͷϙΠϯτɻ લ࡞ͬͨ7PMVNFʹଐ͍ͯͨ͠#SJDLͷதશআ͢Δ͜ͱɻϑΥϧμ͝ͱফ ͢ͷ͕·͍͠ɻͦ͏͠ͳ͍ͱɺʮϑΝΠϧΛஔ͍ͨͷʹݟ͑ͳ͍ʯʮಉ͡ϑΝ Πϧ͕ݸݟ͑ΔʯͳͲͷոݱ͕ൃੜɻ SN rSG FUDHMVTUFS ࣮ߦͯ͠ɺશʹઃఆΛඈͯ͠ɺ SQNrJWI
rGPSDFHMVTUFSGT SQNͰ࠶Πϯετʔϧ͢Δɻ 24 brick1 brick2 brick3 brick4 server1 server2 -
ࠓޙͷల ͏গ͠εϧʔϓοτΛ্͍͛ͨɿ ࠓճͷߏͰɺ֤#SJDLͷσΟεΫ͕4"5")%% ˠ *OGJOJCBOEͷଳҬ͕༨͍ͬͯΔ 'BDFCPPL GMBTIDBDIFͱ44%ΛϚϯτ͢ΔαʔόʔʹՃͯ͠େ෯εϧʔ ϓοτΞοϓͰ͖ͳ͍͔ʁʁ 25 brick1
brick2 brick3 brick4 server1 server2 -
ωοτϫʔΫɺػثߏ 29 brick1 brick2 brick3 brick4 server1 server2 CPU:Core i5-670
Mem:8GB DDR3 PC1333 HDD: HITACHI 2TB 7200rpm SATA x4 M/B Intel DQ57TM CPU:Pentium G620 Mem:4GB DDR3 PC1333 HDD: Intel SSD 80GB M/B Intel DQ67EP CPU:Celeron G1101 Mem:4GB DDR3 PC1333 HDD: HITACHI 1TB SATA HDD M/B HP ML110G6 CPU:Intel Core-i5 2400S Mem:8GB DDR3 PC1333 HDD: HITACHI 1TB SATA HDD x2 M/B WADE-8012 CPU:Celeron G1101 Mem:4GB DDR3 PC1333 HDD: SEAGATE 1TB SATA HDD M/B NEC Express5800 CPU:AMD Athlon 3core Mem:6GB DDR3 PC1333 HDD: SEAGATE 2TB SATA HDD M/B HP ML115 Router: Cisco ASA5505 To internet