自宅スケーラブル・ファイルシステムのご紹介

2012年3月17-18日
Open Source Conference 2012 Tokyo/Spring

Transcript

1. ࣗ୐εέʔϥϒϧɾ ϑΝΠϧγεςϜͷ͝঺հ (MVTUFS'4 *OGJOJCBOEͰ࣮ݱ͢ΔɺແݶͷՄೳੑ ޫ҆ ݈ଠ࿠ 1
2. ΞδΣϯμ ˙·ͣ͸෺݅୳͔͠Βɾɾɾ ˙·ͣ͸ϐΞπʔϐΞ͔Β ˙εΠονߪೖʂʂͱۤ࿑ ˙(MVTUFS'4Λͨ͠ͳΉ ˙$FOU04
   Y@ɺWNXBSF &49JVQ͔ΒͷϚ΢ϯτྫ ˙؀ڥ঺հɾωοτϫʔΫߏ੒঺հ ˙ँࣙ 2
3. ϓϩϑΟʔϧ ޫ҆ ݈ଠ࿠ ໿೥ɺαʔόʔ੡඼ͷ ϓϩμΫτηʔϧεΛ΍͍ͬͯ·͢ɻ (PPHMFͰग़ͯ͘Δओͳաڈͷ࿐ग़ 'BDFCPPL΋΍͍ͬͯ·͢ɻ 3
4. ͳͥࣗ୐4"/ͳͷ͔ɺͲ͜·Ͱ΍Δͷ͔ ̍ɺʮ৮ͬͯΈͳ͍ͱΘ͔Βͳ͍ʯੑ෼͔ͩΒ ̎ɺߴ଎ΠϯλʔϑΣʔε͕ग़ͦΖ͖ͬͯͨ ̏ɺεέʔϥϒϧϑΝΠϧγεςϜʹະདྷΛײ͔ͨ͡Β ̐ɺ׳Ε਌͠Μͩ*"αʔόʔΛ׆༻͢Δ͜ͱ͕Ͱ͖ͦ͏͔ͩΒ εςοϓߴ଎ΠϯλʔϑΣʔεΛϞϊʹ͢Δɻ εςοϓεέʔϥϒϧϑΝΠϧγεςϜΛબͿɻ εςοϓ͍͟ɺݕূʂ 4
5. ·ͣ͸෺݅୳͔͠Βɾɾɾ ৑ஊͰ͢ɻͪͳΈʹεϖοΫ͸ԼهͷΑ͏ͳײ͡ɻ ిݯɿ"ܥ౷ ैྔి౮#ܖ໿ ۭௐ͸ྫྷ๪த৺ ͢΂ͯͷαʔόʔػثΛϩϑτ΁ઃஔɺ഑ઢ 5 生活とスケーラブルストレージの共存＝ロフトIDC Servers in the loft!
6. ·ͣ͸ϐΞπʔϐΞ͔Β ͢΂ͯͷಓ͸ຊ͔Βɾɾɾ ͞·͟·ͳߴ଎ΠϯλʔϑΣʔε͕͋Δதબ୒ͨ͠ͷ͸ɿ*OGJOJCBOE ̍ɺ(C͔Β(C·Ͱɺجຊతʹ࢖͍ํ͸ҰॹͰͭͿ͕͠ޮ͖ͦ͏ ʢେ·͔ʹ͸ʣ ̎ɺΠϯλʔϑΣʔεͷՁ͕͔֨҆ͬͨ ʢதݹͰԁ୆ɺέʔϒϧԁʣ 6 10G/20G SDR Infiniband 40G/56G QDR Infiniband
7. ·ͣ͸ϐΞπʔϐΞ͔Β ݕূͯ͠Θ͔ͬͨ͜ͱɿ ݁ہΧʔυʹ͸*1ΛৼΔ͠ɺৼͬͯ͠·͑͹௨ৗͷ /*$ͱมΘΒͳ͍ ରԠ04͸$FOU04ܥͳΒ໰୊ͳ͠ɺԾ૝ԽϋΠύʔ όΠβʔͷରԠ΋ਐΜͰ͍Δ *OGJOJCBOEωοτϫʔΫ಺ʹ͸4VCOFU
   .BOBHFS͕࠷௿ͭ ىಈ͍ͯ͠ͳ͚Ε͹ͳΒͳ͍ɻ 4VCOFU.BOBHFS͸*OGJOJCBOEεΠονʹ಺ଂͨ͠੡඼΋͋Δɻ 7 <SPPU!NM
   d>
   JGDPOGJH B *GDPOGJH VTFT
   UIF
   JPDUM BDDFTT
   NFUIPE
   UP
   HFU
   UIF
   GVMM
   BEESFTT
   JOGPSNBUJPO
   XIJDI
   MJNJUT
   IBSEXBSF
   BEESFTTFT
   UP
   
   CZUFT #FDBVTF
   *OGJOJCBOE BEESFTT
   IBT
   
   CZUFT
   POMZ
   UIF
   GJSTU
   
   CZUFT
   BSF
   EJTQMBZFE
   DPSSFDUMZ *GDPOGJH JT
   PCTPMFUF
   'PS
   SFQMBDFNFOU
   DIFDL
   JQ JC
   
   
   
   
   
   
   -JOL
   FODBQ*OGJOJ#BOE )8BEES 999999
   
   JOFU BEES
   
   #DBTU
   
   .BTL JOFU
   BEES
   999999999999G
   4DPQF-JOL 61
   #30"%$"45
   36//*/(
   .6-5*$"45
   
   .56
   
   .FUSJD 39
   QBDLFUT
   FSSPST
   ESPQQFE
   PWFSSVOT
   GSBNF 59
   QBDLFUT
   FSSPST
   ESPQQFE
   PWFSSVOT
   DBSSJFS DPMMJTJPOT
   UYRVFVFMFO
   39
   CZUFT
   
   (J#
   
   59
   CZUFT
   
   .J#
8. ϐΞπʔϐΞݕূͰ͸·ͬͨϙΠϯτ ఺ ̍ɺΧʔυ͕յΕ͍ͯͨ FCBZ͔ΒΧʔυΛௐୡ ˠ1$*ͷϒϥέοτ͕͍͍ͭͯͳ͍ ˠ$9ͷίωΫλʔͷ൒ాׂ͕Ε͍ͯͨ ˠͲ͏΍ͬͯ΋ຕ͚ͩϑΝʔϜ͕ ্͕Βͳ͍ ԁߴΛ׆༻͠Α͏ͱ͕ͨ͠ཪ໨ʹʜ 8
9. ϐΞπʔϐΞݕূͰ͸·ͬͨϙΠϯτ ఺ ̎ɺϑΝʔϜ΢ΣΞͷߋ৽ NTU TUBUVTίϚϯυͰσόΠεΛௐ΂ΔˠͦΜͳ΋ͷ͸ͳ͍ʂ ɾɾɾNTU TUBSUͰαʔϏεΛ։࢝͢Δͱ໰୊ͳ͘σόΠεൃݟɻ 9 [root@ml115 ~]# mst status MST modules: ------------ MST PCI module is not loaded MST PCI configuration module is not loaded MST devices: ------------ No MST devices found [root@ml115 ~]# mst start Starting MST (Mellanox Software Tools) driver set Loading MST PCI module - Success Loading MST PCI configuration module - Success Create devices [root@ml115 ~]# mst status MST modules: ------------ MST PCI module loaded MST PCI configuration module loaded MST devices: ------------ /dev/mst/mt25218_pci_cr0 - PCI direct access. bus:dev.fn=12:00.0 bar=0xfdf00000 size=0x100000 Chip revision is: A0 /dev/mst/mt25218_pciconf0 - PCI configuration cycles access. bus:dev.fn=12:00.0 addr.reg=88 data.reg=92 Chip revision is: A0
10. ϐΞπʔϐΞݕূͰ͸·ͬͨϙΠϯτ ఺ ̎ɺϑΝʔϜ΢ΣΞͷߋ৽ ߋ৽ͷίϚϯυͰɺΦϓγϣϯ͕ඞཁͳ৔߹͕͋Δʢಛʹ৽඼ΧʔυΛങͬͨͱ ͖ͷϑΝʔϜਅͬനঢ়ଶ͔Β͸ඞཁͩͬͨͱࢥΘΕΔʣ 10 flint -skip_is -d /dev/mst/mt25218_pci_cr0 -i fw-25218-5_3_000- MHEA28-XTC_A1-A3.bin burn
11. ϐΞπʔϐΞݕূͰ͸·ͬͨϙΠϯτ ఺ ̏ɺωοτϫʔΫઃఆʢ*1ΞυϨεʣ ॳΊ͸TZTUFNDPOGJHOFUXPSLUVJͰઃఆ͕ͨ͠ɺͭͳ͕Βͣɻ ݁ہԼهͷઃఆͰಈ࡞Λ֬ೝͨ͠ɻ 11 /etc/sysconfig/network-scripts/ifcfg-ib0 # Please read /usr/share/doc/initscripts-*/sysconfig.txt # for the documentation of these parameters. DEVICE=ib0 BOOTPROTO=static NETMASK=255.255.255.0 TYPE=Infiniband #HWADDR=80:00:04:04:fe:80 IPADDR=192.168.50.60 BROADCAST=192.168.50.255 NETWORK=192.168.50.0 ONBOOT=yes
12. */'*/*#"/%εΠονߪೖ ͋Δ೔ɺԿؾͳ͘ϠϑΦΫΛͷ͍ͧͯΈΔͱɺϙʔτͷ*OGJOJCBOE εΠονΛൃݟɺఔ౓΋Αͦ͞͏ͳͷͰଈམࡳɻ 'MFYUSPOJDT
    '9ͱ͍͏੡඼ɻˇ Ͱམࡳɻ ͍͍ങ͍෺Λͨ͠ʜ͸͕ͣͩͬͨɺ ͜Ε͸൥͍ͷͰ༗໊ͳεΠονͩͬͨʜ Ҏޙɺϩϑτ͸ຊ෺ͷσʔληϯλʔͱಉ౳ͷଘࡏײɻ 12
13. (-645&3'4Λͨ͠ͳΉ ͓͞Β͍ ̍ɺεέʔϥϒϧͳετϨʔδͷͨΊͷ൚༻෼ࢄϑΝΠϧγεςϜ ̎ɺ*"αʔόʔΛར༻ͯ͠ɺJOGJOJCBOEͷ3%."΍5$1*1ΠϯλʔίωΫ τͳͲͷ֤छετϨʔδΛू໿͠ɺେن໛ฒྻωοτϫʔΫϑΝΠϧγες ϜΛߏஙՄೳ ̏ɺαʔόʔ಺ͷྖҬΛlCSJDLzͱ͍͏୯ҐͰڞ༗ઃఆ͢Δ 13
14. ͭͷಈ࡞Ϟʔυ 14 Server1 Server2 Server3 Server1 Server2 Server3 Distributed Striped ϑΝΠϧసૹ଎౓ɿ֤ϊʔυ୯ମͷೳྗʹ ࠨӈʹଞͷαʔόʔͷ*0ͱ଍ฒΈΛͦΖ͑ Δඞཁ͕ͳ͍ ϑΝΠϧసૹ଎౓ɿ֤ϊʔυ୯ମͷೳྗͱ ετϥΠϓʹࢀՃ͢Δ୆਺ʹࠨӈʹ*0ͷ஗ ͍αʔόʔʹ଍ΛҾͬுΒΕΔ
15. ৑௕ઃఆɿଟॏͷྫ 15 Server1 Server2 Server3 Distributed Server1 Server2 Server3
16. (-645&3'4Λͨ͠ͳΉ ݕূϝχϡʔɿ EJTUSJCVUF SFQMJDBUFɺ4USJQFE SFQMJDBUFͷݕূ ˠ ࠷௿୆αʔόʔ͕ඞཁ Ϛ΢ϯτ͢Δଆ΋JOGJOJCBOE͕ඞཁɺ·ͨෳ਺͔Βͷಉ࣌Ϛ΢ϯτ΋ࢼ͍ͨ͠ /'4ͱ3%."ͷϚ΢ϯτ΋ಉ࣌ʹͰ͖Δ͔ࢼ͍ͨ͠ ˠ ͞Βʹ୆௥Ճ ߹ܭ୆ͷαʔόʔͱɺJOGJOJCBOE )$" έʔϒϧɺεΠονΛۦ࢖ͨ͠ݕূ͕࢝·Δ 16 brick1 brick2 brick3 brick4 server1 server2
17. ֤ϊʔυͷߏ੒ 17 brick1 brick2 brick3 brick4 server1 IB IP: 192.168.50.10 IB IP: 192.168.50.20 IB IP: 192.168.50.1 IB IP: 192.168.50.60 IB IP: 192.168.50.70
18. 70-6.&ͷ࡞੒ 18 #SJDLdɿ 044DJFOUJGJD -JOVY
    
    Y@ HMVTUFSHMVTUFSGT CFUB JOGJOJCBOEPGFE )$".FMMBOPY .)&"95$
    (C 4FSWFS 04$FOU04
    Y@ )$".FMMBOPY .)&"95$
    (C EJTUSJCVUFEEVQMJDBUFͷ৔߹ͷίϚϯυ HMVTUFS WPMVNF
    DSFBUF
    HMVTUFSWPM SFQMJDB
    
    USBOTQPSU
    UDQ SENB EBUBCSJDL
    EBUBCSJDL
    IPNFCSJDL
    IPNFCSJDL 4USJQFEEVQMJDBUFͷ৔߹ͷίϚϯυ HMVTUFS WPMVNF
    DSFBUF
    HMVTUFSWPM TUSJQF
    
    SFQMJDB
    
    USBOTQPSU
    UDQ SENB EBUBCSJDL
    EBUBCSJDL
    IPNFCSJDL
    IPNFCSJDL
19. %*453*#65&3&1-*$"5& 19 brick1 brick2 brick3 brick4 server1
20. 453*1&%3&1-*$"5& 20 brick1 brick2 brick3 brick4 server1
21. ෳ਺ಉ࣌Ϛ΢ϯτ ͭͷϘϦϡʔϜʹରͯ͠ɺͭͷϚγϯ͔ΒϚ΢ϯτͰ͖Δ͔ɻ ʢͲͪΒ΋HMVTUFSωΠςΟϒΫϥΠΞϯτʣ ˠ ໰୊ͳ͘Ϛ΢ϯτͰ͖͕ͨɺ౰વͳ͕ΒϩοΫػೳ͸ͳ͠ɻ 21 brick1 brick2 brick3 brick4 server1 server2 gluster volume start glustervol mount -t glusterfs -o mountproto=rdma 192.168.50.10:/glustervol /root/ibtest mount -t glusterfs -o mountproto=rdma 192.168.50.10:/glustervol /root/ibtest
22. /'4ͰͷϚ΢ϯτ ͭͷϘϦϡʔϜʹରͯ͠ɺͭͷϚγϯ͔ΒϚ΢ϯτͰ͖Δ͔ɻ ʢ$FOU04͸HMVTUFSωΠςΟϒΫϥΠΞϯτɺ&49J͸/'4ʣ ˠ EJTUSJCVUF SFQMJDBUFͷ࣌͸໰୊ͳ͘Ϛ΢ϯτͰ͖͕ͨɺ4USJQFE SFQMJDBUF ࣌ʹ͸/'4Ϛ΢ϯτͷΈͰ͖ͳ͔ͬͨɻ 22 brick1 brick2 brick3 brick4 server1 server2 gluster volume start glustervol mount -t glusterfs -o mountproto=rdma 192.168.50.10:/glustervol /root/ibtest
23. (-645&3'4ݕূ·ͱΊ EJTUSJCVUF SFQMJDBUFɺ4USJQFE SFQMJDBUFͷݕূ ˠ σϑΥϧτ͸EJTUSJCVUFͰɺTUSJQFE͸ߴ଎Խ͕ظ଴Ͱ͖Δ͕༻్Λબͼͦ͏ɻ EJTUSJCVUF͸ϑΝΠϧ࣮ମ͕֬ೝͰ͖ɺ࠷ѱͷ৔߹αϧϕʔδͰ͖ͦ͏ɻ ෳ਺͔Βͷಉ࣌Ϛ΢ϯτ΋ࢼ͍ͨ͠ /'4ͱ3%."ͷϚ΢ϯτ΋ಉ࣌ʹͰ͖Δ͔ࢼ͍ͨ͠ ˠ ಉ࣌઀ଓ΋Մೳ͕ͩɺ4USJQFEΛ૊Μͩ࣌ͷ/'4ͷڍಈΛ΋͏গ͠ݕূ͢Δඞཁ͋Γɻ 23 brick1 brick2 brick3 brick4 server1 server2
24. ݕূͷ5*14 7PMVNFΛ࡞Γ௚͢৔߹ͷϙΠϯτɻ લ࡞ͬͨ7PMVNFʹଐ͍ͯͨ͠#SJDLͷத਎͸׬શ࡟আ͢Δ͜ͱɻϑΥϧμ͝ͱফ ͢ͷ͕๬·͍͠ɻͦ͏͠ͳ͍ͱɺʮϑΝΠϧΛஔ͍ͨͷʹݟ͑ͳ͍ʯʮಉ͡ϑΝ Πϧ͕ݸݟ͑ΔʯͳͲͷոݱ৅͕ൃੜɻ SN rSG FUDHMVTUFS
    ΋࣮ߦͯ͠ɺ׬શʹઃఆΛඈ͹ͯ͠ɺ SQN
    rJWI rGPSDF
    HMVTUFSGT SQN
    Ͱ࠶Πϯετʔϧ͢Δɻ 24 brick1 brick2 brick3 brick4 server1 server2
25. ࠓޙͷల๬ ΋͏গ͠εϧʔϓοτΛ্͍͛ͨɿ ࠓճͷߏ੒Ͱ͸ɺ֤#SJDLͷσΟεΫ͕4"5"
    )%%୆ ˠ *OGJOJCBOEͷଳҬ͕༨͍ͬͯΔ 'BDFCPPL GMBTIDBDIFͱ44%ΛϚ΢ϯτ͢Δαʔόʔʹ௥Ճͯ͠େ෯εϧʔ ϓοτΞοϓͰ͖ͳ͍͔ʁʁ 25 brick1 brick2 brick3 brick4 server1 server2
26. ࣗ୐Ͱεέʔϥϒϧ ϑΝΠϧγεςϜ *"αʔόʔΛ࢖ͬͯɺεέʔϥϒϧετϨʔδ͕༰қʹ૊ΊΔΑ͏ʹ ͳ͖ͬͯ·ͨ͠ɻ ͥͻɺ΍ͬͯΈ·͠ΐ͏ɻ 26 brick1 brick2 brick3 brick4 server1 server2
27. ँࣙ ϠϑΦΫͰJOJGJOJCBOEؔ࿈෦ࡐΛڙڅ͠ଓ͚͓ͯΒΕΔVDIJEB@
    ༷ʹ͓ྱΛ ਃ্͛͠·͢ɻ 27
28. #"$,61 28
29. ωοτϫʔΫɺػثߏ੒ 29 brick1 brick2 brick3 brick4 server1 server2 CPU:Core i5-670 Mem:8GB DDR3 PC1333 HDD: HITACHI 2TB 7200rpm SATA x4 M/B Intel DQ57TM CPU:Pentium G620 Mem:4GB DDR3 PC1333 HDD: Intel SSD 80GB M/B Intel DQ67EP CPU:Celeron G1101 Mem:4GB DDR3 PC1333 HDD: HITACHI 1TB SATA HDD M/B HP ML110G6 CPU:Intel Core-i5 2400S Mem:8GB DDR3 PC1333 HDD: HITACHI 1TB SATA HDD x2 M/B WADE-8012 CPU:Celeron G1101 Mem:4GB DDR3 PC1333 HDD: SEAGATE 1TB SATA HDD M/B NEC Express5800 CPU:AMD Athlon 3core Mem:6GB DDR3 PC1333 HDD: SEAGATE 2TB SATA HDD M/B HP ML115 Router: Cisco ASA5505 To internet