HDFS RAID

an introduction of HDFS RAID

Transcript

1. HDFS RAID ᇙത Friday, December 21, 12
2. Outline • Overview • Some detail • How to use Friday, December 21, 12
3. Why? • $$$$$......and $$$$$.... Friday, December 21, 12
4. How? • 䫩খ෭ຊ਺+҆શੑෆ߱௿=Erasred Code Friday, December 21, 12
5. Overview Friday, December 21, 12
6. DRFS • ࢖༻Raidम䐾ޭೳแ૷ྃHDFSతDFS • ౰读औHDFSதblock਺ਾੋ发ੜ CorruptionException҃DecomissionException时 ࠜਾparityจ݅对䇗ৗblock进ߦम䐾 Friday, December 21, 12
7. RaidShell • Ұ࿽Client୺؅ཧ޻۩ɼఏڙraidFileɼunraidɼ fixBlock, checkStatus౳ޭೳ • ௨过RPC请ٻ޲RaidNodeɼNN౳节఺发ى请ٻ Friday, December 21, 12
8. RaidNode • DRFSதআNNɼJT֎ୈࡾ࿽Master • 维护Ұ组తdaemon线ఔɼࠜਾ༻户ఏަతraid 请ٻʢraid policyʣ׬੒raidɼfixɼpurge౳֤㜎 ૢ࡞ Friday, December 21, 12
9. Threads on RaidNode • TriggerMonitorɿ֩৺线ఔɼࠜਾpolicy഑ஔ调౓׬੒raid • BlockFixerɿम䐾corruptจ݅తworker线ఔ • BlockCopierɿम䐾decomissionจ݅తworker线ఔ • PurgeParityThreadɿਗ਼ཧݽၰparityจ݅త线ఔ • PlacementThreadɿ෼ࢄdatanode্过౓ूதతparity blockత线ఔ • HarThreadɿক过ظతparity进ߦharҎ䫩গparityจ݅਺త线ఔ Friday, December 21, 12
10. Some detail • raid file • fix file Friday, December 21, 12
11. Raid File • hadoop raidshell -raidFile /path • TriggerMonitor Friday, December 21, 12
12. Friday, December 21, 12
13. RaidNode.doRaid Friday, December 21, 12
14. Encode Friday, December 21, 12
15. Erasred Code • RS: 10:4 • XOR: 10:1 Friday, December 21, 12
16. Fix File • DRFS readั获䇗ৗ • BlockIntegrityMonitor检查౸䇗ৗจ݅ Friday, December 21, 12
17. DRFS read Friday, December 21, 12
18. BlockIntegrityMonitro • local • dist • sorry...຅图ྃ... Friday, December 21, 12
19. Local LocalBlockIntegrity线ఔత֩৺ੋपظ调༻doFixํ๏म䐾corruptจ݅ɼओཁ ྲྀఔ೗Լɿ 1. ௨过DFSck获औcurrputจ݅৴ଉʢHTTP访问ʣ 2. 过滤ᎃෆೳ伭䐾తcorruptจ݅ʢ຅༗parityจ݅తʣ 3. কcorrputతจ݅ഉংɼഉং规则೗Լ ◦ parityจ݅优ઌɼsourceจ݅ࡏ޳ ◦ parityจ݅தcodec.priorityߴతࡏઌʢcodec.priority௨过JSONத coder_priority഑ஔʣ 4. 对ഉং߸తcorruptจ݅ྻදґ࣍௨过BlockRecontsturerདྷ伭䐾ɻ Friday, December 21, 12
20. Dist 1. 检查౰લਖ਼ࡏ运ߦతम䐾job਺ɼ೗Ռ౰લjobቮ经େဋjob্ݶɼ则౳଴೭લత job运ߦ׬ʢ该্线ՄҎ௨过raid.blockfix.maxpendingjobsདྷ഑ஔɼᘍ认ੋ100Lʣ 2. ௨过DFSck获औ损ᆀతจ݅৴ଉɼblockfixer线ఔ获औcorruptจ݅৴ଉɼ blockCopier获औdecomissionจ݅৴ଉ 3. 计ࢉ获ಘత损ᆀจ݅త优ઌ级 4. ক计ࢉ޷优ઌ级తจ݅ྻද҈优ઌ级ഉংɼ࡞为ࢀ਺构ݐम䐾Jobɻ 5. Jobత输ೖੋॴ༗धཁम䐾తจ݅pathతsequence fileɻձࠜਾ raid.blockfix.filespertask഑ஔత值进ߦsyncɼଈࡏjobతsplit阶ஈձ҈র该值设ஔ త进ߦsplitɼᘍ认ੋ20 6. JobతMapperओཁੋ௨过Reconstruterࡏtaskص্׬੒㠳应จ݅త伭䐾ɻ Friday, December 21, 12
21. 优ઌ级计ࢉ 1. corrputจ݅త优ઌ级೗Լ(R为จ݅෭ຊ਺ɼC为该จ݅corrputత block਺)ɿ ◦ ᘍ认为LOW ◦ R>1 && C>0时ɿ HIGH ◦ R==1 && C>1时ɿ HIGH ◦ parityfile corrput && C>0时ɿ HIGH 2. decomission优ઌ级计ࢉ规则೗ԼʢD为decomissionతblock਺): ◦ ᘍ认为LOW ◦ D>4时ɿ HIGH Friday, December 21, 12
22. BlockReconstructor • BlockIntegrityMonitor࿨RaidShell对จ݅తम䐾࠷终౎௨过 BlockReconstructorདྷ׬੒ɻ • BlockReconstructorम䐾จ݅过ఔओཁ෼为ࡾ㜎ɿHar parityจ݅ɼparity จ݅࿨ݯ਺ਾจ݅ɻ Friday, December 21, 12
23. Har parity file 1. 获औharจ݅తجຊ৴ଉٴindex 2. 获औharจ݅தతlost blockɼ对㑌࿽block进ߦ೗Լ处ཧɿ 3. ࡏຊ஍จ݅ܥ统创ݐ该blockత临时จ݅ɼ 4. 对该block㐪ٴతॴ༗parityจ݅ɼ获औ对应తsourceจ݅ɼ௨过 Encoderॏ৽encodeɼࡏຊ஍ੜ੒parity਺ɻ 5. কຊ஍ੜ੒తblock਺ਾ发ૹ౸Ұ࿽datanode্ɼdatanodeత选औ 规则ੋဓू܈தআݪblockॴଐ节఺֎ਵص选औҰ࿽ɻ发ૹ过ఔಉ 时ੜ੒blockతmetaจ݅ɻ Friday, December 21, 12
24. Parity File parityจ݅తम䐾处ཧ૬对简单ɿ 1. ࡏຊ࣍创ݐlost blockత临时จ݅ 2. 获औparityจ݅తݯจ݅ɼ௨过Encoderॏ৽encodeɼࡏຊ ஍ੜ੒parityจ݅తblock 3. 选औҰ࿽dnʢ选औ规则࿨har parityจ݅म䐾Ұகʣɼক block਺ਾ发ૹ౸该dn্ɼኂಉ时ੜ੒metaจ݅ Friday, December 21, 12
25. ਺ਾจ݅ ݯจ݅త伭䐾༩parityจ݅తम䐾૬൓ɼੋҰ࿽decode过ఔɿ 1. 对ဋfileத丢ࣦత㑌࿽block执ߦम䐾ૢ࡞ 2. ࡏຊ஍创ݐblockత临时จ݅ 3. ௨过Decoder伭䐾block਺ਾ 4. 选औҰ࿽target dnɼকblock਺ਾ发ૹ给target dnɼኂಉ时ੜ੒meta จ݅ɻ Friday, December 21, 12
26. Decoder Friday, December 21, 12
27. How to use http://wiki.aliyun-inc.com/projects/apsara/wiki/Yunti1/HDFS-Raid-deploy http://wiki.apache.org/hadoop/HDFS-RAID Friday, December 21, 12
28. Ұ࿽问题 • େྔจ݅ڊܕจ݅తraid Friday, December 21, 12
29. Resouces http://wiki.apache.org/hadoop/HDFS-RAID http://hadoopblog.blogspot.com/2009/08/hdfs-and-erasure-codes-hdfs-raid.html http://jiangbo.me/blog/2012/12/21/hdfs-raid/ Friday, December 21, 12
30. THX Friday, December 21, 12