HDFS

Transcript

1. HDFS ᇙതʢ镜圆ʣ Monday, October 22, 12
2. Outline • Overview • NameNode • DataNode • SecondaryNode • Client Monday, October 22, 12
3. Overview Monday, October 22, 12
4. NN Data Structure • Valid fsname -> block list (keep on disk) • Set of all valid blocks (inverted #1) • block -> machinelist (keep in memory, rebuilt from datanode blocksreport) • machine -> blocklist (inverted #2&#3) • LRU cache of updated-heartbeat machines Monday, October 22, 12
5. NN-FSDirectory Monday, October 22, 12
6. NN-FSDirectory • FSDirectory༻ဋ维护౰લܥ统தతจ݅树 • INodeDirectoryදࣔ树தతҰ࿽໨录 • INodeDiretoryWithQuotaੋINodeDirectoryత扩ల㜎ɼଈ带഑额తจ݅ ໨录 • INodeFileදࣔINode树தతҰ࿽จ݅ɼଖத༗࿽BlockInfo਺ਾɼ维护 该จ݅ॴ༗తBlock৴ଉ Monday, October 22, 12
7. NN-BlocksMap Monday, October 22, 12
8. NN-BlocksMap • BlocksMap༻ဋ维护Block -> { INode, datanodes, self ref } తөࣹ䎔 ܥ • BlockදࣔҰ࿽Blockతجຊ৴ଉ • BlockInfo扩లࢠBlockɼআجຊ৴ଉ֎还แׅ该blockతinodeҾ༻ɼ࿨ ॴଐతdatanode৴ଉ Monday, October 22, 12
9. NN-BlockInfo 1. DN1ɼDN2ɼDN3෼䫲දࣔଘ༗վblockతࡾ࿽datanodeతҾ༻ (DataNodeDescriptorʣ 2. DN1-prev-blkදࣔࡏDN1্blockྻදத౰લblockతલஔblockҾ༻ 3. DN1-next-blkදࣔࡏDN1্blockྻදத౰લblockత޳ஔblockҾ༻ Monday, October 22, 12
10. NN-FSImage • FSImage༻ဋදࣔจ݅ܥ统త镜૾ɼ༻ဋ࣋ٱԽจ݅ܥ统৴ଉ࿨启动时 伭䐾จ݅ܥ统结构 Monday, October 22, 12
11. Monday, October 22, 12
12. NN-FSEditLog • FSEditLog༻ဋܥ统启动޳อଘจ݅ܥ统্తमվ记录ɻFSEditLogத తૢ࡞记录ձपظੑ߹ኂ౸FSImageத Monday, October 22, 12
13. NN-ଖଞ结构 1.CorruptReplicasMapɿ௨过Ұ࿽TreeMap维护corruptঢ়态blockతblocks– >datanodedescriptor(s)өࣹɻ 2.recentInvalidateSetsɿ 维护࠷ࣦۙᏈతblockू߹ɼmapத为storageId- >ArrayList 3.datanodeMapɿ维护datanode->blockతөࣹ 4.neededReplicationsɿ௨过Ұ࿽优ઌ级队ྻདྷ维护౰લधཁ备㟨తblockू߹ 5.PendingReplicationBlocksɿ维护౰લਖ਼ࡏ备㟨తblockू߹ 6.overReplicatedBlocksɿ౰લधཁ检查ੋ൱备㟨过ଟతblockू߹ 7.excessReplicateMapɿ维护ܥ统தdatanode༩ଖ্త௒额备㟨blockతू߹ɼ这 ࠣ௒额త备㟨কඃ删আɻ Monday, October 22, 12
14. NN运ߦ໛ࣜ NameNodeଘࡏࡾ䝅运ߦ໛ࣜɿ 1. Normalɿ NameNodeਖ਼ৗ෰务తঢ়态 2. Safe modeɿNameNodeॏ启时进ೖSafe modeɼ该໛ࣜԼ੔࿽ܥ统ੋ୞读 తɼҎศဋNameNodeखصDataNode৴ଉ 3. Backup modeɿ备㟨NameNode处ဋBackup modeɼඃ动త઀Ꮕओ NameNodeత检查఺৴ଉ Monday, October 22, 12
15. NN启动 ࢀ⻅见ɿhttp://jiangbo.me/blog/2012/10/18/hdfs-namenode-startup/ Monday, October 22, 12
16. NN-Safe Mode NameNode进ೖSafeMode޳ձ৽ݐҰ࿽SafeModeMonitor线ఔདྷ检测ੋ൱ೳ 䭧㩂䇖SafeModeɻ 㩂䇖҆શ໛ࣜత标।༗䫆࿽ɿ 1.达౸࠷௿෭ຊ਺ཁٻతblock઎ॴ༗blockతൺྫ 2.进ೖ҆શ໛ࣜత时间ੋ൱达౸࠷௿ཁٻ ࢀ⻅见ɿhttp://jiangbo.me/blog/2012/10/18/hdfs-namenode-safe-mode/ Monday, October 22, 12
17. NN-线ఔ ࡏNameNodeதଘࡏ೗Լ႓䝅线ఔɿ 1. DataNode ݈߁检查؅ཧ线ఔ 2. ෭ຊ؅ཧ线ఔ 3. ે约؅ཧʢlease Managementʣ 4. IPC Handler 线ఔ Monday, October 22, 12
18. NN-Heartbeat • NameNodeத维护ྃҰ࿽datanode৺௓检测త਺组heartbeatsɼཬ໘维护 ྃ㑌࿽datanodeత࠷৽时间ፎɼdatanodeधཁपظੑ޲NameNode发ૹ ৺௓请ٻɼߋ৽࠷৽时间ፎɻ • NameNodeத HeartbeatMonitor线ఔपظߦ检查heartbeatsྻදதੋ൱༗ ௒时ະߋ৽తdatanodeɼ೗༗则认为该节఺ቮ经deadɼ删আ该节఺ɼኂ 删আ该节఺্ॴ༗తblock৴ଉɻ • ೗过༗༗Ꮘతblockɼক这ࠣ༗ᏈblockՃೖ౸धཁ备㟨తྻදதɼ进ߦ备 㟨ɻ • ೗Ռ删আblock导கू܈த达౸备㟨ܥ਺త༗ᏈblockతൺྫԼ߱౸进ೖ safemodeత阀值ɼকձ导கNameNode进ೖSafeModeɼ௚ࢸ备㟨਺满଍ ࠷௿ཁٻ ࢀ⻅见ɿhttp://jiangbo.me/blog/2012/10/18/hdfs-heartbeat/ Monday, October 22, 12
19. NN-෭ຊ؅ཧ • HDFSதత෭ຊ؅ཧ௨过FSNameSystem.javaதతReplicationMonitor线 ఔདྷ׬੒ɻ • 该线ఔपظੑว历 neededReplicationsྻදɼ҈র优ઌ级查ፙ࠷ߴ优ઌ 级ঘະ备㟨తblock • ࢖༻ replicator为该block选औҰ࿽ଘ์备㟨త໨标datanodeʢ实际త备 㟨ૢ࡞ੋࡏ该datanodeԼҰ࣍৺௓检测时༝NN௨஌ଖ҈੒ʣ • ক该blockՃೖ౸ਖ਼ࡏ备㟨తྻදதʢ pendingReplicationsʣɼኂဓ neededReplicationsҠআ • ಉ时该线ఔձ检测ਖ਼ࡏ备㟨తblockத௒时ঘະ׬੒备㟨తɼকଖဓ pendingReplicationsҠআɼॏ৽Ճೖ౸neededReplicationsத ࢀ⻅见ɿhttp://jiangbo.me/blog/2012/10/18/hdfs-namenode-replica-management/ Monday, October 22, 12
20. NN-෭ຊଘ์ࡦུ 1. ୈҰ࿽෭ຊଘ์ࡏ౰લdatanodeతຊ஍ 2. ୈೋ࿽෭ຊଘ์ࡏ༩ୈҰ࿽෭ຊॴࡏdatanodeෆࡏಉҰصՍ্తҰ࿽ datanode্ 3. ୈࡾ࿽෭ຊଘ์ࡏ༩ୈೋ࿽෭ຊಉҰصՍୠෆಉdatanode্ Monday, October 22, 12
21. NN-ે约؅ཧ • LeaseManagementੋHDFSதతҰ࿽ಉ㑊ص੍ɼ༻ဋอ证ಉҰ时ࠁ୞ ༗Ұ࿽client对Ұ࿽จ݅进ߦࣸ҃创ݐૢ࡞ɻ • HDFSதે约؅ཧ௨过LeaseManagerདྷ׬੒ɼओཁ职责แׅɿ • 1.client发ىcreateɼwrite౳ૢ࡞ੋ发์leaseɼ续约࿨ճᏅɻ • 2.௨过Ұ࿽޳୆线ఔճᏅ过ظతlease • Client୺௨过DFSClient.LeaseChecker 对 ঘࡏ࢖༻ୠેظ 过 ൒త lease 进 ߦ 续约 ࢀ⻅见ɿhttp://jiangbo.me/blog/2012/10/18/hdfs-namenode-lease-management/ Monday, October 22, 12
22. NN-ճᏅࢉ๏ 1) NameNode 查 ፙlease৴ଉ 2) 对 ဋleaseதత㑌࿽จ݅fɼྩb 为 fత࠷޳Ұ࿽blockɼ࡞೗Լૢ࡞ɿ 2.1) 获 औbॴࡏతdatanodeྻද 2.2) ྩଖதҰ࿽datanode࡞ 为 primary datanode p 2.3) p ဓNameNode 获 औ࠷৽త 时间 ፎ 2.4) p ဓ㑌࿽DataNode 获 औblock৴ଉ 2.5) p 计 ࢉ࠷খతblock 长 ౓ 2.6) p ༻࠷খతblock 长 ౓࿨࠷৽త 时间 ፎདྷߋ৽۩༗༗Ꮘ 时间 ፎతdatanode 2.7) p ௨஌NameNodeߋ৽ 结 Ռ 2.8) NameNodeߋ৽BlockInfo 2.9) NameNodeဓleaseத 删 আfɼ೗Ռࠑ 时该 leaseதॴ༗จ݅౎ቮඃ 删 আɼক 删 আ 该 lease 2.10) NameఏަमվతEditLog Monday, October 22, 12
23. Secondary NameNode • SecondaryNameNodeࡏHDFSதతओཁ࡞༻ੋ㢦ॿmaster NameNodeपظੑʢᘍ认5෼钟ʣ 执ߦcheckpointૢ࡞ɻ • ଖத༗䫆࿽ओཁతՄ഑ஔଐੑɿ 1. checkpointPeriod: 䫆࣍检查఺త间ִ时间ɼՄ௨过fs.checkpoint.period഑ஔ 2. checkpointSize: EditLogจ݅త࠷େ值ɼ౰EditLog௒过这࿽࠷େ值时ձ㖘੍೭ߦ checkpointɼՄ௨过fs.checkpoint.size഑ஔɼᘍ认ੋ64M Monday, October 22, 12
24. SDN-checkpoint Monday, October 22, 12
25. DN-਺ਾ结构 • HDFSதDataNodeओཁ 负责维护 block->stream bytesతөࣹ䎔ܥɼଈ 实际 block਺ਾతଘ 储 ɻ data/ ├── blocksBeingWritten ├── current │ ├── VERSION │ ├── blk_-1148021215131449924 │ ├── blk_-1148021215131449924_1001.meta │ ├── blk_-8598609183581346893 │ ├── blk_-8598609183581346893_1002.meta │ ├── blk_6693595845022390257 │ ├── blk_6693595845022390257_1003.meta │ └── dncp_block_verification.log.curr ├── detach ├── storage └── tmp Monday, October 22, 12
26. DN-FSDataSet Monday, October 22, 12
27. DN-FSDataSet • FSVolumne༻ဋ进ߦblockจ݅ॴଐతრ؅ཧɼ统计ଘ储໨录额࢖༻৘ႎ • FSVolumeSetੋFSVolumeతू߹ɼఏڙྃॴ༗༰ྔɼႫ༨ۭ间౳ํ๏ɻ • FSDataSetੋࡏFSVolumeSet೭্进ߦ෧૷实现FSDatasetInterfaceआޱɼ޲֎ఏڙ块查询࿨ૢ ࡞ํ๏ɻ • FSDir༻ဋ构ݐblock块ࡏdatanode࣓盘্త层࣍结构ɼᘍ认৘ႎԼ㑌࿽໨录Լ࠷ଟ64࿽ࢠ໨ 录ɼ࠷ଟೳଘ储64࿽块ɻ໨录ॳ࢝Խ时ձ递归扫ඳ໨录Լతॴ༗ࢠ໨录࿨จ݅ɼ构ݐҰ࿽树ܗ 结构ɻaddBlock时ɼटઌ尝试ࡏ౰લ໨录৽Ճ块ɼ೗Ռ౰લ໨录຅༗ۭ闲ۭ间ɼ则尝试ࡏࢠ໨ 录தఴՃɼ೗Ռ຅༗ࢠ໨录ɼ则৽ݐҰ࿽ࢠ໨录ɻ Monday, October 22, 12
28. NN&DN-DN஫ၽ • DataNodeࡏ启动时ձ޲NameNode஫ၽɼఏަ೗Լ৴ଉɿ • nameɿصث໊ʢओص໊+෰务୺ޱ߸ʣ • infoPort: ঢ়态৴ଉ෰务୺ޱ޷ • ipcPortɿ ఏڙipc෰务త୺ޱ߸ • NameNode为DataNode෼഑Ұ࿽storageId Monday, October 22, 12
29. Monday, October 22, 12
30. NN&DN-৺௓检测 Monday, October 22, 12
31. NN&DN-blockReport Monday, October 22, 12
32. DN-offerservice Monday, October 22, 12
33. Client-୅码结构 Monday, October 22, 12
34. Client-DFSClient Monday, October 22, 12
35. Client-DFSClient 1. LeaseCheckerओཁ༻ဋlease检查࿨续约ɻ 2. DFSOutputStream༻ဋఏڙ带bufferతࣈ节ྲྀࣸೖޭೳɻclientࡏࣸೖ਺ਾ时ઌক਺ਾ缓ଘࡏ ຊ஍ɻኂক਺ਾ੾෼੒ଟ࿽packetʢᘍ认㑌࿽packet为64Kʣɻ㑌࿽packetຢඃ፥෼੒ଟ࿽ chunkʢᘍ认512Byteʣɼ㑌࿽chunk౎༗Ұ࿽checksumɻclientࣸ满Ұ࿽packet޳ձক该 packetՃೖ౸Ұ࿽dataqueueதɻ༝DataStreamer线ఔ负责ক㑌࿽packet发ૹ给datanode pipelineɻ发ૹ׬Ұ࿽pakcetɼstreamerձকଖဓdataqueueҠࢸackqueueதɻ ResponseProcessor负责઀Ꮕdatanode发ճతack৴ଉɼ㑌੒ޭ઀ᏅҰ࿽packetతack৴ ଉɼResponseProcessorձকackqueueத该packet删আɻ 3. DFSInputStream༻ဋఏڙࣈ节ྲྀత读औɼଖ಺෦෧૷ྃ༩NN࿨DNతަޓ 4. DataStreamer: 负责޲datanode pipeline发ૹpacketɻଖຊ਎ੋҰ࿽Daemon线ఔɼဓ namenode获औblockId࿨blockଘ์Ґஔɼকpacket发ૹ给pipelineதతdatanodeɼ㑌࿽ packet౎༗Ұ࿽seqIdɼ㑌࿽packet发ૹ׬时౎ձᏅ౸datanodeతack৴ଉɻ౰Ꮕ౸ॴ༗ packetతack৴ଉ޳ʢදࣔ该blockቮ发ૹ׬ʣɼstreamer䎔闭该blockɻ 5. ResponseProcessor:༻ဋ઀Ꮕdatanodeฦճack৴ଉɼኂক㠳应ackqueueதతpacket删আ Monday, October 22, 12
36. ࣸೖྲྀఔ Monday, October 22, 12
37. 读औྲྀఔ Monday, October 22, 12
38. DataXceiverServer • DataNodeࡏ 启动时 ձ௨ 过 DataXceiverServer䇖 启 Ұ࿽Socket୺ޱɼ 负责 block਺ਾత 读 ࣸɻ DataXceiverServerຊ਎࡞ 为 Ұ࿽क 护线 ఔɼ 监 ჶdfs.datanode.address഑ஔత਺ਾ 读 ࣸ෰ 务 ୺ ޱɻ౰༗ 请 ٻདྷ 时 ɼ৽ݐҰ࿽DataXceiver 线 ఔ 处 ཧ 请 ٻɻ • DataXceiver 线 ఔ༻ဋ 处 ཧҰ࿽ 读 /ࣸ਺ਾྲྀ 请 ٻɼଖrunํ๏ೖԼओཁੋࠜਾ 请 ٻதෆಉత 请 ٻ㜎ܕɼ 调 ༻㠳 应 త 处 ཧํ๏ Monday, October 22, 12
39. readBlock() readBlock()ओཁဓdisk读औblock਺ਾɼ构ݐҰ࿽DataOutputStream਺ਾྲྀɼኂ৽ݐҰ࿽ BlockSenderক这࿽਺ਾྲྀ发ૹग़ڈʢdatanode҃ऀclientʣɻ BlockSender.sendBlock()发ૹతBlockతྲྀఔେମ೗Լɿ 1. 读औblockతmeta৴ଉɼ获ಘchecksumኂ发ૹ 2. 发ૹ਺ਾ读औతภҠྔ 3. কblock਺ਾ੾෼为packetɼ发ૹ给client 4. ॴ༗packet发ૹ׬೭޳ɼ䎔闭checksumจ݅࿨blockจ݅ Monday, October 22, 12
40. writeBlock() 1.BlockReceiverဓ্҈packetҰ节఺读औ਺ਾɼࣸೖ౸ຊ஍disk 2.೗༗ԼҰ备㟨节఺ɼক该packet转发给ԼҰ节఺ 3.ক该packetՃೖ౸ackqueue队ྻத౳଴ackফଉ 4.౰ԼҰ节఺׬੒该packetࣸೖ޳ձฦճ该packet对应తack৴ଉ 5.PakcetResponder઀Ꮕ౸ack৴ଉ޳ɼকackqueueத该packet删আɼኂ޲લஔ节఺发 ૹack৴ଉ Monday, October 22, 12
41. • 详细㩘记ࢀ⻅见ɿhttp://jiangbo.me/blog/ categories/hdfs/ Monday, October 22, 12
42. THX Monday, October 22, 12