大規模本番環境における HDFS Erasure Coding の運用について / Using HDFS Erasure Coding for Growth

大規模本番環境における HDFS Erasure Coding の運用について / Using HDFS Erasure Coding for Growth

内田早俊 (LINE)
LINE Developer Meetup #68 - Big Data Platformでの登壇資料です
https://line.connpass.com/event/188176/

53850955f15249a1a9dc49df6113e400?s=128

LINE Developers

September 17, 2020
Tweet

Transcript

  1. USING ERASURE CODING FOR GROWTH Toshihiko Uchida Site Reliability Engineer

    Data Platform Data Science and Engineering Center
  2. • Name • Toshihiko Uchida • Role • Site Reliability

    Engineer • Data Infrastructure Team / Data Platform / Data Science and Engineering Center • History • Working at LINE since 02/2018 • Hadoop ecosystems, Kafka and Flink • Interest • Distributed systems • Program verification theory • Dependent type systems • Model checking • Facebook: https://www.facebook.com/toshihiko.uchida.7 PROFILE
  3. • LINE Data Platform • Unifying Hadoop Clusters • Erasure

    Coding • Erasure Coding Reconstruction Agenda
  4. LINE DATA PLATFORM

  5. • Provide the data platform as a service to internal

    users • Storage • Computation • Query • Pipeline • Governance • Self-service portal MISSION Make the company data-driven
  6. PRODUCTS HDFS HBase Elasticsearch YARN Kubernetes Flink Spark Hive Presto

    Ranger Yanagishima OASIS Jupyter Tableau LINE Analytics Aquarium RStudio Portal Portal Storage Computation Tool & API Kafka Data governance Pilot
  7. https://github.com/yanagishima/yanagishima YANAGISHIMA

  8. https://engineering.linecorp.com/en/blog/data-system-opens-its-doors-to-all-liners/ OASIS

  9. SCALE SERVER:
 2865 EV CPU:
 81200 VCORES RAM:
 602.5 TB

    STORAGE:
 240 PB INCOMING RECORDS:
 600 B/day
 12 M/s (peak) STORAGE USED:
 169 PB WORKLOAD:
 100K+/DAY TABLES:
 17800+ ENGINEERS:
 41
  10. • 10+ Hadoop cluster at 2018 • Multitenancy • Resource

    isolation • Data governance • Cost governance • Efficient utilization of the limited hardware resources • Increasing data size • Jobs requesting excessive resources • Reliability of large-scale distributed systems • … CHALLENGE
  11. UNIFYING HADOOP CLUSTERS

  12. • Merge small clusters into big clusters • Connect big

    clusters by HDFS Federation + ViewFs • LINE DEV DAY 2019 • 100+PB scale Unified Hadoop cluster Federation with 2k+ nodes UNIFICATION Small HDP 2.6 Small Small HDP 2.6 Small Apache Hadoop 3 HDFS Federation + ViewFs
  13. • A new server room for Hadoop • Spine-leaf architecture

    • 25Gbps • Moving existing servers to the room • LINE Engineering Blog • μ΢ϯλΠϜͳ͠ͰHadoopΫϥελΛҠߦͨ࣌͠ͷ࿩ DATA CENTER
  14. ERASURE CODING

  15. • HDFS Erasure Coding (EC) is a technique to store

    data efficiently, keeping similar fault-tolerance with 3x replication. ERASURE CODING d1 d2 d3 d4 d5 d6 p1 p2 p3 d7 … … d1 d2 d3 d4 d5 d6 d7 … storage block logical block encoding cell
  16. REPLICATION VS EC Replication (3x) EC (RS-6-3) Durability 2 3

    Storage efficiency 33% 67% Recovery Replication Reconstruction CPU-bound Network-bound Locality optimization Possible Impossible Write performance Disk-bound Parallel CPU-bound Network-bound Read performace Disk-bound Parallel Network-bound Small files problem Severe Very severe • Cold data • 10PB used
  17. ERASURE CODING RECONSTRUCTION How it can be throttle

  18. • EC Reconstruction is a mechanism to recover lost blocks

    EC RECONSTRUCTION DN1 DN2 DN3 DN4 DN5 DN6 DN7 DN8 DN9 d1 d2 d3 d5 d6 p3 d4 p1 p2 DN12 DN13 DN11
  19. • Network congestion between data centers NETWORK CONGESTION New server

    room Old server room DN1 DN2 DN3 DN4 DN5 DN6 DN7 DN8 DN9 d1 d2 d3 d5 d6 p3 d4 p1 p2 DN12 DN13 DN11
  20. EC METRICS EcReconstructionRemoteBytesRead EcReconstructionTasks

  21. StripedBlockReconstruction DatanodeDescriptor HOW IT WORKS NamenodeRPCServer BPServiceActor DatanodeManager BPOfferService ErasureCodingWorker

    StripedBlockReconstruction DatanodeDescriptor RedundancyMonitor 1. computeDatanodeWork 2. EC reconstruction tasks 1’. sendHeartBeat( xmitsInProgress) 2’. handleHeartBeat( maxTransfer) 3’. EC reconstruction tasks BlockECReconstructionCommands 5’. Update xmitsInProgress 4’. submit
  22. where • repl = # of replication tasks • recon

    = # of reconstruction tasks • weight = dfs.datanode.ec.reconstruction.xmits.weight (= 0.5 by default) Definition XMITSINPROGRESS xmitsInProgress = repl + Σrecon i=1 max(max(sources, targets),1) * weight Meaning For each Datanode, xmitsInProgress represents the weighted number of running replication/reconstruction tasks
  23. • Examples • 0 + max(max(6, 2), 1) * 0.5

    + max(max(6, 1), 1) * 0.5 = 6 Example XMITSINPROGRESS xmitsInProgress = repl + Σrecon i=1 max(max(sources, targets),1) * weight
  24. where • maxSreams = dfs.namenode.replication.max-streams (= 2 by default) Definition

    MAXTRANSFER maxTransfer = maxStreams − xmitsInProgress Meaning For any Namenode and Datanode, maxTransfer represents the number of replication/reconstruction tasks can be sent from Namenode to Datanode
  25. • Examples • 2 - 3 = -1 -> No

    replication/reconstruction task • 2 - 0 = 2 -> Two replication/reconstruction tasks • 1 - 0 = 1 -> One replication/reconstruction tasks Example MAXTRANSFER maxTransfer = maxStreams − xmitsInProgress Note Namenode does not take dfs.datanode.ec.reconstruction.xmits.weight into consideration Let’s set maxStreams=1!
  26. • xmitsInProgress went negative • maxTransfer became too large •

    Remember maxTransfer = maxStreams - xmitsInProgress • Known, but open issue in the community • HDFS-14353: Erasure Coding: metrics xmitsInProgress become to negative. • Fixed the unit test xmitsInProgress become to negative OPEN ISSUE StripedBlockReconstruction BPServiceActor BPOfferService ErasureCodingWorker StripedBlockReconstruction The weight was not taken into consideration Update xmitsInProgress
  27. Component Default New dfs.namenode.replication.max-streams NN, REPL, EC 2 1 dfs.namenode.replication.work.multiplier.per.iteration

    NN, REPL, EC 2 1 dfs.namenode.redundancy.interval.seconds NN, REPL, EC 3s 6s dfs.datanode.ec.reconstruction.threads DN, EC 8 2 dfs.datanode.ec.reconstruction.xmits.weight DN, EC 0.5 0.5 How to throttle CONFIGURATIONS
  28. StripedBlockReconstruction DatanodeDescriptor MAPPING NamenodeRPCServer BPServiceActor DatanodeManager BPOfferService ErasureCodingWorker StripedBlockReconstruction DatanodeDescriptor

    RedundancyMonitor redundancy.interval.seconds replication.work.multiplier.per.iteration replication.max-streams ec.reconstruction.threads ec.reconstruction.xmits.weight
  29. Before and after RESULT Decreased!

  30. • Isolate nameservices for EC from the others • Automate

    archiving cold data by EC • E.g., based on the last access time of files • Use Archival Storage for EC • Develop I/O based throttling of EC reconstruction tasks • HDFS-11023 FUTURE WORK
  31. • Blog • Cloudera: HDFS Erasure Coding in Production •

    Yahoo!: HDFS Erasure Codingͷ঺հͱYahoo! JAPANʹ͓͚Δӡ༻ࣄྫ • Book • Architecting Modern Data Platforms • Designing Data-Intensive Applications REFERENCE
  32. WE’RE HIRING! • Software Engineer • Site Reliability Engineer

  33. THANK YOU