100+PB scale Unified Hadoop cluster Federation with 2k+ nodes

Tianyi Wang
LINE Data Platform Department Engineer
https://linedevday.linecorp.com/jp/2019/sessions/D1-5

Transcript

1. 2019 DevDay 100+PB Scale Unified Hadoop Cluster Federation With 2k+ Nodes > Tianyi Wang > LINE Data Platform Department Engineer
2. Agenda > Introduction of Data Platform > Status, Problems of Data Infrastructure > Our Solution > Next Steps
3. Introduction of Data Platform > What do we do? > Provide data infrastructure > Manage the whole life cycle of data > Services that we provide > Logging SDK, Ingestion Pipeline, Query Engine, BI Tools > Mission > Provide a governed, self-service data platform > Make Data Driven easy
4. Data Infrastructure Ingestion Pipeline Computational Resource Storage BI Tools Machine Learning Security & Management Tools Data Center & Verda
5. Heterogeneous Data Different Domains, Data BI Tools Machine Learning YARN HDFS Catalog ETL Monitoring Security . . .
6. The Good Status Yea Number of Nodes 2000+ Daily Jobs/Queries 100000+ Yea Size of Data 100PB+ Yea
7. Heterogeneous Data Different Pipeline, Use Case Sqoop Database Native log Front log Server log General log Tracking Service Web Tracking Service Datachain Kafka Kafka HDFS HDFS HDFS
8. The Bad Status Nay Number of Clusters 10+ Number of Tables 17800+ Nay Number of ETLs 1000+ Nay
9. Disconnected Unscalable Tooling Infrastructure Storage CPU/RAM Operation Data Batch Data Data Driven Is Not Easy
10. Disconnected Unscalable Tooling Infrastructure Storage CPU/RAM Operation Data Batch Data Make Data Driven Easy
11. Goals One Standard, One Interface, One Cluster > One interface > Determinate API, tooling > No more configuration mess > One cluster > Hadoop 3 is preferred > Reduce operation burden > One standard > Best practice of managing the lifecycle of data
12. Pay Technical Debt != Throw Them Away > High Security Level > Minimum Risk > No Compulsive Schedule > Incremental Migration Migration à la Carte No Compromise > Cost-Effectiveness > Upgrade Hadoop in Place > Minimum Breaking Changes > Minimum Downtime Minimum User Effects Lean Criteria
13. Solution A
14. We Could … Create a new cluster Move everything there - Simplest but most troublesome - Long transition period - Need to double the nodes Merge into the biggest cluster - // Not fully secured - Major version upgrade remains - i ԫ New in B i ԫ New
15. Create & Move - Simplest but most troublesome - Long migration period - Need to double the nodes i ԫ New Create a new cluster Move everything there
16. Merge Into the Biggest - The biggest cluster is not secured by Kerberos - We have two big clusters that users use heavily - Still needs to upgrade Hadoop after merging Merge into the biggest cluster - // Not fully secured - Major version upgrade remains - i ԫ New in B
17. Can We Do Better?
18. With VIEWFS HDFS Federation <property> <name>fs.defaultFS</name> <value>viewfs://line-cluster</value> </property> <configuration> <property> <name>fs.viewfs.mounttable.line-cluster.link./apps</name> <value>hdfs://ns1/apps</value> </property> <property> <name>fs.viewfs.mounttable.line-cluster.link./user</name> <value>hdfs://ns2/user</value> </property> <property> <name>fs.viewfs.mounttable.line-cluster.link./logs</name> <value>hdfs://ns3/logs</value> </property> <property> <name>fs.viewfs.mounttable.line-cluster.link./tmp</name> <value>hdfs://ns3/tmp</value> </property> </configuration> Block Pool Block Pool Block Pool Nameservice Nameservice Nameservice Datanode Datanode Datanode Datanode Datanode VIEWFS
19. Ideal Solution Existing Hadoop Clusters
20. Ideal Solution Existing Hadoop Clusters Important ones < 100 600 1200
21. Ideal Solution < 100 600 1200 Existing Hadoop Clusters Important ones Unified Cluster Merge into Connect
22. Roadmap Merge Small Clusters Connect HDFS Using HDFS Federation Merge YARN Sync DDL And Move Data HDFS YARN Metastore Cleanup
23. Roadmap Merge Small Clusters Connect HDFS Using HDFS Federation Merge YARN Sync DDL And Move Data HDFS YARN Metastore Cleanup
24. Connect Multiple Hadoop Namenode Namenode Resourcemanager Resourcemanager HDP 2.6 HDP 2.6 + Kerberos Datanode
 Nodemanager Datanode
 Nodemanager 1200 600 1200 600
25. Connect Multiple Hadoop Namenode Namenode Resourcemanager Resourcemanager HDP 2.6 HDP 2.6 + Kerberos Datanode
 Nodemanager Datanode
 Nodemanager Apache Hadoop 3.1 + Kerberos Namenode Resourcemanager Datanode
 Nodemanager 1200 600 300+n+m 600 1200 300+n+m
26. Connect Multiple Hadoop Namenode Namenode Resourcemanager Resourcemanager HDP 2.6 HDP 2.6 + Kerberos Datanode
 Nodemanager Datanode
 Nodemanager Apache Hadoop 3.1 + Kerberos Namenode Resourcemanager Datanode
 Nodemanager 1. Decommission 2. Install 3.1 3. Service in 1200 600 300+n+m 600-m 1200-n 300+n+m
27. Connect Multiple Hadoop Namenode Namenode HDP 2.6 HDP 2.6 + Kerberos Datanode
 Nodemanager Datanode
 Nodemanager Apache Hadoop 3.1 + Kerberos Namenode Resourcemanager Datanode
 Nodemanager 300+1200+600 600 Resourcemanager 600-600 1200 Resourcemanager 1200-1200 300+1200+600
28. Connect Multiple Hadoop Apache Hadoop 3.1 + Kerberos Namenode Resourcemanager Datanode
 Nodemanager Namenode HDP 2.6 1200 Namenode HDP 2.6 + Kerberos 600 2100 2100
29. Sounds Great, but
30. Prerequisites of HDFS Federation > Same Security level > Same KDC(realm) > Same RPC version > Same Cluster ID(CID) > Namenode/Datanode > Journalnode @REALM.A KERBEROS @REALM.B KERBEROS
31. Patch It! Make HDFS Federation Possible Add StorageType to RPC Kerberos realm
32. Multiple HDFS Shares a Single YARN HDP 2.6 HDP 2.6 + Kerberos Apache Hadoop 3.1 + Kerberos Namenode Namenode Namenode Resourcemanager Hiveserver2 Hiveserver2 Hiveserver2 Spark Spark Spark Hive Metastore Hive Metastore Hive Metastore
33. Roadmap Merge Small Clusters Connect HDFS Using HDFS Federation Merge YARN Sync DDL And Move Data HDFS YARN Metastore Cleanup
34. Multiple Clients Shares a Single YARN HDP 2.6 HDP 2.6 + Kerberos Apache Hadoop 3.1 + Kerberos Namenode Namenode Namenode Resourcemanager Hiveserver2 Hiveserver2 Hiveserver2 Spark Spark Spark Hive Metastore Hive Metastore Hive Metastore
35. Connect YARN YARN Old Client NS 2 HDP 2.6 NS 3 Get delegation token Submit application <configuration> <property> <name>fs.viewfs.mounttable.line-cluster.link./user</name> <value>hdfs://ns2/user</value> </property> <property> <name>fs.viewfs.mounttable.line-cluster.link./logs</name> <value>hdfs://ns3/logs</value> </property> </configuration> <property> <name>fs.defaultFS</name> <value>viewfs://line-cluster</value> </property> <property> <name>fs.defaultFS</name> <value>hdfs://hdp2.6</value> </property> spark.kerberos.access.namenodes=ns1,ns2,ns3 mapreduce.job.hdfs-servers=ns1,ns2,ns3 tez.job.fs-servers=ns1,ns2,ns3
36. Roadmap Merge Small Clusters Connect HDFS Using HDFS Federation Merge YARN Sync DDL And Move Data HDFS YARN Metastore Cleanup
37. Unify Hive Metastore > Self-service ETL tools to synchronize DDL and data automatically > Use Hive hook to collect changes and apply to the new cluster > Use Presto to analyze data across different Metastores
38. For the Components That Didn’t Support Hadoop 3.x Other Problems > Fixed by our patches > Fix an order of field numbers for HeartbeatResponseProto > Flink cannot truncate file that have ViewFS path > Allow WebHDFS accesses from insecure NameNodes to secure DataNode > Webhdfs backward compatibility (HDFS-14466 reported by LINER) > Support Hive metastore impersonation(Presto#1441 by LINER) > File Merge tasks fail when containers are reused (HIVE-22373 by LINER) > Backporting 10+ community patches
39. What We Have Achieved > Provide an easy way for users to migrate > Backward-compatibility > Flexible migration schedule > Build the next generation data platform based on Hadoop 3 > Using Erasure Coding > Using Docker on YARN > Build a unified cluster based on old clusters > Upgrade Hadoop at the same time > Storage, computational resources merged
40. We Are Hiring!
41. Thank You