• Joined Yahoo! JAPAN as a new graduate in FY2023 • Working on large-scale Hadoop platform operations • Involved in cross-cluster interoperability after the merger
LINE Corporation and Yahoo Japan Corporation ◦ Services: messaging, search, media, e-commerce, etc… • Independent large-scale Hadoop platforms on both sides ◦ Combined capacity of main HDFS clusters exceeded 1EB ◦ Multi-region, multi-tenant compliance constraints • Full integration was not immediately feasible ◦ Major differences in account models, ACL management, locations, etc • Initial goal: safe and practical interoperability
Separate permission management prevented consistent governance • Users and operators needed to understand and follow two different rule sets Challenge 2: Data Exchange • Operational overhead from using intermediate systems • Security and governance constraints
suited for governance operations ◦ Review, inventory, and audit • We wanted to introduce the ex-LINE model to the ex-Yahoo! JAPAN side Perspective ex-LINE ex-Yahoo! JAPAN Access Control model Role-based (Apache Ranger) POSIX like permissions (User / Group) Governance unit DB / Table HDFS Path Operational Trade-off Better for governance and review Flexible, but harder to govern at scale
Key Decision Factors: ◦ Avoid mixing different management units ◦ Minimizing user impact ex-LINE Permission Management System ex-Yahoo! JAPAN Hadoop Existing Area (HDFS Path Based) Existing data staged migration ex-LINE Hadoop New Managed Area (Table Based) Existing Area (Table Based) New / Migrated data Existing data
API and limited to eligible targets • Ranger: Limited to the new sub HDFS cluster ex-LINE Permission Management System Group update(via API) LDAP location lookup Policy update Ranger Hive Metastore DB Enforce policies Existing storage (sub hdfs clusters) New Managed storage (sub hdfs cluster) ex-Yahoo! JAPAN Hadoop
copy with DistCP ◦ Easy to adopt in the existing Hadoop environment ◦ Enables efficient transfer of large-scale data ex-LINE Hadoop ex-Yahoo! JAPAN Hadoop submit job via DistCP queue DistCP Client RM assign job NM metadata NN data DN (Dedicated) metadata NN data DN
platform kept its existing Kerberos KDC ◦ Configuration is required to enable DistCP execution • Control of Execution Identity and Transfer Path ◦ Minimize network impact from cross-DC communication ◦ Control data exchange in an auditable manner due to policy constraints
→ ex-Yahoo! JAPAN • auth_to_local to map Source principals to Destination local accounts ex-LINE Realm ex-Yahoo! JAPAN Realm Trust KDC KDC 3. Cross-Realm TGT 1. Req for Dest NN Service Ticket 2. Cross-Realm TGT 4. Service Ticket(NN) NN DistCP Client 5. Access auth_to_local
Queue • Only dedicated NodeManagers can • Placement Control via NodeLabel reach the destination cluster YARN Approved Account RM Dedicated NodeManagers DistCP Queue Non-approved Account General Account General Queue General NodeManagers
◦ Safe and sustainable operation matters • Do not force different assumptions into one model ◦ A staged approach reduced impact on existing systems • Data transfer must be controllable end to end ◦ Authentication, execution accounts, execution nodes, and network paths
• NameNode stores all the metadata • NameNode handles all the metadata requests # of requests become 3x in 5 years • Used by Spark and Trino • Scale-out is MUST Number of HDFS requests per month (ex-Yahoo! JAPAN)
• ViewFS-based Federation • Mount table is configured in client-side • Production in ex-LINE • Router-based Federation (RBF) • • • • DFSRouter Mount table is in server-side Requests are routed by DFSRouter Available from Hadoop 3.3.0 Production in ex-Yahoo! JAPAN Active NameNodes
requests • Observer NameNodes work as a read-replica, handles only read requests • Observer NameNodes and RBF can be used at the same time since Hadoop 3.4.0 • Production in ex-Yahoo! JAPAN since 2023 Active NameNode writes … reads DFSRouter Observer NameNodes
consistency • Possible to read old information = stale reads Active NameNode 3. Read the file before transfer → Fail! 1. Write a file Observer NameNode 2. Write EditLog EditLog is transferred (async) JournalNodes
guarantee the information is up-to-date 4. Return the response after processing the EditLog corresponding to the state ID 1. msync Active NameNode 2. Return state ID 3. Read with the state ID Observer NameNode EditLog (+state ID) is transferred (async) JournalNodes
active NameNode, does not scale • Need to reduce the frequency Spark Driver, Trino Coordinator Always call msync (auto msync period = 0ms) Spark Executor, Trino Worker Call msync if it’s not been called for > 1 sec (auto msync period = 1000ms) • Result ◦ # of msync become 1/10 (35k → 3k/sec)
ID which will lead to inconsistent reads • We faced ~1% stale reads even after the msync call • Fixed RPC client and merged into upstream • HDFS-17362: Implement RouterObserverReadConfiguredFailoverProxyProvider • To support auto msync period via DFSRouter
newly added DNs (new NW limitation) • Enabled RPS (Receive Packet Steering) to distribute NW packet processing to multi cores • Enabled RFS (Receive Flow Steering) to optimize cache locality • softirq is distributed to CPU cores, teragen become 3x faster
true interoperability • HDFS Fine-grained locking (HDFS-17366) • 7x potential performance gain • Phase 1 is available from Hadoop 3.5.0 • Consider using object storages
lycorptech_jp
layerx
keio_smilab
yokawasa
matsu7874
kaen
classmethod_kaz
rindrics
onenashev
.jpg){kind=avatar}
hiro_torii
news_it_enj
muehara
mfonobong
codingconduct
keithpitt
rakko
caitiem20
aleyda
schroneko
ammeep
jct
jlugia
konakalab
morganepeng
