Upgrade to Pro — share decks privately, control downloads, hide ads and more …

JavaOne 2013 Presentation: Next Generation Hado...

Alex Holmes
September 25, 2013
Technology
6
1k

JavaOne 2013 Presentation: Next Generation Hadoop: It's Not Just Batch!

Hadoop is rapidly becoming the kernel for distributed computing. Hadoop is known as the de facto tool for anything related to batch processing, but the little-known secret is that in recent years, it has become much more than batch. The introduction of tools and technologies such as HBase, Impala, and the next-generation MapReduce architecture have brought real-time capabilities to Hadoop, and it now offers a complete ecosystem that can be used to address any challenge. This presentation explores several real-world big data problems and identifies key parts of Hadoop that can be used to solve these challenges. It also examines how real-time and batch processing can be fused to play to their strengths.

Alex Holmes

September 25, 2013
Tweet

Other Decks in Technology

See All in Technology
BLADE: An Attempt to Automate Penetration Testing Using Autonomous AI Agents
bbrbbq
0
300
SREによる隣接領域への越境とその先の信頼性
shonansurvivors
2
520
【若手エンジニア応援LT会】ソフトウェアを学んできた私がインフラエンジニアを目指した理由
kazushi_ohata
0
150
AIチャットボット開発への生成AI活用
ryomrt
0
170
スクラム成熟度セルフチェックツールを作って得た学びとその活用法
coincheck_recruit
1
140
【Pycon mini 東海 2024】Google Colaboratoryで試すVLM
kazuhitotakahashi
2
500
Terraform CI/CD パイプラインにおける AWS CodeCommit の代替手段
hiyanger
1
240
Security-JAWS【第35回】勉強会クラウドにおけるマルウェアやコンテンツ改ざんへの対策
4su_para
0
180
適材適所の技術選定 〜GraphQL・REST API・tRPC〜 / Optimal Technology Selection
kakehashi
1
170
Lexical Analysis
shigashiyama
1
150
Incident Response Practices: Waroom's Features and Future Challenges
rrreeeyyy
0
160
Lambdaと地方とコミュニティ
miu_crescent
2
370

Featured

See All Featured
Rebuilding a faster, lazier Slack
samanthasiow
79
8.7k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
28
2k
Documentation Writing (for coders)
carmenintech
65
4.4k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
93
16k
Building a Scalable Design System with Sketch
lauravandoore
459
33k
VelocityConf: Rendering Performance Case Studies
addyosmani
325
24k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
47
2.1k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
191
16k
The Language of Interfaces
destraynor
154
24k
A Philosophy of Restraint
colly
203
16k
YesSQL, Process and Tooling at Scale
rocio
169
14k
BBQ
matthewcrist
85
9.3k

Transcript

  1. Next Generation Hadoop: It’s Not Just Batch! Alex Holmes @grep_alex

    Credit: Johan Swanepoel

  2. Agenda • Why Hadoop • Problems • What's new •

    Blueprints

  3. Who am I? • Alex Holmes • Software engineer •

    Working with Hadoop since 2008 • @grep_alex / grepalex.com

  4. Why Hadoop

  5. never throw out your data tweets likes upload video photographs

    web pages deposits/ withdrawals reviews clickstream - How are users responding to your search results? web server logs - Who's hacking into your site? - What are legitimate users doing? application logs - What's the state of your system? - Any errors being generated right now? system logs - CPU/disk/network utilization

  6. store join index analytics aggregate graph processing A Data Mountain

  7. Hadoop GFS (distributed storage) MapReduce (batch distributed compute) GFS HDFS

    (distributed storage) Map Map Map Reduce Reduce Reduce Outputs Shuffle Inputs Inputs Inputs Outputs Outputs Page Rank Trending Searches Indexing Analytics Pig / Hive (MapReduce DSL's)

  8. MapReduce HDFS Pig Hive Cascading Crunch Sqoop RHIPE Flume HBase

    Mahout Solr Cascalog Hue Ambari WebHDFS Oozie Azkaban Impala Splunk Blur

  9. Use Cases • ETL • Complex - large data volumes,

    data adapters for disparate data, N-way joins, coordination • Hadoop provides a scalable architecture and rich ecosystem • Data Warehousing • System of record • HDFS is a scalable, fault-tolerant, tried and tested storage solution • Batch ingress/egress • Research/BI

  10. Problems Credit: Alan Holden

  11. Credit: REUTERS/Mian Khursheed Hadoop v1 > 4K nodes

  12. Credit: Alex Holmes Rigid slot design

  13. MapReduce == batch (and it's the only thing you can

    run)

  14. Data is silo'd

  15. What's new

  16. YARN (resource management) HDFS (storage) Hadoop 1 Hadoop 2 Big

    Data Kernel MapReduce Tez Giraph Storm ... MapReduce (resource management and computational processing) HDFS (storage)

  17. Resource Manager (resource arbitrater) Client (submits work) submit application 1

    ask for containers: - priority - hostname - resources - number of containers 3 Node Manager (container management) Containers create containers 4 5 application-specific communication Application Master (framework-specific resource negotiator) a new application master is created for each application 2

  18. One cluster, distributed storage, distributed scheduler, many types of applications.

  19. Blueprints • NoSQL with HBase • Stream processing with Storm

    • Graph processing with Giraph • SQL-on-Hadoop with Impala • Columnar Data Formats

  20. Blueprint 1: NoSQL with HBase

  21. HBase • Based on Google BigTable • Low-latency, persistent store

    • Distributed, sorted, multi-dimensional map • Massively parallel reads and writes

  22. SLAVE NODE HRegionServer Region Region Region HFiles HFiles HFiles HBase

    architecture MapReduce Mapper Mapper Mapper MapReduce Reducer Reducer HFiles HFiles HDFS Client Container YARN Node Manager Hoya Application Master YARN Resource Manager

  23. column family: raw 1 <!doctype ... com.twitter:/posts/bgates 1 <html><bod... <!DOCTYPE

    ... com.example.www:/index.html com.example.blog:/a/b/c html dirty column family: meta application/ xhtml+xml text/html application/ octet-stream 200 301 202 status code rowkey content type Rows are lexicographically sorted text/xml application/ xhtml+xml Cells can have multiple "Versions" Column families are stored in different files Data model

  24. Indexing in MapReduce MapReduce (batch distributed compute) Internet Crawler (download

    web content) Indexing Inverted index HDFS (distributed storage) Data is usually buffered and written by an intermediary Latency involved in running MapReduce jobs. Crawl data and metadata only available in HDFS

  25. HBase (real-time column store) Internet Crawler (download web content) Inverted

    index MapReduce (batch distributed compute) coprocessors Analytics Dashboard Joins MapReduce Indexing using HBase

  26. see also ... • Accumulo • NSA-developed secure BigTable derivative

    • Eventually open-sourced as an Apache project • Offers HDFS storage, cell-level security • ElephantDB • Developed by Nathan Marz to support his Lambda Architecture • A simple read-only KV store that sources data from HDFS • Only 3K lines of code

  27. Use cases • Capturing system metrics • User-interaction data (messages,

    impressions, clickstream, ...) • Facebook selected HBase for messaging and user analytics over Cassandra • Content serving • Google uses BigTable for GMail, analytics, personalized search

  28. Blueprint 2: Stream processing with Storm

  29. Stream processing • Data enters our systems in real-time, and

    we want to make real-time decisions (e.g. data aggregations, anomaly detection) • Approach used to be custom MQ's and workers, but complex and hard to maintain • Stream processing offers simple programming interfaces, with the framework taking care of fault tolerance and reliability

  30. Storm • Most mature of the systems available • Created

    by Twitter, maintained by Nathan Marz • Used at Twitter, Yahoo!, slew of others • https://github.com/nathanmarz/storm/wiki/Powered-By

  31. HBase Search App I saw a pussy cat I did,

    I did! Kafka search search Storm search Tokenize the search strings and emit a stream of words. <spout> search Maintain a sliding window of words and their occurrences. word word <bolt> 1 sliding window 43 0 51 3 63 cat 72 18 52 821 2 91 dog <bolt> dog top N 43 cat 18 ... 43 Keep a top N list of word frequencies. bolt bolt Trending Search

  32. Storm-on-YARN • Yahoo uses Storm for ad targeting, fraud detection,

    trending topics • Wanted to integrate it into existing YARN infrastructure, and use HDFS as a data source and sink • Added 2 features missing in Storm • Auto-scaling for load balancing • Security to allow Storm to access secure HDFS data

  33. see also ... • Samza • Developed by LinkedIn •

    Leverages Kafka for reliable messaging • Morphlines • A library for streaming ETL • Flume, HBase and MapReduce implementations • Can use SolrCloud as a data sink

  34. Use cases • Trending topics (e.g. Google Zeitgeist, Twitter trending

    hashtags) • Analytics aggregations (e.g. Ad providers) • Image processing (e.g. panorama image generation in Google Street View)

  35. Blueprint 3: Graph Processing with Giraph

  36. PageRank www www www www www www Map Map Reduce

    Reduce Reduce write barrier Map Map Reduce Reduce Reduce write barrier write barrier write barrier

  37. Giraph • Inspired by Google's Pregel, a graph processing architecture

    • Based on the Bulk Synchronous Parallel model of distributed computing • Runs on MapReduce v1, and YARN

  38. MapReduce Giraph Map Map Reduce Reduce Reduce write barrier Map

    Map Reduce Reduce Reduce write barrier write barrier write barrier Map Map sync barrier

  39. PageRank algorithm public void compute(Iterable<DoubleWritable> messages) { double pageRank; if

    (getSuperstep() == 0) { pageRank = 1.0 / getTotalNumVertices(); } else { double rankFromNeighbors = MathUtils.sum(messages); double dampingFactor = ((1.0 - DAMPING_FACTOR) / (double) getTotalNumVertices()); pageRank = dampingFactor + (DAMPING_FACTOR * rankFromNeighbors); } setValue(pageRank); for (Edge<LongWritable, FloatWritable> edge : getEdges()) { sendMessage(edge.getTargetVertexId(), new DoubleWritable(pageRank / getNumEdges())); } } www www www www www www

  40. Use Cases • Analyze user social graphs (popularity, personalized rankings,

    shared connections, shortest path) • Web graphs - PageRank and variants • Networking/transportation (shortest path)

  41. Blueprint 4: SQL-on-Hadoop with Impala

  42. MapReduce (batch distributed compute) Analytics HDFS (distributed storage) Pig /

    Hive (MapReduce DSL's)

  43. Impala • Cloudera’s implementation of Google Dremel • Interactive SQL

    on HDFS and HBase • Written in C++; up to 100x faster than Hive

  44. Impala impalad HDFS / HBase Query Executor Query Coordinator Query

    Planner impalad HDFS / HBase Query Executor Query Coordinator Query Planner impalad HDFS / HBase Query Executor Query Coordinator Query Planner Client 1 2 3 3 4 4 Submit query 1 Push plan fragments 2 MPP distributed 3 Stream intermediary results 4 5 Stream results 5

  45. see also ... Apache Drill (incubating)

  46. Use Cases • Data science research • Exposing data to

    organization • Find the next killer feature in your data!

  47. Blueprint 5: Columnar Data Formats

  48. Parquet and ORC File • Both support repeating nested data

    • And bit packing • And run length encoding • And dictionary encoding • And compression • And are extensible aaaaaabbbccccc 6a3b5c 00000000 00000000 00000000 00000001 00000000 00000000 00000000 00000010 00000000 00000000 00000000 00000011 00000000 00000000 00000000 00000100 00000000 00000000 00000000 00000101 00000000 00000000 00000000 00000110 00000000 00000000 00000000 00000111 00000101 00111001 01110111 00000000 00000000 00000000 00000000

  49. Credit: Martin SoulStealer http://en.wikipedia.org/wiki/File:Mexican_Standoff.jpg Mexican Standoff between Parquet and ORC

    File • Compatible Systems - ORC supports Hive, Parquet MR, Impala, Hive, Pig - winner Parquet • Query optimization - ORC has stripe metadata for min/max/sum/count for each column - winner ORC • Parquet is on version 1.0, already deployed in production @ Twitter with 30% space savings - winner Parquet

  50. Summary

  51. Conclusion • MapReduce still relevant for ETL, data warehousing, bulk

    ingest/egress • With Hadoop 2 we have a Big Data Kernel, where multiple systems can use HDFS as common storage, and YARN for resource management • Fusion of batch + real-time workloads in the same cluster

  52. What else? • Hadoop 2.2 (GA) expected in a few

    weeks • More apps coming soon on YARN (e.g. Tez) • Mesos • Lambda Architecture + Summingbird • Spark, Spark Streaming, Shark, GraphX, ...

  53. grepalex.com for slides and further reading Credit: rhardholt