JavaOne 2013 Presentation: Next Generation Hadoop: It's Not Just Batch!

Next Generation Hadoop: It’s Not Just Batch! Alex Holmes @grep_alex
Credit: Johan Swanepoel

Agenda • Why Hadoop • Problems • What's new •
Blueprints

Who am I? • Alex Holmes • Software engineer •
Working with Hadoop since 2008 • @grep_alex / grepalex.com

Why Hadoop

never throw out your data tweets likes upload video photographs
web pages deposits/ withdrawals reviews clickstream - How are users responding to your search results? web server logs - Who's hacking into your site? - What are legitimate users doing? application logs - What's the state of your system? - Any errors being generated right now? system logs - CPU/disk/network utilization

store join index analytics aggregate graph processing A Data Mountain

Hadoop GFS (distributed storage) MapReduce (batch distributed compute) GFS HDFS
(distributed storage) Map Map Map Reduce Reduce Reduce Outputs Shufﬂe Inputs Inputs Inputs Outputs Outputs Page Rank Trending Searches Indexing Analytics Pig / Hive (MapReduce DSL's)

MapReduce HDFS Pig Hive Cascading Crunch Sqoop RHIPE Flume HBase
Mahout Solr Cascalog Hue Ambari WebHDFS Oozie Azkaban Impala Splunk Blur

Use Cases • ETL • Complex - large data volumes,
data adapters for disparate data, N-way joins, coordination • Hadoop provides a scalable architecture and rich ecosystem • Data Warehousing • System of record • HDFS is a scalable, fault-tolerant, tried and tested storage solution • Batch ingress/egress • Research/BI

Problems Credit: Alan Holden

Credit: REUTERS/Mian Khursheed Hadoop v1 > 4K nodes

Credit: Alex Holmes Rigid slot design

MapReduce == batch (and it's the only thing you can
run)

Data is silo'd

What's new

YARN (resource management) HDFS (storage) Hadoop 1 Hadoop 2 Big
Data Kernel MapReduce Tez Giraph Storm ... MapReduce (resource management and computational processing) HDFS (storage)

Resource Manager (resource arbitrater) Client (submits work) submit application 1
ask for containers: - priority - hostname - resources - number of containers 3 Node Manager (container management) Containers create containers 4 5 application-specific communication Application Master (framework-speciﬁc resource negotiator) a new application master is created for each application 2

One cluster, distributed storage, distributed scheduler, many types of applications.

Blueprints • NoSQL with HBase • Stream processing with Storm
• Graph processing with Giraph • SQL-on-Hadoop with Impala • Columnar Data Formats

Blueprint 1: NoSQL with HBase

HBase • Based on Google BigTable • Low-latency, persistent store
• Distributed, sorted, multi-dimensional map • Massively parallel reads and writes

SLAVE NODE HRegionServer Region Region Region HFiles HFiles HFiles HBase
architecture MapReduce Mapper Mapper Mapper MapReduce Reducer Reducer HFiles HFiles HDFS Client Container YARN Node Manager Hoya Application Master YARN Resource Manager

column family: raw 1 <!doctype ... com.twitter:/posts/bgates 1 <html><bod... <!DOCTYPE
... com.example.www:/index.html com.example.blog:/a/b/c html dirty column family: meta application/ xhtml+xml text/html application/ octet-stream 200 301 202 status code rowkey content type Rows are lexicographically sorted text/xml application/ xhtml+xml Cells can have multiple "Versions" Column families are stored in different files Data model

Indexing in MapReduce MapReduce (batch distributed compute) Internet Crawler (download
web content) Indexing Inverted index HDFS (distributed storage) Data is usually buffered and written by an intermediary Latency involved in running MapReduce jobs. Crawl data and metadata only available in HDFS

HBase (real-time column store) Internet Crawler (download web content) Inverted
index MapReduce (batch distributed compute) coprocessors Analytics Dashboard Joins MapReduce Indexing using HBase

see also ... • Accumulo • NSA-developed secure BigTable derivative
• Eventually open-sourced as an Apache project • Offers HDFS storage, cell-level security • ElephantDB • Developed by Nathan Marz to support his Lambda Architecture • A simple read-only KV store that sources data from HDFS • Only 3K lines of code

Use cases • Capturing system metrics • User-interaction data (messages,
impressions, clickstream, ...) • Facebook selected HBase for messaging and user analytics over Cassandra • Content serving • Google uses BigTable for GMail, analytics, personalized search

Blueprint 2: Stream processing with Storm

Stream processing • Data enters our systems in real-time, and
we want to make real-time decisions (e.g. data aggregations, anomaly detection) • Approach used to be custom MQ's and workers, but complex and hard to maintain • Stream processing offers simple programming interfaces, with the framework taking care of fault tolerance and reliability

Storm • Most mature of the systems available • Created
by Twitter, maintained by Nathan Marz • Used at Twitter, Yahoo!, slew of others • https://github.com/nathanmarz/storm/wiki/Powered-By

HBase Search App I saw a pussy cat I did,
I did! Kafka search search Storm search Tokenize the search strings and emit a stream of words. <spout> search Maintain a sliding window of words and their occurrences. word word <bolt> 1 sliding window 43 0 51 3 63 cat 72 18 52 821 2 91 dog <bolt> dog top N 43 cat 18 ... 43 Keep a top N list of word frequencies. bolt bolt Trending Search

Storm-on-YARN • Yahoo uses Storm for ad targeting, fraud detection,
trending topics • Wanted to integrate it into existing YARN infrastructure, and use HDFS as a data source and sink • Added 2 features missing in Storm • Auto-scaling for load balancing • Security to allow Storm to access secure HDFS data

see also ... • Samza • Developed by LinkedIn •
Leverages Kafka for reliable messaging • Morphlines • A library for streaming ETL • Flume, HBase and MapReduce implementations • Can use SolrCloud as a data sink

Use cases • Trending topics (e.g. Google Zeitgeist, Twitter trending
hashtags) • Analytics aggregations (e.g. Ad providers) • Image processing (e.g. panorama image generation in Google Street View)

Blueprint 3: Graph Processing with Giraph

PageRank www www www www www www Map Map Reduce
Reduce Reduce write barrier Map Map Reduce Reduce Reduce write barrier write barrier write barrier

Giraph • Inspired by Google's Pregel, a graph processing architecture
• Based on the Bulk Synchronous Parallel model of distributed computing • Runs on MapReduce v1, and YARN

MapReduce Giraph Map Map Reduce Reduce Reduce write barrier Map
Map Reduce Reduce Reduce write barrier write barrier write barrier Map Map sync barrier

PageRank algorithm public void compute(Iterable<DoubleWritable> messages) { double pageRank; if
(getSuperstep() == 0) { pageRank = 1.0 / getTotalNumVertices(); } else { double rankFromNeighbors = MathUtils.sum(messages); double dampingFactor = ((1.0 - DAMPING_FACTOR) / (double) getTotalNumVertices()); pageRank = dampingFactor + (DAMPING_FACTOR * rankFromNeighbors); } setValue(pageRank); for (Edge<LongWritable, FloatWritable> edge : getEdges()) { sendMessage(edge.getTargetVertexId(), new DoubleWritable(pageRank / getNumEdges())); } } www www www www www www

Use Cases • Analyze user social graphs (popularity, personalized rankings,
shared connections, shortest path) • Web graphs - PageRank and variants • Networking/transportation (shortest path)

Blueprint 4: SQL-on-Hadoop with Impala

MapReduce (batch distributed compute) Analytics HDFS (distributed storage) Pig /
Hive (MapReduce DSL's)

Impala • Cloudera’s implementation of Google Dremel • Interactive SQL
on HDFS and HBase • Written in C++; up to 100x faster than Hive

Impala impalad HDFS / HBase Query Executor Query Coordinator Query
Planner impalad HDFS / HBase Query Executor Query Coordinator Query Planner impalad HDFS / HBase Query Executor Query Coordinator Query Planner Client 1 2 3 3 4 4 Submit query 1 Push plan fragments 2 MPP distributed 3 Stream intermediary results 4 5 Stream results 5

JavaOne 2013 Presentation: Next Generation Hadoop: It's Not Just Batch!

JavaOne 2013 Presentation: Next Generation Hadoop: It's Not Just Batch!

Other Decks in Technology

Featured

Transcript