An Introduction to Google Percolator

An Introduction to Google Percolator, what is it and what is it used
for ? How is it different from Apache Hadoop MapReduce ?

Transcript

1. Google Percolator • What is it ? • What is it used for ? • Percolator Vs MapReduce • Architecture www.semtech-solutions.co.nz info@semtech-solutions.co.nz
2. Percolator – What is it ? • Incremental updates to Big Data • Developed by Google • Based on Google File System ( GFS ) • Provides transactions and locking • Faster than comparable Map Reduce • Developed by Google due to MapReduce limitations www.semtech-solutions.co.nz info@semtech-solutions.co.nz
3. Percolator – What is it used for ? • Iterative updates • No need to batch process • Update as data received • Data in multi petabyte range • Strong consistency needed • Improved latency ( 100 x ) • Reduced document age ( 50 % ) • Random access to big data repository www.semtech-solutions.co.nz info@semtech-solutions.co.nz
4. Percolator Vs MapReduce Map Reduce • Batch Processing • No transactions • Latency A • Run time scales with data • Code in C++ • Open source • Uses HDFS Percolator • Iterative • Transactions • Latency 100 x A • Incremental updates • Code in Java ( mainly ) • Google owned • Uses GFS www.semtech-solutions.co.nz info@semtech-solutions.co.nz
5. Percolator – Architecture • Applications are a sequence of observers • An observer is called via a notification • A notification is triggered when table data changes • Application calls TabletServer via RPC • TabletServer calls GFS ChunkServer
6. Percolator – Architecture • Applications – Series of observers • Observer – Completes task – Updates table • Next Observer called – Via notification • Percolator worker – Scans for changes – Sends notifications
7. Percolator – Architecture Actual worker diagram including time stamping and locking via Chubby lock server
8. Contact Us • Feel free to contact us at – www.semtech-solutions.co.nz – info@semtech-solutions.co.nz • We offer IT project consultancy • We are happy to hear about your problems • You can just pay for those hours that you need • To solve your problems