An Introduction to Apache Flume

An Introduction to Apache Flume, what is it used for and
how does it work ? How does it fit into the Hadoop tool
set ?

Transcript

1. Apache Flume • What is it ? • How does it work ? • Architecture • Reliability www.semtech-solutions.co.nz info@semtech-solutions.co.nz
2. Flume – What is it ? • A data collection service for Hadoop • For distributed systems • Open source • Scaleable • Reliable • Manageable • Fault tolerant www.semtech-solutions.co.nz info@semtech-solutions.co.nz
3. Flume – How does it work ? • Flumes uses agents which have – A source • Listen for events • Write events to channel – A channel • Queue event data as transactions – A sink • Write event data to target i.e. HDFS • Remove event from queue www.semtech-solutions.co.nz info@semtech-solutions.co.nz
4. Flume – Architecture • A single agent showing its parts • Generally one agent for a given data type www.semtech-solutions.co.nz info@semtech-solutions.co.nz
5. Flume – Architecture • Agents can be chained into flows • Avro can be used for data serialization www.semtech-solutions.co.nz info@semtech-solutions.co.nz
6. Flume – Architecture In complicated flows it may be necessary to think about • Event Data Reliability • Should we have – Complete end to end reliability – Send and forget – Or something in between ? www.semtech-solutions.co.nz info@semtech-solutions.co.nz
7. Flume – Architecture • Complex flows may have many links www.semtech-solutions.co.nz info@semtech-solutions.co.nz
8. Contact Us • Feel free to contact us at – www.semtech-solutions.co.nz – info@semtech-solutions.co.nz • We offer IT project consultancy • We are happy to hear about your problems • You can just pay for those hours that you need • To solve your problems