Kafka Introduction

Transcript

1. APACHE KAFKA OVERVIEW BY SUKASIT LIANG PORNRATTANA
2. QUEUE - fifty itself :* : ' 4$
3. AESSAGEQUEVE.ly / ' ' / l l I I I
4. WHY ? - SOURCE SOURCE SOURCE SOURCE SOURCE SYSTEM SYSTEM SYSTEM SYSTEM SYSTEM Tara ! . . SYSTEM SYSTEM SYSTEM SYSTEM SYSTE - . - . T - - . - USER PAYMENT NEWS I I y I - DISTRIBUTED - I - REPLICATION KAFKA ACTION I , ACTION 2 , ACTIONS # I - FAULT TOLE RENT I ANALYTICS EMAIL DATABASE - FAST SYSTEM (
5. ECOSYSTEM - KAFKA ✓ " " " " " \ KAFKA STREAMS § 987¥ En - PRODUCER ILIFFE ? ! consumer - I IT feet I -200 KEEPER
6. KAFKA us RABBIT MQ - KALIKA ( THE Lod ) RABBI-TMQCTH.li QUEUE ) - COMMIT LOG - FIRST IN , FIRST OUT - WRITE - READ - [ READER [ WRITER - APPEND - ONLY , ORDER BY TIME - STORED DATA IS TRANSIENT IN NATURE . STORED DATA IS PERSISTENT - NOT SHARED BETWEEN IN NATURE C RETAINED BY TIME ) APPLICATIONS - SHARED BETWEEN APPLICATIONS , - EXCHANGE ? # I - App i 11--1111 - APP 2
7. APACHE KAFKA
8. Topics - 1¥11 . - KEY C OPTIONAL ) - TIMESTAMP - OFFSET - PARTITION - VALUE
9. PARTIT.CN#i @ GUARANTEED ORDER TOPIC - ONLY WITHIN PARTITION go OFFSET e - I WEEK RETENTION PARTITION ¢ TI4 / PARTITION i µfI' ¥7 .IT?TyTEe , PARTITION 2 - hi PA : ¥970 ¥ PARTITION L OFFSET O OFFSET ONLY REFERS TO A SPECIFIC PARTITION
10. BROKERS - - BOOTSTRAP SERVERS - I I BROKER I . - DNA BROKER 273Mt BROKER 3 - - - TOPIC I TOPIC I TOPLCL PARTITION 0 PARTITION L PARTITION 2 PIE : fond I path : :{ Toni ftp.wmm-ihftottnoaf.sn.AEoiee.se . %) - A BROKER HAS OWN ID - INTERNAL IMPLEMENTATION - 3 BROKERS AT LEAST - NOT DOCUMENTED
11. REPLICATION FACTOR - A- 1. CREATES 2 COPIES OF PARTITIONS REPLICATION FACTOR = 21 2. ELECTED A LEADER PARTI TONG AND PARTITION =L By ZOOKEEPER BROKER 't FACE BROKERED * BROKER 3 * TOPIC I ftp.ITOPLCI TOPIC I * PARTITION Of T PARTITION 01 PARTITION I CA -1205£ IN - SYNC TOPIC I REPLL PARTITION I IF A SERVER ⑥ TOLERATE Up TO N - 1 IS DOWN ? Ne REPLICATION FACTOR
12. IFA BROKER IS DOWN ? - BROKER 2 ~ BROKER 2 :& ' Topper AT TOPIC ' n ¥AEij# ' PARTITION L PARTITION 't BROKERS a ⇐i±mIE¥tom BROKER 2 Diii * BROKER 's TOPIC I TOPIC I in 2 . 3 REPLICAS PARTITION Of ⑤ PARTITION 't 9 AT LEAST TOPIC I f' ! PARTITION I ¥ X
13. PRODUCERS - tacks ACK 's = O } IIT / PRODUCER - BROKER ✓ ✓ i ✓ ACKS . 1 SOURCE PRODUCER KAFKA / PRODUCER → BROKER - LEADER ✓ SYSTEMS - ACK 's . all MESSAdEK~# P¥TeapER# key : cost key : cus . a PARTITION of REPLICA PARTITION I key : cus -2 -
14. CONSUMERS if MAX i NOE PARTITIONS - PARTITION 4 E' E ' El - Ant - El - i is JW PARTITION I He EI ✓ # y KAFKA CONSUMER g-Y-grff.tn ; PARTITION 2 ④ → 2,45 - no key : cus -1 key : cus -1 key : cus -2 ] t I €4 ¥4 # key : cus -2 key : cus -3
15. CONSUMER GROUPS # GROUPS BROKEN # ④ El El patron " nor " : ;; : :/ ↳ PARTITION '' 7%97,31! ④ #fAR" Tt ONZ GROUP 't ④ ④ REBALANCE tYY¥8"w Groups ' OFFSET
16. ZOOKEEPER You ,sHAkNoT Pass ! ! - STILL AWAKE ? O \ - HB⇒E3 EEEo I IT S 'T , NENG O HEY , BROKER'd ! ! BROKERL JUST DIED YOU ARE THE LEADER ! \ I ex TO # " * " *
17. MESSAGE DELIVERY SEMANTICS - AT MOST ONCE - MESSAGES MAY BE LOST BUT ARE NEVER DELIVERED FAST / MAY BE LOST AT LEAST ONCE - MESSAGES ARE NEVER LOST BUT MAY BE REDELIVERED SAF E/MAY BE DUPLICATED EXACTLY ONCE - EACH MESSAGE IS DELIVERED ONCE AND ONLY ONCE VERY DIFFICULT TO ACHIEVE ) NEED STRONG ENGINEERING
18. QSA