Kafka - A Little Introduction

This is a presentation I gave on Kafka (http://incubator.apache.org/kafka/)- a persistent, distributed messaging system developed at LinkedIn.

I gave it during one of our monthly tech presentation evenings in our office.

Transcript

1. Kafka
   A little introduction
   

   View Slide

2. View Slide

3. Pub-Sub Messaging System
   

   View Slide

4. View Slide

5. View Slide

6. View Slide

7. View Slide

8. Distributed
   

   View Slide

9. View Slide

10. Performance
    

    View Slide

11. Disk/Memory Performance
    Source: http://queue.acm.org/detail.cfm?id=1563874
    Disk SSD Memory
    1
    10
    100
    1,000
    10,000
    100,000
    1M
    10M
    100M
    1000M
    Random access
    Sequential Access
    Read values/second
    

    View Slide

12. Disk/Memory Performance
    Source: http://queue.acm.org/detail.cfm?id=1563874
    Disk SSD Memory
    1
    10
    100
    1,000
    10,000
    100,000
    1M
    10M
    100M
    1000M
    Random access
    Sequential Access
    Read values/second
    

    View Slide

13. Disk/Memory Performance
    Source: http://queue.acm.org/detail.cfm?id=1563874
    Disk SSD Memory
    1
    10
    100
    1,000
    10,000
    100,000
    1M
    10M
    100M
    1000M
    Random access
    Sequential Access
    Read values/second
    

    View Slide

14. Disk/Memory Performance
    Source: http://queue.acm.org/detail.cfm?id=1563874
    Disk SSD Memory
    1
    10
    100
    1,000
    10,000
    100,000
    1M
    10M
    100M
    1000M
    Random access
    Sequential Access
    Read values/second
    Sequential disk read
    faster than random
    memory read
    

    View Slide

15. Persistent
    

    View Slide

16. View Slide

17. View Slide

18. View Slide

19. Length Magic Value Checksum Payload
    4 bytes 1 byte 4 bytes n bytes
    

    View Slide

20. View Slide

21. View Slide

22. View Slide

23. Token
    Offset: 0
    Broker: kafka.local
    Topic: Testing
    Input
    MR Job
    Output
    Offset: 130098
    Broker: kafka.local
    Topic: Testing
    Output
    Sequence File
    

    View Slide

24. Token
    Offset: 0
    Broker: kafka.local
    Topic: Testing
    Input
    MR Job
    Output
    Offset: 130098
    Broker: kafka.local
    Topic: Testing
    Output
    Sequence File
    

    View Slide

25. View Slide

26. Useful Things
    • http://incubator.apache.org/kafka/
    • https://github.com/pingles/clj-kafka
    

    View Slide