Kafka - A Little Introduction

Kafka - A Little Introduction

This is a presentation I gave on Kafka (http://incubator.apache.org/kafka/)- a persistent, distributed messaging system developed at LinkedIn.

I gave it during one of our monthly tech presentation evenings in our office.

3aa708adb3ecf15173f7e6f2f9eefc6c?s=128

Paul Ingles

May 01, 2012
Tweet

Transcript

  1. Kafka A little introduction

  2. None
  3. Pub-Sub Messaging System

  4. None
  5. None
  6. None
  7. None
  8. Distributed

  9. None
  10. Performance

  11. Disk/Memory Performance Source: http://queue.acm.org/detail.cfm?id=1563874 Disk SSD Memory 1 10 100

    1,000 10,000 100,000 1M 10M 100M 1000M Random access Sequential Access Read values/second
  12. Disk/Memory Performance Source: http://queue.acm.org/detail.cfm?id=1563874 Disk SSD Memory 1 10 100

    1,000 10,000 100,000 1M 10M 100M 1000M Random access Sequential Access Read values/second
  13. Disk/Memory Performance Source: http://queue.acm.org/detail.cfm?id=1563874 Disk SSD Memory 1 10 100

    1,000 10,000 100,000 1M 10M 100M 1000M Random access Sequential Access Read values/second
  14. Disk/Memory Performance Source: http://queue.acm.org/detail.cfm?id=1563874 Disk SSD Memory 1 10 100

    1,000 10,000 100,000 1M 10M 100M 1000M Random access Sequential Access Read values/second Sequential disk read faster than random memory read
  15. Persistent

  16. None
  17. None
  18. None
  19. Length Magic Value Checksum Payload 4 bytes 1 byte 4

    bytes n bytes
  20. None
  21. None
  22. None
  23. Token Offset: 0 Broker: kafka.local Topic: Testing Input MR Job

    Output Offset: 130098 Broker: kafka.local Topic: Testing Output Sequence File
  24. Token Offset: 0 Broker: kafka.local Topic: Testing Input MR Job

    Output Offset: 130098 Broker: kafka.local Topic: Testing Output Sequence File
  25. None
  26. Useful Things • http://incubator.apache.org/kafka/ • https://github.com/pingles/clj-kafka