$30 off During Our Annual Pro Sale. View Details »

Kafka - A Little Introduction

Kafka - A Little Introduction

This is a presentation I gave on Kafka (http://incubator.apache.org/kafka/)- a persistent, distributed messaging system developed at LinkedIn.

I gave it during one of our monthly tech presentation evenings in our office.

Paul Ingles

May 01, 2012
Tweet

More Decks by Paul Ingles

Other Decks in Technology

Transcript

  1. Kafka A little introduction

  2. None
  3. Pub-Sub Messaging System

  4. None
  5. None
  6. None
  7. None
  8. Distributed

  9. None
  10. Performance

  11. Disk/Memory Performance Source: http://queue.acm.org/detail.cfm?id=1563874 Disk SSD Memory 1 10 100

    1,000 10,000 100,000 1M 10M 100M 1000M Random access Sequential Access Read values/second
  12. Disk/Memory Performance Source: http://queue.acm.org/detail.cfm?id=1563874 Disk SSD Memory 1 10 100

    1,000 10,000 100,000 1M 10M 100M 1000M Random access Sequential Access Read values/second
  13. Disk/Memory Performance Source: http://queue.acm.org/detail.cfm?id=1563874 Disk SSD Memory 1 10 100

    1,000 10,000 100,000 1M 10M 100M 1000M Random access Sequential Access Read values/second
  14. Disk/Memory Performance Source: http://queue.acm.org/detail.cfm?id=1563874 Disk SSD Memory 1 10 100

    1,000 10,000 100,000 1M 10M 100M 1000M Random access Sequential Access Read values/second Sequential disk read faster than random memory read
  15. Persistent

  16. None
  17. None
  18. None
  19. Length Magic Value Checksum Payload 4 bytes 1 byte 4

    bytes n bytes
  20. None
  21. None
  22. None
  23. Token Offset: 0 Broker: kafka.local Topic: Testing Input MR Job

    Output Offset: 130098 Broker: kafka.local Topic: Testing Output Sequence File
  24. Token Offset: 0 Broker: kafka.local Topic: Testing Input MR Job

    Output Offset: 130098 Broker: kafka.local Topic: Testing Output Sequence File
  25. None
  26. Useful Things • http://incubator.apache.org/kafka/ • https://github.com/pingles/clj-kafka