Using Apache Kafka with Golang - Moath Qasim, Ligatus

Using Apache Kafka with Golang - Moath Qasim, Ligatus

Transcript

1. Using Apache Kafka with Golang A quicklook at sarama, sarama-cluster and a bit more Moath Qasim (@mqmazz) Software Engineer at Ligatus GmbH
2. Agenda - Apache Kafka Quick Overview - Shopify/Sarama Kafka Client - BSM/Sarama-Cluster, Cluster Extension for Sarama - Shopify/Sarama Group Consumer - Q/A Round
3. Apache Kafka Apache Kafka, is a distributed streaming platform, which: - Publish and subscribe to streams of records - Store streams of records - Process streams of records as they occur. Important concepts about Apache Kafka: - Kafka is run as a cluster on one or more servers - Each record consists of a key, a value, and a timestamp. - The Kafka cluster stores streams of records in categories called topics.
4. Apache Kafka Apache Kafka Core APIs: - Producer API - Consumer API App App - Streams API Process Kafka Cluster Kafka Cluster Kafka Cluster - Connector API Kafka Cluster DB DB
5. Apache Kafka Topics and Partition Logs 0 4 5 3 1 2 Partition 0 Partition 1 0 4 3 1 2 Writes Old New 0 4 5 3 1 2 6 Producers And Consumers Producer Consumer
6. Apache Kafka Kafka Cluster Server 1 Server 2 Partition 0 Partition 1 C1 C2 C3 C1 C2 Consumer Group A Consumer Group B
7. Apache Kafka Kafka Use Cases App App App DB Messaging Monitoring App App App Stream Processing App App App
8. Sarama Package sarama is a pure Go client library for dealing with Apache Kafka (versions 0.8 and later). The package includes: - High-level API for easily producing and consuming messages. - Low-level API for controlling bytes on the wire when the high-level API is insufficient. - Sync and Async messages producers. - Kafka messages consumer.
9. Sarama Example config := sarama.NewConfig() config.Producer.RequiredAcks = sarama.WaitForAll config.Producer.Return.Successes = true producer, err := sarama.NewSyncProducer([]string{"localhost:9092"}, config) if err != nil { log.Fatalln(err) } defer func() { if err := producer.Close(); err != nil { log.Fatalln(err) } }() type SyncProducer interface { SendMessage(msg *ProducerMessage) (partition int32, offset int64, err error) SendMessages(msgs []*ProducerMessage) error Close() error } import "github.com/Shopify/sarama"
10. Sarama event := &sarama.ProducerMessage{Topic: "test_topic", Value: sarama.StringEncoder("testing data")} type Event struct { Name string `json:"name"` Type string `json:”type"` } e := &Event{ Name: "Awesome_Berlin", Type: "go_days", } data, err := json.Marshal(e) if err != nil { panic(err) } JSON Protocol Buffers message Event { required string name = 1; required string type = 2; } e := pb.Event { Name: "Awesome_Berlin", Type: "go_days", } data, err := proto.Marshal(e) if err != nil { panic(err) } partition, offset, err := prod.SendMessage(&sarama.ProducerMessage{ Topic: "test", Key: sarama.StringEncoder("test_key"), Value: sarama.ByteEncoder(data), }) if err != nil { log.Printf("FAILED to send message: %s\n", err) } else { log.Printf("> message sent to partition %d at offset %d\n", partition, offset) }
11. Sarama type AsyncProducer interface { AsyncClose() Close() error Input() chan<- *ProducerMessage Successes() <-chan *ProducerMessage Errors() <-chan *ProducerError } Example config := sarama.NewConfig() config.Producer.RequiredAcks = sarama.WaitForLocal producer, err := sarama.NewAsyncProducer([]string{"localhost:9092"}, config) if err != nil { log.Fatalln(err) } defer func() { if err := producer.Close(); err != nil { log.Fatalln(err) } }()
12. Sarama go func() { for err := range producer.Errors() { log.Print(err) } }() type ProducerError struct { Msg *ProducerMessage Err error } producer.Input() <- &sarama.ProducerMessage{ Topic: "test", Key: sarama.StringEncoder("test_key"), Value: sarama.StringEncoder("Awesome-Berlin"), } select { case s.Producer.Input() <- &sarama.ProducerMessage{ Topic: "test", Key: sarama.StringEncoder("test_key"), Value: sarama.StringEncoder("Awesome_Berlin"), }: case err := <-s.Producer.Errors(): log.Println("Failed to produce message", err) // or handle the failed message differently case success := <-s.Producer.Successes(): log.Println("succeeded to produce message", success) }
13. Sarama-Cluster Package cluster provides cluster extensions for Sarama, enabling users to consume topics across from multiple, balanced nodes. The package includes: - High-level API for consuming messages from multiple topics through a multiplexed channel. - High-level API for consuming messages from multiple topics through individual partitions. - Extends sarama.Config with Group specific namespace(e.g: creating consumers with topic whitelists.
14. Sarama-Cluster Consumer Mode: ConsumerModeMultiplexed config := cluster.NewConfig() config.Consumer.Return.Errors = true brokers := []string{"127.0.0.1:9092"} topics := []string{"test_topic_1", "test_topic_2"} consumer, err := cluster.NewConsumer(brokers, "multiplexed-consumer-group", topics, config) if err != nil { panic(err) } defer consumer.Close() go func() { for err := range consumer.Errors() { log.Printf("error: %s\n", err.Error()) } }() for { select { case msg, ok := <-consumer.Messages(): if ok { fmt.Printf(os.Stdout, "%s\t%s", msg.Key, msg.Value) consumer.MarkOffset(msg, “consumer-current-state“) // mark message as processed } } }
15. Sarama-Cluster Consumer Mode: ConsumerModePartitions config := cluster.NewConfig() config.Group.Mode = cluster.ConsumerModePartitions brokers := []string{"127.0.0.1:9092"} topics := []string{"test"} consumer, err := cluster.NewConsumer(brokers, "my-consumer-group", topics, config) if err != nil { return nil, err } defer consumer.Close() for partition := range pc.consumer.Partitions() { go func() { for msg := range partition.Messages() { fmt.Printf("%s\n", msg.Value) } }() }
16. Sarama-Cluster Handling Notifications and Errors config.Group.Return.Notifications = true type Notification struct { Type NotificationType Claimed map[string][]int32 Released map[string][]int32 Current map[string][]int32 } Notification Types: RebalanceStart RebalanceOK RebalanceError config.Consumer.Return.Errors = true func (pc *partitionedConsumer) run() { for partition := range pc.consumer.Partitions() { go func() { for not := range consumer.consumer.Notifications() { fmt.Printf("cluster status: %v", not) } }() go func() { for err := range partition.Errors() { fmt.Printf("%s\n", err) } }() } }
17. Sarama Consumer Group type ConsumerGroup interface { Consume(ctx context.Context, topics []string, handler ConsumerGroupHandler) error Errors() <-chan error Close() error } client, _ := sarama.NewClient([]string{"localhost:9092"}, config) group, _ := sarama.NewConsumerGroupFromClient(“testing_group”, client) go func() { for err := range group.Errors() { fmt.Println("error", err) } }() ctx := context.Background() for { topics := []string{"my-topic"} handler := ConsumerGroupHandler{} err := group.Consume(ctx, topics, handler) if err != nil { panic(err) } }
18. Sarama Consumer Group type ConsumerGroupHandler interface { Setup(ConsumerGroupSession) error Cleanup(ConsumerGroupSession) error ConsumeClaim(ConsumerGroupSession, ConsumerGroupClaim) error } func (handler) Setup(_ sarama.ConsumerGroupSession) error { return nil } func (handler) Cleanup(_ sarama.ConsumerGroupSession) error { return nil } func (h handler) ConsumeClaim(sess sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error { for msg := range claim.Messages() { fmt.Printf("Message:%s topic:%q partition:%d offset:%d\n", msg.Value, msg.Topic, msg.Partition, msg.Offset) sess.MarkMessage(msg, "") } return nil } type handler struct{}
19. References • https://kafka.apache.org/ • https://godoc.org/github.com/Shopify/sarama • https://godoc.org/gopkg.in/bsm/sarama-cluster.v2 • https://godoc.org/github.com/Shopify/sarama#ConsumerGroup