Short before the launch of the METRO Markets' marketplace at https://www.metro.de/marktplatz/ we needed to build an event and user tracking platform to log events relevant to monitor the business KPIs of our site.
Confronted with having also frontend JS applications as well as several backend APIs and services, we wanted to find a quick and elegant solution to reliably track business events on this distributed system.
We summarized the problems that might occur und found a way using Apache Kafka
and tools from the Kafka ecosystem, like Confluent REST Proxy
and Confluent's Kafka Connect HTTP Sink, to easily build a pipeline that gathers all tracking messages in the distributed log and forward them to our analytics providers without writing any line of code.
The slides show the problems and ideas.
The demo is available on GitHub: https://github.com/rk3rn3r/kafka-meetup-2019-09