Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Etsy on Migrating to Kafka (in three short years)

Etsy on Migrating to Kafka (in three short years)

Full post with video here:

Hakka Labs

January 22, 2015
Tweet

More Decks by Hakka Labs

Other Decks in Programming

Transcript

  1. Migrating to Kafka in Three Short Years A look at

    the choices that defined the Etsy analytics stack
  2. Decisions made in the past limit options in the present,

    even if the circumstances under which those past decisions were made are no longer relevant.
  3. In other words, we can’t upgrade the Hadoop cluster until

    we port all of the Cascading.jruby jobs to Scalding.
  4. Sneak Preview ! 1. How Etsy built its original analytics

    stack 2. Handling changes prepared us to rebuild our data pipeline 3. Kafka!
  5. Benefits: ! •Simpler ETL •Visits computed on the client side

    •Easy to reconcile against Google Analytics
  6. Solution: ! 1. App uploads bundles of events to API

    endpoint 2. Backend event logger curls the beacon for every event
  7. Visit Time Logger Event Type 1 12:01 frontend home 1

    12:03 backend login 1 12:03 frontend view listing 1 1:31 backend logout 2 1:31 frontend view listing 2 1:32 frontend search 2 1:33 frontend view listing wrong visit