Etsy on Migrating to Kafka (in three short years)

Full post with video here:

1. Migrating to Kafka in Three Short Years A look at the choices that defined the Etsy analytics stack
3. Path Dependence
4. Decisions made in the past limit options in the present, even if the circumstances under which those past decisions were made are no longer relevant.
5. In other words, we can’t upgrade the Hadoop cluster until we port all of the Cascading.jruby jobs to Scalding.
6. Sneak Preview ! 1. How Etsy built its original analytics stack 2. Handling changes prepared us to rebuild our data pipeline 3. Kafka!
7. Starting from scratch
8. Choice #1 ! Acquire Adtuitive
11. Before you can work on search, you need real analytics
12. Choice #2 ! Build a zero-impact analytics stack
13. Etsy is not a cloud company but the first analytics stack was cloud-based
14. (illustration here) browser CDN EMR S3 mysql FTP
15. Legacy effects: ! 24 hour latency on events 48 hour latency on visits
16. Choice #3 ! Cascading.jruby
17. Hadoop Cascading Cascading.jruby
18. Choice #4 ! Use GA _utma cookie to define visits
19. Benefits: ! •Simpler ETL •Visits computed on the client side •Easy to reconcile against Google Analytics
20. Choice #5 ! Using existing feature library for A/B tests
21. Leveraged existing experience with operational ramp-ups
22. Low impact: just required a logging change
23. Choice #6 ! Build analytics stack around visit-level metrics
24. Great for search and ads, less great for measuring engagement
25. Changing the tires without stopping the car
26. How do we instrument the iOS app? Summer 2012
27. 1. Native app visits should have the same structure as Web visits
28. 2. Native app events should use the existing data pipeline
29. 3. The native app should buffer events and send them when convenient
30. Solution: ! 1. App uploads bundles of events to API endpoint 2. Backend event logger curls the beacon for every event
31. Side effect: ! We have a backend event logger that is now used all over the place
32. CDN diversification project Fall 2012
34. Migrated to our own beacon infrastructure
35. Data pipeline based on Apache, PHP, logrotate, and cron
36. We built our own Hadoop cluster: Etsydoop Fall 2012
37. We hired the Scalding guy Fall 2012
38. Hadoop Cascading Cascading.jruby Scalding
40. Uh oh, the Google Analytics JS hurts performance Fall 2012
41. The event logger’s GA dependency precluded async loading, hurting performance
42. First idea: duplicate the _utma functionality in our own code
43. The trouble with backend events
44. Visit Time Logger Event Type 1 12:01 frontend home 1 12:03 backend login 1 12:03 frontend view listing 1 1:31 backend logout 2 1:31 frontend view listing 2 1:32 frontend search 2 1:33 frontend view listing wrong visit
45. Complete rewrite of our ETL jobs Spring/Summer 2013
47. Backend page-view events Fall 2013
49. 2014: the next phase
50. EventPipe goals
51. Use POST rather than multiple GET requests to prevent data loss
52. Use JSON rather than query strings for comprehensibility
53. Validate beacon data before it enters the data pipeline
54. Use a binary serialization format for long-term storage
55. Use Kafka for data transfer to escape the batch paradigm
56. Eliminate individual beacon servers as points of failure
57. How do we handle the impedance mismatch between Apache/PHP and Kafka?
58. Wrote a server in Go to serialize beacons in Thrift and send them to Kafka
59. Use Apache for SSL termination
60. Still to come
61. Real-ish time ETL
62. Streaming infrastructure
63. Offline processing for more products
64. Other Kafka applications
65. Takeaways
66. Every choice you make has long-term implications
67. Fixing stuff creates new opportunities
68. @rafeco http://rc3.org