Elastic on Aisle 5: Building Kroger's Digital Data Pipeline

How does a large enterprise company change the way it thinks about and measures the success of its digital endeavors, from customer experience to developer quality of life? In this talk, we discuss Kroger's migration from tailing log files on servers to creating a large-scale data pipeline for log and metric aggregation and analysis. We’ll share some of the challenges faced, what our current production pipeline looks like and take a peek at the road map.
Dan Frohlich | Software Architect | Kroger

Transcript

1. Elastic on Aisle 5: Building Kroger’s Digital Data Pipeline Dan Frohlich Software Architect | Kroger October 6, 2016
2. 2 © Copyright 2014 The Kroger Co., Confidential About Kroger
3. 3 © Copyright 2014 The Kroger Co., Confidential About Digital
4. 4 © Copyright 2014 The Kroger Co., Confidential The Problem
5. 5 © Copyright 2014 The Kroger Co., Confidential Our Goals Modular Flexible
6. 6 © Copyright 2014 The Kroger Co., Confidential Our Path
7. 7 © Copyright 2014 The Kroger Co., Confidential Challenges
8. 8 © Copyright 2014 The Kroger Co., Confidential Solution: Project Echo
9. 9 © Copyright 2014 The Kroger Co., Confidential Data Flow Client Software Clients (Java, GO, JS, h?p) • Forma?ing • batching Topbeat Intake API (Java) • Masking • Filtering • Rate limiting Stream KaPa Rollup (Clojure) View ES / Kibana Alerting (GO) Echo Hosts
10. 10 © Copyright 2014 The Kroger Co., Confidential ES Integra4ons Elasticsearch Kibana Shield Index Optimizer Alerting Custom Views
11. 11 © Copyright 2014 The Kroger Co., Confidential Echo System Metrics Kafka throughput: 80Mb/sec / 30k mps (peak) Shards: 9118 ES Nodes: 23 ES Index Latency < .3ms (average) ES Search Latency < 2-4 ms (average) Indexes:1851 Documents: 9 Billion+ (12TB)
12. 12 © Copyright 2014 The Kroger Co., Confidential Impact [I have] visibility into my app like never before! Echo has made me more confident and better looking! [It has] Charts and Pretty Colors [We now] determine fault domain isolation in less than 10 minutes where it [formerly] took us several hours... It has been totally game changing!!
13. 13 © Copyright 2014 The Kroger Co., Confidential The Future ?
14. 14 © Copyright 2014 The Kroger Co., Confidential 14 © Copyright 2014 The Kroger Co., Confidential Thank You.