Powering a BI Application with Elasticsearch at Yodle

Yodle recently transitioned from using Hadoop/Scalding to Elasticsearch to power a tailored BI product for franchises. This talk will detail the impact this transition has had on the product, our customers, and engineer efficiency.
Mark Drago | Director of Engineering | Yodle

Transcript

1. Mark Drago, Director of Engineering Powering a BI Application with Elasticsearch Yodle, a web.com company October 19th, 2016
2. Supporting local business since 2005. 2 Yodle helps more than 50,000 local business owners in 250 different industries find and keep customers simply and profitably. a company
3. 3 Document Search •  Contacts in CRM •  Email Marketing List Criteria Elastic Stack (beats, logstash, kibana, elasticsearch) •  Centralized Log Store •  225 Microservices •  HTTP/JSON •  Docker Containers •  Marathon / Mesos •  Many non-microservice apps •  6 Environments •  dev, multiple QAs, production BI Product •  Website Traffic •  Marketing Data (SEM) •  Phone Call Data •  Search Engine Ranking •  Online Reviews •  etc. @ Yodle 2013 2014 2015
12. 2010/2011
15. Fall 2013 Centermark 1
16. 16 Sample Aggregation Franchise ID Business ID Date Rating F1 B1 September 15 Good F2 B2 September 15 Good F1 B3 October 12 Good F1 B1 October 12 Good F2 B2 October 19 Good F2 B2 October 19 Good Franchise ID Month Good Calls F1 September 1 F2 September 1 F1 October 2 F2 October 2
19. 19 Challenges •  Need ability to backfill historical data at the start •  Need ability to re-backfill as we add features or fix bugs •  Must handle fresh data coming in every day •  Must handle historical data changing
23. 23 Sample Aggregation Franchise ID Business ID Date Rating F1 B1 September 15 Good F2 B2 September 15 Good F1 B3 October 12 Good F1 B1 October 12 Good F2 B2 October 19 Good F2 B2 October 19 Good Franchise ID Month Good Calls F1 September 1 F2 September 1 F1 October 2 F2 October 2 Problem Problem
34. Development Speed Matters
35. Old New
36. 36 5 nodes Each node: -  m4.2xlarge -  8 cores -  32G ram (½ to ES, ½ to page cache) -  EBS backed storage (general SSD) Data size: -  1TB (2 replicas -> 3TB) -  Largest index: 400G (replicas -> 1.2T) -  Index with most docs: 1.6B Elastic setup: -  Version 2.4.1 (excited about 5.x) -  Instant aggregations, Painless scripting, Lucene 6.2 performance improvements -  Use doc values almost exclusively (non analyzed strings / ints) -  Single availability-zone for now -  Novel backup/restore process to get prod data to test environment (see our blog)
43. S5ll fast, even with 2K loca5ons and spanning mul5ple years (2 - 3 seconds)
45. 45 Product Engagement is Through the Roof Product Launched 3x Engagement
46. yodle.com yodlecareers.com yodletechblog.com
47. www.elastic.co 47