Part I - History

Part I of MeetUp @DigitecGalaxus

Transcript

1. The Road to Elastic at Digitec Galaxus Author Martin Gasser Speaker Renato Bosshart Engineering
2. Timeline Search 07.02.18 The Road to Elastic at Digitec Galaxus 2 • 2001 – 2014 Search on SQL Server (old digitec shop) • 2010 – 2016 Custom search (galaxus and digitec beta) • Dec 2014 Remove deleted products from Search • Jan 2016 Prototype Search with elasticsearch • Q1 2016 Implementation Search with elasticsearch • Q4 2017 Suche+
3. Search in digitec classic 07.02.18 The Road to Elastic at Digitec Galaxus 3 • SQL Server Full Text Catalog • Issues: • High load on SQL Server (bad scalability) • Limited functionality
4. Custom search for Galaxus 07.02.18 The Road to Elastic at Digitec Galaxus 4 • Trie + • Damerau–Levenshtein distance = • Our custom search
5. Reasons for switch to elasticsearch 07.02.18 The Road to Elastic at Digitec Galaxus 5 • Custom search not ready for larger product range (going from 300k to 1 Mio products) • 7 GB Memory usage • Performance Issues with Garbage Collection • Search site was the slowest page in our online shops • Hard to understand why some things can not be found • «ironman bluray» did not find «Iron Man Blu-Ray» • Improving search quality not possible without a lot of work Custom search Elasticsearch prototype Elasticsearch current Index size 7 GB 0,27 GB 13 GB Index initialization duration ~3 h 5 min ~8h Average Search time 3 ms 4 ms 200 ms
6. Timeline product filter 07.02.18 The Road to Elastic at Digitec Galaxus 6 • 2001 – 2014 Filter with SQL Query (old digitec shop) • 2010 – 2016 In-Memory filtering with HashSets • Dec 2016 Prototype filtering with elasticsearch • Q1 2017 Implementation filtering with elasticsearch
7. Product filtering in digitec classic 07.02.18 The Road to Elastic at Digitec Galaxus 7
8. Product filtering in digitec classic 07.02.18 The Road to Elastic at Digitec Galaxus 8
9. Product filtering in Galaxus 07.02.18 The Road to Elastic at Digitec Galaxus 9 • Showing all the counts increases complexity • Move load from SQL Server to webservers • Easier to scale • SQL Server just returns the options of each product • Counts are calculated with HashSets (union and intersect) // Pseudo code var lcdProductIds = new HashSet<int>(productIdsByFilterOption["Full HD"]); lcdProductIds.UnionWith(productIdsByFilterOption["UHD"]); lcdProductIds.IntersectWith(productIdsByFilterOption["LCD"]); var lcdCount = lcdProductIds.Count;
10. Reasons for switching Filter to elasticsearch 07.02.18 The Road to Elastic at Digitec Galaxus 10 • Hard to scale to more products • Increase in data transmitted between SQL Server and webservers • HashSet Intersect and Union biggest CPU Consumer on webservers • High maintenance cost • Changes from Product properties not propagated to optimized tables for filtering • Updater for filter tables crashed quite often • Wrong counts when there are products with variants
11. Filter with elasticsearch 07.02.18 The Road to Elastic at Digitec Galaxus 11  Counts are correct with product variants  ETL to elasticsearch is much better due to complete rewrite of the code  Because we have no sorting information in the index, we cannot use pagination and have to load large datasets.  Aggregation-Query which is sent to elasticsearch is very complex