Elasticsearch Aggregations

Transcript

1. Elasticsearch Aggregation Kanji Yomoda (@k-yomo) May 2022
2. Confidential & Proprietary 2021 Agenda • Aggregation types • random_sampler aggregation • How to build facets in e-commerce
3. Confidential & Proprietary 2021 Type of aggregations • Metric aggregations => calculate metrics, such as a sum or average, from field values. • Bucket aggregations => group documents into buckets, based on field values, ranges, or other criteria. • Pipeline aggregations => take input from other aggregations.
4. Confidential & Proprietary 2021 Metrics aggregations
5. Confidential & Proprietary 2021 Metrics aggregations • Avg • Boxplot • Cardinality • Extended stats • Geo-bounds • Geo-centroid • Geo-Line • Matrix stats • Max • Median absolute deviation • Min • Percentile ranks • Percentiles • Rate • Scripted metric • Stats • String stats • Sum • T-test • Top hits • Top metrics • Value count • Weighted avg
6. Confidential & Proprietary 2021 Bucket aggregations
7. Confidential & Proprietary 2021 Bucket aggregations • Adjacency matrix • Auto-interval date histogram • Categorize text • Children • Composite • Date histogram • Date range • Diversified sampler • Filter • Filters • Geo-distance • Geohash grid • Geohex grid • Geotile grid • Global • Histogram • IP prefix • IP range • Missing • Multi Terms • Nested • Parent • Random sampler • Range • Rare terms • Reverse nested • Sampler • Significant terms • Significant text • Terms • Variable width histogram • Subtleties of bucketing range fields
8. Confidential & Proprietary 2021 Pipeline aggregations
9. Confidential & Proprietary 2021 Pipeline aggregations • Extended stats bucket • Inference bucket • Max bucket • Min bucket • Moving function • Moving percentiles • Normalize • Percentiles bucket • Serial differencing • Stats bucket • Sum bucket • Average bucket • Bucket script • Bucket count K-S test • Bucket correlation • Bucket selector • Bucket sort • Change point • Cumulative cardinality • Cumulative sum • Derivative
10. Confidential & Proprietary 2021 How aggregation works
11. Confidential & Proprietary 2021 random_sampler aggregation
12. Confidential & Proprietary 2021 random_sampler aggregation
13. Confidential & Proprietary 2021 random_sampler aggregation = roughly seeing only 0.1% of the documents (1 in every 1000th doc) = needed to get consistent result
14. Confidential & Proprietary 2021 How random_sampler aggregation works
15. Confidential & Proprietary 2021 random_sampler metrics 64 million documents spreaded across many shards
16. Confidential & Proprietary 2021 How to build facets in e-commerce
17. Confidential & Proprietary 2021 Spec • Facet group: (name: colors, facets [{white:10}, { black: 6},...]) • Show top N facets for each metadata (category, brand, color, and etc) • Show all facet counts for it when filtered by itself • Show filtered facet count by the other applied filters
18. Confidential & Proprietary 2021 Filters aggregation × post_filter Search Category aggs - brand filter - color filter Brand aggs - color filter post_filter - brand filter - color filter Color aggs - brand filter Response
19. Confidential & Proprietary 2021 Example queries http://brunozrk.github.io/building- faceted-search-with-elasticsearch -for-e-commerce-part-4/
20. Confidential & Proprietary 2021 References • Aggregations | Elasticsearch Guide [8.2] • Aggregate data faster with new the random_sampler aggregation • Building faceted search with elasticsearch for e-commerce: part 1
21. Confidential & Proprietary 2021 Thanks!