A Bag-of-Documents Model for Query Specificity

Presented at the 2026 ACM Special Interest Group on Information Retrieval (SIGIR) Workshop on e-Commerce

Search queries differ in specificity, and this variation directly affects retrieval quality and the type of search experience required: broad queries often require exploratory experiences, whereas narrow queries permit more precise ranking. Existing measures of query specificity, such as query length and inverse document frequency (IDF), provide only limited signals. We introduce Bag-of-Documents Specificity (BoDS), a two-stage approach that operationalizes query specificity as intent coherence: how tightly a query’s engaged documents cluster in embedding space. In the first stage, we compute specificity directly from engagement-derived document vectors using the mean resultant length of a query’s associated document vectors. In the second stage, we generalize to all queries by fine-tuning a BERT-based regression model on these scores, achieving an 𝑅2 of 0.84 on the hold-out set. Using logs from a large English-language e-commerce platform, BoDS achieves stronger correlations with click-through rate (CTR; 𝑟 = 0.20) and minimum click rank (MCR; 𝑟 = −0.24) than baselines based on query length or IDF. These results demonstrate that BoDS offers a more robust and holistic measure of query specificity, with applications to query performance prediction, adaptive ranking strategies, and query suggestion.

Transcript

1. A Bag-of-Documents Model for Query Specificity Aritra Mandal (eBay) Daniel Tunkelang (Algolia) Zhe Wu (eBay)
2. What is query specificity and why does it matter? • Intuitively, we recognize some queries as broad and others as narrow. ◦ “shoes” vs. “men’s nike air max” • Broad queries call for exploratory browsing, while narrow queries call for frictionless results. ◦ whether to show refinement options above top-ranked results • But broad vs. narrow is a crude dichotomy. Specificity is a continuous spectrum. ◦ is “running shoes” broad or narrow? • What applications does computing query specificity unlock? ◦ Query performance prediction – which can improve query suggestion. ◦ Tuning retrieval — how much results should drift from the core intent. • Query specificity is intuitive and has practical applications – if we can define and compute it.
3. Existing proxies for query specificity are weak. • Query length is a simple but unreliable proxy for specificity. ◦ "mens clothing" (broad) vs "crocs" (narrow). • Inverse document frequency (IDF) is more sophisticated but has two failure modes. ◦ Rare words are not necessarily specific: “inexpensive” (rare) vs. “sneakers” (common) ◦ Aggregation across terms is challenging: sum over-counts; average and max under-count. • Taxonomies and category entropy are at best a partial solution. ◦ Can compare “pants” to “jeans” but not “pants” to “crocs”. • Query clarity is promising as a measure of coherence, but it predates modern embeddings. ◦ “Quantifying Query Ambiguity” [Cronen-Townsend and Croft, 2002] • Our contribution combines the idea of query clarity with the bag-of-documents model.
4. What is the bag-of-documents model? • Represent a query as the mean (centroid) of its associated document vectors. ◦ “Semantic Equivalence of e-Commerce Queries” [Mandal et al, 2023] • Model is trained using a seed set of queries and their associated document vectors. ◦ Seed queries use actual means; fine-tuning generalizes to all queries. • Intuition: a query “is” the distribution of relevant documents searchers engage with. ◦ The associated documents can come from engagement for explicit relevance judgments. • Prior work focused on the direction of the bag-of-documents query vector. ◦ The cosine between two queries measures the similarity of their intents. • Query specificity focuses on how tightly the bag concentrates around that mean direction.
5. Query specificity measures the coherence of the search intent. • Query specificity measures the “variance” of the result distribution, complementing its mean. ◦ But variance is only defined for scalars, not vectors. • We could use the mean of pairwise cosine similarities among document pairs. ◦ Reasonable approach, but expensive (quadratic) to compute. • Again, we can draw an analogy to the definition of variance. ◦ Var(X) = E[X2] - E[X]2 = E[(X − E[X])²] • We can measure query specificity in terms of the expected similarity to the mean query vector. ◦ Remember, we are working with cosine similarity, not Euclidean distance. • Query specificity operationalizes the cluster hypothesis in an embedding space. ◦ Information Retrieval [van Rijsbergen, 1979]
6. The mean resultant length of the bag measures query specificity. • The bag of documents contains a set of unit vectors. ◦ Normalize if base model does not produce unit vectors. • Compute the mean of the documents in the bag. ◦ This provides the direction of the query vector. • But now compute the magnitude of the query vector. ◦ Higher coherence means higher specificity. • Alternatively compute this value from cosines to the query vector. ◦ Mean cosine(centroid, doc) = length of the mean vector. • This statistic is related to von Mises–Fisher concentration. "men’s black t-shirts" "shirts" high R_q low R_q
7. Examples of query specificity from the Amazon ESCI dataset. Query Tokens Specificity What it shows "rxbar" 1 0.92 short but highly specific "ti84" 1 0.91 short but highly specific "10.2 inch ipad case" 4 0.86 long + specific (aligned) "nike shoes for girls" 4 0.80 long yet only moderate — length overstates "kindle" 1 0.48 brand, but intent diffuse (reader / case / e-books) "free" 1 0.41 short and genuinely broad / ambiguous Note how token length is uninformative: single-token queries span specificity values from 0.41 to 0.92.
8. Compute specificity directly for head queries, then generalize. • For head and torso queries, compute query specificity directly using the documents in the bag. ◦ This relies only on the document vectors. • Then use these seed query specificity values to generalize to a model that serves all queries. ◦ Fine-tune a BERT regression head on the previously computed query-specificity pairs. • Alternatively, could use a KNN approach, computing specificity for unknown queries based on their nearest-neighbors. ◦ Simpler approach, but quality is dependent on the density of the known queries.
9. Query specificity correlates with searcher engagement. • Computed click through rate (CTR), mean click rank (MCR), and various specificity measures for 3M queries on an English e-commerce platform. • Bag-of-documents specificity measure beats query length and inverse document frequency (IDF) to predict both query performance signals. • Absolute correlations are modest: engagement is dominated by ranking and presentation. • Bag size correlates with query frequency, but we can debias to control for query frequency, and correlations still hold. Specificity measure Click through Rate Mean Click Rank Query length 0.17 −0.04 Sum IDF 0.15 −0.21 Mean IDF 0.09 −0.21 Bag of Docs 0.20 −0.24 Band Queries Specificity Debiased Head (top 1%) 22,007 −0.14 −0.15 Torso (1–10%) 198,468 −0.19 −0.20 Tail (>10%) 1,979,559 −0.22 −0.23 All 2,200,034 −0.22 −0.22
10. Results generalize to tail queries and other corpora. • Regression model achieves R² = 0.84 on 200K hold-out queries with known specificity, showing that specificity is recoverable from query text alone. • Results also generalized to Amazon ESCI dataset. ◦ Used MiniLM as the document model and dataset judgments to create bags. ◦ Queries consistently exhibit much higher specificity than random query sets. ◦ Computing the bags using Exact or Substitute vs. just Exact lowers specificity for 89% of queries. ◦ Bag-size dependence disappears for even modest-sized bags (n≥15).
11. Query specificity does have some known limitations. • Two different ways to think about bag size, adding complexity: ◦ Number of distinct documents in the bag. ◦ Query frequency, which makes intuitive sense for engagement. • Embedding anisotropy compresses dynamic range. ◦ Mean-centering or other transformations can help. • Specificity conflates query breadth, ambiguity, and coherence. ◦ It measures coherence directly, ignoring the reasons for low coherence.
12. Conclusion and Future Work • Computing query specificity using the bag-of-documents model allows us to model specificity in terms of the intent coherence of relevant document using a modern embedding space. • This approach outperforms query length and inverse document frequency (IDF) as a query performance predictor, based on click through rate (CTR) and mean click rank (MCR). • Simple to compute: computed directly for head and torso queries, and then generalized by using those query-specificity pairs as training data to fine-tune a regression model. • Unfortunately, there is no agreed-upon standard ground truth for specificity. • We hope to establish a public intent-coherence benchmark with human judgments, perhaps based on the Amazon ESCI dataset.
13. Thank you!