Keywords to Conversations. 50 years of searchandising evolution: how software engineering and information science unite to solve the real challenges of online merchants.
ElasticSuite solutions Launched in 2016, ElasticSuite has grown to become the premier open-source search and merchandising solution for Magento 2.. Romain Ruaud CREATOR Creator of Gally and ElasticSuite. Spurring search and merchandising innovation in the Magento ecosystem since 2011. #1 Search Solution: Over 4,700,000+ total downloads worldwide. Open-Source Core: Built entirely on open technologies Elasticsearch / OpenSearch). Sovereign Control: Powering merchant data sovereignty, privacy, and software ownership natively. • • • About Romain Ruaud & ElasticSuite
THE COMMERCE ANALOGY An internal search bar is a salesman. How do we turn an algorithm into an expert adviser? We have continuously adapted software to bridge the bottleneck separating human expression (often imprecise) from the rigid structure of a database.
Ruaud 04 // ACADEMIC FOUNDATIONS The Problem The Need: Computationally model the human, nuanced concept of "relevance" when interpreting queries. The Challenge: Early computer science systems treated documents binarily. A file either contains the searched term 1) or doesn't 0. No nuance of degree or probability of match could be handled. The Technical Solution At the University of Cambridge, Stephen Robertson and Karen Spärck Jones revolutionize text processing. They establish the foundations of probabilistic Information Retrieval IR. Document relevance becomes a mathematical probability that varies according to the relative importance of each term. 1976 Theoretical foundations
Rank results so that the most interesting items rise to the top of the list. The Challenge: In a query like "red dress", how do we make the machine understand that the essential term is "dress" and not the color ? Without weighting, products matching only "red" pollute the entire top of the search results page. The Solution: TFIDF Theorized by Karen Spärck Jones, it balances local term frequency with its global scarcity: Term: "dress" (Rare, high IDF) Weight: 85% Term: "red" (Common, low IDF) Weight: 15% TFIDF Decoding the user intent CHAPTER 02 // 1976 - 1994 : THE ORIGINS
Prevent ranking manipulation by poor writing practices or keyword spamming. The Challenge: TFIDF is linear. Some authors or copywriters artificially repeat the word "shoes" 50 times in their product page to hijack the top spot. This is the toxic era of keyword stuffing, which severely degrades buyer trust. The Solution: Okapi BM25 Introduction of logarithmic frequency saturation TREC3, 1994. Whether a word appears 3 or 50 times, its score reaches a ceiling. It also accounts for average product description length. 1994 Okapi BM25 CHAPTER 02 // 1976 - 1994 : THE ORIGINS
Need: Launch the world's first dynamic e-commerce stores Amazon, eBay). The Challenge: An online catalog is alive: stock levels change, prices fluctuate. Early academic indexing engines required massive, offline, asynchronous batch calculations. They were physically unable to integrate live inventory updates into the search experience. The Pragmatic Choice E-commerce makes a radical engineering compromise: sacrificing linguistic relevance for the operational robustness of real-time transactions. Catalogues are deployed on transactional relational databases. BM25 remains confined to academic labs while the retail web rolls out SQL. 1995 The great schism CHAPTER 03 // 1995 - 1998 : THE EARLY E-COMMERCE ERA
and lack of score: SQL queries slow down dramatically as the product database grows. SQL operates on pure binary logic 0 or 1. No relevance score is computed. Additionally, scanning the table sequentially Full Table Scan) destroys server performance. Algorithmic Complexity ON The Sequential scan As the catalog grows, query response time degrades linearly. An absolute technical dead end for transactional search. SQL LIKE The binary dead end CHAPTER 03 // 1995 - 1998 : THE EARLY E-COMMERCE ERA
Need: Retrieve products instantly without putting load on the primary transactional database. The Challenge: As transaction volumes and concurrent users rise, SQL bases lock up because they must process text searches and checkout payments on the same tables. The Solution: Lucene & Inverted Index Doug Cutting creates Lucene 1999. Text search becomes a robust, open-source library. Instead of scanning rows, Lucene splits records into unique tokens pointing directly to document IDs (postings lists): Term (Token) Occurrence Documents (Postings List) shoes Doc 1, Doc 4, Doc 12, Doc 34, Doc 89 red Doc 4, Doc 12, Doc 55, Doc 80 heel Doc 12, Doc 55, Doc 89 1999 The inverted index CHAPTER 04 // 1999 - 2012 : THE LINGUISTIC EXECUTION
The Need: Avoid losing shoppers who make spelling mistakes on mobile devices (e.g. typing "coffe" instead of "coffee" or "jakt" instead of "jacket"). The Challenge: Calculating edit distances on the fly against the entire catalog for every query crashes the CPUs. The Solution: Lucene 4.0 DFA The catalog dictionary is pre-compiled into a state graph DFA. Evaluating spell tolerance becomes a fast, linear operation ON based on the query size. Search speed goes up 100x. Input Target Distance Action magnto magento 1 Insert 'e' coffe coffee 1 Insert 'e' jakt jacket 2 Insert 'c' & 'e' 2012 The Levenshtein DFA barrier CHAPTER 04 // 1999 - 2012 : THE LINGUISTIC EXECUTION
Need: Align search result relevance with actual merchant business goals. The Challenge: Standard search engines only understand linguistics. If an out-of-stock product happens to contain the target keyword 10 times, it jumps to the top spot, hiding available, high-margin alternatives. Pure matching hurts the merchant's financial conversion rate. The Solution: function_score Elasticsearch 0.90 introduces the function_score API. It unifies text matching scores with catalog signals: Text Score BM25 : 2.5 In-stock multiplier : × 1.5 High-margin multiplier : × 1.2 Promoted! × = • • • 2013 The Function Scoring era CHAPTER 04 // 1999 - 2012 : THE LINGUISTIC EXECUTION Final Score = 2.5 1.5 1.2 4.5 Final Rank BM25 Score Business Weight
The Need: Bring the power of dynamic searchandising into the popular Magento 1 ecosystem. The Challenge: Magento 1's native catalog search relied on Solr or on limited database LIKE structures. It was incapable of leveraging Elasticsearch fine-grained function scoring, depriving merchants of advanced merchandising strategies. The Solution: smile-elasticsearch After a few other R&D products (the 1st MongoDB adapter for Magento, or the 1st Varnish cache tags implementation), we jump in the search playground . We publish the historical module that bypasses native search, connecting Magento directly to the Elasticsearch API. This is the genesis of our searchandising vision. 2013 Beginning of our journey on Magento 1 CHAPTER 05 // 2013 - 2019 : THE RISE OF SEARCHANDISING We give power to the merchants to deliver their business strategy.
Need: Allow business teams to manage relevance and visual search rules without editing code. The Challenge: Designing and adjusting Elasticsearch relevance scoring or JSON structures requires writing immense, fragile, and nested query trees. This is a highly repetitive task that only specialized back-end engineers can perform. The Solution: ElasticSuite The launch of Magento 2 reshapes the framework. We completely rebuild the architecture and release ElasticSuite. We encapsulate all the raw technical logic inside ElasticSuite and decide to expose directly all the related configurations embedded right in the back-office. Business merchants finally regain full control of their digital shelves. 2016 The birth of ElasticSuite CHAPTER 05 // 2013 - 2019 : THE RISE OF SEARCHANDISING
SaaS Lock-in The Trend: During the mid-2010s, the market shifted heavily toward cloud-based proprietary search engines. The Issue: Black-box algorithms took away merchant autonomy, hiding ranking parameters while introducing high per-query operational costs. The Open-Source Path We chose the opposite path: building fully open-source solutions on top of Elasticsearch (and later OpenSearch). Why it mattered: Customer search behaviors, click logs, and sensitive catalog metadata stayed entirely within the merchant's hosting infrastructure.. No API toll booths, complete operational sovereignty. 2016 Sovereign open-source choice CHAPTER 05 // 2013 - 2019 : THE RISE OF SEARCHANDISING
Need: Understand buying intent, even if the visitor does not use the exact words written on the product page. The Challenge: BM25 remains a literal matching engine. If a customer searches for "outerwear" and the catalog only contains the term "jacket", the product remains hidden without manually managing painful synonym lists. The Solution: Conceptual Matching Information science steps away from standard inverted indexes to model words as geometric concepts. We no longer evaluate exact character matches, but rather the proximity of ideas. This opens the path for dense and sparse vectors. 2020 Semantics - Moving beyond keywords CHAPTER 06 // 2020 - 2025 : THE SEMANTIC SHIFT
Need: Surface relevant products based on logical analogies and conceptual meaning. The Challenge: How do we mathematically project and calculate this similarity at search-time across thousands of items without destroying response latency? The Solution: Embeddings (vectors) Words and products are projected as analysis vectors into multi-dimensional semantic spaces. The vector search revolution CHAPTER 06 // 2020 - 2025 : THE SEMANTIC SHIFT
Need: Maintain exact technical precision SKUs, part numbers) while offering conceptual, semantic search tolerance. The Challenge: Standard BM25 text scores and Vector similarity scores use completely different scales, such as a 1.5 text-matching score vs a 0.85 vector similarity score. They cannot simply be added together. The Solution: Reciprocal Rank Fusion Instead of adding incompatible raw scores, we mathematically merge the ranking positions RRF Unified Match Example: • Product A ranks 1st in exact text search • Product A ranks 3rd in semantic vector search → RRF combines these list positions into a perfectly balanced final rank. Modern hybrid indexing CHAPTER 06 // 2020 - 2025 : THE SEMANTIC SHIFT
Need: Let customers chat naturally to refine their shopping cart selection. The Challenge: Letting a raw LLM answer autonomously risks semantic hallucinations. The AI can invent non-existent products, completely ignoring real-time business and inventory parameters. One Voice: The assistant must respect inventory and merchandising rules to speak with one unified, strategically aligned voice. The Solution: Intent Translation and Guardrail agent The LLM acts as an intent translator: it extracts strict filters from the conversation to query the transactional index in real time. This ensures 100% accurate, editable search constraints: Customer Prompt: "I need a warm jacket, in-stock in size M." Intent Extraction Filters): Category: Jacket | Size: M | Stock > 0 Merchant Guardrail: Exclude out-of-stock, prioritize active merchandising margin boost. Conversational AI under control CHAPTER 06 // 2020 - 2025 : THE SEMANTIC SHIFT
Core Business Need Software Solution ElasticSuite Product Positioning 19951998 Find strict keywords in dynamic e-commerce catalogs SQL LIKE Binary & Slow) Not existent Pure SQL dependency) 19992012 Rank text relevance and handle typing errors Lucene TFIDF & Levenshtein DFA Pioneering module released for Magento 1 2013 20132019 Align results with margins and stock levels Elasticsearch BM25 & function_score) ElasticSuite Release 2016 with visual cockpit 20202026 Understand natural intent and support secure chat Hybrid BM25 Semantic vectors with RRF Open-Source Sovereignty, Privacy & Controlled AI rules 50 years of searchandising
Need: Reach customers where they formulate buying intents outside of your site. The Challenge: Users browse traditional web structures less and less. Instead, they prompt external AI engines directly ChatGPT Search, Perplexity, Siri). If your catalogue cannot be interpreted by these external bots, your products cease to exist. Our Shifting Mission We must migrate from "on-site search box" to a reliable Product Intelligence Provider. Our goal is to structure and expose catalog business logic cleanly for digestion by external AI agents. Standard keyword matching is no longer enough. 2026 The new barrier CHAPTER 08 // TOWARDS FUTURE
Need: Scale traffic without letting platform API costs devour e-commerce profitability. The Challenge: Under massive request volumes generated by automated AI agents, transactional billing-per-query of SaaS systems becomes financially unsustainable. Furthermore, piping customer transaction history out to third-party APIs violates modern privacy laws Privacy-First). The Return of True Ownership Merchants in 2026 demand digital sovereignty and predictable infrastructure costs. ElasticSuite's open architecture (on-premise or cloud hosting on top of Elasticsearch/OpenSearch) stands as the natural answer: Full data control, with zero premium billing per query. 2026 End of the SaaS mirage CHAPTER 08 // TOWARDS FUTURE
core product data and merchant business intelligence (live inventory, multi-tiered prices, fine variations) in formats designed for AI ingestion. Deploy rich semantic data schemas alongside API delivery via real-time feeds. The Gatekeeper AI Data Gateway) Regain control over scraping. ElasticSuite acts as a governance gateway for incoming AI crawlers. The merchant dictates exactly what AI engines can read (stock status, margins, exclusions) while monitoring generated value. AIReady Catalog & Gateway CHAPTER 08 // TOWARDS FUTURE
away YESTERDAY (KEYWORDS) Structuring data so a machine can understand it. TODAY (SEARCHANDISING) Weighting data so it serves the merchant's business goals. TOMORROW (CONVERSATIONS) Protecting data so it remains the merchant's exclusive property. The winner of the AI era will be the merchant who owns their tech stack and retains absolute control over their data. — Romain Ruaud, Creator of ElasticSuite & Gally “Technology changes, the battle remains the sameˮ
romainruaud
marktimemedia
panda_program
akihiro_kokubo
aleyda
rkaga
reverentgeek
zakiyama
tamaranovitovic
baasie
mojombo
mlcsv
bethany
