Oliver Yee - Brighton SEO October 2026

Too Many URLs, Too Little Value: The Crawl Bloat Problem

Transcript

1. Too Many URLs, Too Little Value: The Crawl Bloat Problem Oliver Yee Head of SEO @ MediaVision linkedin.com/in/oliver-yee-2b184199/
2. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Your website is probably creating URLs faster than you realise www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
3. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Faceted navigation and action parameters accounted for approximately 75% - February 2026: Search Off the Record, Gary Illyes www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
4. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Source of crawling issues 60% 50% 40% 30% 20% 10% 0% Faceted navigation www.wearemediavision.com Action parameters Irrelevant parameters Plugins and widgets Other unusual URL issues linkedin.com/in/oliver-yee-2b184199/
5. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Google’s crawlers might not explore the rest of the site or increase the site’s crawl budget. - https://developers.google.com/crawling/docs/crawl-budget www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
6. BrightonSEO · Too Many URLs, Too Sites with more than one million unique pages that change around weekly www.wearemediavision.com Little Value: The Crawl Bloat Problem Sites with more than 10,000 pages that change daily Sites with significant numbers of “Discovered, currently not indexed” URLs linkedin.com/in/oliver-yee-2b184199/
7. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Crawl bloat can affect more sites than you might think. www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
8. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem What does crawl bloat actually mean? www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
9. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Search engines spend time crawling large numbers of unnecessary or lowvalue URLs on your website. www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
10. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem But these URLs aren’t indexed…so what’s the problem? www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
11. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Crawl bloat is not the same as index bloat www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
12. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem The two often overlap, but a URL does not need to be indexed to consume crawling resources. Crawl bloat Index bloat Search engines spend time requesting low-value URLs Low-value URLs enter the search index Can affect discovery and recrawling Can affect quality signals and search appearance URLs may never be indexed Indexed pages are part of the visible problem www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
13. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Google identifies “perceived inventory” as the factor site owners can most directly control. - https://developers.google.com/crawling/docs/crawl-budget www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
14. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem What counts as a low-value URL? www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
15. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem “Low value” is not simply “no organic traffic”. Low Value Criteria Example Empty or near-empty Zero product category pages Permanently out of stock No product retirement process Internally discoverable but not intended for search Pages built for CRM or PPC www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
16. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Some examples of what this may look like: URL Issue https://www.musclefood.com/collections/hampers/products/theessential-stock-up-hamper?variant=56347728740727 Out of the box Shopify issue which creates collection and variant URLs from PLPs with the product canonical resolving to: https://www.musclefood.com/products/the-essential-stockup-hamper https://www.newlook.com/uk/womens/sale/accessories/c/ukwomens-saleaccessories?comp=NavigationBar|mn|sale|womens|accessori es|homeware Tracking parameters added to internal links with canonical resolving to https://www.newlook.com/uk/womens/sale/accessories/c/ukwomens-sale-accessories https://www.riverisland.com/c/women/tops?f-design=floral&fcat=bardot-and-cold-shoulder-tops&f-cat=halter-neck-tops&fcat=sequin-tops&f-cat=shirts&f-cat=t-shirts&f-cat=vests Deep combinations of facets and filters being discovered through internal linking www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
17. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem How different teams can contribute to bloat: Decision Typical teams Intended outcome SEO consequence Add more filters Ecommerce, UX, Product and Merchandising Improve product discovery More crawlable combinations Preserve old products Ecommerce, Trading and Customer Experience Help returning customers and retain product information Growing legacy URL estate Create campaign pages Marketing, Trading, Content and CRM Support campaigns and seasonal trading activity Thin or expired landing pages remain accessible Add tracking parameters Analytics, Performance Marketing and CRM Improve campaign attribution Duplicate crawl paths and URL variations Launch a new taxonomy Ecommerce, Product, UX and Merchandising Improve navigation and product categorisation Old and new URL routes may coexist www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
18. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Why is it therefore important to resolve? www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
19. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Valuable pages may be discovered later, refreshed less often or left outside the index. www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
20. BrightonSEO · Too www.wearemediavision.com Many URLs, Too Little Value: The Crawl Bloat Problem linkedin.com/in/oliver-yee-2b184199/
21. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Google will tighten crawling resources to improve efficiency as AI investment grows. Crawl bloat will become a bigger liability. www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
22. BrightonSEO · Too www.wearemediavision.com Many URLs, Too Little Value: The Crawl Bloat Problem linkedin.com/in/oliver-yee-2b184199/
23. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem How you can identify, classify and resolve? www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
24. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem If you see in your investigation something like this, then you likely have an issue to resolve: www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
25. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Conducting further governance: Action What to assess Review Google Search Console Examine discovered, crawled and indexed URL patterns. Crawl the site Identify parameters, filters, duplicate pages, canonical signals and crawl paths. Analyse server logs Understand where Googlebot is genuinely spending its time. Segment by URL pattern Group similar URLs rather than evaluating hundreds of thousands individually. Measure organic value Compare search demand, impressions, backlinks, conversions and content differentiation. www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
26. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Deciding what to do next: URL type Crawl? Index? Internal links? Typical treatment Valuable landing page Yes Yes Yes Self-canonical and optimised Useful to users, not search Sometimes No Controlled Noindex or access controls Duplicate URL Limited No No Canonical or redirect Removed page without replacement Temporarily No No 404 or 410 www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
27. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Problems with faceted navigations www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
28. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Common complications: Common mistake Example Potential consequence Every filter creates a crawlable URL /dresses?colour=black /dresses?size=10 Googlebot encounters thousands of filter combinations, including pages with little or no search value. Multiple filters can be combined without limits /dresses?colour=black&siz e=10&brand=x&price=50100 The number of possible URLs grows exponentially as more facets are selected. Filter order creates different URLs /dresses?colour=black&siz e=10 /dresses?size=10&colour= black Identical results become accessible through multiple URL variations. www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
29. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem It can be difficult to get faceted navigations right www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
30. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Some approaches to handle and prevent common issues with faceted navigations: Approach Example implementation What it does URL fragments /dresses#colour=black Everything after # is not sent to the server as part of the URL request, so it does not create a conventional crawlable page. Buttons instead of HTML links <button datafilter="black">Black</button > Removes crawlable <a href> links from the filter interface. JavaScript event handlers A button triggers an AJAX request and updates the product grid Changes the product selection without exposing a new linked URL. www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
31. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Even then, implementing customisable fields can create significant development overhead: Page element Ideal state Example: Black Dresses Clean URL structure Use a readable URL that remains consistent /dresses/black/ Title tag Write a unique title targeting the facet’s search demand `Black Dresses Meta description Describe the specific product range and encourage clicks Shop black dresses in mini, midi and maxi styles... H1 Use a unique heading that accurately describes the filtered range Black Dresses Category Copy Add useful, specific content rather than repeating the parent category copy Content covering core phrase, products, fits and styles www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
32. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Introducing Accelerate MediaVision’s Metis’, long-tail PLP product to help brands capture searches for categories they don’t have but should have. (01) Unlimited AI-generated PLPs, opening categories where demand already exists, selected and crafted by people (02) Hosted seamlessly, with no dev resource required (03) Provides the exact same shopping experience for users www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
33. BrightonSEO · Too www.wearemediavision.com Many URLs, Too Little Value: The Crawl Bloat Problem linkedin.com/in/oliver-yee-2b184199/
34. BrightonSEO · Too www.wearemediavision.com Many URLs, Too Little Value: The Crawl Bloat Problem linkedin.com/in/oliver-yee-2b184199/
35. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Keeping the problem fixed www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
36. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Stop crawl bloat coming back. Build URL controls into everyday decisions. Prevent Monitor Assign ownership Include SEO in Alert on unexpected growth platform and navigation in URLs and parameters changes Define responsibilities across SEO, development, UX and trading Set rules for which filters create crawlable URLs Review logs to see where bots spend their time Agree what happens when campaigns end or products are discontinued Validate sitemaps before changes go live Check that sitemaps stay aligned with intended indexable pages Make URL checks part of the release process www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
37. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Which URLs deserve Google’s attention? www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
38. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Take control of what gets crawled. 1. Find the waste 2. Protect the value 3. Stop it returning Use crawl data, GSC and server logs to identify unnecessary URL patterns. Keep useful search landing pages discoverable. Control duplicate and low-value combinations. Build URL rules, monitoring and ownership into everyday decisions. www.wearemediavision.com linkedin.com/in/oliver-yee-2b184199/
39. BrightonSEO · Too Many URLs, Too Little Value: The Crawl Bloat Problem Get in touch > > www.wearemediavision.com Oliver Yee linkedin.com/in/oliver-yee-2b184199/ linkedin.com/in/oliver-yee-2b184199/