Crawled but Not Indexed: Why Googlebot Leaves — A Tragedy in 3 Acts

Why did Googlebot crawl my page but not index it? This 14-slide deck answers the question every webmaster asks when Search Console shows "Crawled – currently not indexed": a crawl is a fetch, not a verdict.
Act I is discovery and fetch (crawl budget, the 2 MB Googlebot fetch limit, the mobile crawler), Act II is parsing, rendering, canonicalization and evaluation, Act III is the indexing decision — with the difference between crawled but not indexed and Discovered – currently not indexed spelled out in Google's own words.

https://en.speedyindex.com/

Transcript

1. S P E E DY I N D E X · P R E S E N TAT I O N Googlebot came. Googlebot saw. Googlebot left without indexing. A tragedy in three acts: why Google crawls a page, evaluates it, and still keeps it out of the index — and how to rewrite the ending. ACT I · CAME SpeedyIndex · en.speedyindex.com AC T I I · SAW ACT III · LEFT Linda Bjorkvin, Project Manager · 2026
2. THE QUESTION EVERY WEBMASTER ASKS Why did Googlebot crawl my page but not index it? ACT I · CAME Discovery and fetch. Evidence: a Googlebot hit with HTTP 200 in your server logs — and a “last crawl” date in URL Inspection. ONE-LINE ANSWER AC T I I · SAW A crawl is a fetch, not a verdict. Google downloads the page (Act I), evaluates it (Act II) and decides (Act III). “Crawled – currently not indexed” means the decision was “not now”. Change what Google evaluates and the verdict can change too. Parse, render, canonicalize, evaluate. Evidence: URL Inspection shows the rendered page and the canonical Google selected. ACT III · LEFT Google’s own docs: Search works in three stages — crawling, indexing, serving — and not every page makes it through each one. SpeedyIndex · en.speedyindex.com The indexing decision. Evidence: Page indexing report → “Crawled – currently not indexed”. Source column: “Google systems”. 02 / 14
3. ACT I Googlebot came. How the crawler decides to knock on your door — and what its visit proves S E R V E R L O G · I L L U S T R AT I V E L I N E 66.249.66.1 - - [14/Aug/2026:03:12:07 +0000] "GET /new-guide/ HTTP/1.1" 200 41288 "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) … Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" What the 200 proves: the fetch happened. That is all it proves. Act I is attendance, not applause — Google downloaded the page; it hasn’t formed an opinion yet. How it found you Links from indexed pages, an XML sitemap, or a URL it already knew. No link and no sitemap means nobody sent the invitation. How much it crawls Crawl budget = crawl capacity limit (what your server can take) × crawl demand (how much Google wants your URLs). Every site starts on the same conservative default limit; it rises only with demand and a fast, healthy server — and it is shared across all Google crawlers. How often it returns Popularity, freshness and staleness drive demand. A page nobody links to and nobody updates gets a rare, low-priority visit — even on a small site. Source: Google Search Central — crawl budget documentation for large sites, updated July 22, 2026. SpeedyIndex · en.speedyindex.com 03 / 14
4. ACT I · SCENE 2 The fine print of the visit What Googlebot actually downloads — per Google’s crawler documentation, updated February–March 2026 2 MB 64 MB 1 by 1 Mobile per URL for PDF files per resource is the crawler that counts The maximum Googlebot fetches for Google Search, HTTP headers included. Larger pages are not rejected: the fetch stops at the cutoff and only the downloaded part goes on to indexing. The PDF exception. Other Google crawlers and fetchers default to 15 MB unless their product sets its own limit — Googlebot is one crawler among many. Each CSS and JavaScript file referenced in the HTML is fetched separately, under the same 2 MB rule. Heavy bundles are cut the same way pages are. Googlebot Smartphone does the reading. Mobile-first indexing means the mobile fetch is the version Google evaluates and indexes. Sources: Google Search Central Blog, “Inside Googlebot” (March 2026); Googlebot documentation update (February 3, 2026). Typical HTML pages sit far below the limit — the cutoff bites pages with huge inline scripts or data blobs. SpeedyIndex · en.speedyindex.com 04 / 14
5. ACT II Googlebot saw. Between the fetch and the verdict: the audition Parse The raw HTML is read: title, headings, links, meta robots, canonical, hreflang. Links found here send Googlebot to other URLs — their Act I. Render → The page is queued for rendering in an evergreen Chromium. Content that appears only after JavaScript runs waits for this step — which can come later. Canonicalize → Similar pages are clustered and Google picks one representative URL. Your rel=canonical is a strong hint, not an order. Evaluate → Is this page worth a place in the index? Depth, originality, site-level quality and demand for the topic feed the decision. Googlebot “sees” twice — first the raw HTML, later the rendered DOM. Anything that lives only in the rendered version is invisible until rendering happens, so a JavaScript-only page auditions with an empty script. Sources: Google Search Central — JavaScript SEO basics; canonicalization documentation. SpeedyIndex · en.speedyindex.com 05 / 14
6. ACT II · SCENE 2 Six reasons the audition fails Why Google reads a page and still passes on it Thin or templated The page adds nothing over what Google already indexed: boilerplate, a few generic lines, a category with no distinct value. Weak internal signals An orphan page known only from the sitemap: no internal links, no context, no vote of confidence from the rest of the site. SpeedyIndex · en.speedyindex.com Near-duplicate Sorts, filters, print views, http/https twins. Google keeps one representative and leaves the rest waiting — under a “Duplicate” status or here. Rendering gap Main content injected by JavaScript that Googlebot has not rendered yet. On first reading the page looks empty. Site-level quality Thousands of low-value URLs (facets, sessions, tags) lower the site’s overall assessment. New pages then get less benefit of the doubt. Soft-404 patterns A 200 page that says “no results”, “out of stock” or shows an empty template. Google files it under Soft 404 — or simply passes. 06 / 14
7. ACT III Googlebot left without indexing. The verdict, as Google Search Console spells it out Crawled – currently not indexed Google fetched and evaluated the page and decided not to index it — for now. The report says it may or may not be indexed later and that there is “no need to resubmit this URL for crawling”. In the report, the Source column reads “Google systems”: the call came from Google’s quality and duplication systems, not from a tag on your site. Discovered – currently not indexed Google found the URL but never crawled it. Typically it wanted to, expected the crawl to overload the site, and rescheduled it — which is why the last-crawl date is empty. Act I never happened: the play was postponed, not reviewed. Fixing the page text changes little here; crawl paths, internal links and server headroom do. Neither status is a penalty. Both say “currently”. Pages leave them when the reason changes — the whole game is telling which act failed. Source: Google Search Console Help — Page indexing report (status definitions). SpeedyIndex · en.speedyindex.com 07 / 14
8. INTERLUDE Read the script: a five-minute diagnostic Before you fix anything, find out which act failed 1 2 3 URL Inspection Last crawl date, “crawled as” Googlebot smartphone, crawl allowed, fetch successful, indexing allowed — and user-declared vs Google-selected canonical. Live test vs view-source Compare the rendered HTML with the raw HTML. If the main content exists only in the rendered version, Act II is your suspect. Server logs Filter Googlebot hits (verify against Google’s published IP ranges): status codes, bytes served, and whether mobile Googlebot returned after your last change. SpeedyIndex · en.speedyindex.com 4 5 Duplicate check Search a distinctive sentence from the page in quotes. Compare siblings: sorts, filters, print views, http/https and slash variants. Internal-link count How many indexed pages link here? Zero means the sitemap is the only witness for the defense. No Search Console access? For third-party URLs — backlinks, guest posts, tier-2 — a bulk index checker answers question zero (is it indexed at all?) for 100 to 100,000 URLs at once: en.speedyindex.com/google-index-checker 08 / 14
9. THE REWRITE Rewrite the ending Eight moves that change what Google evaluates Be the best answer to one intent — add what is missing (data, examples, comparisons); cut boilerplate. Kill soft-404 patterns — real 404/410 for empty pages, real content for the rest. Merge near-duplicates — 301 or canonical to the strongest version. One intent, one URL. Clean the sitemap — only canonical, indexable, 200 URLs, with honest lastmod dates. Ship the main content in raw HTML — SSR/SSG, or verify in URL Inspection that it renders. Cut URL sprawl — facets, sorts, sessions: robots.txt, canonicals or noindex, so site-level quality signals improve. Link from relevant indexed pages — hubs, related guides, menus where the page belongs. Then bring Googlebot back — internal links, the updated sitemap, or an indexing service that triggers a real mobile Googlebot recrawl. Only after the change is live. Anti-pattern: “Request indexing” every day on an unchanged page. Once, after a real fix and a clean live test — fine. Daily — theatre. SpeedyIndex · en.speedyindex.com 09 / 14
10. PLOT TWISTS Six endings that look like Act III but aren’t Check these before you rewrite anything Stale report The Page indexing report lags. URL Inspection may already say “URL is on Google”. Check the URL, not the count. Yesterday’s rewrite You fixed the page; Google has not recrawled it yet. The status reflects the last reading, not the current script. SpeedyIndex · en.speedyindex.com Correct exclusions /page/2/, feeds, tag archives, filter combos: “not indexed” is often the right ending. Don’t fix what should stay out. The 2 MB cutoff Inline JSON or state blobs push the main content past what Googlebot downloads. Check raw HTML size and where the content sits. Wrong protagonist “Duplicate, Google chose different canonical than user” is a separate status. You are polishing a URL Google already replaced. Open backstage doors Staging, dev, parameter and login URLs left crawlable inflate the count — and expose things. Block them at the source. 10 / 14
11. EPILOGUE · TIMING What to expect — in Google’s own numbers Timing statements from Google’s documentation, not estimates ~1 week up to ~2 weeks 0 guarantees to start crawling and indexing to validate a fix to crawl, index or serve What Google says to expect for a brandnew page or site. Wait a few days before diagnosing a fresh URL. Typical time after you click “Validate fix” — in some cases much longer. Patience is part of the workflow. Google states it does not guarantee crawling, indexing or serving of any page — even one that follows Search Essentials. Search Console Help — Page indexing report Search Console Help — Page indexing report Google Search Central — How Search works “Currently” is a status, not a sentence. Pages move into the index once the reason changes and Googlebot has read the new version. SpeedyIndex · en.speedyindex.com 11 / 14
12. T H E S TA G E H A N D Getting Googlebot back for a second reading Where SpeedyIndex fits — after Act II is fixed, not instead of it What it does You submit URLs — your own pages or thirdparty links (backlinks, guest posts, tier-2). SpeedyIndex triggers real Googlebot Smartphone crawls. No Search Console verification needed. What you pay for Pay-per-Result: 100 tokens per URL that gets indexed. Links are checked on Day 7 (Google) / Day 15 (Yandex); tokens spent on non-indexed URLs are auto-refunded to your balance. Limits, plainly: no 100% indexing guarantee — Google’s algorithm makes the final call. SpeedyIndex guarantees the process (submission, checking, refund), not the verdict. Fix Act II first: a recrawl of an unchanged page is a repeat performance. The optional Pre-Indexing Link Check drops 404/410/451, robots/noindex-blocked, media and already-indexed URLs before they cost tokens. SpeedyIndex · en.speedyindex.com What you get back A report with indexed links, unindexed links, indexing errors — including “Crawled – Currently Not Indexed” — and the titles of indexed pages. Standard mode (bulk, up to 100,000 links) or Drip-Feed (spread over hours or days). 200 free tokens for new users · app.speedyindex.com 12 / 14
13. B O N U S · FA Q Four questions from the comments Short answers for the questions this status always raises Is “Crawled – currently not indexed” a penalty? No. It is a status, not a manual action — manual actions live in their own report. Its Source column reads “Google systems”: a quality and duplication call, not a sanction. Will a new sitemap fix it? A sitemap solves discovery (Act I). It can nudge a re-evaluation of fixed URLs, but it does not rewrite Act II. Content, duplication and internal links do. SpeedyIndex · en.speedyindex.com Should I keep clicking “Request indexing”? Once, after a real change and a clean live test. Google’s own status text says resubmitting is not needed; the recrawl of improved content is what changes the outcome. Does the 2 MB fetch limit affect me? Rarely. It hits pages with giant inline scripts or data blobs. Check the byte size of your raw HTML if the main content sits low on the page. 13 / 14
14. SPEE DY I N DE X · C TA Rewrite the ending. Diagnose which act failed. Fix the page. Then bring Googlebot back for a second reading — with a Day-7 report that tells you which links made it and refunds the ones that did not. How to fix “Crawled – currently not indexed” → en.speedyindex.com/fix-crawled-currently-not-indexed Bulk index checker (Google / Bing / Yandex) → en.speedyindex.com/google-index-checker Drip-Feed indexing → en.speedyindex.com/drip-feed-indexing 200 free tokens for new users → app.speedyindex.com Scan for the fix guide Crawled – currently not indexed: causes and fixes on en.speedyindex.com SPEEDYINDEX · OFFICIAL CHANNELS Site: en.speedyindex.com · Blog: blog.speedyindex.com · Telegram bot: t.me/SpeedyIndexBot · YouTube: @SpeedyIndex_com X: x.com/speedyindexbot · Facebook: facebook.com/speedyindexbot · Reddit: reddit.com/user/SpeedyIndexbot SpeedyIndex · Linda Bjorkvin, Project Manager