Guides / Get Found

By DLVX · 14 min read · Updated 20 October 2026 · Part 2 of 10 in Get Found

Get indexed: Search Console, sitemaps and IndexNow

  • Google Search indexing is a prerequisite for its organic results and supporting links in its AI features. Other discovery routes, direct visits and sales can still exist, so check the channel you actually want to improve.
  • Sitemaps with honest lastmod dates, internal links from pages Google already crawls, and IndexNow for Bing and friends do most of the work.
  • Request indexing is a nudge for a handful of URLs, the Indexing API is not for normal pages, and blocking the wrong bot can quietly remove you from AI answers.
On this page

Why indexing comes before everything else

Every other guide in this track assumes one thing: the page exists in the search engine's index. If it does not, nothing else matters. No ranking, no rich result, no citation in an AI answer.

Google is plain about this for its AI features. A page must be indexed and eligible to show with a snippet, and Google says there are no additional technical requirements beyond that. Inclusion is still not guaranteed. That is from Google's AI features page, last updated 10 December 2025, and the newer AI optimization guide, last updated 10 July 2026, repeats it: you do not need new machine readable files, AI text files or special markup to appear.

So when an owner tells us "ChatGPT never mentions us" or "we vanished from AI Overviews", our first move is boring on purpose. We open Search Console and check whether the pages that should answer the question are even in the index.

We launched the new site three weeks ago and most pages still show "Discovered, currently not indexed". Is something broken?

Usually not broken, usually unconvinced. Google found the URLs but has not decided they are worth crawling and storing yet. Google's own Page indexing report help says "not indexed" is not necessarily bad, and that small sites under about 500 pages probably do not need to watch the report closely. The fix is rarely a button. It is usually internal links from pages Google already crawls often, a sitemap with accurate dates, and pages that give a reason to be indexed. We come back to each of these below.

50,000URLs or 50MB uncompressed per sitemap file, Google's limit (updated 8 July 2026)
10,000URLs per IndexNow POST, per the protocol documentation
10 to 12manual "request indexing" submissions per property per day in our own use. Google does not publish the number

Sitemaps: the boring file that does the most

A sitemap is a list of the canonical URLs you want indexed. It does not force anything. Google's sitemap overview says a sitemap helps discovery but does not guarantee crawling or indexing, and that a well linked site of about 500 pages or fewer may not need one at all. We ship one anyway on every site, because it costs nothing and it is the only place you can tell engines what changed and when.

The part most sites get wrong is lastmod. Google's build and submit a sitemap page, last updated 8 July 2026, says it uses the lastmod value only "if it's consistently and verifiably accurate", for example when it matches the page's real last change. It also says it ignores priority and changefreq. Bing says the same thing from its side: in a post from 31 July 2025 it asks sites not to set lastmod to the time the sitemap was generated unless the content actually changed, and to pair sitemaps with IndexNow.

The lastmod trap we see most

Many site builders stamp every URL with today's date each time the sitemap regenerates. Engines learn that your dates mean nothing and stop trusting them. Set lastmod from the content's real change date, and leave it alone when only the footer or a script changed.

Sitemap field or habitWhat engines do with itOur rule
Canonical, absolute URLs onlyGoogle asks for exactly thisWorks Never list redirects, noindex pages or parameter duplicates
Accurate lastmodUsed by Google when it proves reliable; Bing asks for real change datesWorks Date of the last real content change
lastmod set to "now" on every buildTeaches engines to ignore your datesDon't Fix the generator
priority and changefreqIgnored by GoogleSkip it Harmless, pointless
One giant sitemap past 50,000 URLsOver Google's limitDepends Split into a sitemap index by section

Submit the sitemap once in Search Console and once in Bing Webmaster Tools, and reference it in robots.txt. After that, the engines fetch it on their own schedule.

Search Console: inspect, request, then stop clicking

Search Console is free and it is the only first party view of how Google sees your pages. Three parts matter for indexing.

  1. URL Inspection. Paste one URL and you get Google's indexed version and a live test. The two lines we read first are "Page indexing" and the Google selected canonical. If Google picked a different canonical than you declared, your page is being treated as a duplicate of something else. We have seen guides that were live, sitemapped and well written sit unindexed for exactly that reason.
  2. Request indexing. After the live test passes you can ask Google to queue the URL. Google's recrawl guide is clear that there is a quota, that repeat requests do not speed anything up, and that for many URLs you should submit a sitemap. The URL Inspection help page adds that you cannot request indexing if the live test says the page is not indexable. In our own work the quota has been about 10 to 12 requests per property per day. Google does not publish that number, so treat ours as an observation, not a rule.
  3. Page indexing report. This groups every known URL by status: indexed, or not indexed with a reason. Read the reasons, not the totals. "Crawled, currently not indexed" on your money pages is a quality or duplication problem. "Excluded by noindex" on pages you want found is a deployment mistake.

Can't we just use the Indexing API and push everything through instantly?

No, and we would not want you to try. Google's Indexing API guide, updated 16 October 2026, says the API can only be used for pages with JobPosting, or BroadcastEvent embedded in a VideoObject. That is job ads and livestreams. The quota page, updated the same day, sets a default of 200 publish requests per project per day, and the guide warns that spam policies apply to what you submit and that you must not get around the limits with extra accounts. Tools that sell "instant indexing" through this API for ordinary pages are using it outside its stated purpose.

Changed in 2026

Google updated both Indexing API pages on 16 October 2026 to reflect current limits. The scope line, jobs and livestreams only, did not loosen.

IndexNow: tell Bing and others the moment a page changes

IndexNow is an open protocol for pinging search engines when a URL is added, updated or deleted. Bing describes it on its IndexNow page as a free, open source protocol. You prove you own the site with a key file, then send URLs.

The mechanics, from the IndexNow documentation:

  1. Create a key. 8 to 128 characters, letters, digits and dashes. Host it as a UTF 8 text file named after the key, at the root of your site, for example https://example.com/<your-key>.txt.
  2. Submit. One URL with a simple GET, or a JSON POST with your host, key and a urlList of up to 10,000 URLs.
  3. Read the response. 200 means submitted, 202 means received while the key is still being checked, 403 means the key is wrong, 422 means the URLs do not match the host, 429 means you are sending too much.

Which engines take part? The IndexNow FAQ names Amazon, Bing, Naver, Seznam.cz, Yandex and Yep, and says a submission to one endpoint is shared with all of them. The protocol's own list of endpoints shows Bing, Yandex, Seznam, Naver, Yep, the Internet Archive and Amazonbot.

Does IndexNow help with Google?

Google is not on either list. We looked for an official Google statement adopting IndexNow and could not find one in Search Central documentation. So we use IndexNow for Bing, which Copilot draws on, and for the other engines that read the shared feed, and we use sitemaps and Search Console for Google. Scale is not the issue: in a 2023 anniversary post, Bing said 60 million websites were joining daily and 1.4 billion URLs were submitted each day. That is a 2023 Bing figure; we have not seen a newer verified one.

Page changes Sitemap lastmod IndexNow ping Google Bing and others
Two routes for the same change. Google reads sitemaps and Search Console; IndexNow reaches Bing and the engines that share its feed.

How our engine runs indexing every night

We do not leave indexing to memory. Our Growth Engine runs a fixed sequence after every release, and the order matters more than any single step.

  1. Deploy and verify first. No ping goes out until the new version is live and rendered on the real domain. Pinging a URL that still serves the old page wastes the crawl and can teach an engine that your signals are early.
  2. Diff the sitemap. The step compares the live sitemap with the last accepted state and keeps only URLs that are new, or whose lastmod really changed.
  3. Check the key file. If the IndexNow key file is not reachable yet, the step does nothing and marks nothing as sent. The first ping after the key goes live then carries every pending URL. In our experience a fresh key file can return 403 for a few minutes after a deploy.
  4. POST to IndexNow. In batches, within the 10,000 URL limit, and we log the response code.
  5. Queue the Google side. Money pages and new guides go into a small indexing queue. We inspect each, request indexing within the daily quota, and recheck a few days later. A URL that stays out goes back to triage, not back into the queue.

These rules are in our engine operating notes. None of it needs a paid tool.

How long should we wait before we worry?

Google's recrawl guide says crawling can take days to weeks. Our working rule: a new page on a healthy site that is linked from a crawled page should show in URL Inspection within one to two weeks. If it does not, stop requesting and start diagnosing: canonical, duplication, internal links, quality.

Robots rules and AI crawlers: decide on purpose

robots.txt controls crawling, not indexing. Google's robots.txt guide says it is mainly for managing load, that it is not a way to hide a page, and that a disallowed page can still be indexed if other pages link to it. To keep something out of results, use noindex or a login.

Two rules we never break. First, never block the scripts and stylesheets a page needs to render. Google renders pages, and a blocked asset folder can make a working page look empty to the crawler. Second, decide AI crawler access bot by bot, because the big providers now split training from search.

BotWhat it is forBlock it and you lose
OAI-SearchBot (OpenAI)Surfacing sites in ChatGPT searchAppearing in ChatGPT search answers, beyond navigational links
GPTBot (OpenAI)Content that may train foundation modelsNothing in search, per OpenAI. A training choice
ChatGPT-User (OpenAI)User triggered visits; OpenAI says robots rules may not applyNot the search control
Claude-SearchBot, Claude-User (Anthropic)Search indexing and user requested fetchesAnthropic says blocking may reduce visibility in Claude's web search
ClaudeBot (Anthropic)Training dataNothing in search. A training choice
PerplexityBotPerplexity's search results; not training, per PerplexityPerplexity citations
Google-ExtendedGemini training and grounding in Gemini Apps and Vertex AINothing in Google Search; Google says it is not a ranking signal
Applebot-ExtendedApple's opt out of model training; it does not crawlNothing in Spotlight, Siri or Safari search

Sources: OpenAI's crawler overview (undated, read 20 October 2026), Anthropic's crawler article (updated 7 April 2026), Perplexity's crawler docs (undated), Google's common crawlers (updated 15 October 2026), Apple's Applebot page (published 4 September 2026). OpenAI and Perplexity both say robots changes can take about 24 hours. Perplexity also says its user triggered fetcher generally ignores robots rules, because a person asked for the page.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /
Disallow: /account/

User-agent: *
Allow: /
Disallow: /account/

Sitemap: https://example.com/sitemap.xml

That example keeps you in most AI search while opting out of the training crawlers named. One caveat: blocking Google-Extended also stops your content grounding answers in Gemini Apps, so if Gemini matters to your buyers, leave that line out. A bot that matches its own group ignores the * group, which is why the private folder is repeated under OAI-SearchBot. It is a business decision, not a ranking trick, and it only works if your CDN or firewall does not block the search bots before they ever reach robots.txt. Check the firewall too.

Changed in 2026: a Google switch for AI features

Search Console now has a "Search generative AI" control. Google announced it on 3 June 2026 and the page notes it rolled out to all sites on 31 August 2026. The help page says excluded sites get no traffic or impressions from AI Overviews and AI Mode, that the change takes about one to two days, and that it does not stop training (that is Google-Extended) or remove you from Search (that is noindex). Almost no business that wants customers should flip it. We mention it so nobody flips it by accident.

Worked example: a guide that was live but invisible

A pattern we see often enough to describe without naming anyone. A business publishes a set of helpful guides on a subdomain. They are in the sitemap, they return 200, they read well. Three months later Search Console shows almost no impressions for them.

  1. Inspect, don't guess. URL Inspection on each guide showed only a few indexed. One page had a Google selected canonical pointing at a different site entirely, which meant Google treated it as someone else's duplicate.
  2. Find who links to them. The guides were linked only from each other and from the sitemap. No page that Google crawls often pointed at them.
  3. Fix the lever, not the copy. We added links from the main site's crawled pages, in the body and with descriptive anchors, fixed the canonical, and corrected lastmod so it showed real edit dates.
  4. Then request indexing for the few most important URLs, within the quota, and pinged IndexNow for the rest.

The lesson stays the same each time. A sitemap tells engines a page exists. Internal links from pages they already trust tell them it matters. Rewriting a page that was never indexed fixes nothing. The internal linking guide goes deeper on that lever.

For a look at our own numbers: across 19 July to 16 October 2026, 42 percent of the clicks dlvx.co got from Google search (22 of 52) came from queries Search Console does not list at all (our Search Console record). Indexing is the part you can see. A lot of what happens after it, you cannot.

What is hype, and what is a waste of an afternoon

Claim or habitVerdictWhy
"Instant indexing" services for normal pagesAgainst the rulesThe Indexing API is for job postings and livestreams only, per Google
Requesting indexing for the same URL every daySkip itGoogle says repeat requests do not speed crawling
Changing lastmod to look freshHypeGoogle and Bing both ask for real change dates; fake ones get ignored
An llms.txt file to get indexed by AIHypeGoogle says Search does not use it; see our tactic triage guide
IndexNowWorksFree, fast for Bing and partner engines. Not a Google route
Blocking GPTBot to "protect" contentDependsA fair training choice; it does not remove you from ChatGPT search if OAI-SearchBot stays allowed
Blocking everything AI at the CDNSkip itIt often catches the search bots too, and then nobody can cite you

Mistakes we fix most, and a checklist

The common mistakes, in the order we usually find them: a staging noindex that shipped to production; a sitemap listing redirects and parameter URLs; lastmod reset on every build; canonical tags pointing at the wrong host; a CDN bot rule blocking search crawlers; robots rules blocking the folder that holds scripts; new pages linked from nowhere; and people hammering request indexing instead of asking why a page is excluded.

Also check that the important text is in the HTML the server sends. Google renders JavaScript, but not every crawler or fetcher does, and plain server rendered HTML removes a whole class of doubt. See what a 200 response does not tell you for why a page that loads is not proof a crawler can read it.

  1. Search Console and Bing Webmaster Tools verified, sitemap submitted to both.
  2. Sitemap lists canonical 200 URLs only, lastmod equals the real content change.
  3. URL Inspection on every money page: indexed, and the Google selected canonical is yours.
  4. Page indexing report read by reason, not by total.
  5. IndexNow key file live at the root, pings sent only for new or changed URLs, after the deploy is live.
  6. robots.txt allows search bots, never blocks scripts or styles, and your CDN agrees with it.
  7. The generative AI control in Search Console left on include, unless you have a deliberate reason.
  8. Every new page linked from at least one page Google already crawls.

Next in the track: Structured data and your entity, without the myths.

Sources and dates

Every external fact in this guide was read on 20 October 2026. Each source shows its publisher and the date it was published or last updated, and we flag what changed in 2026. Studies are their authors' measurements, usually correlations, not promises of results. Our own numbers come from our own Search Console and tools and say so where they appear.

Want this run on your own site? Start with the AI visibility diagnostic, or see the whole Get Found track.