The problem everyone ignores

Publishing is not the same as being found. A crawler first has to discover the URL, then fetch it, then decide it is worth storing. New sites and sites with few links can wait a long time at each step, and nothing on your dashboard tells you which step you are stuck on.

Three steps from URL to index

  • Discovery: the crawler learns the URL exists, from a link, a sitemap or a direct submission.
  • Crawling: it fetches the page, if your server answers quickly and your robots.txt allows it.
  • Indexing: it stores the page, if the content is unique, useful and not marked noindex.

Start with Google Search Console

Verify your site, submit your sitemap, and use URL Inspection on important new pages. The Request Indexing button puts a page in a priority queue. It has a daily limit, so save it for pages that matter.

Then read the Pages report. "Discovered, currently not indexed" usually means Google knows the URL but hasn't spent crawl time on it. "Crawled, currently not indexed" means it fetched the page and wasn't convinced, which is a content problem, not a technical one.

Use IndexNow for Bing and others

IndexNow is a simple protocol: you host a key file on your site and send a short request listing the URLs that changed. Bing, Yandex, Naver and Seznam accept it. Google does not.

It is worth the ten minutes. Bing's index also feeds several AI assistants, so being in Bing quickly helps you appear in AI answers as well as in Bing itself.

Make your sitemap honest

  • List only pages you want indexed: no redirects, no noindex pages, no duplicates.
  • Set lastmod to the date the content really changed. Crawlers learn to ignore sitemaps that say everything changed today.
  • Reference the sitemap in robots.txt.
  • Split very large sitemaps and use a sitemap index.

Links still do most of the work

Crawlers follow links. A new page linked from your homepage or a busy hub page gets found within hours; an orphan page with no links may never be. Add new articles to a hub, link them from related pages, and use descriptive link text.

One genuine link from another site does more for discovery than any submission tool.

Let AI bots read your pages

AI assistants use their own crawlers. Some gather training data, some build a search index, and some fetch a page only when a user asks about it. Each has a name you can allow or block in robots.txt, for example GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. Google-Extended is a control token rather than a separate crawler: it governs whether Google's AI models may train on your pages.

Two practical points. Many AI crawlers don't run JavaScript, so content that only appears after scripts load is invisible to them: serve real HTML. And if you want to be cited in AI answers, don't block the search and user-triggered bots, even if you choose to block the training ones.

Common blockers

  • A leftover noindex tag or a robots.txt rule from development.
  • Canonical tags pointing at a different URL.
  • Slow or failing server responses during crawling.
  • Thin or duplicated content across many near-identical pages.
  • Redirect chains and soft 404 pages.

A word on paid indexing services

Services that promise to get any URL indexed in hours are selling crawler visits, not indexing. A visit is only the second of the three steps. No one outside the search engine can guarantee the third, and services built on networks of throwaway sites linking to your page can put you on the wrong side of spam policies. Fix the page and the links first; they are free.

A simple weekly routine

  • Publish, add internal links, and check the page renders without JavaScript.
  • Send the URL through IndexNow and request indexing in Search Console.
  • After a week, check the Pages report and fix whatever status the page is stuck in.
  • Ask an AI assistant with web search a question your page answers, and see whether you are cited.