Your shop is beautiful, well stocked and open. It is also not in the phone book, the front gate has a padlock the delivery driver cannot open, and your building manager has quietly instructed security to turn away anyone in an unfamiliar uniform.
None of these are shop problems. All of them are why nobody comes.
The four gates, in the order they fail
Gate one — robots.txt
Covered in Chapter 2, repeated here because it is the most common single failure. Open yourdomain.com/robots.txt. You are looking for anything that blocks OAI-SearchBot, PerplexityBot, Claude-SearchBot, or a blanket User-agent: * with a broad Disallow.
A reasonable default for a business that wants AI visibility but does not want to donate to model training:
Allow the search bots — OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot. These are how you get cited.
Block the training bots if you wish — GPTBot, Google-Extended, Applebot-Extended, CCBot. This is a genuine choice with no direct visibility cost.
You will see claims that blocking any given bot costs a precise percentage of citations. Those numbers are invented. The honest statement is: blocking a search bot removes you from that engine; blocking a training bot does not.
Gate two — your firewall and CDN
This is the invisible one. Cloudflare, AWS WAF, Sucuri and similar services block traffic they consider non-human, and AI crawlers frequently qualify. Your robots.txt can be perfect while your edge quietly returns 403 to every bot that matters. Nothing in your analytics will tell you.
The landscape here is also shifting under you commercially. Cloudflare moved through 2026 from a pay-per-crawl model toward paying based on whether content was actually used in an answer, and announced that from 15 September 2026 its default settings would block mixed-use crawlers from pages hosting ads unless the site owner changes the setting. TIER B The specifics will keep moving. The lesson will not: somebody other than you may be deciding whether AI can read your website, and the default may not be what you want.
Gate three — Bing
Because ChatGPT and Copilot retrieve substantially through Bing's index, a site that is well-indexed in Google and poorly indexed in Bing is invisible to a large share of AI users. This is the single most under-done piece of technical work in the field, because Bing has been irrelevant for so long that nobody checks it.
- Verify the site in Bing Webmaster Tools. You can import directly from Google Search Console, which takes minutes.
- Submit your sitemap.
- Turn on IndexNow — an open protocol that pushes new and changed URLs to Bing instantly rather than waiting for a crawl. Most CMS platforms have a plugin. It is free.
- Confirm
Bingbotis not blocked at the firewall. It very often is.
One honest caveat: IndexNow tells Bing a URL changed. It does not guarantee the page is indexed or surfaced. It removes a delay; it does not manufacture relevance.
Gate four — sitemap and URL hygiene
Given that roughly a third of AI-crawler fetches hit 404s, an accurate sitemap is worth more now than it was two years ago. Practical rules:
- Your sitemap should list only live, indexable, canonical URLs. No redirects, no 404s, no
noindexpages. - When you move a page, 301 it. Permanently. Do not delete it.
- Keep
lastmoddates truthful. Falsifying them across the whole site to fake freshness is a well-known and easily detected pattern.
- Read
robots.txtline by line. - Compare
site:yourdomain.comcounts on Google and Bing. A large gap is a finding. - Ask whoever manages your hosting to grep the server logs for
OAI-SearchBot,PerplexityBotandClaudeBotover the last 30 days. Zero visits means a gate is shut. - Open your sitemap and click ten URLs at random. Count how many are not a live 200 page.
- Check your CDN or WAF bot-management settings for anything set to "block AI bots" or "block unknown bots".
Get someone to look at your raw server logs for AI crawler hits. It is the only way to know for certain that the machine is being let in, and it answers in one query what a month of guessing will not.