There is a mistake I think we are going to see a lot with AI search.
People will add one line to robots.txt, decide their site is now “optimized for AI” and move on.
Crawler access matters, but access is only the front door. Once a crawler gets inside, the website still has to make sense.
That is one reason I made the Marketur AI Readability Check look beyond robots.txt. I want to know what a machine actually receives when it requests the page.
Put the important content in the HTML
A browser can do a lot after a page initially loads. JavaScript can fetch content, assemble interfaces and replace half the screen.
That can be great for users, but when I care about discovery I also care about what exists in the HTML response itself.
Your main explanation of what the page is about should not depend on a fragile client-side interaction if it does not need to.
Make the page topic obvious
The boring SEO basics are still useful.
A descriptive title, a useful meta description, a clear main heading and a sensible canonical URL give machines straightforward signals about the page. None of these guarantees that an AI system will cite you. They simply remove unnecessary ambiguity.
I would rather make a page obvious than clever.
Tell machines who the business is
This matters even more on business websites.
If a page says “we build websites” but gives weak signals about who “we” are, where the business lives online or which social profiles belong to it, a machine has more identity resolution to do.
Organization and LocalBusiness structured data can provide explicit information such as the organization’s name, URL and related profiles. That is not magic AI optimization. It is simply cleaner machine-readable identity.
Do not confuse AI controls with Google Search controls
Some AI-related robots.txt tokens control specific uses rather than ordinary search indexing.
Google, for example, says Google-Extended controls certain uses of content for Gemini training and grounding and does not affect whether the site appears in Google Search. Google documents the distinction here.
OpenAI likewise tells publishers that OAI-SearchBot access matters for content they want included in ChatGPT search summaries and snippets. OpenAI’s current publisher guidance is here.
There is no magic AI SEO file
You will also hear a lot about files and markup specifically created for language models. Some ideas may become useful standards and some may disappear.
I would not let a shiny new file distract me from the fundamentals: accessible pages, useful content, clear identity, clean metadata, sensible internal links and crawler rules that match what I actually want.
See what a crawler gets from your site
If you want a quick starting point, run your URL through the free Marketur AI Readability Check.
It checks crawler access and the page signals that help explain what the page and business are. Then you can fix the things that are actually missing instead of guessing at a new kind of SEO.