Open your website in Chrome or Safari and everything might look perfect.
That tells you what you can see.
It does not automatically tell you what a crawler can see.
I did not think enough about that difference until I built the Marketur AI Readability Check and started testing websites I had built myself.
Some of the results surprised me enough that I think every person building websites with AI should understand what is happening underneath the design.
A browser and a crawler are not the same visitor
Your browser is incredibly capable.
It downloads HTML, CSS and JavaScript. It executes code. It makes additional network requests. It changes the page after it loads. Modern web applications depend on that behavior.
A crawler may not do all of those things the same way.
Googlebot is unusually sophisticated. Google’s current documentation says it renders JavaScript and has updated its guidance to make clear that JavaScript-loaded content is not automatically a problem for Google Search.
That is good news, but Googlebot is not every crawler.
Different search engines, AI services, agents and fetchers have different purposes and capabilities.
Access comes before readability
Before an AI system can understand a page, it has to be able to reach it.
That sounds obvious, but there are multiple places access can fail.
Your robots.txt file can disallow a crawler. A CDN or security layer can reject the request. A server can return an error. A bot policy can make a decision before your application ever receives the request.
OpenAI, for example, says publishers who want content discoverable and surfaced in ChatGPT search should not block OAI-SearchBot. OpenAI’s current publisher guidance explains this directly.
But allowing a crawler through the door is only step one.
The next question is what HTML it receives
This is where my own AI-built websites got interesting.
A website can send a relatively small application shell and depend on JavaScript to construct most of what a human eventually sees.
That architecture can work. Single-page applications are not automatically bad for SEO.
But if your important content only becomes available after client-side code runs, you are depending on the crawler to process the site the way you expect.
I would rather know.
That is why our checker looks at the readable text in the HTML it receives instead of taking a screenshot and declaring the site fine.
Multiple URLs can hide the same underlying problem
This one really caught my attention.
You can have links that appear to lead to separate pages:
/about/pricing/features/contact
To a person using the application, they can look completely different.
But depending on the architecture, an outside fetch can receive nearly the same initial document for multiple routes, with JavaScript responsible for turning them into different screens.
Again, that can be implemented correctly. The problem is not knowing whether yours was.
I ran into this question while working with AI builders, which is why I wrote about AI builders creating pages crawlers may not read the way you expect.
Then there are the SEO signals around the content
Readable text is not the entire job.
I also want important pages to have useful titles, headings, language information, structured data where appropriate and clear internal links.
Google says structured data helps it understand page content and can make pages eligible for certain search features. Google’s structured data documentation goes into the details.
For JavaScript sites, Google also documents issues around crawlable links, status codes and canonical URLs. Its JavaScript SEO guide is a good technical reference.
Why I built a checker instead of another SEO score
I wasn’t interested in making another tool that gives a website 83 out of 100 because a title is two characters too long.
I wanted answers to more basic questions.
Can an AI crawler get in?
Is there meaningful content when it does?
Do deeper pages actually expose different readable content?
Is infrastructure sitting in front of the site that could affect bot access?
Does the site expose enough structure to make the content understandable?
Those questions became much more important to me after the tool exposed things on my own sites that I had not noticed while building them.
AI search and Google search overlap, but they are not identical
I think this is where people can get themselves in trouble.
They see a page indexed by Google and assume every AI system must therefore be able to fetch and understand it.
Or they see an AI bot allowed in robots.txt and assume that proves the site is readable to that system.
Neither assumption tells the whole story.
Google Search, AI search crawlers, training crawlers and user-triggered fetchers can have different purposes. I broke down one important example in GPTBot vs OAI-SearchBot and Why the Difference Matters.
See what your own site exposes
This is one of those things I would rather test than argue about.
Run your website through the Marketur AI Readability Check.
It is free to try and it checks the public site from the outside. It cannot see private settings inside your hosting account or CDN, and it does not pretend it can. When something cannot be verified externally, the report should tell you that.
That is the whole reason I built it.
I didn’t know some of my own sites had these problems until the tool brought them to my attention.
You might want to check yours.