Can an AI actually read your site?

In 2024 a lot of people blocked "the AI bots". Since then the crawlers have split in two: one set collects content to train models, a different set builds the index behind the answers ChatGPT, Claude and Perplexity give. Most sites still have the setting they chose before that split, which often means they are invisible to AI search while still being trained on.

This reads your robots.txt properly, tells you which of those two things you are actually doing, and checks whether your content is even in the HTML a crawler receives. Then it writes the rules for whichever outcome you want.