Your robots.txt file is the single most powerful lever you have over how crawlers, including AI bots, interact with your website. Get it right and you control your AI visibility. Get it wrong and you can disappear from AI answers overnight.
In this guide you will learn how to allow citation bots while blocking training scrapers. We will keep it practical, with clear steps, visual breakdowns, and specific actions you can take today. The first step in any AI visibility project is to run a free AI crawler check on your website so you know exactly where you stand against the 196 bots we track across 8 categories.
Key Takeaways
- Block AI Training but Allow AI Search: The Hybrid Setup is a practical, repeatable process, not a one-time fix.
- Most AI visibility problems trace back to access, not content.
- You can verify every change with the free AI crawler check and the robots.txt validator.
- Document your approach so the whole team applies it consistently.
How to Change robots.txt Without Breaking Anything
robots.txt failures are rarely subtle in effect and almost always subtle in appearance, so the sequence is built around proving each change before the next one.
Read the file that is actually served
Request robots.txt from the live domain rather than opening the copy in your repository or your CMS settings screen. Those three frequently disagree, and only one of them is what crawlers see.
Identify which rule matches which agent
Group rules by user-agent block and work out which block a given crawler will obey, remembering that the most specific matching block wins and that a crawler obeys exactly one. This is where most misdiagnosis happens.
Change one thing, in one block
Broad rewrites make attribution impossible when something breaks. Edit a single directive, note what you expect to change, and generate the replacement with a robots.txt generator rather than by hand.
Test the rule against real paths before publishing
Run the edited file through a robots checker with the specific URLs you care about, including at least one you intend to block and one you intend to allow. A rule that passes syntax checking can still match nothing.
Verify from outside, then record what you changed
Fetch the live file again and re-run an AI crawler checker. Then write down the change and the reason, because the next person to debug this file will otherwise be guessing at your intent.
Why the Training and Citation Split Is Harder Than It Looks
The position most publishers and businesses actually want is specific: do not use my content to train a model, but do cite me when answering a live question. robots.txt has no syntax for that distinction, so the entire policy has to be expressed as a list of which named crawlers you permit.
That makes the work factual rather than strategic. You need to know which agent belongs to which purpose, and the answer changes as vendors add crawlers. A policy that was correct when written becomes wrong through no action of yours, which is why this needs a review interval rather than a decision.
Where People Get the Training and Citation Split Wrong
Learn which agent does which job, per vendor
OpenAI operates separate crawlers for training and for answering live queries, and Google separates its search crawler from its AI training signal. Blocking the vendor by guessing at names catches the wrong one about as often as the right one. The list is short, it is documented, and reading it is the highest-value ten minutes in this whole exercise.
A named block must restate the wildcard protections you still want
Because a crawler follows only its most specific matching block, adding a named block for a training bot removes it from every rule under User-agent: *. If those wildcard rules protected paths you care about, they have to be repeated inside the new block. Skipping this is how a tightening change becomes a loosening one.
Verify from outside, because your firewall may disagree
A robots.txt that permits retrieval crawlers means nothing if a CDN bot-management rule blocks them at the edge. These are configured by different people and neither is visible from the other, so the only reliable check is an external fetch that reports what a crawler actually receives.
Expect the list to change and schedule the review
New crawlers appear, existing ones split into specialised agents, and vendors rename them. A policy is only as current as its last review, and nothing notifies you when it goes stale. A recurring calendar entry is a more effective control than any amount of care taken on the day.
The Training and Citation Split Mistake That Costs Most
Blocking a vendor wholesale to stop training
When the concern is training, the instinct is to block everything belonging to that vendor. This also blocks the retrieval agent that would have cited you with a link, so you give up the traffic you wanted while frequently failing to stop the training you objected to, because the training crawler was named differently. The loss appears as absence and nothing flags it.
The One Training and Citation Split Check That Settles It
The check that matters here: For each major vendor, name the crawler you are blocking and the crawler you are allowing. If you cannot name both, the policy is a guess rather than a decision.
Where to Go From Here
Training and Citation Split is one piece of a larger picture. The full list of AI crawlers documents every crawler we track with its operator, purpose and safety rating, and the multi-site crawler check audits many sites in one pass if you manage a portfolio.
Syntax errors in robots.txt fail silently, so validate before you publish. Use the robots.txt check, then confirm the result with an AI crawler checker.
Your Training and Citation Split Action Checklist
Five concrete steps, specific to what this guide covered. Work through them in order, changing one thing at a time so you can tell which change produced the result.
- □Back up your current robots.txt before editing, so you can revert if a rule behaves unexpectedly.
- □Apply the pattern from this guide to a copy first, then diff it against the live file.
- □Run the robots checker to confirm the syntax resolves the way you intend.
- □Check that no Disallow rule shadows a path you meant to allow, which is the most common silent failure.
- □Re-crawl after the change, because engines cache robots.txt and the effect is not instant.
