Verify AI Bots Are Real with Reverse DNS

Blocking the wrong bots can cost you AI visibility. Allowing the wrong bots can cost you bandwidth and content. This guide helps you draw the line in the right place.

In this guide you will learn how to confirm a crawler is genuine and not spoofed. We will keep it practical, with clear steps, visual breakdowns, and specific actions you can take today. The first step in any AI visibility project is to run a free AI crawler check on your website so you know exactly where you stand against the 196 bots we track across 8 categories.

Blocking Bad Traffic Without Blocking Citations

Security controls and AI visibility pull in opposite directions, so the sequence is designed to keep the two decisions separate rather than letting one silently override the other.

Why Bot Identity Verification Is Harder Than It Looks

A user agent string is a claim, not an identity. Anyone can send any user agent they like, which means the header that tells you a request came from a major AI crawler is exactly as trustworthy as the sender chose to make it. Treating that header as identification is the root of most bot policy that does not work, because both halves fail at once: the impostors get the access you granted to the real crawler, and your logs credit them for it.

The textbook answer is a forward-confirmed reverse DNS lookup, and it is the right answer when it is available. What the guides tend to leave out is that it often is not. Only a small minority of the operators in the bot database publish the address ranges or hostname conventions that make the check possible. So a verification policy built on the assumption that every claim can be confirmed will quietly fail open on most traffic, which is worse than having no policy, because it produces a record that looks like verification and is not.

Four Verification Rules Worth Knowing

Confirm in both directions, because one direction proves nothing

The check has two halves and the second is the one people skip. Resolve the request address back to a hostname, then resolve that hostname forward to an address and confirm it matches the address you started with. Reverse records are controlled by whoever holds the address, so a reverse lookup on its own can claim any name at all. Only the round trip ties the address to a domain the operator actually controls, and only that is evidence.

Decide what happens when verification is impossible, not just when it fails

A rule that handles verified and unverified traffic still has a gap, because most claims fall into neither. When an operator publishes no ranges and follows no hostname convention, the check cannot return a verdict at all. Write the third branch explicitly. Reasonable answers include treating the traffic by behaviour rather than identity, or granting it the access you would give an anonymous visitor. What matters is that the case is decided in advance rather than resolving to accidental trust.

Prefer the operator published ranges when they exist, and cache them

Several operators publish their address ranges as a file that is meant to be fetched and cached. Where that exists it is more reliable than a name lookup and much cheaper, because it needs no per-request resolution. Fetch it on a schedule, keep the previous copy, and alert if it fails rather than falling back to allowing everything. The failure mode to design against is the fetch breaking silently and the rule degrading into an unconditional allow.

Never verify on the hot path without a cache

A name lookup on every request turns an outside service into a hard dependency of your own latency, and a slow resolver becomes a slow site. Verify once per address, cache the verdict with an expiry, and make sure a resolver timeout produces your explicitly chosen default rather than whatever the surrounding code happens to do on error. Signed agent authentication is starting to appear and will eventually make some of this unnecessary, but it is present on very few entries today and cannot be relied on yet.

The Most Expensive Misunderstanding About Bot Identity Verification

Logging the user agent and calling the traffic identified

The expensive version of this mistake is not being fooled once. It is building a reporting pipeline on the assumption that the header is true, then making decisions from the output for months. Crawler dashboards grouped by user agent read as fact, get quoted in reviews, and drive access changes, while the underlying figure is a mixture of the real crawler and anything that copied its string. The correction is unglamorous: record verification status as its own field next to the claimed agent, and treat any figure that lacks that field as an upper bound rather than a count.

What to Test About Bot Identity Verification Before Moving On

Where to Go From Here

Verification is one piece of a larger picture. The AI bot directory documents every crawler we track with its operator, purpose and safety rating, and the batch checker audits many sites in one pass if you manage a portfolio.

Blocking hostile scrapers while staying visible to search-and-cite bots is a balance, not a switch. Verify both sides with an AI crawl checker.

Your Verification Action Checklist

Five concrete steps, specific to what this guide covered. Work through them in order, changing one thing at a time so you can tell which change produced the result.

• □Establish a baseline with an AI crawler test and write down the score before you change anything.
• □Apply the single highest-impact change from this guide, on its own, so you can attribute the result.
• □Validate the change with the robot checker before it reaches production.
• □Re-measure and compare against your baseline rather than against expectation.
• □Schedule a recurring re-check, because redesigns and security updates quietly undo this work.