Twitterbot (X (Twitter)): User Agent & robots.txt Rules

Q: What is the official user-agent string for Twitterbot?

The official user-agent string for Twitterbot is: Twitterbot. This is the exact string you must use in robots.txt, Nginx, Apache, or Cloudflare firewall rules to target this bot. User-agent matching in robots.txt is case-insensitive, but the string must be spelled correctly. You can verify that a request genuinely comes from Twitterbot by performing a reverse-DNS lookup on the source IP - legitimate bots resolve back to their operator's domain.

Q: How do I block Twitterbot in robots.txt?

Add the following lines to your /robots.txt file: User-agent: Twitterbot Disallow: / This instructs Twitterbot not to crawl any path on your site. The Disallow: / directive covers the entire domain including subfolders. To only block specific sections, replace / with the path (e.g., Disallow: /blog/). Note: robots.txt is publicly readable - any bot or human can inspect it at yourdomain.com/robots.txt.

Q: How do I verify if Twitterbot is crawling my site?

Search your web server access logs for the string Twitterbot (case-insensitive grep: grep -i "Twitterbot" /var/log/nginx/access.log). You can also check Google Search Console → Coverage → Crawl Stats for Googlebot variants. For Twitterbot specifically, filter by user-agent in your log analysis tool (GoAccess, AWStats, etc.).

Quick Facts

User-Agent:: Twitterbot
Category:: Social Media Bots
Operator:: X (Twitter)
Safety:: Safe
Blocking Impact:: Varies - Evaluate before blocking
SEO Impact Score:: 0/10
Official docs:: developer.twitter.com

What is Twitterbot?

Twitterbot (now X) fetches content to generate 'Cards' (previews) when links are posted on the X platform.

Twitterbot (now X) fetches content to generate 'Cards' (previews) when links are posted on the X platform. Twitterbot is operated by to generate rich link previews when URLs are shared on their platform. It sends GET requests to your URL, reads <meta property="og:..."> and <meta name="twitter:..."> tags, and caches the result. Blocking Twitterbot means all links to your domain shared on appear as raw text without thumbnail, title, or description. This can reduce CTR from social referrals but has zero SEO impact.

What happens if you block Twitterbot?

❓ **Impact Unknown** - The SEO consequences of blocking Twitterbot are not fully documented. Before blocking, check your analytics to confirm whether this bot generates referral traffic, review your server logs for crawl frequency, and test in a staging environment if possible.

Generally safe to allow; provides legitimate crawling value.

How to block Twitterbot with robots.txt

<code>User-agent: Twitterbot</code> - Matching is case-insensitive. Robots.txt is fetched from the root of each subdomain separately.

Block completely (robots.txt)

User-agent: Twitterbot
Disallow: /

Allow all (robots.txt)

User-agent: Twitterbot
Allow: /

Block private only (robots.txt)

User-agent: Twitterbot
Disallow: /private/
Disallow: /api/
Disallow: /admin/
Allow: /

Nginx server block

# Nginx: Hard-block Twitterbot
if ($http_user_agent ~* "Twitterbot") {
    return 403 "Bot blocked";
}

Apache .htaccess

# Apache: Hard-block Twitterbot
SetEnvIfNoCase User-Agent "Twitterbot" bad_bot
Order Allow,Deny
Allow from all
Deny from env=bad_bot

Meta robots tag

X-Robots-Tag header

X-Robots-Tag: noindex, nofollow

Is Twitterbot safe to allow?

Yes, Twitterbot is a **safe and legitimate** crawler. It is operated by , which publicly documents its crawler at an official URL and follows the Robots Exclusion Protocol (RFC 9309). The user-agent string Twitterbot is verifiable via reverse-DNS lookup on the crawling IP addresses. You can safely allow it unless you have a specific reason to block (e.g., AI training opt-out or SEO tool visibility).

Verify by reverse-DNS lookup: legitimate Twitterbot requests resolve to 's domain.

What does Twitterbot do?

Understanding Twitterbot's purpose helps you decide whether to allow or block it.

Generates link previews for social media shares
Fetches Open Graph and Twitter Card metadata
Creates rich snippets when links are posted
Validates and displays website thumbnails
Improves user experience on social platforms

Frequently Asked Questions

What is the official user-agent string for Twitterbot?

The official user-agent string for Twitterbot is: Twitterbot. This is the exact string you must use in robots.txt, Nginx, Apache, or Cloudflare firewall rules to target this bot. User-agent matching in robots.txt is case-insensitive, but the string must be spelled correctly. You can verify that a request genuinely comes from Twitterbot by performing a reverse-DNS lookup on the source IP - legitimate bots resolve back to their operator's domain.

Is Twitterbot safe?

Yes, Twitterbot is a **safe and legitimate** crawler. It is operated by , which publicly documents its crawler at an official URL and follows the Robots Exclusion Protocol (RFC 9309). The user-agent string Twitterbot is verifiable via reverse-DNS lookup on the crawling IP addresses. You can safely allow it unless you have a specific reason to block (e.g., AI training opt-out or SEO tool visibility).

Will blocking Twitterbot hurt my SEO?

❓ **Impact Unknown** - The SEO consequences of blocking Twitterbot are not fully documented. Before blocking, check your analytics to confirm whether this bot generates referral traffic, review your server logs for crawl frequency, and test in a staging environment if possible.

How do I block Twitterbot in robots.txt?

Add the following lines to your /robots.txt file:

User-agent: Twitterbot
Disallow: /

This instructs Twitterbot not to crawl any path on your site. The Disallow: / directive covers the entire domain including subfolders. To only block specific sections, replace / with the path (e.g., Disallow: /blog/). Note: robots.txt is publicly readable - any bot or human can inspect it at yourdomain.com/robots.txt.

Does Twitterbot respect robots.txt?

Yes - Twitterbot is a well-behaved bot operated by X (Twitter). It fetches and parses /robots.txt before crawling any page, following RFC 9309.

How do I verify if Twitterbot is crawling my site?

Search your web server access logs for the string Twitterbot (case-insensitive grep: grep -i "Twitterbot" /var/log/nginx/access.log). You can also check Google Search Console → Coverage → Crawl Stats for Googlebot variants. For Twitterbot specifically, filter by user-agent in your log analysis tool (GoAccess, AWStats, etc.).

What is the crawl frequency of Twitterbot?

Twitterbot crawls at a moderate rate. If you notice excessive traffic in your logs, you can add a Crawl-delay directive:

User-agent: Twitterbot
Crawl-delay: 10

(10 second delay between requests).

Can I block Twitterbot from specific pages only?

Yes. Instead of a global Disallow: / you can restrict Twitterbot to specific paths:

User-agent: Twitterbot
Disallow: /private/
Disallow: /staging/
Allow: /

This allows Twitterbot everywhere except the listed paths. Path matching in robots.txt uses prefix matching - Disallow: /private/ blocks /private/page.html but NOT /public/private/.

Why is my link preview broken when shared on 's platform?

If links to your site appear without a preview image or title on , it's likely because: 1. Twitterbot (Twitterbot) is blocked in your robots.txt. 2. Your Open Graph meta tags are missing or malformed. 3. 's cache is stale - request a refresh using 's debugger tool. Fix: Remove any block rule for Twitterbot and ensure your pages include <meta property="og:title">, og:description, and og:image.