search
format_align_left
JSON Formatter CSS Formatter HTML Formatter JavaScript Formatter XML Formatter SQL Formatter Exception Formatter Markdown Preview & HTML to Markdown
data_object
JSON to Excel/CSV JSON to TypeScript, C#, Go, Java, Kotlin, Python & Dart JSON ⇄ YAML / TOML Converter JSON Diff – Compare Two JSON JSON Viewer & JSONPath Tester
table_chart
Excel/CSV to JSON Excel/CSV to SQL Excel/CSV to HTML
description
Compare text Text to Hex Text to Base64 Case Converter Regex Tester Vietnamese Accent Remover & Slug Generator Word Counter Line Tools: Remove Duplicates, Sort Lines Lorem Ipsum Generator
image
Convert Image Image to Base64 Converter Image Compressor Resize & Crop Image Favicon Generator Color Palette from Image EXIF Viewer & Remover
qr_code_2
QR Code Generator QR Code Scanner VietQR Bank Transfer QR Generator Wi-Fi QR Code Generator vCard QR Code Generator (Business Card) Barcode Generator (EAN-13, Code 128, Code 39)
lock
Convert Token Convert Hash Generate Password HMAC Generator (SHA-256, SHA-512) AES Encrypt / Decrypt Online Bcrypt Hash Generator & Checker
schedule
Convert datetime Timezone Info Vietnamese Lunar Calendar Converter Date Calculator Cron Expression Parser Countdown Timer
link
Encode and Decode URLs Shorten URL URL Parser & UTM Builder HTTP Header Checker
storage
Statistics Parser Statistics Compare SQL INSERT to JSON Converter SQL Table to C# / TypeScript / Java Class
travel_explore
Create Sitemap robots.txt Generator Meta Tag Generator Open Graph Preview
terminal
Number Base Converter UUID / ULID / NanoID Generator Chmod Calculator HTTP Status Codes
palette
Color Picker
help_outline
About UsPrivacy PolicyTerms of Service
THUTOOL
search

robots.txt Generator

🔒 Processed 100% in your browser – your data never leaves your device
All crawlers (User-agent: *)

Bing and Yandex respect Crawl-delay; Google ignores it.

AI crawlers

Blocks bots that collect content for AI training or AI answers. Search engines are not affected.

GPTBotOpenAI (training)
ChatGPT-UserOpenAI (ChatGPT browsing)
OAI-SearchBotOpenAI (SearchGPT)
ClaudeBotAnthropic
Claude-UserAnthropic (user fetch)
anthropic-aiAnthropic (legacy)
Google-ExtendedGoogle Gemini / Vertex AI
Applebot-ExtendedApple Intelligence
CCBotCommon Crawl
PerplexityBotPerplexity
BytespiderByteDance
AmazonbotAmazon
Meta-ExternalAgentMeta AI
cohere-aiCohere
DiffbotDiffbot
Search engine bots
GooglebotGoogle Search
Googlebot-ImageGoogle Images
BingbotMicrosoft Bing
SlurpYahoo
DuckDuckBotDuckDuckGo
BaiduspiderBaidu
YandexBotYandex
ApplebotApple
coccocbot-webCốc Cốc
YetiNaver
Sitemap

Upload the file to the root of your domain, e.g. https://example.com/robots.txt.


How to create a robots.txt file

  1. Choose the rule for all crawlers: allow crawling (and list the paths to block) or block everything.
  2. Add paths to block, one per line, for example /admin/ or /*?sort=, and exceptions under "Allowed paths".
  3. Tick Block AI crawlers to stop GPTBot, ClaudeBot, CCBot, Google-Extended and other AI bots, or set a rule for each search engine bot.
  4. Add your sitemap URL, then copy or download robots.txt and upload it to the root of your domain.

Example

User-agent: *
Disallow: /admin/

User-agent: GPTBot
User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

FAQ

Does robots.txt hide a page from Google? No. It only stops crawling; a blocked URL can still be indexed if other sites link to it. Use a noindex meta tag to keep a page out of search results.

If I add a rule for Googlebot, does it still follow the * rules? No. A crawler uses only the most specific group that matches it, so the tool copies your blocked paths into the group of allowed bots.

Does blocking AI crawlers affect my Google ranking? No. Google-Extended only controls use in Gemini; Googlebot still crawls your site for search.

Related tools

  • Create SitemapEffortlessly create a sitemap for your website by crawling and capturing all URLs with our powerful online sitemap generator. Our tool will crawl through your website, collecting all accessible URLs, and generate a comprehensive sitemap file for easy submission to search engines. With our user-friendly interface and efficient crawling process, you can quickly create a sitemap that ensures better indexing and visibility for your website. Start using our sitemap generator now and improve your website's search engine optimization (SEO) and overall online presence.
  • Meta Tag GeneratorGenerate title, meta description, canonical, Open Graph and Twitter Card tags with length counters and Google, Facebook and X previews. Free.
  • Open Graph PreviewEnter a URL to preview how it looks on Facebook, Zalo and X. Checks og:title, og:image, twitter:card, canonical and title/description length.
  • JSON to Excel/CSVConvert JSON to Excel (XLSX) or CSV online for free. Paste or upload JSON and download a ready-to-use spreadsheet in seconds.
  • Excel/CSV to JSONConvert Excel (XLS, XLSX) or CSV files to JSON instantly. Free online converter, your file is processed in the browser and never uploaded.
  • Convert datetimeConvert Unix timestamps (seconds or milliseconds) to readable dates and dates back to timestamps, with ISO 8601 and RFC 2822 output.
@Made by THUTOOL