Free SEO Tool
Control how search engine bots crawl your site and protect your private directories.
Configure Rules
Crawler Access
Block These Paths
Add Custom Path
Seconds between each request. Leave blank unless your server needs throttling.
robots.txt Preview
2 linesUser-agent: *
Disallow:How It Works
Allow all, block all, or custom rules
Quick-fill chips or add your own
Helps crawlers find your sitemap
Place the file at yoursite.com/robots.txt
FAQ
robots.txt is a plain text file placed at the root of your website that tells search engine crawlers which pages or sections they are allowed or not allowed to crawl.
No. Disallowing a URL only stops crawling — Google may still index the URL if other pages link to it. To remove a page from Google's index, use a noindex meta tag or the URL Removal Tool in Search Console.
Upload it to the root of your domain so it is accessible at yourdomain.com/robots.txt. It must be at the top-level domain, not a subdirectory.
That is your choice. Blocking AI crawlers prevents bots like GPTBot (OpenAI) and CCBot from using your content to train AI models. It has no effect on Google, Bing, or your SEO rankings.