100% Free · Runs In Your Browser
Free Robots.txt Generator
Answer a few simple questions on the left, and get a clean, ready-to-upload robots.txt file on the right — updated instantly as you go. Nothing you type is sent anywhere; this tool runs entirely on your own device.
Robots.txt Generator
Copy or download the file above.
Name the file exactly robots.txt (all lowercase).
Upload it to your site's root folder — via your host's File Manager or FTP — so it's reachable at yourdomain.com/robots.txt, not inside a subfolder.
Double-check it by visiting that address in your browser after uploading.
The Basics
What Is Robots.txt, In Plain Words?
Think of robots.txt as a small note pinned to the front door of your website, written for search engine robots instead of people. It doesn't lock any doors — it simply says "please come in here" or "please don't go in there," and well-behaved crawlers like Google's read it before they start browsing your site.
Most websites don't need to block anything at all — the safest and most common setup is simply allowing every search engine to crawl every page, and pointing crawlers to your sitemap so they can find your content faster. Blocking becomes useful mainly for things visitors don't need indexed, like an admin login page or an internal search results page.
One important thing to know: robots.txt is a request, not a lock. It relies on crawlers choosing to respect it — reputable ones like Googlebot do, but it isn't a security tool, so never rely on it to hide anything truly private.
Watch Out For These
Common Robots.txt Mistakes
Small errors here can quietly stop your whole site from showing up on Google.
Blocking the Entire Site by Accident
Leaving a stray "Disallow: /" rule in place after a site launches.
Uploading It to the Wrong Location
Placing the file inside a subfolder instead of the site's root directory.
Using It to Hide Private Content
Assuming a Disallow rule keeps a page secret or secure.
Forgetting the Sitemap Line
Publishing a robots.txt file without linking to the sitemap.
Copying Another Site's File Blindly
Reusing a robots.txt found online without checking if its rules fit your own site.
Never Checking It Again
Uploading the file once and never revisiting it as the site changes.
FAQ
Robots.txt: Frequently Asked Questions
What is a robots.txt file?
Robots.txt is a small text file placed at the root of your website that tells search engine crawlers which parts of your site they're allowed to visit and which parts to leave alone.
Do I need a robots.txt file?
Not strictly — search engines will crawl your site without one. But having a simple, correct robots.txt file that allows crawling and points to your sitemap is considered good practice and removes any ambiguity for crawlers.
Where do I upload my robots.txt file?
It must be placed in the root folder of your website so it's reachable at yourdomain.com/robots.txt — not inside a subfolder. Most hosting control panels (like Hostinger's File Manager) or an FTP client can upload it there directly.
Can robots.txt stop my pages from ranking on Google?
Yes, if it's set up incorrectly. Blocking an important page or folder in robots.txt can prevent Google from crawling it at all, which can quietly remove it from search results. Always double-check what you're blocking before uploading.
What does blocking AI crawlers actually do?
It adds rules asking AI companies' crawlers (like OpenAI's GPTBot or Google's Google-Extended) not to use your content to train their models. Reputable AI companies respect these requests, though robots.txt relies on voluntary compliance rather than being technically enforced.
Will this tool submit my website anywhere?
No. This generator runs entirely in your browser — nothing you type is sent to a server or stored anywhere. You copy or download the result and upload it to your own site yourself.
Is it better to block or allow everything?
For almost every live website, allowing search engines to crawl everything is the right default. Blocking specific folders only makes sense for things like admin areas, internal search results, or duplicate content you don't want indexed.
How do I check if my robots.txt file is working?
After uploading, visit yourdomain.com/robots.txt directly in your browser to confirm it loads and shows the correct rules. You can also check it inside Google Search Console's settings.