Robots.txt Generator
A robots.txt generator produces a correctly formatted robots.txt file — the plain-text file at a site's root that tells search engine crawlers which paths they may or may not crawl, and where to find the XML sitemap. A robots.txt file only asks well-behaved crawlers not to fetch certain paths — it has no way to enforce that request, and a disallowed page can still appear in search results, with no preview, if other pages link to it, which is why blocking indexing reliably requires a noindex meta tag instead.
How it works
- Enter a user-agent (or leave as * for all crawlers).
- List paths to disallow and, optionally, paths to explicitly allow — one per line.
- Optionally add your sitemap URL.
- Copy the result into a file named robots.txt at your site's root.
Formula Template assembly: User-agent line, Disallow/Allow lines, optional Sitemap line
Frequently asked questions
Does robots.txt stop a page from appearing in search results?
Not reliably — it stops crawling, but a disallowed URL can still be indexed (with no content shown) if other pages link to it. Use a noindex meta tag instead to prevent indexing.
What if I want to block everything?
Add "/" as a disallow path — but be certain, since this blocks your entire site from search engines.
What if I list no disallow paths at all?
The generator emits a bare "Disallow:" line, which means "disallow nothing" — the whole site is crawlable, the standard default-open configuration.
Where does this file need to go?
At the root of your domain, e.g. https://example.com/robots.txt — it is not recognized anywhere else.
Is my configuration uploaded anywhere?
No. Generation runs entirely in your browser.