What does Disallow: / do?

It blocks that user-agent from crawling your entire site. Leave paths empty to allow crawling everywhere.

Where do I put robots.txt?

At the root of your domain, e.g. https://example.com/robots.txt. It will not work in a subfolder.

Generate a robots.txt file for search engines.

Build a robots.txt file that controls how search engine crawlers access your site.

User-agent

Disallow (one path per line)

Allow (one path per line)

Sitemap URL (optional)

User-agent: *

Crawl your site and generate an XML sitemap automatically.

Generate a manifest.json for your PWA.

Generate SEO meta tags for any page.

This tool runs entirely in your browser. Your data is never uploaded or sent to our servers.