Free · No sign-up

Robots.txt Generator

Build a robots.txt file with rules per crawler, your sitemap, and a one-click preset that blocks AI crawlers.

Your robots.txt

How to use it

A robots.txt in three steps.

  1. 01

    Choose the crawlers

    Start with * for every crawler, then add a group for any crawler that needs its own rules.

  2. 02

    Set allow and disallow paths

    List one path per line. A group with no rules allows everything. Use the preset to block the main AI crawlers.

  3. 03

    Upload it to your site root

    Save the text as robots.txt at the root of your domain, then check that the right crawlers can reach you with the AI Crawler Checker.

Go further · AI Visibility

Track whether AI assistants cite you.

AI Visibility shows where ChatGPT, Gemini and Perplexity mention your brand, so you can decide which crawlers to welcome.

See AI Visibility

FAQ

Questions, answered.