Skip to content
Kiln
Tools

SEO

Robots.txt tester

Fetch one site's robots.txt and see whether a path is allowed or disallowed for a crawler user agent.

This one uses a server lookup because the job cannot finish on the device alone.

A public hostname or website URL. Kiln fetches /robots.txt for that site.

Leave this blank to use the path from the website URL, or / when you entered only a hostname.

Leave this blank to test Googlebot. The longest matching user-agent group is used.

Examples

Load a sample into the form, then run it.

How to use it

  1. 1Enter a public hostname or website URL.
  2. 2Enter the path you want to test. Leave it blank to use / or the path already in the URL.
  3. 3Enter a crawler user agent, or leave it blank to test Googlebot.
  4. 4Read the matching rule, the user-agent group, and the sitemap lines. Copy the result if you need it.

When it helps

  • Check whether a documentation path is open to Googlebot.
  • See which Allow or Disallow rule wins.
  • List the sitemap addresses published in the file.

Good to know

  • The match follows the longest-rule behavior documented for Google's robots.txt parser. The most specific user-agent group is used, the longest matching path wins, and Allow wins when the lengths are equal.
  • A trailing $ anchors the end of the path. * matches any sequence of characters. The comparison is case-sensitive.
  • Crawl-delay, Host, and Request-rate are listed and not applied. Other crawlers can read the same file differently.
  • A missing, refused, or unreachable robots.txt is not reported as allowed. Sitemap addresses are listed and not fetched.
  • The file is limited to 500 KB. The address is sent to Kiln so the file can be fetched, and it is not stored.

Questions

Does an allowed path mean the page will be indexed?

No. robots.txt asks cooperative crawlers not to fetch a path. It does not control indexing, and it is not a privacy or authentication control.

What if the file cannot be fetched?

Kiln says the check could not be completed. It does not treat a timeout, a refusal, or a private address as permission to crawl.

Related tools

  • XML sitemap analyzer

    XML sitemap analyzer

    Report the size, URL distribution, missing dates, and duplicates in one sitemap. Child sitemaps are listed, not fetched.

    SEO

  • Meta tag checker

    Meta tag checker

    Read the title, description, canonical URL, and robots meta tag from one public page.

    SEO