Robots.txt Tester – Check Allowed & Blocked URLs | NetworkUstad

A robots.txt file tells search crawlers which parts of a site they may visit, but a rule that looks right can still block the wrong pages: a broad Disallow can swallow a section you meant to keep open, and an Allow that should carve out an exception may lose to a longer Disallow. The only reliable way to know is to test paths against the rules before publishing. This tester parses pasted robots.txt rules, applies the standard longest-match logic, and tells you for any path whether it is allowed or blocked, with the winning rule named. It tests rules; it does not fetch or publish your robots.txt file.

How to Use the Robots.txt Tester

1. Paste your robots.txt rules into the rules box.
2. Type the URL path to test, such as /admin/login.
3. Click Test.
4. Read Allowed or Blocked and the rule that decided it.
5. Test the exceptions too, such as /admin/public/page.
6. Fix the rule that surprises you and re-test before publishing.

Key Features

• Parses User-agent groups, including the wildcard group.
• Applies longest-match wins, with Allow winning ties.
• Supports wildcards and end anchors in patterns.
• Tests one path at a time with a clear verdict.
• Names the deciding rule so fixes are obvious.
• Runs entirely in the browser; nothing is fetched or published.

What the Result Means

Allowed means a crawler following your rules may visit the path; blocked means it should not. The deciding rule matters more than the verdict: if a broad Disallow wins over a narrow Allow, the Allow pattern needs to be longer or the Disallow narrower. Remember that robots.txt is a request, not a security control: well-behaved crawlers obey it, but it does not hide a URL from anyone who knows it, so private content needs real authentication, not just a Disallow line.

Worked Example

A site wants search engines out of its admin area but needs its public help pages inside /admin/public indexed. The rules Disallow: /admin/ plus Allow: /admin/public look right, but only testing proves it: /admin/login comes back Blocked, /admin/public/guide comes back Allowed thanks to the longer Allow, and /blog comes back Allowed because no rule matches it. With those three verdicts, the site owner can publish confidently instead of discovering the mistake in search results weeks later. The same tester also catches the opposite error: a rule intended to block only a staging directory that accidentally blocks the live product pages, which would otherwise remove the site from search results entirely until someone notices the traffic drop.

Best Practices and Tips

• Test every exception path, not just the blocked section itself.
• Keep Disallow patterns as narrow as the section they protect.
• Use the wildcard group for rules that apply to all crawlers.
• Check that your sitemap and key landing pages are not accidentally blocked.
• After publishing, verify with a search console robots tester as a second opinion.

Common Mistakes to Avoid

• Assuming robots.txt hides private pages from the public.
• Writing a Disallow that is broader than the section it should cover.
• Expecting a short Allow to beat a longer Disallow.
• Forgetting that rule order does not decide; match length does.

Frequently Asked Questions

What does longest-match mean?

When several rules match a path, the rule with the longest pattern wins. If an Allow and a Disallow tie, Allow wins.

Does this test a live robots.txt?

No. It tests the rules you paste, so you can check a draft before it goes live.

Why is /admin/public allowed when /admin/ is blocked?

Because the Allow pattern is longer than the Disallow pattern, so it wins for paths under /admin/public.

Can robots.txt block a specific crawler?

Yes, with a User-agent group naming that crawler, but this tester applies the wildcard group.

Related Tools

• Robots.txt Generator – Build a valid robots.txt file
• .htaccess Redirect Generator – Build Apache redirect rules
• llms.txt Generator – Build an llms.txt file for AI crawlers