All free tools
Free tool · No signup

Robots.txt Tester

Paste robots.txt and test a URL for a crawler product token. See the matching allow or disallow rule and its line number.

Processed on your device. No upload, account or daily limit.

Runs locally on pasted rules, up to 500 KB and 5,000 rules. Supports exact product-token groups, wildcards and end anchors. Does not fetch robots.txt or check indexing, response status or meta directives.

Keep going

Put your data to work.

Explore scrapers for products, business listings, jobs and more.

Explore scrapers