All free toolsFree tool · No signup
Robots.txt Tester
Paste robots.txt and test a URL for a crawler product token. See the matching allow or disallow rule and its line number.
Runs locally on pasted rules, up to 500 KB and 5,000 rules. Supports exact product-token groups, wildcards and end anchors. Does not fetch robots.txt or check indexing, response status or meta directives.
Keep going
Explore scrapersPut your data to work.
Explore scrapers for products, business listings, jobs and more.