Robots.txt checker

Paste your robots.txt, enter a URL or path and pick a crawler. The checker shows whether that crawler may fetch the URL and which line decides it. Everything runs in your browser; nothing is sent or saved.

How the result is decided

The checker follows the robots.txt standard (RFC 9309) as Google documents it: the crawler uses the most specific User-agent group that matches its name, or the * group if none does. Inside that group the longest matching Allow or Disallow path wins, and when an Allow and a Disallow rule are the same length, Allow wins. * matches any characters and $ marks the end of the URL.

Robots.txt controls crawling, not indexing. A blocked URL can still appear in search if other pages link to it; to keep a page out of results use a noindex tag and leave the page crawlable.

Guide: Robots.txt for a small business blog: what to block · All PilotaSEO guides