What the robots.txt tester does
It fetches https://<domain>/robots.txt and shows its user-agent groups, their Allow and Disallow rules, any Crawl-delay and the Sitemap lines. Enter a user agent and a path to see whether that crawler may fetch it, and which rule decided.
Type the user agent token as it appears in the file, such as Googlebot or Bingbot. When no group names it, the * group applies.
How crawlers read the rules
A crawler follows only the group that names it, ignoring the * group when a specific one exists. Within that group, the most specific matching rule wins, which in practice is the longest one, and Allow wins a tie with Disallow. * matches any sequence of characters and $ anchors the end of the path.
robots.txt controls crawling, not indexing. A blocked page can still appear in search results if other sites link to it; to keep a page out of the index, let it be crawled and add a noindex meta tag instead.
Frequently asked questions
How do I block a folder in robots.txt?
Add Disallow: /folder/ under the user agent you want to block, for example under User-agent: *. The trailing slash limits the rule to that folder and leaves /folder-other alone.
Does Google respect Crawl-delay?
No. Googlebot ignores Crawl-delay and adjusts its rate to how fast your server answers. Bing and some other crawlers do honour it.
Why is my page indexed even though robots.txt blocks it?
Because robots.txt stops crawling, not indexing. Google can index a URL from links alone, without fetching it. Remove the Disallow and add noindex if you want it out of the results.
What happens if there is no robots.txt?
Crawlers treat a missing file (a 404) as permission to crawl everything. A server error on robots.txt is different: Google pauses crawling the site until it can read the file.