Tools·SEO & marketing
Parse a domain's robots.txt and simulate a crawler per User-Agent against a given path.