# robots.txt for https://www.burgessniple.com User-agent: * Disallow: /*?q= Disallow: /bin/ Disallow: /areas/ Disallow: /_admin/ Disallow: /job-application/ User-agent: Scrapy/1.0.5 Disallow: / Sitemap: https://www.burgessniple.com/sitemap.xml