User-agent: Scrapy Allow: / User-agent: * Disallow: /stats/ Sitemap: https://crosswater-job-guide.com/sitemap_cwj.xml