User-agent: rogerbot crawl-delay: 5 User-agent: trovitBot Disallow: / User-Agent: * Disallow: /admin Disallow: /api Allow: /blog*?* Disallow: /*?* Disallow: /job-search Disallow: /searches Disallow: /job/*/apply Disallow: /job/*/save_job Disallow: /job/*/unsave_job Sitemap: https://www.computerfutures.com/sitemap.xml