# snaprecruit.com robots.txt # Updated: 2026-11-21 (Wave 6: Explicit AI Agent Disallow Inheritance Fix + /outbound/ Crawl-Budget Guard) # AI Agents & LLMs Directory: https://snaprecruit.com/llms.txt | https://snaprecruit.com/llms-full.txt # Default: Allow all search engines User-agent: * Allow: / Disallow: /login.html Disallow: /register.html Disallow: /autologin.html Disallow: /ajax/ Disallow: /apply.html Disallow: /outbound/ # Allow: /get-blog-image.html (social crawlers need this for og:image previews) Disallow: /list-of-question.html Disallow: /daily-recommended-jobs/ Disallow: /blog/.html Disallow: /superadmin/ Disallow: /developer/ Disallow: /snapboard/ Disallow: /messenger/ Disallow: /contacts/ Disallow: /resumebox/ Disallow: /settings.html Disallow: /wallet.html Disallow: /billing-info.html Disallow: /profile.html Disallow: /manage-subscriptions.html Disallow: /personal-transactions.html Disallow: /identity-validation.html Disallow: /search-candidates Disallow: /applicants.html Disallow: /ai/ Disallow: /ai.html Disallow: /job/*/apply.html Disallow: /*/apply.html Disallow: /*?query= Disallow: /*?city= Disallow: /*?state= Disallow: /*?location= Disallow: /*?country= Disallow: /*?remote= Disallow: /*?employment= Disallow: /*?jobType= Disallow: /*?time= Disallow: /*?pastDays= Disallow: /*?radius= Disallow: /*?sourceType= Disallow: /*?partner= Disallow: /*?company= Disallow: /*?tags= Disallow: /*?sort= Disallow: /*?*query= Disallow: /*?*city= Disallow: /*?*state= Disallow: /*?*location= Disallow: /*?*country= Disallow: /*?*remote= Disallow: /*?*employment= Disallow: /*?*jobType= Disallow: /*?*time= Disallow: /*?*pastDays= Disallow: /*?*radius= Disallow: /*?*sourceType= Disallow: /*?*partner= Disallow: /*?*company= Disallow: /*?*tags= Disallow: /*?*sort= # ============================================================================== # AI Search & Generative Engine Optimization (GEO) Agents (Allow - Drive Traffic) # Per RFC 9309, explicit User-agent groups do NOT inherit User-agent: * Disallow # rules, so private/apply/outbound/faceted-query exclusions are repeated here. # ============================================================================== User-agent: GPTBot User-agent: ChatGPT-User User-agent: OAI-SearchBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: bingbot User-agent: Microsoft-SearchBot User-agent: Copilot User-agent: GrokBot User-agent: xAI-Bot User-agent: Googlebot User-agent: Google-Extended User-agent: Claude-SearchBot User-agent: Claude-User User-agent: Applebot User-agent: Applebot-Extended User-agent: meta-webindexer User-agent: Meta-ExternalAgent User-agent: Meta-ExternalFetcher User-agent: FacebookBot User-agent: Amazonbot User-agent: NV-DataGatherer User-agent: NvidiaBot User-agent: YouBot User-agent: DuckAssistBot User-agent: cohere-ai User-agent: MistralBot User-agent: DeepSeekBot User-agent: CCBot Allow: / Disallow: /login.html Disallow: /register.html Disallow: /autologin.html Disallow: /ajax/ Disallow: /apply.html Disallow: /outbound/ Disallow: /list-of-question.html Disallow: /daily-recommended-jobs/ Disallow: /blog/.html Disallow: /superadmin/ Disallow: /developer/ Disallow: /snapboard/ Disallow: /messenger/ Disallow: /contacts/ Disallow: /resumebox/ Disallow: /settings.html Disallow: /wallet.html Disallow: /billing-info.html Disallow: /profile.html Disallow: /manage-subscriptions.html Disallow: /personal-transactions.html Disallow: /identity-validation.html Disallow: /search-candidates Disallow: /applicants.html Disallow: /ai/ Disallow: /ai.html Disallow: /job/*/apply.html Disallow: /*/apply.html Disallow: /*?query= Disallow: /*?city= Disallow: /*?state= Disallow: /*?location= Disallow: /*?country= Disallow: /*?remote= Disallow: /*?employment= Disallow: /*?jobType= Disallow: /*?time= Disallow: /*?pastDays= Disallow: /*?radius= Disallow: /*?sourceType= Disallow: /*?partner= Disallow: /*?company= Disallow: /*?tags= Disallow: /*?sort= Disallow: /*?*query= Disallow: /*?*city= Disallow: /*?*state= Disallow: /*?*location= Disallow: /*?*country= Disallow: /*?*remote= Disallow: /*?*employment= Disallow: /*?*jobType= Disallow: /*?*time= Disallow: /*?*pastDays= Disallow: /*?*radius= Disallow: /*?*sourceType= Disallow: /*?*partner= Disallow: /*?*company= Disallow: /*?*tags= Disallow: /*?*sort= # Bulk LLM indexing crawler paced cleanly via Crawl-delay (live Claude-User / Claude-SearchBot remain unthrottled above) User-agent: ClaudeBot Crawl-delay: 2 Allow: / Disallow: /login.html Disallow: /register.html Disallow: /autologin.html Disallow: /ajax/ Disallow: /apply.html Disallow: /outbound/ Disallow: /list-of-question.html Disallow: /daily-recommended-jobs/ Disallow: /blog/.html Disallow: /superadmin/ Disallow: /developer/ Disallow: /snapboard/ Disallow: /messenger/ Disallow: /contacts/ Disallow: /resumebox/ Disallow: /settings.html Disallow: /wallet.html Disallow: /billing-info.html Disallow: /profile.html Disallow: /manage-subscriptions.html Disallow: /personal-transactions.html Disallow: /identity-validation.html Disallow: /search-candidates Disallow: /applicants.html Disallow: /ai/ Disallow: /ai.html Disallow: /job/*/apply.html Disallow: /*/apply.html Disallow: /*?query= Disallow: /*?city= Disallow: /*?state= Disallow: /*?location= Disallow: /*?country= Disallow: /*?remote= Disallow: /*?employment= Disallow: /*?jobType= Disallow: /*?time= Disallow: /*?pastDays= Disallow: /*?radius= Disallow: /*?sourceType= Disallow: /*?partner= Disallow: /*?company= Disallow: /*?tags= Disallow: /*?sort= Disallow: /*?*query= Disallow: /*?*city= Disallow: /*?*state= Disallow: /*?*location= Disallow: /*?*country= Disallow: /*?*remote= Disallow: /*?*employment= Disallow: /*?*jobType= Disallow: /*?*time= Disallow: /*?*pastDays= Disallow: /*?*radius= Disallow: /*?*sourceType= Disallow: /*?*partner= Disallow: /*?*company= Disallow: /*?*tags= Disallow: /*?*sort= # ============================================================================== # Sitemaps # ============================================================================== Sitemap: https://snaprecruit.com/sitemap.xml # ============================================================================== # Scrapers & Aggressive Non-User Bots (Block — no search traffic value, waste crawl budget) # ============================================================================== User-agent: Bytespider Disallow: / User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: PetalBot Disallow: / User-agent: Epochal-Search Disallow: / User-agent: Reflectionbot Disallow: / User-agent: ShapBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: jobinv-inventory-spike Disallow: / User-agent: Lightpanda Disallow: / # Foreign search engines (no US market value) User-agent: Baiduspider Disallow: / User-agent: Baiduspider-render Disallow: / User-agent: Sogou Disallow: / User-agent: YandexBot Disallow: /