# Sitemap location (must be top-level, not inside a User-agent block) Sitemap: https://www.cosgrid.com/sitemap.xml Sitemap: https://www.cosgrid.com/video-sitemap.xml # ── All crawlers ──────────────────────────────────────────────── User-agent: * Allow: / # Block non-public paths Disallow: /api/ Disallow: /admin/ # Block query-string duplicates (e.g. UTM params) Disallow: /*?* Disallow: /*?ref= Disallow: /*?utm_ # ── Googlebot – allow JS, CSS, and images for rendering ───────── User-agent: Googlebot Allow: /*.js$ Allow: /*.css$ Allow: /*.webp$ Allow: /*.png$ Allow: /*.jpg$ # ── AI Crawlers – allow for GEO visibility ─────────────────────── User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Google-Extended Allow: / User-agent: PerplexityBot Allow: / User-agent: anthropic-ai Allow: / User-agent: ClaudeBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: CCBot Allow: / User-agent: cohere-ai Allow: / User-agent: YouBot Allow: / User-agent: Applebot Allow: / User-agent: Diffbot Allow: / User-agent: FacebookBot Allow: /