User-agent: *
Allow: /
Disallow: /api/
Disallow: /login
Disallow: /dashboard
Disallow: /r/
Disallow: /p/
Disallow: /settings
Disallow: /new
Disallow: /onboarding
Disallow: /teams

# AI assistants and answer engines are explicitly welcome to read and cite
# Comma's public pages — this is the generative-engine optimization (GEO)
# surface. The rules below MUST mirror the default group above: private and
# app surfaces stay blocked, only public marketing/docs pages are crawlable.
# A crawler matching a specific group does NOT fall back to `*`, so the
# Disallow list is repeated here intentionally.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: DuckAssistBot
User-agent: MistralAI-User
Allow: /
Disallow: /api/
Disallow: /login
Disallow: /dashboard
Disallow: /r/
Disallow: /p/
Disallow: /settings
Disallow: /new
Disallow: /onboarding
Disallow: /teams

# Link-preview (unfurl) bots respect robots.txt, so a blanket Disallow on
# /p/ would break the rich card when someone pastes a share link into
# Slack, X, Discord, etc. These bots render previews from the OG tags; they
# do not index. Search and AI crawlers above stay blocked from /p/ (share
# pages are private by design) and every /p/ response also carries
# X-Robots-Tag: noindex as defense in depth — see src/server.ts.
# SkypeUriPreview is the Teams/Skype unfurler; Embedly and Iframely are the
# preview services behind Notion bookmarks and Atlassian smart links — the
# same three surfaces the /share-html-report-in-teams, /embed-html-report-in-
# notion and /embed-html-report-in-confluence guides send people to.
User-agent: Slackbot
User-agent: Slackbot-LinkExpanding
User-agent: Twitterbot
User-agent: facebookexternalhit
User-agent: LinkedInBot
User-agent: Discordbot
User-agent: TelegramBot
User-agent: WhatsApp
User-agent: SkypeUriPreview
User-agent: Embedly
User-agent: Iframely
Allow: /p/
Allow: /
Disallow: /api/
Disallow: /login
Disallow: /dashboard
Disallow: /r/
Disallow: /settings
Disallow: /new
Disallow: /onboarding
Disallow: /teams

Sitemap: https://commareports.com/sitemap.xml