Fix: AI robots policy

Training, search, and user crawlers are separate. Decide explicitly. Blocking GPTBot does not block OAI-SearchBot.

Search-friendly / training restricted

# AI crawl policy: allow search/user, block training
# Adjust to your legal/product preference.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# Opt out of Gemini training without affecting Googlebot
User-agent: Google-Extended
Disallow: /

Fully open

# AI crawl policy: fully open (train + search + user)
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Google-Extended
Allow: /

Also check CDN/WAF rules: robots.txt allow is useless if the edge blocks AI user-agents.