HitKeep tracks GPTBot, ClaudeBot, PerplexityBot, and other AI crawler fetches when you forward server-side request records into the AI fetch ingest endpoint.
Browser JavaScript is not enough for this job because most AI crawlers do not run hk.js.
What to collect
Forward records from the layer that can see crawler requests. Each record should include the requested path, hostname, status code, user agent, and optional response metadata such as content type, bytes served, and response time.
- CloudFront standard logs.
- nginx or Caddy access logs.
- App server middleware.
- Load balancer logs.
- A static-site edge pipeline.
User agents to watch
Do not treat this table as a consent or robots policy. It is only a reporting filter for requests you already observe at the server or edge.
| Family | Common user agents |
|---|---|
| OpenAI | GPTBot, ChatGPT-User, OAI-SearchBot |
| Anthropic | ClaudeBot, Claude-User |
| Perplexity | PerplexityBot |
| Google AI | Google-Extended, GoogleOther, GoogleOther-Image |
| Apple | Applebot-Extended |
| Meta | meta-externalagent, meta-externalfetcher |
| Amazon | Amazonbot |
Send records to HitKeep
Create a scoped API client for the site, then post matching crawler requests to the AI fetch endpoint.
POST /api/sites/{id}/ingest/ai-fetchWhat appears in the dashboard
AI fetch records appear in AI Visibility Analytics. HitKeep groups requests by assistant, operator family, resource type, path, status code, response time, and bytes served. The browser tracker separately records human visits from AI referrers. Correlation only works when both signals exist.