How to Track GPTBot, ClaudeBot, and PerplexityBot
HitKeep tracks GPTBot, ClaudeBot, PerplexityBot, and other AI crawler fetches when you forward server-side request records into the AI fetch ingest endpoint.
Browser JavaScript is not enough for this job because most AI crawlers do not run hk.js.
What to collect
Section titled “What to collect”Forward records from the layer that can see crawler requests. Each record should include the requested path, hostname, status code, user agent, and optional response metadata such as content type, bytes served, and response time.
- CloudFront standard logs.
- nginx or Caddy access logs.
- App server middleware.
- Load balancer logs.
- A static-site edge pipeline.
User agents to watch
Section titled “User agents to watch”| Family | Common user agents |
|---|---|
| OpenAI | GPTBot, ChatGPT-User |
| Anthropic | ClaudeBot, Claude-Web |
| Perplexity | PerplexityBot |
| Google AI | Google-Extended, GoogleOther, Google-Safety |
| Apple | Applebot-Extended |
| Meta | meta-externalagent, meta-externalfetcher |
| Amazon | Amazonbot |
Do not treat this table as a consent or robots policy. It is only a reporting filter for requests you already observe at the server or edge.
Send records to HitKeep
Section titled “Send records to HitKeep”Create a scoped API client for the site, then post matching crawler requests to the AI fetch endpoint.
POST /api/sites/{id}/ingest/ai-fetchMinimum record shape
Section titled “Minimum record shape”| Field | Why it matters |
|---|---|
| Hostname and path | Groups fetches by the site and page requested |
| User agent | Identifies the AI crawler family |
| Status code | Shows crawler errors and blocked pages |
| Content type | Separates HTML pages, assets, and feeds |
| Response time and bytes | Highlights slow or unexpectedly heavy responses |
Verification checklist
Section titled “Verification checklist”- Generate or wait for matching crawler requests in your logs.
- Confirm the forwarder only sends AI crawler families.
- Open AI Agents, switch to the Crawlers tab, and check fetch volume by assistant family.
- Filter by status code to find 404, 403, 429, and 5xx responses.
- Compare fetched paths with AI-referred human visits after enough time has passed.
- Keep one saved report for crawler fetch volume and one for AI referral visits so clients can see that the two signals answer different questions.
How this differs from robots.txt
Section titled “How this differs from robots.txt”robots.txt tells crawlers what you allow or disallow. HitKeep reporting tells you what matching AI crawler user agents actually requested and how your site responded.
Use both. Keep robots rules and crawler policies in the site configuration, then use HitKeep to verify crawler access, errors, response types, and later AI-referred visit patterns.
What appears in the dashboard
Section titled “What appears in the dashboard”AI fetch records are shown in the Crawlers tab of the AI Agents page. HitKeep groups requests by assistant, operator family, resource type, path, status code, response time, and bytes served. The browser tracker separately records human visits from AI referrers. Correlation only works when both signals exist.