---
title: "CloudFront AI Crawler Tracking for HitKeep | HitKeep"
description: "Track GPTBot, ClaudeBot, PerplexityBot, and other AI fetches from a CloudFront Function without collecting visitor IPs, cookies, or query strings."
canonical: "https://hitkeep.com/guides/tracking/cloudfront-ai-crawler-tracking/"
---

# CloudFront AI Crawler Tracking for HitKeep

CloudFront AI crawler tracking uses a viewer-response function, a filtered CloudWatch Logs subscription, and HitKeep AI fetch ingest.

The edge function writes one compact line only for known AI crawler user agents. Ordinary browser requests are not logged by this path.

Managed option

## Turn filtered edge records into AI Visibility evidence.

Keep the server-side crawler feed separate from browser analytics, then correlate the two signals in HitKeep.

[Choose Business annual](https://cloud.hitkeep.eu/signup?utm_source=hitkeep_docs&utm_medium=ai_visibility&utm_campaign=agency_pilot&utm_content=cloudfront_ai_crawler&plan=business&billing=annual)[Open the full AWS guide](https://hitkeep.com/guides/tracking/ai-fetch-aws/)

## Recommended setup

```
CloudFront viewer response -> filtered CloudWatch log -> Lambda -> HitKeep AI fetch ingest
```

This setup records server-side crawler fetches. It does not replace the browser tracker. Keep `hk.js` installed so HitKeep can also see AI-referred human visits, goals, events, and automatic interactions.

## When to use this guide

Use this path when the site already has a CloudFront viewer-response function or when the CloudFront plan does not include standard access logs. If the site runs behind nginx, Caddy, or application middleware, forwarding origin access logs can provide fuller status and response-time coverage.

## What to configure

- A HitKeep site for the domain.
- An API client with instance role User and an Admin grant only for that site.
- A live viewer-response CloudFront Function that emits `HITKEEP_AI_FETCH` records.
- A CloudWatch Logs subscription restricted to that marker.
- A Lambda function that validates crawler user agents and hostname again.

HitKeep includes a production-oriented CloudFormation template at `infrastructure/ai-crawler-forwarder.yaml`. It creates the retained CloudWatch log group, filtered subscription, scoped Lambda forwarder, retry and dead-letter handling, and alarms. The API token stays in an existing SecureString SSM parameter and is read only at runtime.

Use instance role User plus a Site Admin grant: Site Editor cannot ingest AI fetches, while an instance-wide owner or admin role is unnecessary. The forwarder resolves the permitted site by hostname, so the site UUID is not copied into infrastructure configuration.

## Crawler filter

Filter at both boundaries: the edge function emits only known AI crawler families, and the Lambda checks the user agent and hostname again before forwarding. This keeps the HitKeep AI fetch endpoint focused on crawler visibility instead of becoming a general log sink.

```
user_agent contains GPTBot, ClaudeBot, PerplexityBot, ChatGPT-User,
Google-Extended, Applebot-Extended, Amazonbot, or meta-externalagent
```

## What to forward

- Requested path.
- Hostname.
- Status code.
- Content type.
- Bytes served when `Content-Length` is present.
- User agent.

Do not send request bodies or form data. The edge function selects only these fields from the viewer response. It never reads or writes viewer IP addresses, query strings, cookies, or referrers. CloudFront viewer-response functions do not run for origin responses with status 400 or higher, so use standard or origin logs when complete error coverage is required.

## Deploy the provided stack

```
aws cloudformation deploy \
  --region us-east-1 \
  --stack-name hitkeep-ai-crawler-forwarder \
  --template-file infrastructure/ai-crawler-forwarder.yaml \
  --capabilities CAPABILITY_IAM \
  --no-execute-changeset \
  --parameter-overrides CloudFrontFunctionName=... HitKeepBaseUrl=https://cloud.hitkeep.eu HitKeepApiTokenParameterName=/.../api-token TrackedHostname=hitkeep.com
```

CloudFront Function logs always land in `us-east-1`. This command creates a change set without executing it. Review that generated change set, then execute it as a separate approved action. Do not put the token itself in the command or template; pass only the name of an existing SecureString parameter.

## Common failure modes

- The viewer-response function is updated in DEVELOPMENT but not published to LIVE.
- The CloudWatch subscription filter does not match `HITKEEP_AI_FETCH`.
- The edge function or Lambda accepts all bot traffic instead of AI crawler families.
- The API client token is not scoped to the target site.
- The hostname in the log does not match the site configured in HitKeep.
- Only browser tracking is installed, so crawler fetches never appear.

## How this differs from CloudFront access analytics

CloudFront access logs show every request at the edge. This function-based path writes only responses requested by known AI crawler families and turns those records into AI Visibility reports.

Use full CloudFront or origin logs for security, debugging, and complete error analysis. Use HitKeep for the content questions: which AI systems requested which pages and whether those fetched pages later receive AI-referred visits.

## How to verify

- Open **AI Agents** in HitKeep and switch to the **Crawlers** tab.
- Check that fetch volume appears for the site.
- Filter by assistant family.
- Look for 4xx or 5xx errors.
- Wait for normal `hk.js` traffic before reading the correlation report.

Correlation needs time because it compares fetched paths with later AI-referred human visits.

## Related

- [AI Fetch Ingest](https://hitkeep.com/guides/tracking/ai-fetch-ingest/)
- [AI Fetch on AWS](https://hitkeep.com/guides/tracking/ai-fetch-aws/)
- [Track GPTBot, ClaudeBot, and PerplexityBot](https://hitkeep.com/guides/tracking/track-gptbot-claudebot-perplexity/)
- [AI visibility reporting for SEO agencies](https://hitkeep.com/use-cases/ai-visibility-seo-agencies/)
- [AI Visibility Dashboard Reference](https://hitkeep.com/guides/analytics/ai-visibility/)

[Previous AI fetch on AWS](https://hitkeep.com/guides/tracking/ai-fetch-aws/)[Next Track GPTBot, ClaudeBot, and Perplexity](https://hitkeep.com/guides/tracking/track-gptbot-claudebot-perplexity/)