CloudFront AI Crawler Tracking for HitKeep

CloudFront AI crawler tracking uses a viewer-response function, a filtered CloudWatch Logs subscription, and HitKeep AI fetch ingest.

The edge function writes one compact line only for known AI crawler user agents. Ordinary browser requests are not logged by this path.

Recommended setup

CloudFront viewer response -> filtered CloudWatch log -> Lambda -> HitKeep AI fetch ingest

This setup records server-side crawler fetches. It does not replace the browser tracker. Keep hk.js installed so HitKeep can also see AI-referred human visits, goals, events, and automatic interactions.

When to use this guide

Use this path when the site already has a CloudFront viewer-response function or when the CloudFront plan does not include standard access logs. If the site runs behind nginx, Caddy, or application middleware, forwarding origin access logs can provide fuller status and response-time coverage.

What to configure

A HitKeep site for the domain.
An API client with instance role User and an Admin grant only for that site.
A live viewer-response CloudFront Function that emits HITKEEP_AI_FETCH records.
A CloudWatch Logs subscription restricted to that marker.
A Lambda function that validates crawler user agents and hostname again.

HitKeep includes a production-oriented CloudFormation template at infrastructure/ai-crawler-forwarder.yaml. It creates the retained CloudWatch log group, filtered subscription, scoped Lambda forwarder, retry and dead-letter handling, and alarms. The API token stays in an existing SecureString SSM parameter and is read only at runtime.

Use instance role User plus a Site Admin grant: Site Editor cannot ingest AI fetches, while an instance-wide owner or admin role is unnecessary. The forwarder resolves the permitted site by hostname, so the site UUID is not copied into infrastructure configuration.

Crawler filter

Filter at both boundaries: the edge function emits only known AI crawler families, and the Lambda checks the user agent and hostname again before forwarding. This keeps the HitKeep AI fetch endpoint focused on crawler visibility instead of becoming a general log sink.

user_agent contains GPTBot, ClaudeBot, PerplexityBot, ChatGPT-User,
Google-Extended, Applebot-Extended, Amazonbot, or meta-externalagent

What to forward

Requested path.
Hostname.
Status code.
Content type.
Bytes served when Content-Length is present.
User agent.

Do not send request bodies or form data. The edge function selects only these fields from the viewer response. It never reads or writes viewer IP addresses, query strings, cookies, or referrers. CloudFront viewer-response functions do not run for origin responses with status 400 or higher, so use standard or origin logs when complete error coverage is required.

Deploy the provided stack

aws cloudformation deploy \
  --region us-east-1 \
  --stack-name hitkeep-ai-crawler-forwarder \
  --template-file infrastructure/ai-crawler-forwarder.yaml \
  --capabilities CAPABILITY_IAM \
  --no-execute-changeset \
  --parameter-overrides CloudFrontFunctionName=... HitKeepBaseUrl=https://cloud.hitkeep.eu HitKeepApiTokenParameterName=/.../api-token TrackedHostname=hitkeep.com

CloudFront Function logs always land in us-east-1. This command creates a change set without executing it. Review that generated change set, then execute it as a separate approved action. Do not put the token itself in the command or template; pass only the name of an existing SecureString parameter.

Common failure modes

The viewer-response function is updated in DEVELOPMENT but not published to LIVE.
The CloudWatch subscription filter does not match HITKEEP_AI_FETCH.
The edge function or Lambda accepts all bot traffic instead of AI crawler families.
The API client token is not scoped to the target site.
The hostname in the log does not match the site configured in HitKeep.
Only browser tracking is installed, so crawler fetches never appear.

How this differs from CloudFront access analytics

CloudFront access logs show every request at the edge. This function-based path writes only responses requested by known AI crawler families and turns those records into AI Visibility reports.

Use full CloudFront or origin logs for security, debugging, and complete error analysis. Use HitKeep for the content questions: which AI systems requested which pages and whether those fetched pages later receive AI-referred visits.

How to verify

Open AI Agents in HitKeep and switch to the Crawlers tab.
Check that fetch volume appears for the site.
Filter by assistant family.
Look for 4xx or 5xx errors.
Wait for normal hk.js traffic before reading the correlation report.

Correlation needs time because it compares fetched paths with later AI-referred human visits.