Bot & Crawler Detection

Tag-based analytics only sees clients that execute JavaScript, so non-JS crawlers — GPTBot, Googlebot, ClaudeBot, PerplexityBot, scrapers — are invisible to it. Forward your server or edge requests to Kitbase and they're classified server-side: bot and crawler traffic is stored with full attribution, while human requests are ignored (the JS SDK already counts them).

Pick your platform to get a copy-paste setup guide:

Next.jsEdge middlewareTanStack StartRequest middlewareNode.js / ExpressApp middlewareCloudflare Workersfetch handlerVercelLog Drain · zero codenginxAccess-log shipping

How it works

Every platform does the same thing: it observes the original visitor's request signals (User-Agent, IP, path, …) and POSTs them to Kitbase's ingestion endpoint. Kitbase runs its bot & crawler attribution pipeline — including cryptographic Web Bot Auth verification when present — and stores the bot/crawler requests.

• Endpoint: POST https://ingest.kitbase.dev/ingest/v1/server (Vercel uses a Log Drain instead).
• Auth: your secret API key (sk_kitbase_…) as Authorization: Bearer <key> — never the browser-exposed SDK key. The key resolves your project.
• Target: name the environment (e.g. Production) — resolved within the key's project.
• Humans are ignored, so it's safe to forward every request; only bots/crawlers are stored, and only when events are enabled for the environment.

Privacy — we only keep the bots

Forwarding every request doesn't mean every request is stored. Human visitors' signals are used only to classify the request in memory and are then discarded — only bot and crawler requests are persisted. For those, the raw IP is stored only when IP logging is enabled for the environment; otherwise it's used to derive geolocation (country, region, city) and then dropped.

See the full request schema and response in the API reference.