---
url: 'https://docs.kitbase.dev/crawler-detection.md'
description: >-
  Detect AI crawlers and bots (GPTBot, Googlebot, ClaudeBot, …) that never run
  the JavaScript SDK by forwarding your server or edge requests to Kitbase. Pick
  your platform for a copy-paste setup guide.
---

# Bot & Crawler Detection

Tag-based analytics only sees clients that execute JavaScript, so non-JS crawlers — GPTBot, Googlebot, ClaudeBot, PerplexityBot, scrapers — are invisible to it. Forward your **server or edge requests** to Kitbase and they're classified server-side: bot and crawler traffic is stored with full attribution, while human requests are ignored (the JS SDK already counts them).

## Detection vs. filtering {#detection-vs-filtering}

Kitbase has two bot features that point in opposite directions — this page is about the first one:

| | **Bot & Crawler Detection** (this page) | **Bot filtering** (built into [web analytics](/web-analytics)) |
|---|---|---|
| Goal | Make bots **visible** — see which crawlers and AI bots visit your site | Keep bots **out** of your human analytics |
| Where it runs | Your server or edge, forwarding requests to Kitbase | The JS SDK in the browser, plus the server-side pipeline |
| What happens to bots | Stored as their own dataset with attribution (name, vendor, verified/spoofed) | Silently discarded — never reach your dashboards or billing counts |
| Setup | Opt-in: install a forwarder for your platform (below) | Automatic — [no configuration](/autocapture#bot-filtering) |

Both use the same [attribution pipeline](/track-events#bot-crawler-attribution) to decide *what's a bot* — they just do opposite things with the answer. And neither is [AI Visibility](/ai-visibility), which tracks how AI **answers** mention your brand, not which AI **crawlers** visit your site.

::: tip Guided setup in the dashboard
The first time you open the **Crawlers** page in the dashboard (under **AI visibility** in the project sidebar), a setup wizard walks you through the whole flow — creating a secret API key, installing the forwarder for your platform, and live-verifying that the first request and first detected crawler arrive.
:::

Pick your platform to get a copy-paste setup guide:

## How it works

Every platform does the same thing: it observes the **original visitor's** request signals (User-Agent, IP, path, …) and POSTs them to Kitbase's ingestion endpoint. Kitbase runs its [bot & crawler attribution pipeline](/track-events#bot-crawler-attribution) — including cryptographic [Web Bot Auth](/track-events#bot-crawler-attribution) verification when present — and stores the bot/crawler requests.

* **Endpoint:** `POST https://ingest.kitbase.dev/ingest/v1/server` (Vercel uses a [Log Drain](/crawler-detection/vercel) instead).
* **Auth:** your **secret** API key (`sk_kitbase_…`) as `Authorization: Bearer <key>` — never the browser-exposed SDK key. The key resolves your project, which is the target for everything you send.
* **Humans are ignored**, so it's safe to forward every request; only bots/crawlers are stored, and only when events are enabled for the project.

::: info Privacy — we only keep the bots
Forwarding every request doesn't mean every request is stored. Human visitors' signals are used only to classify the request in memory and are then **discarded** — only bot and crawler requests are persisted. For those, the raw IP is stored only when IP logging is enabled for the project; otherwise it's used to derive geolocation (country, region, city) and then dropped.
:::

## Comparing periods {#compare-periods}

The crawler reports read from the same date window as the rest of your analytics, and compare against an earlier one the same way — against the previous period unless `comparePreset`, or a `compareFrom`/`compareTo` pair, names another. [Compare to an earlier period](/web-analytics#compare-periods) covers what each preset means, the guard rails, and how the extra response fields read; only the shapes below are specific to crawlers.

| Report | What a comparison adds |
|---|---|
| `GET /bots/analytics/timeline` | `comparisonSeries` — the same per-vendor series over the earlier window, at the same interval. A vendor that only crawled in one of the two windows has a series in that one alone |
| `GET /bots/analytics/top` | `previousRequestCount` per bot, with `absoluteChange` and `percentageChange` |
| `GET /bots/analytics/countries` | `previousRequestCount` and `previousBotCount` per country; the change fields describe requests, the metric the list is ranked on |
| `GET /bots/analytics/categories` | `previousRequestCount` per actor category, under the same `search` and `actorType` filters |
| `GET /bots/analytics/top-paths` | `previousRequestCount` per path, under the same `vendor` filter |

A crawler that first showed up inside your selected period reads `previousRequestCount: 0` — it was there to be counted and wasn't — while one that has since stopped crawling is simply not listed, because the ranking is the selected period's.

## Kitbase's own crawler {#kitbaseauditbot}

If your project has a [website connected](/guide/analytics#connect-a-website-optional), Kitbase's own crawler — **KitbaseAuditBot** (`Mozilla/5.0 (compatible; KitbaseAuditBot/1.0; +https://kitbase.dev/audit-bot)`) — fetches your public pages to power Site Audit and page indexing. With a forwarder installed you'll see it in the Crawlers page like any other bot, and it holds itself to the same standard we detect others by:

* Every request is signed with **Web Bot Auth** (RFC 9421 HTTP Message Signatures, Ed25519). Kitbase's detection pipeline verifies the signature, so KitbaseAuditBot always shows as **verified** — and a request that fakes its user-agent without a valid signature is flagged as suspected or spoofed.
* The public key directory is published at `https://kitbase.dev/.well-known/http-message-signatures-directory`, so any third-party detector that implements Web Bot Auth can verify it the same way.
* Verified KitbaseAuditBot requests are stored for visibility but **never count toward your bot-event usage** — you aren't billed for traffic Kitbase generates.

See the full request schema and response in the [API reference](/api-reference#server-side-crawler-detection).