Next.js Bot & Crawler Detection

Detect crawlers and AI bots that never run the JavaScript SDK (GPTBot, Googlebot, ClaudeBot, scrapers, …) directly from Next.js Middleware. The middleware runs in the Edge runtime on every matched request and forwards the visitor's signals to Kitbase, which classifies the actor server-side and stores bot/crawler traffic with full attribution. Human requests are ignored, so it's safe to run on every request.

Privacy — we only keep the bots

Forwarding every request doesn't mean every request is stored. Human visitors' signals are used only to classify the request in memory and are then discarded — only bot and crawler requests are persisted. For those, the raw IP is stored only when IP logging is enabled for the environment; otherwise it's used to derive geolocation (country, region, city) and then dropped.

Prerequisites

Set two environment variables (server-only — don't prefix them with NEXT_PUBLIC_):

• KITBASE_API_KEY — your project's secret API key (sk_kitbase_…), not the browser SDK key.
• KITBASE_ENVIRONMENT — the target environment name, e.g. Production.

Setup

Add a middleware.ts file at your project root (or in src/). Forward with event.waitUntil so the page response is never delayed. (request.ip was removed in Next.js 15, so the client IP is read from x-forwarded-for.)

// middleware.ts
import { NextResponse, type NextRequest, type NextFetchEvent } from "next/server";

export function middleware(request: NextRequest, event: NextFetchEvent) {
  const fwd = request.headers.get("x-forwarded-for");
  event.waitUntil(
    fetch("https://ingest.kitbase.dev/ingest/v1/server", {
      method: "POST",
      headers: { authorization: `Bearer ${process.env.KITBASE_API_KEY}`, "content-type": "application/json" },
      body: JSON.stringify({ environment: process.env.KITBASE_ENVIRONMENT, events: [{
        user_agent: request.headers.get("user-agent"),
        ip_address: fwd?.split(",")[0]?.trim(),
        method: request.method,
        host: request.headers.get("host"),
        path: request.nextUrl.pathname,
        referrer: request.headers.get("referer"),
        signature: request.headers.get("signature"),
        signature_input: request.headers.get("signature-input"),
        signature_agent: request.headers.get("signature-agent"),
      }]}),
    }).catch(() => {}), // never block the response
  );

  return NextResponse.next();
}

export const config = {
  // Page requests only — skip static assets and Next internals to avoid noise.
  matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};

The signature* headers are part of Web Bot Auth — forwarding them lets Kitbase cryptographically verify a crawler's claimed identity when present.

Hosted on Vercel?

You can skip middleware entirely and use a Vercel Log Drain — it captures crawler requests (static + edge) with zero code.

Next steps

• API reference — full request schema, response, and attribution fields.
• All platforms — setup guides for other frameworks and hosts.