Node.js & Express Bot & Crawler Detection

Detect crawlers and AI bots that never run the JavaScript SDK by adding a small middleware to your Express (or any Node.js) app. It forwards each request's signals to Kitbase, which classifies the actor and stores bot/crawler traffic with attribution. Human requests are ignored.

Privacy — we only keep the bots

Forwarding every request doesn't mean every request is stored. Human visitors' signals are used only to classify the request in memory and are then discarded — only bot and crawler requests are persisted. For those, the raw IP is stored only when IP logging is enabled for the environment; otherwise it's used to derive geolocation (country, region, city) and then dropped.

Prerequisites

Set two environment variables:

• KITBASE_API_KEY — your project's secret API key (sk_kitbase_…), not the browser SDK key.
• KITBASE_ENVIRONMENT — the target environment name, e.g. Production.

Setup

Mount the middleware early, before your routes. The IP comes from x-forwarded-for (when behind a proxy/load balancer) and falls back to the socket address. Make sure trust proxy is set if you terminate TLS at a load balancer.

app.use((req, res, next) => {
  fetch("https://ingest.kitbase.dev/ingest/v1/server", {
    method: "POST",
    headers: { "authorization": `Bearer ${process.env.KITBASE_API_KEY}`, "content-type": "application/json" },
    body: JSON.stringify({ environment: process.env.KITBASE_ENVIRONMENT, events: [{
      user_agent: req.headers["user-agent"],
      ip_address: req.headers["x-forwarded-for"]?.split(",")[0]?.trim() || req.socket.remoteAddress,
      method: req.method,
      host: req.headers.host,
      path: req.path,
      referrer: req.headers.referer,
      signature: req.headers["signature"],
      signature_input: req.headers["signature-input"],
      signature_agent: req.headers["signature-agent"],
    }]}),
  }).catch(() => {}); // fire-and-forget; never block the response
  next();
});

For high-traffic apps, buffer observed requests and send them in batches (up to 500 per call) on a short interval instead of one request per call.

The signature* headers are part of Web Bot Auth — forwarding them lets Kitbase cryptographically verify a crawler's claimed identity when present.

Next steps

• API reference — full request schema, response, and attribution fields.
• All platforms — setup guides for other frameworks and hosts.