Observability Without Lock-In: Structured Logs and OpenTelemetry-Compatible Tracing

# A real observability stack failure mode, lightly fictionalized:
#
# - "We have logs." Yes, in three different formats, one of them is
#   console.log("got here"), and none of them carry a request id.
# - "We have tracing." There's an SDK pinned in package.json. It was
#   installed in 2023. Nobody knows which spans it actually emits.
# - "We have metrics." prom-client is in two services and StatsD in
#   the third. The dashboards agree on nothing.
# - "We're going to clean it up." Yes. After this quarter. With the
#   migration. After the migration. After the next one.
#
# The lock-in problem is that every observability SDK wants to OWN your
# app: register globally, monkey-patch fetch, become the only logger.
# Then you can't run on Workers, you can't run on Edge, and you can't
# switch vendors without a rewrite. DaloyJS goes the other way:
# small contracts, your choice of implementation, zero global patching.

// src/log.ts — boot a structured JSON logger.
import { createLogger } from "@daloyjs/core";

export const log = createLogger({
  level: process.env.LOG_LEVEL ?? "info",
  bindings: {
    service: "books-api",
    env: process.env.NODE_ENV ?? "development",
    region: process.env.FLY_REGION ?? process.env.AWS_REGION ?? "local",
  },
});

// One record per line, always JSON, always with level + time:
//
//   {"level":"info","time":"2026-06-03T08:42:11.108Z",
//    "service":"books-api","env":"production","region":"arn1",
//    "event":"boot","msg":"server starting"}
//
// Pipe stdout into Loki / CloudWatch / Datadog / your terminal,
// whatever. The structure is the same everywhere.

// Want pino in production for performance? createLogger is just one
// implementation of the Logger interface. Anything matching the shape
// works — same constructor signature, no framework changes.
import pino from "pino";
import type { Logger } from "@daloyjs/core";
import { App } from "@daloyjs/core";

// pino implements .child() and .info/.warn/.error/etc. already.
// The shape is intentionally compatible, so this just works:
const log = pino({ level: "info" }) as unknown as Logger;

const app = new App({ logger: log });

// In tests:
//   new App({ logger: false })   // → noopLogger, silent
//   new App({ logger: noopLogger })

// src/app.ts — request IDs are the spine of observability.
import { App, requestId, timing } from "@daloyjs/core";
import { log } from "./log.js";

const app = new App({ logger: log });

// Order matters. requestId() runs in beforeHandle and stamps
// ctx.state.requestId + sets x-request-id on the response.
// trustIncoming defaults to false because clients can spoof headers
// unless your edge proxy strips/rewrites them.
app.use(requestId({ trustIncoming: false }));

// timing() adds Server-Timing: app;dur=12.34 so the browser DevTools
// Network tab shows your handler time without any client code.
app.use(timing());

// Inside a handler, use ctx.log (the framework already created a
// child logger bound to the request id) instead of the top-level log.
app.route({
  method: "GET",
  path: "/books",
  operationId: "listBooks",
  responses: { 200: { description: "ok" } },
  handler: async (ctx) => {
    ctx.log.info({ event: "list_books" }, "listing books");
    // ↑ every log line for this request carries requestId in the JSON
    return { status: 200, body: { items: [] } };
  },
});

// Output for one request — note the shared requestId across lines.
{"level":"info","time":"2026-06-03T08:42:11.108Z","service":"books-api",
 "requestId":"01HZQ4M8Z1F7E0SE0EH7E3WJW2","method":"GET","path":"/books",
 "msg":"request received"}

{"level":"info","time":"2026-06-03T08:42:11.110Z","service":"books-api",
 "requestId":"01HZQ4M8Z1F7E0SE0EH7E3WJW2","event":"list_books",
 "msg":"listing books"}

{"level":"info","time":"2026-06-03T08:42:11.114Z","service":"books-api",
 "requestId":"01HZQ4M8Z1F7E0SE0EH7E3WJW2","status":200,"durationMs":6,
 "msg":"request handled"}

// Server-Timing on the wire (DevTools Network tab → Timing):
HTTP/1.1 200 OK
content-type: application/json; charset=utf-8
x-request-id: 01HZQ4M8Z1F7E0SE0EH7E3WJW2
server-timing: app;dur=5.91

// src/tracing.ts — bring your own OTel tracer.
// DaloyJS does NOT import @opentelemetry/api. You install it, you
// configure it, you pass the tracer in. That's the whole API surface.
import { trace } from "@opentelemetry/api";
import { App, otelTracing } from "@daloyjs/core";

// 1. Wire your OTel SDK exactly as the OTel docs say (node SDK, edge
//    SDK, sdk-trace-base + a custom exporter, etc.). DaloyJS does not
//    care which one — it never touches the global provider.
import "./otel-bootstrap.js"; // ← your code; calls trace.setGlobalTracerProvider(...)

const tracer = trace.getTracer("books-api", "1.0.0");

const app = new App({
  hooks: otelTracing({
    tracer,
    // Use your routing knowledge to template the span name instead of
    // letting it be "GET /books/abc123" (high-cardinality nightmare).
    spanName: (req) => {
      const url = new URL(req.url);
      const m = url.pathname.match(/^\/books\/[^/]+$/);
      return m ? `${req.method} /books/:id` : `${req.method} ${url.pathname}`;
    },
  }),
});

# The four-hook lifecycle, in order, for a single request:
#
# 1) onRequest     — starts a SERVER span and stamps HTTP semantic
#                    attributes:
#                       http.request.method, url.path, url.scheme,
#                       server.address, url.query, user_agent.original
# 2) beforeHandle  — stores the span on ctx.state.otelSpan so handlers
#                    can add events / create child spans / set custom
#                    attributes ("user.id", "tenant.id", ...).
# 3) onError       — calls span.recordException(err) and sets status
#                    to ERROR with err.message. Runs ONCE.
# 4) onSend        — sets http.response.status_code, escalates to ERROR
#                    for 5xx if onError didn't already, and calls
#                    span.end() exactly once. Even if the handler threw.
#
# Everything is delivered through the standard Hooks contract from
# src/types.ts — no monkey-patching of fetch, no async-hooks magic,
# no surprise globals. The same hooks composition you'd write by hand.

// Add app-specific semantic attributes inside a handler.
// Use the OTel convention names so your existing dashboards work.
import type { Context } from "@daloyjs/core";
import type { TracingSpan } from "@daloyjs/core";

app.route({
  method: "GET",
  path: "/books/:id",
  operationId: "getBook",
  responses: { 200: { description: "ok" } },
  handler: async (ctx) => {
    const span = (ctx.state as { otelSpan?: TracingSpan }).otelSpan;
    span?.setAttribute("http.route", "/books/:id");
    span?.setAttribute("books.id", ctx.params.id);
    span?.setAttribute("tenant.id", ctx.state.tenantId as string);

    // Bonus: tie the request id and the trace together. Anyone reading
    // logs in Datadog can jump to the matching trace in Jaeger.
    ctx.log.info({ event: "get_book", bookId: ctx.params.id });

    return { status: 200, body: { id: ctx.params.id } };
  },
});

// You can use otelTracing() WITHOUT installing @opentelemetry/api.
// The framework only depends on the small TracingTracer/TracingSpan
// interfaces. Roll your own collector — useful on Workers, in tests,
// or for emitting OTLP HTTP directly from an edge function.
import { App, otelTracing } from "@daloyjs/core";
import type { TracingTracer, TracingSpan } from "@daloyjs/core";

const collected: object[] = [];

const tinyTracer: TracingTracer = {
  startSpan(name, options) {
    const startedAt = Date.now();
    const attrs: Record<string, unknown> = { ...(options?.attributes ?? {}) };
    let status: { code: number; message?: string } = { code: 0 };

    const span: TracingSpan = {
      setAttribute(k, v) { attrs[k] = v; },
      setAttributes(o) { Object.assign(attrs, o); },
      setStatus(s) { status = s; },
      recordException(err) {
        attrs["exception.type"] = (err as Error)?.name ?? "Error";
        attrs["exception.message"] = (err as Error)?.message ?? String(err);
      },
      end() {
        collected.push({ name, durationMs: Date.now() - startedAt, status, attrs });
        // Ship to your OTLP endpoint, console, KV, whatever:
        // fetch("https://otlp-http.example/v1/traces", { ... })
      },
    };
    return span;
  },
};

const app = new App({ hooks: otelTracing({ tracer: tinyTracer }) });

// W3C traceparent propagation — bring your own propagator.
// DaloyJS deliberately does NOT import a propagator (they pull in
// the OTel context API and aren't edge-safe in every runtime).
// Pass any function that returns the parent context you want.
import { propagation, ROOT_CONTEXT } from "@opentelemetry/api";

const app = new App({
  hooks: otelTracing({
    tracer,
    contextFromRequest: (req) => {
      // Build a header carrier from the Web Request:
      const carrier: Record<string, string> = {};
      req.headers.forEach((v, k) => { carrier[k] = v; });
      return propagation.extract(ROOT_CONTEXT, carrier);
    },
  }),
});

// Now upstream services that already started a trace (your gateway,
// a frontend RUM library, another service) get the child-of relation
// recorded automatically, with no per-handler code.

// otelTracing() is just hooks. You can compose your own exporter at
// the same layer — wrap startSpan to also push to your sink, or wrap
// otelTracing's onSend hook to also emit a log line.
import { otelTracing } from "@daloyjs/core";
import type { Hooks } from "@daloyjs/core";

const tracing = otelTracing({ tracer });

const tracingWithAccessLog: Hooks = {
  ...tracing,
  onSend(res, ctx) {
    tracing.onSend?.(res, ctx);             // keep span termination
    if (!ctx) return;
    ctx.log.info({
      event: "access",
      status: res.status,
      method: ctx.request.method,
      path: new URL(ctx.request.url).pathname,
    });
  },
};

app.use(tracingWithAccessLog);

// tests/observability.test.ts — verify spans without a real backend.
import { test } from "node:test";
import assert from "node:assert/strict";
import { App, otelTracing } from "@daloyjs/core";
import type { TracingTracer } from "@daloyjs/core";

test("emits one SERVER span per request with HTTP attributes", async () => {
  const spans: Array<{ name: string; attrs: Record<string, unknown>; status: number }> = [];

  const tracer: TracingTracer = {
    startSpan(name, options) {
      const attrs = { ...(options?.attributes ?? {}) } as Record<string, unknown>;
      let statusCode = 0;
      return {
        setAttribute(k, v) { attrs[k] = v; },
        setStatus(s) { statusCode = s.code; },
        recordException() {},
        end() { spans.push({ name, attrs, status: statusCode }); },
      };
    },
  };

  const app = new App({ hooks: otelTracing({ tracer }) });
  app.route({
    method: "GET",
    path: "/ping",
    operationId: "ping",
    responses: { 200: { description: "ok" } },
    handler: async () => ({ status: 200, body: { ok: true } }),
  });

  const res = await app.request("/ping");
  assert.equal(res.status, 200);
  assert.equal(spans.length, 1);
  assert.equal(spans[0]!.name, "GET /ping");
  assert.equal(spans[0]!.attrs["http.request.method"], "GET");
  assert.equal(spans[0]!.attrs["http.response.status_code"], 200);
});

# Runtime-by-runtime observability checklist.
#
# Node / Bun (long-lived):
# - createLogger to stdout, ship via fluent-bit / vector / your agent.
# - @opentelemetry/sdk-node + the OTLP exporter you prefer.
# - requestId({ trustIncoming: true }) ONLY if your reverse proxy
#   strips and rewrites x-request-id.
#
# Cloudflare Workers:
# - createLogger writes via console.log (the Workers runtime captures
#   it). Logpush ships to R2/S3/Splunk.
# - For tracing: roll a small TracingTracer that POSTs to an OTLP/HTTP
#   collector (or directly to vendor APIs like Honeycomb). The "no OTel
#   SDK" example above is the template.
# - Inherit traceparent via contextFromRequest — propagation only needs
#   reading the header; you can do that without the OTel propagator.
#
# Vercel Edge / Functions:
# - Same as Workers for Edge runtime. For Node functions, the standard
#   @opentelemetry/sdk-node works if the cold-start budget allows.
# - Vercel's Observability tab already reads structured stdout JSON.
#
# AWS Lambda:
# - createLogger to stdout (CloudWatch). Add bindings: { aws_request_id }.
# - @opentelemetry/sdk-node + the Lambda layer the OTel project ships,
#   or roll a tiny tracer + flush at the end of each invocation.
#
# Same App, every runtime: same Logger, same TracingTracer interface,
# different transports. The framework never assumes which one.

# Pre-flight observability checklist for any service.
#
# 1) requestId() at the top of the chain. trustIncoming=false unless
#    your edge proxy validates and rewrites the header.
# 2) createLogger with bindings: { service, env, region, version }.
#    Use ctx.log inside handlers, not the top-level instance.
# 3) timing() so the Network tab shows handler latency without
#    instrumentation in the frontend.
# 4) otelTracing({ tracer, spanName }) — TEMPLATE the span name. Raw
#    URLs are the most common source of cardinality explosions in
#    every observability bill on earth.
# 5) Wire contextFromRequest to your propagator. A trace that starts
#    at the frontend and ends in the database is the only kind worth
#    paying for.
# 6) Record exceptions and stamp http.response.status_code. The
#    framework does this for you — just verify in dev.
# 7) Tie logs to traces. Stamp the trace id on every log line; jump
#    from a log entry to the matching span in your UI of choice.
# 8) Keep the SDK out of @daloyjs/core. The portability story dies
#    the day you accept a hard dep on @opentelemetry/api.