Tracing

Every call, run, and step can record a span. The format is W3C Trace Context and OTLP, so a HighX trace joins whatever you already run — Jaeger, Tempo, Honeycomb, Braintrust — rather than asking you to adopt something HighX-shaped.

Nothing is recorded until you ask for it. Telemetry costs latency, bytes, and somebody's bill, so it is opt-in rather than opt-out.

For environment-driven setup, telemetryFromEnvironment() selects OTLP when OTEL_EXPORTER_OTLP_ENDPOINT is set, uses the console exporter locally otherwise, and honors OTEL_SDK_DISABLED, OTEL_SERVICE_NAME, resource attributes, and OTLP headers.

Turning it on

import { OtlpTraceExporter, SpanRecorder } from "@highx/observability";

const recorder = new SpanRecorder({
  exporter: new OtlpTraceExporter({
    endpoint: "https://collector.example.com",
    resource: { serviceName: "notes" },
    headers: { "x-api-key": OTEL_API_KEY },
  }),
  onError: (error) => console.error("telemetry", error),
});

export const TenantObject = createTenantObject<Env>({
  schema: databaseSchema,
  functions: serverFunctions,
  telemetry: {
    tracer: recorder,
    // Called when a unit of work ends. Exporting on every call would add a
    // round trip to every request, so batch until it is worth sending.
    flush: () => (recorder.queued >= 64 ? recorder.flush() : Promise.resolve()),
  },
});

The object flushes when a call returns, when a workflow is started or resumed, when an app request finishes, and before its alarm hands back control — the last moment before it may hibernate.

What gets recorded

Span names are the same for every function and every step; identity lives in attributes. One name per function would make the backend's index as large as your application.

The function span covers authorization and validation as well as the handler, so a denied caller is a failed call rather than a call that never appears.

Agent runs

An agent run is a workflow, so it needs no separate instrumentation — but a turn is a model call and the tool calls that follow from it, which are separate steps. The loop groups them:

highx.workflow.drive              9.41s   run.id=run_7 workflow.name=review
└ highx.agent.turn                8.90s   agent.phase=publishing agent.turn=1
   ├ gen_ai.chat                  8.10s   gen_ai.request.model=claude-opus-5
   │                                      gen_ai.usage.input_tokens=1200
   │                                      gen_ai.usage.output_tokens=340
   │                                      highx.usage.reasoning_tokens=180
   │                                      highx.usage.cost_micros=4200
   └ highx.tool.call              0.62s   highx.tool.name=publish
                                          highx.tool.approval=granted

The model call uses OpenTelemetry's gen_ai.* names, so an agent run renders in a tracing backend with no HighX-specific configuration.

Cost is on the call itself. The usage ledger answers "what has this tenant spent today"; only the span answers "which turn was expensive", and no aggregate can be taken apart into that afterwards.

Every tool call appears, including the ones that did not run

A refused call never becomes a step, so it would otherwise leave no trace at all — and "the model kept asking for a tool it may not use" is exactly what someone reading a disappointing run needs to see.

Replay records nothing

A workflow resumes by replaying its recorded steps from the top. Replayed steps emit no spans at all. A run driven ten times would otherwise emit its first step ten times over, and the trace would claim the work happened again when all that happened was a resume.

Suspension is not failure either. A run that sleeps is working, and marking those spans as errors would make every long workflow look like an incident — the drive span records highx.run.outcome: "sleeping" and stays unset.

The same holds for an agent waiting on an approval. Its earlier turns replayed when a person answered; only the new turn is recorded, so the trace shows the model being called once because it was called once.

A run outlives the request that started it

A run is linked to the call that started it, not nested inside it. Its trace ID is stored on the run, so a workflow that sleeps for three days resumes still knowing which story it belongs to.

Each drive attempt is its own span within that trace. Ten wakes read as ten spans rather than one that appears to have taken ten days.

await tenant.startWorkflow("review", args, identity, runId);
// The run's spans share ctx.trace's trace ID, as siblings rather than children.

Trace context in your code

ctx.trace is the handler's place in the trace. Forward it on anything that leaves HighX, and the far side joins the trace instead of starting its own.

import { formatTraceparent } from "@highx/core";

export const publish = action({
  args: publishArgs,
  returns: publishResult,
  auth: true,
  handler: async (ctx, args) => {
    const response = await ctx.fetch("https://api.example.com/publish", {
      method: "POST",
      headers: {
        "content-type": "application/json",
        ...(ctx.trace === undefined ? {} : { traceparent: formatTraceparent(ctx.trace) }),
      },
      body: JSON.stringify(args),
    });
    return response.json();
  },
});

Inbound, forward the caller's header into the object so a request that crosses into a tenant stays one trace:

const result = await tenant.execute(
  path,
  args,
  identity,
  crypto.randomUUID(),
  request.headers.get("traceparent") ?? undefined,
);

A missing or malformed traceparent starts a new trace rather than failing the call. A bad header is the caller's instrumentation problem, not a reason to refuse their work.

Local development

Running a collector is reasonable to ask of a deployment and unreasonable to ask of somebody who wants to see why their agent took nine seconds.

import { ConsoleSpanExporter, SpanRecorder } from "@highx/observability";

const recorder = new SpanRecorder({ exporter: new ConsoleSpanExporter() });

· highx.workflow.drive 9.41s
  · highx.agent.turn 8.90s
    · gen_ai.chat 8.10s
    · highx.tool.call 0.62s

What spans will not carry

Attribute values are string, number, or boolean, and strings are truncated at 1KB. An attribute is a label, not a payload — anything richer belongs in a log line that the span's IDs point at.

Attribute names that read as secrets — apiKey, sessionToken, authorization — are redacted by the same rule the logger uses. Span attributes leave for a third-party backend, which is a worse place for a credential to land than a log file.

When a step fails, the span keeps both the wrapper and the cause: statusMessage names the step that failed and error.cause says what actually went wrong. A trace carrying only the first sends you to the logs to learn anything useful.

Failure is never yours to handle

Telemetry never breaks the work. A tracer that throws is caught, a failed export is reported to onError rather than raised at the caller, and a full queue drops spans and counts them rather than growing without bound.

A failed batch is dropped rather than retried forever — requeueing lets a broken collector grow the queue until the object is evicted, trading a gap in the traces for a memory problem in production. The exporter still retries transient failures (429, 5xx, network) within one attempt budget, and never retries a 401 or a 400 it would only earn again.