Guides

OpenTelemetry

Send a span for every call, plus duration metrics and connection logs, from your agent to your own backend through OpenTelemetry.

The SDK plugs into the OpenTelemetry setup your service already has. Every call the gateway sends your agent becomes a span in your own tracing backend (Datadog, New Relic, Grafana, or anything that accepts OTLP), with a duration histogram and connection logs alongside it. The gateway exports nothing and never holds a credential for your backend: the data leaves your process through the exporters you configure, and nowhere else.

Use it when you want to see what your handler did inside a call: which database query was slow, which downstream request failed. For who called what, and for the call log itself, use the dashboard (see Observability).

#The three places to look

The layers are independent. Layer 2 never touches the gateway. Two ids tie them together: the request id and the trace id (below).

#How it works

The SDK uses only the OpenTelemetry API. The API does nothing until your service registers an OpenTelemetry SDK, so:

• No OpenTelemetry SDK registered: every instrument is a no-op. There is no overhead and nothing is sent.
• OpenTelemetry API package not installed (Node.js): the SDK switches the bridge off. Nothing breaks.
• An OpenTelemetry SDK registered: spans, metrics and logs leave through your exporters, to whatever your pipeline points at.

Instrumentation can never break a call: your handler runs exactly once whatever happens to the instrumentation around it.

#Set it up

#Node.js

1. Install the API packages. Add @opentelemetry/api-logs as well if you want logs.

npm install @opentelemetry/api @opentelemetry/api-logs

2. Register an OpenTelemetry SDK in your service. If you already export traces from this service, skip to step 3. Otherwise this is a minimal setup that exports all three signals over OTLP/HTTP:

npm install @opentelemetry/sdk-node @opentelemetry/sdk-metrics @opentelemetry/sdk-logs \
  @opentelemetry/exporter-trace-otlp-http @opentelemetry/exporter-metrics-otlp-http \
  @opentelemetry/exporter-logs-otlp-http

import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { OTLPLogExporter } from '@opentelemetry/exporter-logs-otlp-http';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { BatchLogRecordProcessor } from '@opentelemetry/sdk-logs';

export const sdk = new NodeSDK({
  serviceName: 'orders-agent',
  traceExporter: new OTLPTraceExporter(),
  metricReaders: [new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter() })],
  logRecordProcessors: [new BatchLogRecordProcessor({ exporter: new OTLPLogExporter() })]
});

sdk.start();
process.on('SIGTERM', () => void sdk.shutdown());   // flushes what is buffered

3. Start OpenTelemetry first, then the agent.

import { sdk } from './tracing.ts';   // starts OpenTelemetry on import
import { Agent } from '@spacefix/core';
import { telemetry } from '@spacefix/telemetry';

const agent = new Agent({ apiKey: process.env.SPACEFIX_API_KEY! });
agent.use(telemetry());                 // also publishes process metrics through OpenTelemetry
agent.register(tools);
void agent.connect();

That is all. The agent needs no OpenTelemetry options. To see spans for the database and HTTP calls your handler makes, your service also needs the instrumentation for those libraries (for example @opentelemetry/auto-instrumentations-node, loaded before anything imports your database driver). The SDK makes the handler's span the parent. The instrumentation creates the children.

Two options on the agent:

new Agent({ apiKey, otel: { argKeys: true } });   // also record argument names
new Agent({ apiKey, otel: false });               // turn the bridge off

For process metrics through OpenTelemetry the switch is telemetry({ otel: false }).

#Go

The Go adapter is a separate module, so the core SDK stays free of third-party code. You add it only if you want OpenTelemetry. It needs Go 1.26 or newer, because the OpenTelemetry Go libraries do.

go get github.com/softmode/spacefix-go/otel \
  go.opentelemetry.io/otel/sdk go.opentelemetry.io/otel/sdk/metric go.opentelemetry.io/otel/sdk/log \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp \
  go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp \
  go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp

Register the OpenTelemetry SDK, create the adapter, and attach it with UseObserver before Run:

package main

import (
	"context"
	"log"
	"os"
	"os/signal"
	"syscall"

	"github.com/softmode/spacefix-go/core"
	spacefixotel "github.com/softmode/spacefix-go/otel"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp"
	"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	logglobal "go.opentelemetry.io/otel/log/global"
	sdklog "go.opentelemetry.io/otel/sdk/log"
	sdkmetric "go.opentelemetry.io/otel/sdk/metric"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

// setupOTel registers tracing, metrics and logging that export over OTLP/HTTP.
// The destination comes from OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS.
func setupOTel(ctx context.Context) (shutdown func(context.Context), err error) {
	res := resource.NewSchemaless(semconv.ServiceName("orders-agent"))

	traceExp, err := otlptracehttp.New(ctx)
	if err != nil {
		return nil, err
	}
	tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(traceExp), sdktrace.WithResource(res))
	otel.SetTracerProvider(tp)

	metricExp, err := otlpmetrichttp.New(ctx)
	if err != nil {
		return nil, err
	}
	mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(sdkmetric.NewPeriodicReader(metricExp)), sdkmetric.WithResource(res))
	otel.SetMeterProvider(mp)

	logExp, err := otlploghttp.New(ctx)
	if err != nil {
		return nil, err
	}
	lp := sdklog.NewLoggerProvider(sdklog.WithProcessor(sdklog.NewBatchProcessor(logExp)), sdklog.WithResource(res))
	logglobal.SetLoggerProvider(lp)

	return func(ctx context.Context) {
		_ = tp.Shutdown(ctx) // flushes what is buffered
		_ = mp.Shutdown(ctx)
		_ = lp.Shutdown(ctx)
	}, nil
}

type OrderInput struct {
	ID string `json:"id" desc:"Order id"`
}

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()

	shutdown, err := setupOTel(ctx) // first, so the providers exist before the agent runs
	if err != nil {
		log.Fatal(err)
	}
	defer shutdown(context.Background())

	agent, err := core.New(core.Options{APIKey: os.Getenv("SPACEFIX_API_KEY")})
	if err != nil {
		log.Fatal(err)
	}

	obs, err := spacefixotel.New(spacefixotel.Options{}) // set ArgKeys: true to also record argument names
	if err != nil {
		log.Fatal(err)
	}
	if err := agent.UseObserver(obs); err != nil {
		log.Fatal(err)
	}

	err = core.AddTool(agent, "get_order", "Look up an order by id.",
		func(ctx context.Context, in OrderInput) (map[string]string, error) {
			return map[string]string{"id": in.ID, "item": "Widget"}, nil
		})
	if err != nil {
		log.Fatal(err)
	}

	if err := agent.Run(ctx); err != nil { // blocks until the signal arrives
		log.Fatal(err)
	}
}

The adapter uses the global tracer, meter and logger providers by default. spacefixotel.Options also accepts TracerProvider, MeterProvider and LoggerProvider if you would rather pass them explicitly. Inside a handler, the ctx you receive carries the active span, so an instrumented HTTP client or database driver nests its spans under the call.

The adapter does not publish the process metrics that the Node.js telemetry package does. For Go process metrics, use the telemetry package (agent.Use(telemetry.New(...))) (for the dashboard) or OpenTelemetry's own runtime instrumentation (for your backend).

#Point it at your backend

Every backend below accepts OTLP, so only the environment changes. The code above does not.

# An OpenTelemetry Collector: the most flexible option, it fans out to any backend.
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
OTEL_SERVICE_NAME=orders-agent

# Enable OTLP ingest on the Datadog Agent, then point the SDK at it.
OTEL_EXPORTER_OTLP_ENDPOINT=http://<datadog-agent-host>:4318
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta
OTEL_SERVICE_NAME=orders-agent

# Use otlp.eu01.nr-data.net for EU accounts.
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.nr-data.net:4318
OTEL_EXPORTER_OTLP_HEADERS=api-key=<your-license-key>
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta
OTEL_SERVICE_NAME=orders-agent

Grafana (Tempo for traces, Prometheus or Mimir for metrics, Loki for logs), Honeycomb and others use the same two variables with their own endpoint and header. Vendors change their endpoints, so check each one's current OTLP guide. Keep the key in your secrets manager and in the environment, never in code.

#What is emitted

#Spans

One SERVER span per call that reaches your agent, from the instrumentation scope @spacefix/core.

A failed call also gets span status ERROR with the error message, and an exception event with the error's details.

Your own spans. Handlers run inside the call's span, so spans you create nest under it:

import { trace } from '@opentelemetry/api';
const tracer = trace.getTracer('orders');

agent.tool('get_order', { inputSchema: z.object({ id: z.string() }) }, ({ id }) =>
  tracer.startActiveSpan('orders.lookup', async span => {
    try { return await orders.find(id); } finally { span.end(); }
  })
);

tracer := otel.Tracer("orders")

func(ctx context.Context, in OrderInput) (Order, error) {
	ctx, span := tracer.Start(ctx, "orders.lookup")
	defer span.End()
	return orders.Find(ctx, in.ID)
}

#Metrics

From the SDK, scope @spacefix/core (Node.js and Go):

Only tools carry a name on the histogram. Resource uris and prompt names stay off metrics to keep them manageable, and are on the spans instead.

From the Node.js telemetry package, when it is attached and its otel option is on (the default), scope @spacefix/telemetry:

The histogram gives you percentiles the dashboard's average cannot. Useful questions:

• 95th percentile per tool: the 95th percentile of mcp.server.operation.duration, grouped by gen_ai.tool.name.
• Error rate: calls with error.type set, divided by all calls.
• Is it the code or the process? Compare nodejs.eventloop.delay.mean with the duration. Slow handlers with a quiet loop point at your dependencies. A high loop delay means the process itself is saturated.

#Logs

With a logger provider registered (Node.js also needs @opentelemetry/api-logs), the agent emits log records from the logger @spacefix/core:

Handler failures are emitted inside the call's span, so the record carries the trace id and your backend links the log to the trace.

#Follow one call end to end

A call has two ids, and both are visible in the dashboard and in your backend.

To investigate a failure, open the call's row on Logs, copy its Trace id, and search for it in your backend. You see the whole request: the client's span (if the client traces), your agent's span, and everything your handler did. In the other direction, a span's spacefix.request_id tells you which Logs row it is.

#Trace propagation

1. The MCP client may send the W3C traceparent (and tracestate) headers on its request.
2. The gateway keeps a valid traceparent as it is, or starts a trace if the client sent none or sent something malformed, and passes it to your agent along with the request id.
3. The agent starts its span as a child of that context and runs your handler inside it.
4. The agent reports its span id and the handler's duration back with the result, so the Traces page can show your agent's bar.

When the gateway started the trace, your span carries that trace id and the gateway's invented span id as its parent.

#Privacy

• Argument values and results are never recorded. Not on spans, not on metrics, not in logs.
• Argument names are off by default. Turn them on for debugging. They appear as spacefix.arg_keys, capped at 20.
• Error messages are recorded, on the span status, the exception event and the log body, because you need them to debug. They go only to your backend, and they contain whatever your handlers put in them. Do not put secrets or personal data in error messages.
• The gateway stores the trace id and request id on its call log. Neither is sensitive.
• Nothing leaves your process except through the exporters you configured.

#Verify it works

The quickest check in Node.js is to print spans to the console. Swap the trace exporter while you try it:

import { NodeSDK } from '@opentelemetry/sdk-node';
import { ConsoleSpanExporter } from '@opentelemetry/sdk-trace-base';

new NodeSDK({ traceExporter: new ConsoleSpanExporter() }).start();

Spans are batched, so they appear a few seconds after the call.

Make a call from a real MCP client, or with curl and your own traceparent:

curl -X POST https://api.spacefix.io/mcp/<project-id> \
  -H "Authorization: Bearer <client-key>" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"get_order","arguments":{"id":"1"}}}'

You should see a span named tools/call get_order with trace id 4bf92f3577b34da6a3ce929d0e0e4736 and parent span id 00f067aa0ba902b7, and the same trace id on the call's Logs row.

Calls from the dashboard's Playground also get a span, but they carry no traceparent and no spacefix.request_id, so use a real client to check propagation.

#Troubleshooting

#Limitations

• Only calls that reach your agent are traced. A call the gateway refuses, or that times out before your agent answers, produces no span. Those appear in Logs only. A call that times out and is then answered late still produces a span, because the handler ran.
• A trace the gateway starts has a parent span that no backend has. When the client sends no traceparent, some backends mark your span as having a missing parent. Instrumenting the client removes this.
• Clock skew affects the "estimated" bars. spacefix.gateway.queue_ms and the transit time on the Traces page compare the gateway's clock with your host's. Hosts with synced clocks agree within a few milliseconds. A skewed host reads off by the skew. Handler time is measured on one clock and is exact.
• The OpenTelemetry logs API is still evolving, so the log signal depends on a pre-1.0 package in Node.js. Traces and metrics depend on stable APIs.
• Not recorded: argument values, results, your API key, and anything about calls the gateway answered itself (cache hits).

#Next steps

• Observability: the dashboard pages that sit next to your own backend.
• Node.js SDK and Go SDK: the rest of each SDK's options.
• Troubleshooting: when something does not show up where you expect.