Guides

Observability

What you can see about your traffic in the dashboard, and how to follow one call from a gateway log row into your own tracing backend.

Every call that reaches your agent is recorded, and the dashboard turns those records into pages you can use to answer four questions: is my agent healthy, who is calling what, why did that call fail, and where did the time go. This page walks through each page, says what it needs from you, and ends with how to follow one call from a row in Logs into Datadog, New Relic, Grafana or any other tracing backend you already use.

#Where to look

All times on these pages are UTC.

#Dashboard

/dashboard is the first page after you sign in. It refreshes itself every few seconds. If you have more than one project, filter chips at the top narrow every number to one project.

#Headline cards

#Process tiles

Heap used, RSS (resident memory), CPU (a share of one core) and Event loop (mean delay, with how busy the loop was). The event loop delay is the number that predicts tool latency: a handler can be fast while the process is saturated, and then every call queues.

These tiles and the heap, CPU and loop columns of the Per project table need telemetry in your agent (see Telemetry from your agent). Until an agent reports, the page says Nothing reporting yet.

#Traffic flow

A live diagram of calls moving from clients, through your agents, to the tools, resources and prompts they run. It draws one movement per real call and nothing else moves.

The eye button hides the moving traffic, and your choice is remembered. On a narrow screen the diagram scrolls sideways inside its card.

The lock button next to it holds the diagram in place. Unlock it to drag the diagram around, scroll to zoom, and double-click (or press reset) to put it back. The lock is remembered; the position is not.

The diagram is a short-lived picture, not a record: it keeps about a minute of events and empties when the gateway restarts. Logs is the record.

#Per project

One row per project: Status, Agents, Tools, Heap, CPU, Loop, Uptime. Status is one of:

#Traffic & reliability

Switch between 24h (hourly buckets) and 7d (daily buckets). It shows four figures with the change against the previous period (Calls, Success rate, Median latency and the 95th percentile latency), a chart of calls with the failed share in red, the busiest tools, and What stands out, a short list of things worth acting on.

This is computed from your call log, so it works without telemetry, and it only reaches back as far as your plan keeps logs: on a plan with 3 days of history the week toggle reads 3d. The notes appear after about 20 calls:

#Telemetry from your agent

Process health is optional. It comes from a separate package that you attach to the agent. Reports ride the connection the agent already has, over the same key, so there is nothing else to configure.

npm install @spacefix/telemetry

# Built into the core module: nothing more to install.

Then attach it to the agent before it connects:

import { telemetry } from '@spacefix/telemetry';

agent.use(telemetry());   // reports every 10 seconds
void agent.connect();

// Before agent.Run.
err := agent.Use(telemetry.New(telemetry.Options{
	Interval: 10 * time.Second, // the default
}))

Metrics appear within one interval. A failing report is swallowed, so a telemetry problem never takes a tool down.

Each sample covers one window and reports:

A Go agent has no event loop, so it reports the closest equivalents: the mean time goroutines wait to be scheduled, and how busy the runtime was. They rise when the process is overloaded, like event loop delay does in Node.js.

The gateway keeps the most recent samples in memory only and drops them when a project's last agent disconnects, so the process tiles are a live view, not history. For history of every number, send it to your own backend with OpenTelemetry.

#Agents

/agents lists the processes holding a connection to the gateway right now: a project with three replicas has three agents. Each row shows the project, how long it has been connected, how many tools, resources and prompts it registered, and its calls and errors since it connected. A green dot means connected and healthy, a red dot means at least 5% of its calls have failed, and an amber dot means it is Disabled.

Open one to see everything it registered, its process figures from telemetry, and its last 30 calls. A Disable button takes an agent out of rotation without disconnecting it: it keeps its connection but gets no new calls, and calls already running finish. Disabling lasts until the agent disconnects, so a restarted agent comes back enabled.

An agent's page is live, not history. Its id exists only while it is connected, and its counters start at zero each time it connects. For history use Usage, Logs and Traces.

#Usage

/usage shows calls across all your projects (filter to one with the chips). It covers the last 30 days.

• This month is your monthly allowance as a bar: calls used of the limit, resetting on the 1st (UTC). Playground runs are not counted. See Plans and limits.
• Tiles: Calls, Errors, Success rate and Mean latency.
• Calls per day for the last 14 days, with failed calls in red.
• The table: one row per day, project and tool, with calls, errors, error rate and average time. Resource reads are listed under their uri and prompts under their name.
• Busiest tools and, when you have more than one project, By project.

• Where time goes, Why calls fail and Clients. These three come from the call log, so they cover your plan's log window rather than 30 days:

  • Where time goes splits the mean call time between your handler and everything else (the gateway and the network), for calls where your agent reported its handler time.
  • Why calls fail counts failures by type (see Logs). Each links to the Logs page filtered to that type.
  • Clients counts calls and errors by the MCP client's reported name (claude-code), or by the User-Agent of a caller that reported none.

Export CSV downloads the table with the columns day, project, tool, calls, errors, avg_ms. By default it covers the last 30 days. Cells that start with =, +, - or @ are prefixed with an apostrophe so a spreadsheet does not run them as formulas. You can run 6 exports a minute.

#Logs

/logs lists every call, newest first, updating while you are on the first page. A call is a tool call, a resource read or a prompt that reached your agent, whether it succeeded, failed or timed out.

#Columns and details

The table shows Time (UTC), Name (the tool name, or the resource uri or prompt name with a resource or prompt tag), Project, Status (ok or error), Duration, Size in to out, and Error. Click a row for details:

#Filters

Project, Status (Any status, Succeeded, Failed), Type (Tools, Resources, Prompts), Source (Clients or Playground) and a tool name box. When you filter to failures, a failure type filter appears. Page sizes are 10, 20, 50 and 100. The tiles above the table follow the filters: calls, success rate, 50th and 95th percentile latency, and data moved.

#Failure types

A request refused before it reaches an agent is not logged and not counted: a bad key, a rate limit, a suspended project, an allowance already spent, arguments that fail the tool's schema, or no agent connected at all. It never ran your code, so look for it on the client side. See Troubleshooting and Errors.

#Retention and export

Rows are kept for your plan's history: 3 days on Free. Longer plans keep more; see Plans and limits. A row's expiry is fixed when it is written, so moving to a bigger plan keeps new rows longer but does not extend old ones.

Export CSV writes time_utc, project, tool, kind, ok, ms, arg_bytes, result_bytes, error, request_id, trace_id, source, client, handler_ms, agent_inbound_ms, error_kind. It is limited to your retention and to 100,000 rows, and shares the 6-a-minute export limit.

#Traces

A trace is the path of a call, or of several calls that share a trace id, between the MCP client, the gateway and your agent. /traces lists them, newest first: Started (UTC), Trace id, First call, Project, Calls, Status, Duration and Started by. Filters are Project, Status, Source and tool name.

Client traces and gateway traces. If the MCP client sent a traceparent header, the call joins the client's trace and is labelled a client trace. One client trace can hold many calls. If it sent none, the gateway starts a trace and the call is labelled a gateway trace. Either way every call that reaches an agent has a trace id. Playground runs get a trace of their own, so they appear here too.

Open a trace for the timeline. Select any row to expand it for where it ran, when it started, how long it took, its share of the trace, its span id and notes.

• A marker for the client's own span, when there was one. The gateway cannot know how long the client's span ran. Only the client's tracing backend does.
• A gateway bar for each call: the time the gateway spent on it end to end (red if it failed, with the reason beside the name). With several calls, each is numbered (#1, #2, ...) and parallel calls overlap.
• Three legs inside it for your agent: gateway → agent (the call travelling to you), handler (your code, with the agent that ran it) and agent → gateway (the result coming back).
• Each bar carries its duration, and the longest single step is tagged slowest.

Around the timeline the page adds:

• What stands out: plain-language findings, such as why a call failed (timed out, agent error, or no agent connected), whether the time is mostly in your handler or outside it, and whether calls ran in parallel.
• Where the time went: one bar splitting the total into your handlers, gateway and transit, time the agent gave no breakdown for, and calls that failed before an answer.
• Trace details: trace id, start and finish, who started it, project, the agents that ran the calls and the SDK versions they reported, credential, client and the data moved.
• A Spans table (where, start, duration, share, span id, notes) and the calls in the trace with size in and out, the agent, request id, credential, client and, for a failed call, the error.

Why some bars say "estimated". Two things are measured exactly: the gateway's time for the whole call, and how long your handler ran (measured by your agent on its own clock). Where the handler sits inside the call is inferred from the gateway-to-agent latency your agent observed, and that figure compares two different computers' clocks. So the legs on either side of the handler are drawn lighter and tagged estimated. If your agent did not report that latency, the transit time is split evenly between the two legs. If it reported no timing at all, you see only the gateway bar.

The queries and HTTP requests your handler makes are not on this page. They are spans in your backend, under the same trace id.

#Activity

/activity is the audit trail of your account: sign-ins, key and project changes, tool setting changes, two-factor changes, webhooks and more. It is kept for at least 30 days. See Security.

#Following a call into your own backend

The gateway exports no telemetry and never holds credentials for your backend. Instead your agent emits spans, metrics and logs through your own OpenTelemetry setup, and the gateway hands each call's trace context to the agent so the two line up. Setup is in OpenTelemetry. In short:

• If your MCP client sends traceparent, your agent's span is a child of the client's span, in the same trace.
• If it sends none, the gateway starts the trace and your agent's span carries that trace id.
• Your handler runs inside that span, so queries and HTTP calls made by instrumented libraries nest under it.

To go from a gateway log row to a trace in Datadog, New Relic or similar:

1. Open the call's row on Logs (or the trace under Traces) and copy the Trace id: 32 hexadecimal characters, for example 4bf92f3577b34da6a3ce929d0e0e4736. On a trace's page, the Copy trace id button does it.
2. In your tracing backend, search for that trace id. The field name and query syntax depend on the vendor, so check its documentation for searching by OTLP trace id.
3. You see the whole request: the client's span (if the client traces), your agent's tools/call <name> span, and everything your handler did.

In the other direction, a span's spacefix.request_id attribute is the Request id in the Logs row details, so from a slow span you can find the gateway's row.

Only calls that reach your agent produce a span. A call the gateway refuses, or one that times out before your agent answers, appears in Logs only. And when the client sent no traceparent, the span's parent is a span the gateway invented, which no backend has, so some backends mark your span as having a missing parent. Instrumenting the client removes that.

#What is stored, and what is not

Error messages are stored, and they are whatever your handler threw, so do not put secrets or personal data in them. For the full picture see Security.

#Things worth knowing

• Writes are buffered for about two seconds. Calls are written in batches, so a call can take a moment to show up. The pages flush before they read, so your own just-made calls are there when you load a page.
• Cache hits are invisible. If you set a cache time on a tool, a call answered from the cache never reaches your agent and is not logged, traced or counted.
• Per-tool figures on an agent's page are per project. The call log does not record which replica ran a call, so with several replicas the per-tool numbers cover all of them.
• The week toggle is bounded by your plan. The dashboard can only show what your log retention keeps.
• Playground runs are logged (marked playground, credential dashboard session) but never counted toward your allowance, and have no trace.

#Next steps

• OpenTelemetry: send spans, metrics and logs from your agent to your own backend.
• Alerts and webhooks: get told when an agent drops or your allowance runs low.
• Plans and limits: how long logs are kept and how calls are counted.
• Troubleshooting: when a call fails and you do not know why.