Guides

Running replicas

Run more than one copy of your service: how calls are shared, why state must live in a database, how to deploy without dropping calls, and what happens when a replica dies.

Each copy of your service that runs the SDK opens its own connection to your project, so one project can have many agents. This is how your tools scale: exactly as your service does. This page explains how calls are shared between replicas, the one rule you must follow (do not keep state in process memory), how to deploy without errors, and what happens when a replica stops.

#How calls are shared

• Calls are spread round-robin, per tool, across the project's connected, enabled agents that have the requested tool (or resource or prompt). Each tool keeps its own place in the rotation, so a mix of tools still spreads evenly. There is no sticky routing: do not rely on the same replica answering next time.
• A replica that disconnects leaves the rotation at once, and calls it was running fail immediately with the reason, instead of waiting for the timeout.
• A disabled agent stays connected but is skipped (see Disabling an agent).
• The tool list a client sees comes from one enabled agent: the one that has been connected longest. This assumes every replica registers the same tools. Give every replica the same tools.
• Replicas may be written with different SDKs. A Node.js agent and a Go agent can serve the same project as long as they register the same tools.

You can see every replica on the Agents page, and on the project page under Connected agents: its id, how long it has been connected, how many tools, resources and prompts it registered, and its calls and errors since it connected.

#The rule: each replica has its own memory

Because calls are spread across replicas, a tool that keeps state in a variable gives different answers depending on which replica received the call. Handlers must be stateless: keep anything that must be seen by more than one call in a database, a cache or a queue that every replica can reach.

Anything you hold per process (counters, in-memory caches, the "already seen" set that makes a write idempotent) is per replica.

#A worked example: the shared-state mistake

This agent keeps its list in an array. It works perfectly with one replica.

const items = []; // each replica has its own copy of this array

agent.tool(
  'add_item',
  { description: 'Add an item to the list.', inputSchema: z.object({ item: z.string() }) },
  ({ item }) => {
    items.push(item);
    return `added ${item} on ${replica}`;
  }
);

agent.tool('get_items', { description: 'List every item.' }, () => ({ replica, items }));

var (
	mu    sync.Mutex
	items []string // each replica has its own copy of this slice
)

func add(ctx context.Context, in AddItemInput) (string, error) {
	mu.Lock()
	defer mu.Unlock()
	items = append(items, in.Item)
	return "added " + in.Item + " on " + replica, nil
}

Run it as two replicas, A and B, of one project and call it through the gateway. This is the real output:

add_item   {"item":"x"}  ->  added x on A
get_items                ->  {"replica":"A","items":["x"]}
get_items                ->  {"replica":"B","items":[]}      <- B never saw the write

The write landed on A. The next read was served by A and looked right; the one after was served by B and the list looked empty. Nothing is broken in the tool or the gateway: the data was never shared. The mistake is invisible with a single replica, which is how it reaches production.

The fix is to move the state into storage every replica can see:

import pg from 'pg';

const pool = new pg.Pool(); // reads PGHOST, PGDATABASE and friends from the environment

agent.tool(
  'add_item',
  { description: 'Add an item to the list.', inputSchema: z.object({ item: z.string() }) },
  async ({ item }) => {
    await pool.query('insert into items (name) values ($1)', [item]);
    return `added ${item}`;
  }
);

agent.tool('get_items', { description: 'List every item.' }, async () => {
  const { rows } = await pool.query('select name from items order by id');
  return rows.map(r => r.name);
});

func registerItems(agent *core.Agent, db *sql.DB) error {
	if err := core.AddTool(agent, "add_item", "Add an item to the list.",
		func(ctx context.Context, in AddItemInput) (string, error) {
			_, err := db.ExecContext(ctx, "insert into items (name) values ($1)", in.Item)
			return "added " + in.Item, err
		}); err != nil {
		return err
	}

	type none struct{}
	return core.AddTool(agent, "get_items", "List every item.",
		func(ctx context.Context, _ none) ([]string, error) {
			rows, err := db.QueryContext(ctx, "select name from items order by id")
			if err != nil {
				return nil, err
			}
			defer rows.Close()
			items := []string{}
			for rows.Next() {
				var name string
				if err := rows.Scan(&name); err != nil {
					return nil, err
				}
				items = append(items, name)
			}
			return items, rows.Err()
		})
}

Now every replica reads and writes the same table, and it does not matter which one answers. A tool that is a thin adapter over the service layer you already have (which already uses your database) is correct by construction.

#Rolling deploys

In a rolling deploy, new replicas connect before old ones leave, so there is never a moment with nobody connected, provided your platform starts the new replicas before it stops the old ones (on Kubernetes, a rolling update with maxUnavailable set to 0).

#Stop an agent cleanly

When a replica is told to stop, call disconnect() (or cancel the context in Go) so running calls can finish:

for (const signal of ['SIGINT', 'SIGTERM']) {
  process.on(signal, async () => {
    await agent.disconnect(); // waits up to 5 seconds for running calls; disconnect({ graceMs }) changes it
    process.exit(0);
  });
}

ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
// Run returns nil after ctx is cancelled and running calls have finished
// (up to Options.ShutdownTimeout, 10 seconds by default).
if err := agent.Run(ctx); err != nil {
	log.Fatal(err)
}

While it drains, the agent keeps its connection open so running calls can post their results. A call that arrives in that window is answered at once with an error (in the Node.js SDK, Agent is shutting down.) rather than being run. To avoid even that, disable the old agent first, as below, and stop it once In flight reaches 0.

#Changing the tool list during a deploy

The tool list is read from the longest-connected enabled agent. While a deploy that adds or renames a tool is half-rolled, the list clients see is the old replica's. A call is only ever sent to a replica that registered that tool, so a tool that exists on some replicas still works, but one that exists on none fails with No connected agent serves tool "<name>". The list settles when the rollout ends.

If that matters, deploy tool changes in two steps, the way you stage a database migration: ship the handler first, then expose it.

#Disabling an agent

On the Agents page, Disable takes one replica out of rotation without disconnecting it. Use it to drain a replica before stopping it, to pull a misbehaving process out of service, or to see how clients behave with fewer replicas.

Press Disable on the agent's row or on its page, then Disable agent in the dialog (Disable agent <id>?). Enable acts immediately.

To drain safely: Disable, watch In flight reach 0, then stop the process. To switch off one tool for every client instead, see Building tools.

#When a replica dies

If a replica crashes or loses its network while running a call:

• The gateway notices the closed connection and fails the call at once with Agent disconnected before returning a result., instead of waiting for the timeout. The client sees it as an isError result.
• The gateway does not send the call to another replica. Delivery is at most once: it cannot know whether your handler already did its work, so it never runs a call twice by itself. If the caller retries, your tool should be safe to repeat (see Idempotency).
• The replica leaves the rotation immediately; new calls go to the remaining agents.
• If it was the last replica, clients get 503 and No agent connected for server "<project id>".

The same applies when a key is revoked or expires: the agent is disconnected and calls it was running fail (Key revoked., API key expired.).

#Reconnection

A reconnect (a network blip, a restart, a deploy, the gateway restarting) is a new connection: a new agent id, back to enabled, with its Calls and Errors counters starting from zero. Logs, traces and usage keep the history.

Both SDKs reconnect on their own. They wait 1 second after the first failure and double the wait up to 30 seconds, with random jitter so a fleet that lost its connection together does not reconnect in lockstep, and they honour a Retry-After from the gateway. A successful connection resets the wait. A failed connection attempt logs a line such as [spacefix] Connect failed: 503 ...; retrying in 2000ms.

Anything that cuts the connection can make an agent appear, vanish and return with a new id each time: a proxy with a short idle timeout, a platform that freezes idle processes, or a crash loop. The gateway sends a keep-alive every 25 seconds.

#Connection limits by plan

A project can hold a limited number of agents at once: 32 on the Free plan by default. An agent that connects past the limit is refused with 429 and Too many agent connections: the Free plan allows 32., and keeps retrying, so it connects as soon as there is room. Connection attempts are also limited per project (60 a minute on the Free plan); past that, the gateway answers 429 with Retry-After. See Plans and limits for every plan.

#Checklist

1. Give every replica the same tools, resources and prompts.
2. Keep state in a database, cache or queue, not in process memory, and make writes idempotent.
3. Call disconnect() (or cancel the Go context) on shutdown.
4. Start new replicas before stopping old ones.
5. Disable an agent and wait for In flight to reach 0 when you need a zero-error drain.
6. Use a separate project, and a separate key, for each environment.

#Next steps

• Building tools: idempotent writes and stateless handlers.
• Projects and keys: rotate an agent key across replicas without downtime.
• Concepts: the model behind projects, agents and replicas.