Operations & troubleshooting

This page covers the runtime behavior of hermes-bridge — what it does under load and failure — and how to diagnose common problems.

Concurrency model

• Per-chat ordering. Each chat gets its own worker. Messages within a chat are processed strictly in order; only one Hermes call per chat is ever in flight.
• Cross-chat parallelism. Different chats run concurrently, bounded by BRIDGE_MAX_CONCURRENT (default 4).
• Idle retirement. A chat’s worker retires after an idle period and respawns on the next message, so memory doesn’t grow with the number of chats ever seen.
• Backpressure. Each chat buffers a bounded number of pending messages. If a chat floods faster than the agent can answer, the oldest overflow is dropped (and logged) rather than growing without bound. Dropped messages are not marked as processed, so a stream reconnect can still replay and deliver them.

Failure handling

Circuit breaker

A consecutive-failure circuit breaker protects against a down or slow Hermes box:

1. After BRIDGE_BREAKER_THRESHOLD consecutive failures the breaker opens.
2. While open, triggered messages get the BRIDGE_FALLBACK_MESSAGE (rate-limited per chat so an outage can’t flood a chat) instead of hanging.
3. After BRIDGE_BREAKER_COOLDOWN the breaker goes half-open and admits a single probe call. Success closes it; failure re-opens it for another cooldown.

The concurrency slot is acquired before the breaker is consulted, so a granted half-open probe always records its outcome — the breaker can never get stuck open.

Replay and dedupe

The bot stream buffers recent events (≈5 minutes / 100 events per bot). On reconnect the bridge resumes from the last event id it processed, and de-duplicates by message id so a replayed message is never answered twice. Disconnects longer than the replay window lose events permanently.

Timeouts

Each Hermes call is bounded by BRIDGE_REQUEST_TIMEOUT (default 120s) — long enough for tool-using runs, which can legitimately take 10–120s. A reply the agent already produced is still posted even if shutdown begins while waiting.

Typing indicator

While a Hermes call is in flight the bridge emits a typing/“thinking” signal, refreshed periodically (clients auto-expire stale indicators), and explicitly stops it when the call ends. This requires DONUT_TYPING_URL to be set — otherwise the agent works silently.

Agent answers larger than the 4,000-byte bot-message limit are stored as ordered continuation replies in the same thread, so the client displays the complete answer without exposing a downloadable file URL. A single answer is capped at 48,000 bytes (at most 16 replies) to bound memory and database write amplification from untrusted prompts; a visible notice is appended when that safety limit is reached.

Presence

A bot shows online when it holds a live WebSocket connection to the bot stream (i.e. the bridge is connected). Presence is resolved from hub state.

Process-local presence

Presence is currently process-local. In a multi-instance backend deployment it must be backed by a shared store (e.g. Redis) before bot traffic is sharded. Single-instance deployments are unaffected.

Troubleshooting

Symptom	Likely cause	Fix
Bridge exits immediately on start	Missing/invalid required env var	Read the startup error; it names the offending variable. See Configuration.
Bot never replies to @mentions	Bot not in the chat, or trigger mode is manual	Add the bot to the chat; set trigger mode to mention or all.
Bot replies to every message	Trigger mode is all	Switch the chat’s trigger mode to mention.
Replies show raw **markdown**	BRIDGE_REPLY_CONTENT_TYPE is TX	Set it to MD (the default).
No typing indicator	DONUT_TYPING_URL unset	Set DONUT_TYPING_URL (and hermes_bridge_typing_url on ECS).
Every message gets the fallback text	Hermes unreachable → breaker open	Check HERMES_BASE_URL/HERMES_API_KEY and that Hermes is reachable from the bridge’s network.
Bridge can’t connect to the stream	Bad/revoked bot token, or wrong DONUT_STREAM_URL	Verify the donutbot_… token and the WSS URL. A regenerated token must be re-deployed.
Bot answers another bot’s messages	BRIDGE_RESPOND_TO_BOTS=true	Set it to false (the default) to restore the loop guard.

Logs

Set LOG_LEVEL=debug for verbose output. The bridge logs connection lifecycle, dropped messages (with chat and message ids), agent-call failures, and breaker transitions. Message content is never logged at info level.

Security checklist

• Keep the Hermes API server on a private network reachable only by the bridge.
• Store DONUT_BOT_TOKEN and HERMES_API_KEY in a secret manager, never in plaintext task definitions.
• Configure Hermes with a restricted toolset — chat input is an untrusted prompt-injection surface. Never route chat input through the hermes-webhook toolset (it grants terminal access).
• A bot can only post to chats it is a member of — this is enforced server-side.

Related

• Overview — what the integration is and how it fits together.
• Deployment — set it up locally or on ECS.
• Configuration — every environment variable.
• Bot Receive / Bot Send — the underlying bot APIs the bridge uses.