OpenClaw Agent Observability: Detecting Silent Failures

The scariest failure in production isn't the crash. A crash is loud — the process disappears, the supervisor restarts it, you get a notification. The dangerous failure is the one where everything stays green: the cron fires, the agent wakes up, the transcript ends with “Done — report published,” and no report exists. Traditional monitoring can't see this, because every layer it inspects is genuinely healthy. This is the observability problem specific to agentic systems, and it needs its own playbook.

Why Agent Failures Hide

A conventional service fails by returning an error. An agent fails by returning prose. The model is a text generator wrapped in a process, and the process exits 0 whether the model did the work or merely narrated doing the work. Every silent failure I've investigated on this fleet traces back to that one structural fact, expressed through one of four mechanisms:

• Narrated work. The agent describes an action it never took. Exit code 0, transcript reads perfectly, filesystem unchanged.
• Truncated context. The run started with a bootstrap that overflowed the context budget, so the agent silently lost half its instructions and executed the half it still had.
• Swallowed stderr. A wrapper script redirects stderr to /dev/null or an unwatched file. The underlying tool failed loudly; nobody was listening.
• Resource starvation. The host hit a process or memory ceiling, the spawn failed with EAGAIN, the agent interpreted the empty result as “nothing to do,” and reported success.

Only the last one shows up in a CPU graph. The rest are invisible to infrastructure monitoring by construction.

The Three Layers Worth Instrumenting

Useful agent observability stacks into three layers, each answering a strictly harder question than the one below it. Most deployments instrument only the first, then wonder why outages go undetected for days.

Layer 1 — Process Liveness: “Is it running?”

This is the cheap layer, and it's where the diagnostic breadcrumbs live. Two specifics matter more than people expect.

Capture exit status, not just uptime. Under launchd, read LastExitStatus; under systemd, read ExecMainStatus. The distinction between exit -15 / 143 (SIGTERM — something external killed you, usually resource pressure) and exit 1 (your code threw) changes the entire investigation. A supervisor with KeepAlive enabled will restart the process so fast that uptime graphs never dip, which means the restart itself is the only evidence a failure occurred. Log it.

Never discard stderr. The most persistent blind spot on any agent host is a wrapper script that pipes stderr somewhere nobody reads. If your gateway writes to a log file, confirm something actually tails it and alerts. An unwatched log isn't observability; it's a filing cabinet.

Layer 2 — Run Completion: “Did the scheduled run happen?”

Here you invert the default alerting logic. Alerting on errors is useless when the failure mode produces no error. Instead, alert on absence — a dead man's switch. Every scheduled agent, as its final action, writes a heartbeat record: one line of JSON with the job name, an ISO-8601 timestamp, a duration, and a status. A separate watcher checks that each expected heartbeat is newer than its interval plus a grace window. If the daily briefing job hasn't stamped a heartbeat in 26 hours, that fires, regardless of what any transcript claims.

Two rules keep this honest. First, the heartbeat must be written by the same code path that does the work, not by the wrapper that invokes it — otherwise you're monitoring your scheduler, which was never the thing at risk. Second, the watcher should be a plain script on a plain timer. If your failure detector is an agent, it can narrate its own success too. Keep at least one link in the chain dumb.

Layer 3 — Work-Product Assertion: “Did the artifact change?”

This is the layer that catches narrated work, and almost nobody builds it. The principle: assert on the artifact, never on the transcript. Every job needs a machine-checkable definition of done that is independent of anything the model said. If the job publishes a page, assert the route returns 200 and its content hash differs from yesterday's. If it writes a report, assert the file exists, was modified within the run window, and exceeds a minimum byte count. If it commits code, assert HEAD moved. If it sends a message, assert the send API returned a message ID, and record that ID.

The byte-count check matters more than it looks. A common failure is the agent that successfully writes a file containing an apology for being unable to complete the task: it exists, it's fresh, and it's 180 bytes of regret. A minimum-size assertion catches it; an existence check does not. Treat a failed assertion as a failed run even when the agent insists otherwise, and in multi-agent pipelines assert at every handoff rather than only at the end — otherwise you inherit a corrupted intermediate and debug the wrong stage. That coordination discipline is covered in depth in mastering multi-agent coordination.

Resource Pressure Is a Leading Indicator

Layers 1 through 3 are detection. Resource telemetry is prediction, and on a single-host deployment it's the highest-value leading indicator you can collect. Agent runtimes spawn subprocesses generously — shells, browsers, language servers, sub-agents. Every one of those is a slot against the host's process ceiling, and not all of them get reaped. The characteristic pattern is that nothing degrades gracefully: process count drifts upward across a week of normal use, then a spawn-heavy run crosses the limit and several unrelated services take SIGTERM simultaneously. The cluster looks like a mysterious correlated outage. It's a resource ceiling.

Two practices defuse it. Sample the process count owned by your agent user — not the machine total, which is dominated by desktop applications you can't control — and record it alongside every run. Then set a spawn gate: a pre-flight check that refuses to fan out sub-agents above a threshold measured from your own idle baseline. A job that declines to start and says why is far better than four services dying at once.

Log Shape: One Line Per Run

Agent transcripts are enormous and nearly unsearchable. Don't try to make them your observability substrate. Emit one structured line per run to a separate append-only log, carrying: job name, start and end timestamps, duration, exit status, the assertion result from Layer 3, a token or cost figure if your runtime exposes one, and the owned-process count at start. That's it.

This format is small enough to keep for a year and regular enough to answer real questions with grep and jq: which job's duration is creeping up, which one fails every third Tuesday, whether cost spiked before or after a config change. Keep full transcripts too — they're the forensic record once an alert fires — but rotate them aggressively and never query them first. Persisted state, including these logs, follows the same durability rules as everything else in OpenClaw's memory model.

A Realistic Starting Point

You don't need all of this on day one, and building it out of order wastes effort. The sequence that pays back fastest:

• Week one: stop discarding stderr; log exit status on every restart.
• Week two: add heartbeats plus one dumb watcher that alerts on staleness. This alone catches most real outages.
• Week three: write a work-product assertion for your single most important job. Watch how often it disagrees with the transcript — that number is usually the surprise.
• Week four: sample process counts and set a spawn gate from your measured baseline.

The underlying shift is small but hard: stop treating the agent's own account of its work as evidence. Everything above is machinery for getting a second opinion from the filesystem, the scheduler, and the network. Once you have that, agents become genuinely operable — and you can extend the same assertion discipline to scheduled work as you add it, starting with your first OpenClaw cron job.

Related Reading

• OpenClaw Security Best Practices: Securing Your AI Agent Deployment
• Setting Up Your First OpenClaw Cron Job
• Mastering Multi-Agent Coordination in OpenClaw
• OpenClaw Memory Explained: A Beginner's Guide

Frequently Asked Questions

What is a silent agent failure?

A run that completes with a success exit code and a plausible transcript but produces no real change in the world — no file written, no page published, no message sent. It's distinct from a crash because every conventional health signal stays green, which is exactly why it needs work-product assertions rather than process monitoring.

Can I use my existing monitoring stack for OpenClaw agents?

For Layer 1, yes — Prometheus, Grafana, or whatever you already run handles process liveness fine. Layers 2 and 3 are usually custom, because the definition of done is specific to each job, but they're small: a heartbeat writer is a few lines, and an assertion is usually a single check on a file, a route, or a git ref.

Should the failure watcher itself be an agent?

No. At least one link in the detection chain should be a plain script on a plain timer. If your detector is a model-driven agent, it can experience the same narrated-success failure mode as the job it's watching, and you lose the independence that made the check worth having.

What's the single highest-value thing to add first?

A staleness alert on heartbeats. It's the cheapest layer to build and catches the largest share of real outages, including cron jobs that silently stopped firing weeks ago — a failure that otherwise goes unnoticed until someone asks where a report went.