How to watch an agent run

Watching an agent run means following the work from its trigger to its final state. A transcript shows what was said. A run should also show what the agent was asked to do, which tools it called, what those tools returned, what changed in the operating system, and whether a person took over.

Begin with the trigger

Identify what started the run: a new call, an inbound text, a cancelled appointment, a completed visit, or a manual request from the team. Confirm that the trigger belongs to the intended job.

A recovery agent should not start from an appointment that was merely moved. A review request should not start before checkout closes. A front-desk run should not reopen a thread that a person already owns.

Read the plan as a boundary

The plan should state the outcome the agent may pursue and the actions it may take. It should be narrow enough to inspect. “Help the customer” is not a plan. “Find a valid 60-minute appointment at Riverside and book it after confirmation” is.

Inspect every tool call

For availability, check service, duration, location, provider, and time range. For booking, check the selected slot and returned record. For waitlist, check the opening, candidate order, and active hold. For handoff, check the reason and owner.

An agent may produce a perfectly reasonable sentence from incomplete inputs. The tool call is where that mistake becomes visible.

Confirm the side effect

If the run claims an appointment was booked, open the calendar. If it claims a cancellation was filled, confirm the old opening is gone and the queue is closed. If it claims a person owns the thread, verify the alert and assignment.

Do not accept a “success” status as proof of the operating result. The destination system is the source of truth.

Review the stop as carefully as the completion

A stopped run should name the condition, preserve the conversation, and reach the correct person. Check whether the agent stopped too early, too late, or for the wrong reason. Repeated handoffs can expose missing tools or unclear policy. Missing handoffs can expose unsafe scope.

When the agent should hand off to a person provides a practical ownership model.

Watch in context before expanding

Quiet test data hides collisions. Review runs during the period when staff, rooms, calls, and walk-ins compete. Compare the agent’s view with what the person on the floor sees. Then change one rule or mapping at a time and watch the next set of runs.

For the live operating test, use Tuesday will lie to you about whether the agent actually works.

Read the run in chronological order

Start with the event timeline rather than the final summary. The order reveals whether the agent asked for missing information before calling a tool, whether it rechecked a slot after the customer changed the request, and whether it continued acting after a handoff.

A useful timeline distinguishes customer messages, agent decisions, tool calls, system events, and human actions. Timestamps matter when two people compete for one slot or when a delayed integration response arrives after the conversation moved on.

The summary can help navigation, but it should not replace the raw sequence when the team investigates a mistake.

Compare the plan with the permitted job

The plan should stay inside the deployed job. If a front-desk agent was asked to book a standard appointment, the plan should not include changing policy, offering an unapproved discount, or creating a different service to make the time fit.

Look for the exact outcome, constraints, and stop conditions. A strong plan may say: identify the requested service, confirm location, find a valid provider and room, offer up to three options, book after explicit confirmation, and hand off on a complaint or off-menu request.

If the plan expands after a customer objection, inspect whether the new action is still allowed. Helpful language can hide scope drift.

Inspect arguments, not only tool names

Seeing “availability called” proves very little. Open the arguments. Check that duration, service, location, resource, provider preference, and time range match the conversation.

Then inspect the result. Did availability return a genuinely bookable option or a provider-only opening? Did booking return a stable record? Did waitlist create one active hold or message an uncontrolled list?

Follow data back to its source

When a tool argument is wrong, identify where it came from. The customer may have stated it. The agent may have inferred it. A service map may have supplied it. The calendar may have returned stale data.

This source tracing determines the fix. If the service duration is wrong in the POS, changing the prompt will not create a reliable system. If the agent ignored a clear correction from the customer, the conversation logic needs attention. If the field was missing, the plan may need one clarifying question.

Understand retries and duplicate protection

Tool calls can time out after the destination completed the action. A run may retry because it did not receive the response. Check whether the tool uses an idempotency key and whether the retry returned the existing record or created a duplicate.

An unknown result should trigger a destination check before another write. If the run simply repeats booking calls until one returns success, it is not safe for a live calendar.

Review the run beside the destination

Open the calendar, inbox, waitlist, or business profile in a second view. The run record explains intention. The destination proves the side effect.

Compare identifiers, times, duration, provider, resource, and status. If a person corrected the booking later, preserve both the original run and the correction. That history is more useful than rewriting the run to look clean.

Classify failures before fixing them

Use a stable failure taxonomy:

• Trigger: the wrong event started the job.
• Context: identity, service, location, or booking was missing.
• Plan: the agent chose work outside the approved path.
• Tool input: arguments did not match the conversation.
• Tool execution: integration failed or returned uncertainty.
• Policy: the business had no defined answer.
• Handoff: the agent stopped, but ownership failed.
• Destination: a person or system changed state during the run.

Fix the nearest layer. A taxonomy keeps review meetings from ending with “make the agent smarter,” which is not an actionable diagnosis.

Sample successful runs too

Teams naturally review errors and handoffs. Also sample clean completions. A workflow can appear successful while relying on an accidental calendar gap, an overly broad service mapping, or a person quietly correcting the result.

Select runs across services, channels, languages, locations, and busy periods. Confirm that routine success follows the same contract each time.

Watch for silent customer effort

A run may complete only after asking the customer to repeat details already present in the thread or book. Count unnecessary restarts, repeated identity questions, and long option lists.

The goal is not the fewest messages at any cost. It is the shortest safe path using context the business already has permission to use.

Build a weekly review board

Create a compact operating view with five columns: completed, refused, handed off, corrected, and unresolved. Add counts by job and location, then open representative runs rather than reading every transcript.

The board should make change visible. If corrected bookings rise after a service-map update, investigate before adding more volume. If handoffs cluster around one policy, decide whether to clarify the rule or keep that boundary human.

Control changes and compare versions

Record when a prompt, service map, tool contract, or policy changes. Tag runs with the active version. Without that link, the team cannot tell whether a failure happened before or after the fix.

Change one important variable at a time when possible. Watch the next representative set of runs before continuing. Rapid, overlapping edits make improvement impossible to attribute.

Decide when a run needs immediate intervention

Most review can happen after the fact. Some conditions should alert the team immediately: repeated booking conflicts, tool writes with unknown outcomes, high-priority handoffs without an owner, or an agent continuing after a person accepts the thread.

Define these alerts before launch and keep the list short. If every refusal creates an urgent notification, the team will ignore the one that matters.

Use a run review to approve expansion

Before adding another service, channel, or location, review a representative set from the current scope. Confirm clean destination records, stable tool results, correct boundaries, and manageable handoffs.

Expansion is an operating decision supported by run evidence. It should not happen because the demo looked good or the launch date arrived.

Run an incident replay

When a booking, hold, or handoff goes wrong, copy the original inputs into a safe test environment. Preserve the active versions of the service map, tool contract, policy, and agent instructions. Replay the run without changing several variables at once.

Confirm the failure, apply the smallest correction, and run the same fixture again. Then test a nearby success case to ensure the fix did not narrow valid work or create a new handoff.

Document the cause, correction, evidence, owner, and release time. Link the production runs affected by the issue so the team can repair customer or calendar state where necessary.

Keep a pause threshold

Define conditions that pause the job automatically or instruct the operator to pause it: repeated unknown writes, duplicate bookings, destination mismatch, unowned urgent handoffs, or a sudden increase in corrections.

The threshold should be specific enough to act on. The pause procedure should preserve open threads, stop new side effects, alert the owner, and explain the temporary response customers receive.

Watching runs is useful only when the team can intervene before a pattern becomes a larger operating problem.

Preserve examples of correct restraint

Keep runs where the agent refused an impossible slot, skipped a duplicate ask, or handed off before making a protected decision. These examples teach reviewers that safe non-completion is part of quality, not a defect to optimize away.