How to read an email agent's activity log after a bad reply
After an AI agent sends a bad reply, read two records: the activity feed for what it did and the audit log for who changed its setup. Here is the order.
On this page(9 sections)
- What is an agent activity log?
- What should you do first after a bad reply?
- Step 1: find the send in the activity feed
- Step 2: for an Auto send, read the confidence and the grounding
- Step 3: check which rules should have held it
- Step 4: read the audit log for changes before the send
- What should you change afterward?
- A short post-incident checklist
- Bottom line
When an email agent sends a reply you wish it had not, the fastest way to understand it is to read two records in order: the activity feed, which says what the agent did, and the audit log, which says who changed its settings and when. Together they tell you whether the agent misjudged one message or was set up to make this mistake.
This post walks through that review for an agent in Koltrix, where the Auto mode is Beta and off by default. The same questions apply to any agent that sends mail, so the checklist is useful even if you build your own. For the weekly, routine version of this review, see auditing what an AI assistant did in your mailbox.
What is an agent activity log?
An agent activity log is the record of what an email agent did: which messages it drafted, which it sent, which it held for a person, and why. It is different from an audit log, which records changes to the agent itself, such as a setting edited or a key created. You need both, because a bad reply has two possible causes: a decision the agent made, or a setup someone gave it.
In Koltrix the two live in different places:
| Record | Answers | What an entry holds | Who can read it |
|---|---|---|---|
| Activity feed ("Done for you" in the Action center) | What did the agent do? | Time, agent address, kind of event, a short summary, the thread | Members with access to that agent mailbox |
| Audit log (one per agent mailbox) | Who changed what? | Time, actor, action, summary | Owners and admins |
The activity kinds are sent_auto, sent_approved, drafted, held, released, rejected and killed. The actor on an audit entry is a person, an API key (shown by its prefix) or the agent itself.
What should you do first after a bad reply?
Stop further sends before you investigate. If the agent is in Auto, switch it back to Suggest on its settings, or use the workspace kill switch, which puts every agent back to Suggest at once. Neither unsends the message: sent mail cannot be recalled, so the first job is to stop a second one. Then contact the recipient if the reply needs correcting, as covered in apology emails that rebuild trust.
Doing this first costs a minute and keeps the evidence intact. Switching autonomy back is itself written to the audit log, so you will see it later in the timeline.
Step 1: find the send in the activity feed
Open the activity feed and find the entry for the thread. The kind tells you which path the message took:
sent_auto: the agent sent it on its own. This is the case to examine most closely, because every rule it passed was supposed to make that safe.sent_approved: the agent proposed it and a person approved it. The question becomes what the approval screen showed and whether the person edited the text.drafted: the agent only wrote a draft. If the bad text went out anyway, a person sent it from the inbox.
Note the time, the agent address and the thread. Open the thread and read the message the agent was answering, not only its reply. Many bad replies are a reasonable answer to a message that was ambiguous, forwarded or not really a question.
Step 2: for an Auto send, read the confidence and the grounding
Koltrix writes an audit entry, agent.send_auto, for every autonomous send. It carries the agent's confidence, how many knowledge entries the answer was grounded in, and a short reason the agent gave. Read those three things together:
| What you see | What it suggests |
|---|---|
| Confidence only just above the threshold | The agent was near the line. Raise the bar by narrowing what it answers |
| Grounded in zero or one entries | It had little to go on. Add the missing answer to its knowledge |
| Confident, well grounded, still wrong | The knowledge itself is wrong or stale. Fix the source, not the agent |
| Reason does not match the message | It misread the request. Tighten its instructions |
An Auto send requires confidence of at least 0.85 and an answer grounded in what the agent knows. The reason text is written by the agent, so treat it as a clue and not as proof of what happened. It is shown as plain text for that reason.
If a send failed instead, the audit log shows agent.send_auto_failed. That is a delivery problem, not a judgment problem, and the message did not go out.
Step 3: check which rules should have held it
Auto is meant to stop short and ask a person when something looks off. A held item records why, using one of these reasons:
| Held reason | Meaning |
|---|---|
recipient_not_allowed / unknown_recipient |
The recipient was not on the allowed list, or was not already in the conversation |
daily_cap |
The agent had reached its daily send limit |
quiet_hours |
It was inside the hours you blocked |
injection_risk |
The incoming message looked like an attempt to steer the agent |
low_confidence |
Confidence was below the threshold |
duplicate |
An identical message went out in the last 10 minutes |
kill_switch |
Agents were switched back to Suggest |
Now compare. Which of these rules existed when the bad reply went out, and what were their values? A daily cap of 50 on an agent that should send five a day, or an empty allowed list that let it answer anyone in a thread, is a setup problem. Our posts on choosing a daily send cap and choosing quiet hours cover how to pick those numbers.
Step 4: read the audit log for changes before the send
Open the agent's audit log and look at the entries in the hours and days before the bad send. Every change to an agent is recorded: its autonomy level, allowed senders and recipients, daily cap, quiet hours, instructions, keys, and the knowledge entries added or removed. The record keeps ids, names and sizes, not the text itself, so you can see that a knowledge note was edited without the log holding its contents.
You are looking for one of three patterns:
- A recent change. Someone widened the cap, removed a recipient limit or edited the instructions shortly before. The change is the likely cause.
- A change by an unexpected actor. A key prefix you do not recognize, or a person who should not have access. That is a least-privilege problem, and the fix is to revoke the key.
- No change at all. The agent behaved as configured. The setup was permissive enough to allow the mistake, and the fix is a stricter setup, not a search for a culprit.
What should you change afterward?
Decide on one fix per cause, and make it the smallest one that works:
- Wrong or missing knowledge: correct the source the agent answers from.
- Too much latitude: move the agent from Auto to Approve, or narrow what it answers.
- Limits too loose: lower the daily cap, shorten allowed hours, or restrict recipients.
- A one-off misread: keep the agent as is, and add an escalation rule for that kind of message so it asks first.
A reasonable rule is to return to Suggest for a week after any bad Auto send, review the drafts, and promote only when the drafts have been right. What an approval step should show describes what that review screen needs to include.
A short post-incident checklist
- Stop further sends: back to Suggest, or the kill switch.
- Find the send in the activity feed and note its kind, time and thread.
- Read the message the agent was answering.
- For an Auto send, read confidence, grounding and the reason.
- List which rules applied and what their values were.
- Read the audit log for changes in the days before.
- Fix the cause, then write one line on what changed and why.
Bottom line
Read the activity feed to learn what the agent did and the audit log to learn how it was set up, in that order. Most bad replies trace back to one of three things: thin or wrong knowledge, rules that were looser than you meant, or a recent change nobody remembered. Stop the sending first, find the cause second, and change only what the evidence points to. To see how agents are set up in Koltrix, start from the autonomy dial, and keep Suggest as the starting level.
Start with Koltrix
Your domain, one inbox, and an API that sends.
A team inbox where AI sorts and drafts (nothing is sent without your click), plus the transactional API and SMTP relay your product sends with. 7 days free, no card.
