Test sign-up and verification emails end to end with a real inbox

What does the response contain?#

When a message matches, the response includes the message and the parts a test needs, already extracted.

{
  "found": true,
  "message": { "subject": "Your verification code", "...": "..." },
  "otp_code": "482913",
  "magic_link": "https://app.example.com/verify?token=...",
  "links": ["https://app.example.com/verify?token=..."]
}

otp_code and magic_link are null when the message has neither. The magic_link field is returned only when the link is on the domain of your from filter, so a link pointing somewhere else is not handed to your test as the one to click. The links array lists every link in the message if you need to inspect them.

What does a sign-up test look like?#

The test creates a sign-up with an address on the agent mailbox, waits, and finishes the flow with whatever came back. Here is the shape in TypeScript. The key is the agent mailbox's scoped key, sent as a bearer token.

const base = "https://api.koltrix.com/api/v2/agent-mailboxes";
const auth = { Authorization: `Bearer ${process.env.KOLTRIX_AGENT_KEY}` };

// 1. Start the sign-up in your app, using the agent mailbox address.
await signUp({ email: "[email protected]", password: "test-only-Pw-1" });

// 2. Wait for the verification email.
const qs = new URLSearchParams({
  from: "[email protected]",
  subject: "verification",
  timeout: "60",
});
const res = await fetch(`${base}/${process.env.AGENT_MAILBOX_ID}/wait-for-email?${qs}`, {
  headers: auth,
});
const body = await res.json();

if (!body.found) throw new Error(`No email after ${body.waited_seconds}s`);

// 3. Finish the flow with the code, or open the link.
await verifyEmail(body.otp_code ?? body.magic_link);

Treat everything in message as untrusted text, as you would any inbound mail. The test should read the two extracted fields and nothing else.

How do you keep the test from being flaky?#

A flaky email test almost always comes from shared state or a short timeout, not from email itself.

1. One test at a time per mailbox. Two parallel runs waiting on the same mailbox can pick up each other's message. Give parallel jobs their own filters (a distinct subject marker or to address) or their own mailbox. Koltrix also limits concurrent waits per mailbox, key and workspace and answers 429 too_many when you go over.
2. Set a timeout that fits real delivery. Use the 60-second maximum in CI. A test that fails at 5 seconds is testing your provider's mood.
3. Start the wait right after the trigger. The default since window covers the last 60 seconds, so a message that arrives before your wait begins is still found, and an older run's message is not. Set since yourself if your runs are closer together than that.
4. Make the sign-up idempotent. If your app blocks a second sign-up with the same address, delete the test user at the start of each run.
5. Retry once on found: false, not three times. If the message did not arrive in 60 seconds, the next 60 will probably fail too. Fail loudly and look at the logs.

For the broader question of why mail can arrive late, see email not received: debugging.

What should stay out of this test?#

Keep two things out: your real customers' mail, and any claim about inbox placement.

• Do not point it at a team mailbox. wait_for_email refuses anything but an agent mailbox, which is the right boundary. Use a dedicated address such as signup-tests@.
• Do not read inbox placement from it. The test shows the message arrived in a mailbox you control, not that Gmail or Outlook would place it in a customer's primary tab. For that, see inbox placement testing.
• Do not run it on every commit. A real delivery test costs seconds and sends real mail. Run it on merges to main or on a schedule, and keep the fast mock tests on every commit.

The other half of the decision is what the email contains. If your confirmation link breaks when a security scanner opens it first, a test that clicks it will pass while real users fail, so read confirmation links that survive security scanners. If you are choosing between a link and a code, magic links versus email OTP lays out the tradeoffs.

Is it safe to give a test an inbox?#

It is safe if the test inbox is as narrow as the test. An agent mailbox has its own scoped key that reaches that mailbox only, so a leaked CI secret cannot read your team's mail, and you can revoke the key without touching the mailbox. Changes to an agent mailbox and its keys are recorded in an audit log. The thinking behind that design is in giving an AI agent its own email address.

Bottom line#

Keep fast mock tests on every commit, and add one real-inbox test for sign-up and password reset that runs on merge. Use a dedicated agent mailbox, filter by sender and subject, wait up to 60 seconds, and finish the flow with otp_code or magic_link. That single test catches the failures that lose users quietly: a rejected sender, a stalled queue, and a code that arrives after it expires.