KoltrixTesting transactional email in CI without real sends
Assert on the message your code builds, not on an inbox. Layered tests for transactional email: rendering units, contract tests and a few end-to-end checks.
On this page(10 sections)
- Decide what you are actually testing
- Layer 1: rendering tests
- Layer 2: behavior tests with a fake sender
- Layer 3: wire-format tests
- Layer 4: contract tests for HTTP email APIs
- Layer 5: real delivery, outside the main pipeline
- Keeping tests fast and deterministic
- Things to keep out of CI entirely
- Checklist
- Key takeaways
The worst way to test transactional email in CI is to send real messages to a real inbox and poll for them. It is slow, flaky, rate-limited and occasionally emails someone who should not get it.
The better way is to test the message your code builds, at several layers, and keep real delivery out of the pipeline almost entirely.
Decide what you are actually testing
"Does email work?" is several separate questions, each best answered at a different layer:
| Question | Best layer |
|---|---|
| Does the template render with real data? | Unit test |
| Is the right email triggered by the right event? | Unit or integration test with a fake sender |
| Is the MIME message well formed with correct headers? | Integration test against a captured message |
| Does our code call the provider API correctly? | Contract test against a stub |
| Does the provider accept and deliver it? | Occasional smoke test outside the main pipeline |
Most bugs live in the first three rows, and all three can run in milliseconds without a network.
Layer 1: rendering tests
Render every template with fixture data and assert on the output. These tests catch undefined variables, broken syntax, missing text parts and bad links.
- Configure the template engine to fail on undefined variables.
- Snapshot the rendered subject, HTML and text so changes are reviewed deliberately.
- Assert that links are absolute HTTPS URLs on allowed hosts.
- Include awkward fixtures: very long names, non-ASCII characters, empty optional fields, zero and many line items.
These run in the same suite as your other unit tests and need nothing special.
Layer 2: behavior tests with a fake sender
Put sending behind an interface and swap in a fake that records messages instead of sending them:
class FakeMailer:
def __init__(self):
self.sent = []
def send(self, message):
self.sent.append(message)
def test_password_reset_sends_one_email(app, fake_mailer):
app.request_password_reset("[email protected]")
assert len(fake_mailer.sent) == 1
msg = fake_mailer.sent[0]
assert msg.to == ["[email protected]"]
assert "Reset your password" in msg.subject
assert "https://app.example.com/reset?token=" in msg.text_bodyThis layer answers the questions that matter most for product behavior:
- Exactly one email is sent, not zero and not two.
- It goes to the right address and only that address.
- Unverified or suppressed recipients get nothing.
- Retried jobs do not produce duplicates.
- Sensitive values, such as tokens, appear where they should and nowhere else.
Assert on structured fields rather than scraping HTML. If your message object exposes the subject, recipients and text body, tests stay readable and do not break on cosmetic template changes.
Layer 3: wire-format tests
Some bugs only appear once the message is serialized: wrong MIME nesting, missing headers, charset problems, attachments in the wrong place. Capture the bytes your mail library produces and parse them back:
from email import message_from_bytes, policy
def test_receipt_mime_structure(build_receipt):
raw = build_receipt(order_id="A-10293").as_bytes()
msg = message_from_bytes(raw, policy=policy.default)
assert msg.get_content_type() == "multipart/mixed"
body, attachment = list(msg.iter_parts())
assert body.get_content_type() == "multipart/alternative"
assert [p.get_content_type() for p in body.iter_parts()] == ["text/plain", "text/html"]
assert attachment.get_filename() == "receipt-A-10293.pdf"
assert msg["Message-ID"]If your app uses SMTP, you can go one step further and run a local SMTP catcher as a CI service container, send through real SMTP code to it, and fetch messages from the catcher's API. That exercises your SMTP client configuration without any external network.
Layer 4: contract tests for HTTP email APIs
If you send through a provider's HTTP API, test your client against a stub that checks the request shape:
- The method, URL and authentication header are correct.
- The JSON body has the fields the provider documents, with the right types.
- An idempotency key is sent and stays stable across retries of the same logical send.
- Error responses (4xx, 429, 5xx) are handled as intended: retried, surfaced, or dropped.
Tools that record and replay HTTP interactions work, but a hand-written stub with explicit assertions is usually clearer. Keep the stub honest by updating it when the provider's documentation changes.
Layer 5: real delivery, outside the main pipeline
Real delivery testing still has a place, just not on every commit. A scheduled job, perhaps daily or after deploys, can:
- Send a message through the real provider to a dedicated test mailbox you own.
- Check that it arrives within an expected time.
- Verify authentication results in the received headers (SPF, DKIM, DMARC pass).
Keep this job separate so its occasional flakiness, caused by real networks and real filtering, does not block merges. Alert on repeated failures rather than single ones.
Keeping tests fast and deterministic
Email tests become flaky for the same reasons other tests do, plus a few of their own:
- Time. Templates that print "expires in 15 minutes" or today's date produce different output every run. Freeze the clock in tests, or pass the timestamp into the template as data.
- Randomness. Tokens, Message-IDs and MIME boundaries are random by design. Inject a deterministic generator in tests, or normalize those values before comparing snapshots.
- Asynchronous sending. If email is sent by a background job, tests must either run the job inline or wait for the queue to drain explicitly. Sleeping for a second and hoping is how intermittent failures start.
- Locale and time zone. CI machines often run in UTC with a different locale from developer laptops. Set both explicitly in the test environment so dates and currency format the same everywhere.
Fast, deterministic tests get run; slow, flaky ones get skipped, and skipped tests protect nobody.
Things to keep out of CI entirely
- Production credentials. CI should never hold a key that can send to arbitrary recipients from your production domain.
- Real customer addresses. Fixtures use reserved domains such as
example.com,example.netandexample.org, which can never belong to a real person. - Polling real inboxes in pull-request builds. It turns a millisecond assertion into a minute of flakiness.
Checklist
- Rendering tests with strict variables and snapshots for every template.
- Behavior tests with a fake sender asserting count, recipients and key content.
- Wire-format tests parsing serialized MIME for structure and headers.
- Contract tests for the provider API client, including retries and idempotency.
- A separate scheduled smoke test for real delivery and authentication.
- No production keys and no real addresses anywhere in CI.
Key takeaways
- Test the message your code builds, not the inbox it lands in.
- Rendering, behavior and wire-format tests catch most email bugs without any network.
- A fake sender makes "exactly one email to the right person" a simple assertion.
- Keep real-delivery checks in a scheduled job so they cannot slow down or flake your main pipeline.
Start with Koltrix
Your domain, one inbox, and an API that sends.
A team inbox where AI sorts and drafts (nothing is sent without your click), plus the transactional API and SMTP relay your product sends with. 7 days free, no card.
