KoltrixKeeping staging environments from emailing real customers
Copying production data into staging is how customers get test emails. Use allowlists, recipient rewriting and separate credentials to make it impossible.
On this page(11 sections)
Almost every company that runs a staging environment has, at some point, sent a test email to real customers. The fix is not more care; it is making the mistake structurally impossible.
It usually happens the same way: someone copies production data into staging to reproduce a bug, a background job wakes up, and a few thousand people receive "TEST invoice please ignore." The guardrails below make that sequence harmless even when someone gets it wrong.
How it happens
The failure modes repeat across companies:
- Production data restored into staging with real email addresses intact.
- Scheduled jobs (renewal reminders, digests, trial expiry notices) running in staging against that data.
- Shared credentials, where staging uses the same email API key or SMTP account as production.
- Queue replays, where a developer replays a batch of jobs to reproduce a bug.
- Webhook-triggered flows, where a payment provider's test mode still triggers emails to the addresses on file.
- Developer laptops configured with production environment variables "just for a minute."
Each of these is a reasonable action on its own. The problem is that nothing stands between the action and a real inbox.
Layer 1: separate credentials per environment
Staging must never hold credentials that can send as your production domain to arbitrary recipients. Concretely:
- Create separate API keys or SMTP accounts for each environment.
- Scope staging keys narrowly. If your provider supports restricting keys to specific recipients, domains or a sandbox mode, use it.
- Use a separate sending domain or subdomain for staging, such as
staging-mail.example.com, so any message that does escape is clearly not from production and does not affect production reputation. - Store production credentials only where production runs. They should not appear in staging configuration, CI variables for non-production jobs, or developer
.envfiles.
If a staging key leaks or is misused, the damage is limited to what that key can do.
Layer 2: a recipient allowlist enforced in code
The strongest guardrail lives in your own sending code, right before the message leaves your application. In any environment other than production, check every recipient against an allowlist and refuse to send to anything else:
import os
ALLOWED_DOMAINS = {"example.com"} # your company domain
ALLOWED_ADDRESSES = {"[email protected]"}
def guard_recipients(recipients: list[str]) -> list[str]:
if os.environ.get("APP_ENV") == "production":
return recipients
allowed = [
r for r in recipients
if r.lower() in ALLOWED_ADDRESSES or r.lower().rsplit("@", 1)[-1] in ALLOWED_DOMAINS
]
dropped = set(recipients) - set(allowed)
if dropped:
log.warning("blocked non-production email", extra={"dropped": sorted(dropped)})
return allowedApply it to To, Cc and Bcc, and in every code path that sends mail, including background jobs and admin tools. The safest design has exactly one function that hands messages to the provider, with the guard inside it, so no code path can bypass it.
Fail closed. If the environment variable is missing or unrecognized, treat the environment as non-production.
Layer 3: recipient rewriting
Allowlists block mail, which can make testing awkward: QA wants to see what a customer would have received. Recipient rewriting solves that by redirecting instead of dropping:
- Replace every recipient with a shared test inbox, or with a plus-address that preserves the original, such as
[email protected]. - Add a header or a visible banner noting the original recipients and the environment.
- Clear Cc and Bcc, or rewrite them too.
Testers see realistic messages, and nobody outside the company receives anything.
Layer 4: scrub data when copying production
Restoring production data into staging is sometimes necessary, but email addresses should not survive the trip. Add a mandatory step to your restore process that rewrites them:
UPDATE users
SET email = 'user+' || id || '@example.com'
WHERE email NOT LIKE '%@example.com';Do the same for any table that stores addresses: contacts, invoices, notification preferences, mailing lists, webhook configurations. Treat phone numbers and other contact details the same way. Scrubbing also helps with privacy obligations, since staging environments are usually less tightly protected than production.
Make the scrub part of the restore script itself, not a separate manual step someone might forget.
Layer 5: control scheduled jobs
Cron jobs and scheduled workers are behind many incidents because they run without anyone pressing a button. In non-production environments:
- Disable bulk scheduled email jobs by default, and enable them explicitly when testing them.
- Cap the number of messages any single job can send in staging.
- Log a clear warning when a job that sends email starts outside production.
Make it visible
Even with guardrails, make non-production mail obvious:
- Prefix subjects with the environment, such as
[STAGING]. - Use a different From name, such as "Example (Staging)."
- Add a colored banner to the top of HTML templates in non-production builds.
If something does slip through, recipients and support staff can recognize it immediately.
Test the guardrails
Guardrails that are never tested quietly stop working. Add automated tests that:
- Attempt to send to an external address with the environment set to staging and assert that nothing reaches the provider.
- Verify the rewrite logic preserves the original recipient in a header.
- Confirm the restore script leaves no addresses outside your allowed domains.
When it happens anyway
Have a short plan for a staging email incident:
- Stop the job or revoke the staging key immediately.
- Determine who received what, using provider logs.
- Decide with support whether a short apology or clarification is needed, written in plain language.
- Find which guardrail was missing and add it.
Checklist
- Separate credentials and a separate sending subdomain for each non-production environment.
- One sending function with a fail-closed recipient allowlist or rewrite.
- Scrubbed email addresses in every production data copy, enforced by the restore script.
- Scheduled email jobs disabled or capped outside production.
- Visible environment markers in subject, sender name and template.
- Automated tests proving the guardrails work.
Key takeaways
- Staging email incidents come from ordinary actions meeting missing guardrails.
- Separate credentials limit damage; an allowlist in your own code prevents it.
- Recipient rewriting keeps testing realistic without reaching customers.
- Scrub addresses whenever production data is copied, as part of the restore itself.
- Test your guardrails like any other feature.
Start with Koltrix
Your domain, one inbox, and an API that sends.
A team inbox where AI sorts and drafts (nothing is sent without your click), plus the transactional API and SMTP relay your product sends with. 7 days free, no card.
