KoltrixLifecycle Email Metrics That Matter (and Vanity Ones to Drop)
Open rates stopped meaning much once mail clients began prefetching images. The lifecycle email metrics worth tracking instead, email type by email type.
On this page(8 sections)
If your onboarding emails report a healthy open rate and your activation numbers haven't moved in six months, the open rate is not telling you what you think. Lifecycle email exists to change what people do in your product, so that's where you should measure it.
This post covers why the classic email metrics have weakened, what to track instead for each type of lifecycle email, and how to tell whether an email is doing anything at all.
Why open rates became unreliable
An "open" has always been an indirect measurement. The email contains a tiny tracking image, and when the image loads, the sender records an open. That was never perfect, since people with images turned off never counted, but it was a rough signal.
Then mail clients started loading images on the user's behalf. Some privacy features fetch remote content through a proxy shortly after delivery, whether or not the person ever looks at the message. Other clients cache images. The result is that a large and unpredictable share of "opens" are machines, not people.
That doesn't make opens useless for everything. A sudden collapse can still hint at a delivery problem. But as a measure of whether people read your onboarding email, it's noisy enough that you shouldn't make decisions on it.
Clicks are better, but they have their own noise: some security scanners follow links in incoming mail to check them. Treat click counts as directional and look at what happens after the click.
Measure the product event, not the email event
Every lifecycle email is sent because you want someone to do something. Write that something down, and measure it.
| What it's for | Metric that matters | |
|---|---|---|
| Welcome | Bring them back for a second session | Share of new signups with a second session within 3 days |
| Onboarding step nudges | Complete a specific setup step | Completion rate of that step among recipients |
| Activation emails | Reach the activation event | Activation rate within the trial window |
| Trial ending | Convert to paid | Trial-to-paid conversion rate |
| Dunning | Recover failed payments | Share of failed payments recovered, and revenue recovered |
| Team invite reminders | Get a second user active | Invites accepted, and invitees active within 7 days |
| Win-back | Bring churned customers back | Reactivations among recipients over 90 days |
| Product updates | Drive adoption of new features | Feature adoption among recipients |
None of those require trusting an open pixel. They come from your own product data, which you control and can trust.
Add a reply rate where it applies
Some lifecycle emails are meant to start a conversation: founder welcome notes, onboarding check-ins, exit surveys. For those, the reply rate is a strong signal. A real human typing a response is about as clear an indication of engagement as email offers.
Track replies by email type, and read them. The count tells you whether the email works; the content tells you why.
Use holdout groups to prove an email does anything
Here's the uncomfortable question every lifecycle program should face: would these people have done the thing anyway?
If 40 percent of users who receive your "connect your data" email go on to connect their data, that sounds good. But if 38 percent of users who never got the email also connect their data, the email is contributing very little.
The fix is a holdout group:
- Randomly assign a small share of eligible users (say 10 percent) to not receive the email.
- Send to everyone else as normal.
- After enough time, compare the target metric between the two groups.
The difference is the email's real effect. Sometimes it's large. Sometimes it's close to zero, and you've learned that the email can be rewritten or removed.
A few practical notes:
- Randomize properly. Assign by a hash of the user ID, not by signup date or plan, so the groups are comparable.
- Wait long enough. Small products need weeks to collect enough users for a clear answer. Don't call it after three days.
- Don't hold out critical mail. Password resets, receipts and security notices are not experiments.
- Rotate holdouts. Don't keep the same users in every holdout forever.
Metrics to drop or demote
Open rate as a success metric. Keep it, if you like, as a rough deliverability smoke alarm. Stop putting it in reports as evidence that an email works.
Total emails sent. Sending more is not an achievement. If anything, a falling number of emails per activated user can mean your onboarding is getting more efficient.
Click-through rate in isolation. A click that doesn't lead to the product action is a curiosity, not a result.
List growth for lifecycle mail. Lifecycle audiences grow with signups. The number reflects your acquisition, not your email.
Metrics to keep an eye on for health
Some email metrics still matter, just not as measures of persuasion:
- Bounce rate, especially hard bounces. A rising rate points to bad addresses entering your signup flow.
- Spam complaint rate. If people mark your lifecycle email as spam, mailbox providers will notice, and so should you.
- Unsubscribe rate by email. A single email with a much higher unsubscribe rate than the others is a sign it's unwelcome.
- Delivery failures from your provider's webhooks. If your app sends through an API, delivery and bounce events tell you whether mail is arriving at all.
Email APIs report some of these per message through webhooks, and which ones varies by provider, so check before you design a dashboard around them. Koltrix, for example, sends sent, bounced, opened and clicked events (see the webhooks documentation); complaint and unsubscribe rates then come from your own data, such as your preference page and the replies asking you to stop. Either way, tie the events to the same user records as your product metrics.
Putting it together: a monthly review
Once a month, for each lifecycle email:
- What is its target product metric, and did it move?
- What does the holdout comparison say, if one is running?
- Reply rate and a skim of replies, for conversational emails
- Unsubscribe and complaint rates compared with other emails
- Any bounce or delivery anomalies
- One decision: keep, rewrite, retime, or retire
Key takeaways
- Machine opens from privacy features and caching make open rates a poor success measure.
- Tie each lifecycle email to the product event it exists to drive, and measure that.
- Use holdout groups to find out whether an email changes behavior at all.
- Keep bounces, complaints and unsubscribes as health metrics, not as proof of persuasion.
Start with Koltrix
Your domain, one inbox, and an API that sends.
A team inbox where AI sorts and drafts (nothing is sent without your click), plus the transactional API and SMTP relay your product sends with. 7 days free, no card.
