An alert channel that posts every few minutes soon becomes background noise. If it reports routine CPU and queue changes alongside a broken checkout, the important message can go unread.
Alerting fails when it measures machines instead of users.
Start from what users feel
We write alerts around symptoms a customer would notice:
- The site is down or erroring. Error rate above a threshold for five minutes, or an external uptime check failing from two locations.
- The site is slow. 95th percentile response time on key pages above a set limit.
- A core flow is broken. Zero successful checkouts, sign-ups or payments in a window where there should be some.
- Background work is stuck. The oldest job in the queue is older than it should ever be.
These page someone. Everything else is a dashboard.
Causes go on dashboards
High CPU, memory, disk and connection counts are useful when you investigate a problem. They make poor alerts because they fire when nothing is wrong and stay quiet when something is. A server at 90% CPU serving every request on time is fine. A server at 20% CPU returning errors isn't.
The exception is anything that will cause an outage if nobody acts: a disk that will fill within hours, a certificate expiring in seven days, a database nearing its storage limit. Those alert, with enough lead time to fix them during working hours.
Every alert needs an owner and an action
Before an alert goes live we answer three questions. Who gets it? What should they do first? Can it wait until morning? If nobody can say what action to take, it's not an alert yet.
We link each alert to a short runbook entry: what the alert means, where to look, and how to mitigate. The person paged at 2am shouldn't have to guess.
Two levels, not ten
We use two severities. Page means a human responds now, day or night. Ticket means someone looks at it next working day. Warning, notice and info levels in between train people to ignore the channel.
Review the noise monthly
Once a month we look at every alert that fired. If it fired and nobody needed to act, we raise the threshold or delete it. If an incident happened without an alert, we add one. The list stays short because we prune it.
A small starting set
For a typical web product we launch with:
- Uptime check on the home page and one logged-in page
- Error rate on the API
- Response time on checkout or the main workflow
- "No successful orders in 30 minutes" during business hours
- Queue age for background jobs
- Certificate and domain expiry
Six alerts that people trust beat sixty they've muted.
