A monitor that cries wolf is worse than none. After a few false alarms people stop reading the alerts, and the real one gets the same shrug. This guide explains where false alerts come from and what you can do about them.
First, a definition. A false alert is an alert about a problem that your visitors did not have. It is not the same as a short real outage: a site that was unreachable for two minutes really was down, and the alert was right.
Where false alerts come from
A blip between the monitor and the site
The check travels over the public internet. A packet lost on the way, a congested link, or a short routing problem can make one check fail while the site is perfectly healthy for its visitors. This is the most common cause, and it passes by itself.
The monitor is blocked
Firewalls, bot protection, and rate limits are built to stop automated requests, and a monitor sends automated requests. If your protection answers a check with a challenge page or a 403 Forbidden, the monitor sees a failure while real people get through.
UptimeWire treats any final answer outside the 200 to 399 range as a failed check. That includes 401 and 403. If your site shows that answer to automated visitors, allow the monitor. At the time of writing, UptimeWire's requests identify themselves as UptimeWireBot/1.0.
A slow page
A check waits for an answer for a limited time. In UptimeWire a check gives up after 15 seconds. A page that sometimes takes longer, for example while a cache warms up, will fail those checks even though a patient visitor would get a page.
Monitor an address that answers quickly, such as a simple page or a health page, rather than a heavy search or report page.
Too many redirects
UptimeWire follows up to five redirects. A chain longer than that, or a loop between two addresses, counts as a failure, with the reason "more than 5 redirects". Browsers are also limited, so a chain that long is a real fault, but it may only show up in a monitor.
A page-text rule that is too strict
A page-text rule fails the check when the expected text is missing, or when forbidden text appears. UptimeWire reads the raw HTML of the page, up to 1 MB, and matches the text without regard to case. It does not run scripts. Text that the browser adds with JavaScript is not in that HTML, so a rule built on it will fail on a healthy page.
Planned work
A deployment, a database migration, or a server restart makes the site fail on purpose. If the monitor does not know, it reports a real outage that you caused. This is not false in the sense of untrue, but it is noise, and planned maintenance windows exist to remove it.
How repeated checks reduce noise
The simplest defence is not to believe a single failure. When a check fails, UptimeWire checks again. An incident opens only after three failed checks in a row, about 20 seconds apart. It closes after two successful checks.
Where more than one check location is available, a failure must also be seen by more than half of the locations that answered. Where only one location is available, a second server can be asked for an opinion before the incident opens, if one is configured. Without a second opinion, an incident can open on the three failures from a single place.
The trade-off is plain. Repeating a check costs a little time, roughly 40 seconds on top of the check interval. It buys a large fall in false alerts. It does not give instant detection, and no setting does.
Planned maintenance windows
For work you can predict, tell the monitor in advance. In UptimeWire you schedule a planned maintenance window for a website. Checks keep running and the history shows the failures, but:
- Incident alerts that begin inside the window are held. If the outage ends inside the window, the alerts are cancelled. If it runs past the end, they are sent when the window ends.
- The time inside the window is left out of the uptime calculation.
- Warnings about the certificate, the domain, and DNS are not held by the window.
A window must be set up ahead of time, and limits apply to its length and number. It is a tool for work you announce, not a way to hide an outage.
What to tune, in order
- Read the failure reasons. The incident history shows why each check failed: timeout, refused connection, unresolved name, certificate error, or a status code. A pattern in the reasons usually names the cause.
- Allow the monitor through your protection if the reason is
403or a challenge. - Choose a lighter address if the reason is a timeout.
- Review page-text rules. Make sure the text is in the raw HTML.
- Use maintenance windows for planned work.
- Choose where alerts go. Send the first alert to a place where a person will see it, and keep the noisy channels for what matters.
- Send a test notification to confirm the route works. It also shows what the alert looks like at 3 a.m.
What not to do
- Do not lengthen the check interval to reduce alerts. That only delays detection of real outages.
- Do not ignore a repeated pattern. If a monitor fails at the same time every night, something happens at that time. It might be a backup that slows the server.
- Do not turn off a monitor to stop the noise. Fix the cause.
When the alert is real but short
A brief outage can be real and still feel like noise, for example a restart that takes a minute. If it recurs, it is a real problem with a small footprint. Count it. The incident history records start and end times, so you can see how often it happens and for how long.
An HTTP check also cannot see every kind of failure. A 200 answer does not prove that checkout works, and silence from a monitor is not proof that nothing is wrong. Website Uptime Monitoring explains what a check measures and what it misses.
Checklist
- Do the alerts carry a reason? Read it.
- Is the monitor allowed through bot protection and rate limits?
- Does the monitored address answer well within 15 seconds?
- Are page-text rules based on the raw HTML?
- Are planned changes covered by a maintenance window?
- Does the first alert go to someone who will act on it?