16 / INFRASTRUCTURE / MONITORING
A monitor that cries wolf gets muted, and one sharing a server with the sites dies with them.
The challenge
Hosted uptime services charge per monitor and per check interval. The hard part of running your own is not the HTTP request, it is trust. A monitor that fires on a single failed probe gets muted within a week, and one that emails on every failed check through a two-hour outage is worse than nothing. And because mine runs on the same server as the sites it watches, a dead box takes the monitor down with them and nobody gets told.
What I built
I built a Next.js 15 and TypeScript app over SQLite through Prisma, with three tables: Monitor, Check, Incident. Each pass loads the active monitors, probes them with bounded concurrency, and hands every verdict to a state machine that opens an incident only after N consecutive failures and closes it on the first success. That gives exactly one DOWN alert and one RECOVERED alert per outage, with the downtime duration. Alerts fan out to email, Telegram and Slack through a dispatcher where each channel has its own timeout and its own try/catch, so a dead SMTP server cannot stall or silence the others, and around it I put a dashboard, an incident history, CSV import and export, Basic auth, and cron, systemd and pm2 configs.
How it works
The detail
Booting it locally exposed what 279 green tests had hidden. Node's global fetch wraps every network error in a generic TypeError("fetch failed") with the real libuv code nested under .cause, while my mock HTTP layer threw top-level codes, so the suite passed cleanly and every real outage would have read "fetch failed" instead of "connection refused" or "DNS lookup failed". The classifier now walks .cause, and I added five regression tests in the undici shape with it.
I started the Incident table's downNotifiedAt and upNotifiedAt as audit stamps, which quietly made them a failure mode: if SMTP hiccuped at the moment a site went down, the stamp was written anyway and the outage produced no alert, ever. The dispatcher now reports whether at least one channel actually succeeded, and the state machine stamps only on confirmed delivery. A DOWN send that fails on every channel is retried on the next pass rather than lost.
The cron pass takes a flock, but the dashboard's Check now button does not go through it, so a manual re-check can race a scheduled pass holding a stale copy of the monitor row and open a second incident or double-send. I gate both transitions on the persisted open incident instead: down is find-or-create, recovery closes an incident only if one is open. A down monitor with no incident recovers silently rather than emailing a spurious zero-second recovery.
Because the monitor runs on the same box as the sites, it cannot report its own death, so I end every pass with a GET to an external heartbeat service. I put the ping deliberately after the work rather than before, so it means the checker ran and did not hang, not that the box booted. It never throws, and I wrapped the prune step for the same reason: a database hiccup must not skip the heartbeat and page the owner over nothing.
I ran the build from a written specification: interfaces and file ownership frozen in CONTRACTS.md first, then nine parallel lane agents, then ten independent adversarial verifiers pointed at the finished code. I fixed their findings forward and logged them append-only in DECISIONS.md, including the write-only notification stamps and a Telegram Markdown 400 that would have silently dropped any alert for a URL containing an underscore. Two of the corrections above came out of that pass, not out of the tests.
The outcome
A conventional per-module unit suite sits underneath a second layer I organised around named failure gates, each with its own adversarial file that tries to break it: 300 fake sites to check peak concurrency, a scripted 200 to 500 to 200 outage asserting exactly one DOWN and one RECOVERED with the right duration, a hung channel that must not stall the pass. The whole suite runs against an injected HTTP function, fake channels and a temporary database, and today it passes at 284 tests across 18 files with a clean typecheck, while the app boots and serves a dashboard of five seeded monitors, one of them deliberately unreachable and showing an open incident with "connection refused". I have never deployed it. No alert has ever arrived in a real inbox, and the go-live steps in the RUNBOOK, wiring a real heartbeat URL, confirming a real email lands, forcing one real down-and-up, are still outstanding.