Server Monitoring from Scratch: Metrics, Logs, and Health Checks

If you have ever started a project like Server Monitoring from Scratch: Metrics, Logs, and Health Checks and watched it grow from a clean folder structure into an unruly pile of exceptions, this guide is for you. It is the distilled version of the lessons that took years of production work to learn.

Introduction

We start with the principle: monitoring is not dashboards, it is alerts. A dashboard nobody watches is decoration; an alert that pages someone at 3am is a control. The stack is built around the question 'what wakes me up?' and everything else is bonus.

Then the layers, bottom up: health checks (is the service answering?), metrics (is it degrading?), logs (why did it fail?), and alerting (who gets told?). Each layer has a free, boring implementation: a cron + curl, a metrics scrape, journalctl, and a notification channel.

The middle section is the alerting design: thresholds that are not too loud (alert fatigue kills alerts) and not too quiet (a server that 'just felt slow' is a server nobody measured). The design is the hard part; the tools are the easy part.

We finish with the operational habit: the weekly review, the runbook that comes with every alert, and the self-healing seeds — because the best alert is the one that fires, a runbook answers, and the incident becomes a five-minute footnote.

The difference between a server that runs and a server that is run is monitoring: knowing when it is healthy, when it is degrading, and when it has already failed. This article builds a complete monitoring stack for a small server — without a single enterprise tool, and with a budget of zero dollars.

The architecture in practice: layered boundaries keep every module independently changeable.

Why It Matters

A server without monitoring fails twice: the service stops, and nobody knows for hours. The cost of the second failure is usually the greater — the first is an incident, the second is a credibility problem. The monitoring in this article turns 'the site was down for four hours' into 'the site was down for four minutes, and here is the log line'.

Small servers do not need enterprise monitoring — they need the three questions answered: is it up, is it degrading, and why did it fail. Each question has a free, boring answer (curl, metrics, logs), and the boring answer is the one that gets maintained for years. Tooling that requires a budget gets abandoned; tooling that is a cron job survives.

Alert design is the difference between monitoring that works and monitoring that trains people to ignore it. An alert for every blip is noise; an alert for every actual problem is a superpower. The thresholds and the escalation in this article are the design that keeps alerts honest.

• Monitoring is alerts, not dashboards
• The stack: health checks, metrics, logs, alerting — in that order
• Cron + curl is a complete health-check system
• Thresholds need tuning: loud enough to matter, quiet enough to trust
• Every alert ships with its runbook attached
• Free tools, boring tools, maintained for years

The Problem

The beginner failure is monitoring theater: installing Grafana, wiring twenty dashboards, and never once being paged — because the dashboards are beautiful and nobody looks at them. The server fails at 2am, and the monitoring was a mural, not a control.

The second failure is the alert wall: fifteen channels, forty rules, and a notification for every log line — which trains the operator to ignore notifications, which defeats the entire purpose. The fix is the design discipline: every alert must have a threshold, a human, and a runbook.

The Approach

The health-check layer: a cron job that curls the health endpoint every minute and alerts if it is not 200 — the classic uptime check, implemented in ten lines. The app's /health endpoint (from the deployment article) is the contract; the cron is the watchdog; the alert is the bark.

The metrics layer: the kernel's own numbers — load, disk, memory, connections — collected by a cron, kept in a small file or a lightweight tool (telegraf, or simply a script that logs values), and reviewed weekly against the baseline. Metrics do not need to be real-time; they need to be trending.

The logs layer: journald's own journal (or PM2 logs) with the review habit — and the search habit: every alert is followed by a log query that names the cause. The three layers compose: health catches the outage, metrics show the degradation, logs explain it, and the runbook turns the explanation into the fix.

The whole stack in two scripts: health every minute, disk every hour, both alerting to a channel you actually read (Telegram here; email or ntfy work identically). The thresholds are the design: the health check is strict (any failure alerts), the disk check is tuned (90% before it matters).

# the health check (cron: every minute)

# */1 * * * * ~/monitor/health.sh

#!/bin/bash

URL="http://127.0.0.1/health"

if ! curl -fsS --max-time 10 "$URL" > /dev/null; then

  # service down -> notify (example: Telegram bot)

  curl -s "https://api.telegram.org/bot$TG_TOKEN/sendMessage" \

    -d chat_id="$TG_CHAT" -d text="DOWN: $URL at $(date)"

fi



# the disk check (cron: hourly)

# 0 * * * * ~/monitor/disk.sh

#!/bin/bash

THRESHOLD=90

USE=$(df -h / | awk 'NR==2 {print $5}' | tr -d '%')

[ "$USE" -ge "$THRESHOLD" ] && echo "DISK $USE%" | \

  curl -s -d chat_id="$TG_CHAT" -d text="$(cat -)" \

  https://api.telegram.org/bot$TG_TOKEN/sendMessage

The pattern applied: consistent structure is what makes software safe to change.

Alert Channels for Small Servers

The channel matters less than the discipline: one channel, used consistently, checked by the person on call. Telegram (or ntfy) with a bot token is the sweet spot — instant, push-capable, and free. The runbook attached to the alert matters more than the medium that delivers it.

Implementation

Build the stack in the order the failures occur: health checks first (cron + curl, one day), disk and load thresholds second (same pattern, one day), the review habit third (weekly, ten minutes), and the log query habit fourth (every alert ends in a journalctl or log search that names the cause). The stack is four small things, each boring, each maintained.

Tune the thresholds by experiment: start with generous limits (disk 90%, load above core count for ten minutes), then tighten weekly until alerts stop firing on noise and start firing on problems. Alert fatigue is a tuning problem, not a personality problem — the thresholds are the fix.

Write the runbooks as alerts appear: each alert gets a paragraph — 'what this alert means, the three usual causes, the first fix to try'. The runbook file lives in the repo with the monitoring scripts, so the monitoring stack is versioned like the code it watches. A year later, a 3am alert is a script, not a mystery.

• Health check every minute — curl + cron + a notification channel
• Disk and load thresholds checked hourly
• Thresholds tuned weekly until the noise is gone
• Every alert ends with a log query that names the cause
• Runbooks live in the repo next to the scripts
• The weekly review reads metrics trends, not just alerts
• One channel, consistently, for all alerts
• The stack survives being ignored for a month — and works when it matters

Key Decisions

Real-time metrics or periodic checks?

Periodic checks for a small server: a cron every minute is more than enough to catch the failures that matter, and it costs nothing to run. Real-time streaming (Prometheus + Grafana) earns its keep when there are many servers or when performance anomalies need second-level resolution — which is not this article's server.

Who is on call for a hobby server?

You — and the honest design reflects that: alerts go to one channel, at thresholds that matter, with runbooks that assume a tired operator at 3am. The design that assumes a 24/7 NOC is a design that fails for a solo operator. Simplicity is the feature.

Should the health check live on the same server?

The cron-based check can live locally, but a second external uptime monitor (a free service or a second server) is the check that catches the whole box being down — because a dead box cannot run its own health check. The external monitor is the only honest way to know the server is reachable from the internet.

Common Mistakes to Avoid

The most common server mistake is default exposure: a box installed with the distribution defaults — password SSH, all interfaces, no firewall — then left to the internet. The scanners find it within days, and the 'default config' becomes the compromise vector. The hardening article exists because the default is a liability, not a convenience.

The second mistake is the unverified backup: a schedule that has never been restored, discovered at the worst possible moment. The database article's restore test is not ceremony — it is the only way a backup stops being a hope and becomes a capability.

• Default installs with password SSH and open ports left public
• Backups scheduled but never restore-tested
• The database bound to every interface with auth off
• Deploys by hand, undocumented, unreproducible
• Monitoring that is a dashboard rather than an alert

Patterns That Scale

The pattern that carries every server article is the checklist-as-code: hardening steps, deploy steps, and monitoring steps all written down as scripts and documents in the repository. The server becomes a build artifact — provision, document, reproduce — instead of a snowflake maintained by memory.

The second pattern is the layered defense: keys, firewall, least privilege, and patching each protecting the others, so a failure at one layer is contained by the next. The security article is the map of those layers, and every article in this category assumes them.

• The server is documented in the repo and rebuildable from it
• Keys, firewall, patching, and least privilege layer together
• Backups are restore-tested on a schedule
• Monitoring alerts on real thresholds with runbooks attached

Real-World Example

The monitoring on the boxes behind this platform is exactly the stack in this article: a cron health check with a Telegram alert, a disk threshold, and journald as the log layer — plus a free external uptime monitor that catches the whole-box failures. The cost is zero dollars, and the track record includes catching a disk fill at 89% before the nightly backup — the alert's runbook naming the exact cleanup, done in minutes.

The runbook habit earned its keep on the incident that inspired it: a deployment that broke the database connection at 2am, alerted instantly, and resolved in under ten minutes because the runbook said 'check the connection string in .env, restart the app' — and the log line that followed confirmed it. The alert did not save the night alone; the runbook did. That is the whole design in action.

Case Study: Server Monitoring from Scratch: Metrics, Logs, and Health Checks

When TaskFlow Pro hit its first real traffic spike, the architecture described in this article was the difference between an incident and a non-event. The queries were indexed, the reads were cached, and the pages were server-rendered — so the spike showed up as a flat line on the database charts and nothing more.

What made it possible was not a clever library. It was the discipline of applying these patterns consistently from day one: every module shaped the same way, every decision written down, every claim verified with a measurement.

• The lesson that cost the most in servers: measure before changing anything, and let the data pick the fix.
• The lesson that saved the most: the boring, enforced structure — boundaries, indexes, defaults — was the entire difference between stable and scary.
• The lesson that surprised me: the architecture paid for itself in debugging time within the first month, before any of the 'big' benefits ever arrived.

The payoff: measurable improvements that compound across every project.

Putting It Into Practice

Start with the hardening checklist on your next (or existing) box: keys-only SSH, default-deny firewall, automatic updates, and a documented SERVER.md. The pass is an afternoon and converts the box from a liability into a reproducible asset.

Then build the serving stack deliberately: Nginx in front, the app under a process manager, the database with its backup and restore test — each from its article, each documented. The stack is the operating manual of everything this portfolio serves.

How This Applies to Your Stack

The server layer is the run-time of this entire platform: a hardened VPS running the application under systemd, Nginx in front, MongoDB behind, with the monitoring and backup routines from these articles documented in the repository. The articles in this category are not theoretical — they are the actual operating manual of the boxes that serve this site.

Your server stack will differ in tools, not in shape: an LTS distribution, keys and firewalls, a reverse proxy, a process manager, and a database — with monitoring and backups as the constant. The discipline transfers wholesale; only the commands change.

Key Takeaways

• A health check runs every minute and alerts on failure
• An external uptime monitor watches the box from outside
• Disk and load thresholds alert before they become incidents
• The alert channel is one, consistent, and push-delivered
• Thresholds were tuned until the noise disappeared
• Every alert has a runbook paragraph attached
• The weekly review reads metrics trends
• The monitoring stack lives in the repo, versioned with the code

Frequently Asked Questions

Do I need Grafana to have monitoring?

No — Grafana is a dashboard, and dashboards are optional. The controls are the alerts and the logs. For a small server, cron scripts plus a notification channel give you every control this article describes, at zero cost and zero dashboard debt. Add dashboards when the metrics story needs a visual — not before.

What is the cheapest external uptime monitor?

A free-tier uptime service (UptimeRobot, Hetrixtools) or a second server running a cron curl. Either catches the failure that local checks cannot: the box itself being unreachable. The key requirement is only that it is outside the box — that is the entire job.

How do I avoid alert fatigue?

By design: thresholds tuned to actual problems, one channel, and alerts that require action. Every alert that fires without needing action is a vote for ignoring the next one. The weekly tuning pass is the antidote — alerts that have not earned their keep get quieted.

What metrics actually matter for a small server?

Disk usage, load average, memory pressure, and the health of the services themselves. Those four catch the overwhelming majority of small-server failures. Everything else — per-request latencies, database query plans — matters later, when the basics are already covered.

Should monitoring be part of the deploy, or separate?

Both: the deploy script ends with a health check (does the new version answer?), and the standing monitor runs continuously (does the service stay up?). The deploy check catches bad releases immediately; the standing monitor catches the drift that follows. The pair is what 'monitoring integrated' means.

What do I do on the first night I get paged?

Follow the runbook, honestly: read the alert, run its log query, apply its first fix, verify with the health check, and then — the step everyone skips — update the runbook with what actually happened. The first night is the calibration that makes every later night shorter.

What is the single highest-value server task?

The restore test: actually restoring a backup into a scratch environment. It validates the entire backup chain — schedule, encryption, storage, tooling — in one afternoon, and it is the task nobody does until the day it is the only thing that can save them.

How much server security is 'enough'?

Enough is the checklist in the hardening article, maintained: keys-only SSH, a default-deny firewall, automatic patching, least-privilege users, and monitored logs. Everything beyond that (IDS, compliance frameworks) is insurance for specific threats — add it when the threat model justifies it, not before.

Conclusion

Build the health check today, tune the thresholds this week, and write the first runbook when the first alert fires. The quiet confidence of a monitored server is the best thing this article sells, and it is on sale at the price of an afternoon.

Monitoring is the difference between owning a server and being owned by it: alerts instead of dashboards, thresholds instead of noise, and runbooks instead of panic. The stack is four boring pieces — health, metrics, logs, alerts — and the whole thing costs nothing but an afternoon.