Four programs, one word: “monitoring”
“Set up monitoring” is the request. Behind it sit at least three distinct programs that people routinely conflate. node_exporterexposes a machine's metrics on an HTTP port — it measures, it does not store. Prometheus scrapes that port on a schedule and stores the time series — it collects, it does not draw. Grafana queries Prometheus and renders the graphs — it draws, it stores nothing of its own worth losing. Miss which one does what and you end up debugging Grafana for a gap that Prometheus caused, or expecting node_exporter to keep history it never had.
The thesis is therefore boring and load-bearing: the stack is easy to install and easy to leave wide open, and the security story is almost entirely about which ports face the world and which default password you forgot to change. The graphs are the fun part. The bind addresses are the part that matters.
Before you start: what is already listening?
Every piece of this stack wants a well-known port, and a busy server may already have taken one. Collisions here are not subtle — a service that cannot bind its port simply does not start — but they are easy to avoid by looking first.
- node_exporter's 9100 —
ss -tlnH 'sport = :9100'. If something already answers there, an exporter is likely already running, and a second one is noise. - Prometheus' 9090 and Grafana's 3000— the two you will actually reach. Grafana's 3000 is a popular port; a dev server or another app may hold it.
- Is any of this already installed? A half-configured Prometheus from a previous attempt is worth finding before you write a second config file that fights it.
There are two honest ways to run this stack, and the right one depends on the box. Native packages — Grafana from its repository, Prometheus and node_exporter as systemd services — are the most transparent and survive reboots cleanly. A Docker Compose stack puts all three plus wiring in one file, which is faster to stand up and tear down. Neither is wrong; what matters is that you know which one you chose, because the day you debug it you need to know whether to read a unit file or a compose file.
1. node_exporter: the thing being measured
Start at the bottom. node_exporter is a small binary that reads the kernel's view of the machine — CPU, memory, disk, network, load — and exposes it as text on port 9100. It keeps no history; it answers “what is true right now” every time it is scraped. Install the binary, run it as a dedicated unprivileged user under a systemd service, and prove it with the only verification that counts: something is listening on 9100. A version string is not enough; a listening socket is.
And here is the first exposure decision. node_exporter's endpoint reveals a great deal about a machine. It has no authentication of its own, so it should be reachable by Prometheus and nothing else — bound to a private interface, or firewalled to the collector's address. An exporter open to the internet is a free reconnaissance feed for anyone who finds it.
2. Prometheus: the thing that collects
Prometheus is the memory of the stack. On a schedule it scrapes every target you list — node_exporter here, but also application metrics, cAdvisor for containers, anything that speaks the format — and stores the series on disk. Its configuration is a list of scrape targets and intervals, and its verification is not “the service is active” but “the service is ready”: curl -sf http://localhost:9090/-/ready returning a 200 means Prometheus has loaded its config and is actually scraping, which is a stronger claim than a green unit.
Prometheus' own web UI on 9090 is useful and completely unauthenticated. Treat 9090 the way you treat 9100: bind it to loopback, reach it through a proxy that checks who you are, and never publish it raw. The metrics it holds are a map of your infrastructure, and the query interface will happily draw that map for anyone who can load the page.
3. Grafana: the thing that draws — and its one dangerous default
Grafana is the part everyone actually wanted: dashboards, on port 3000, querying Prometheus as a data source. Installing it and confirming it answers — curl against /api/health returning 200 — is the easy part. The part that matters fits in one sentence:
Grafana ships with the login admin / admin, and until you change it, your monitoring stack is world-writable to anyone who reaches the port. This is not a theoretical risk. Default-credential Grafana instances are found and taken over routinely, and a dashboard tool with a data source configured is a query console into your Prometheus. Change the admin password on first login, before you do anything else, and verify you did — a monitoring system you do not control is worse than none, because it lies to you with authority.
4. Wiring it together, and where the wires must not reach
A Compose stack makes the assembly one file: Prometheus scraping node_exporter, Grafana pointed at Prometheus, all three on a private network with only Grafana's port surfaced — and even that behind a reverse proxy with TLS if it faces anything but localhost. The verification that the stack is real is not three separate “active” checks; it is Grafana answering /api/health and a dashboard actually rendering a metric that originated in node_exporter, which proves the whole chain end to end.
For logs rather than metrics, the same shape repeats one layer over: Loki stores, an agent ships (Grafana Alloy now that Promtail is end-of-life), and Grafana draws. The exposure rules are identical — bind private, proxy with auth, change every default — because a log aggregator open to the world leaks more than a metrics one does.
What this stack does not do
Standing up Prometheus and Grafana gives you visibility, not reliability. It does not page anyone — that is Alertmanager's job, and an alert with no routing is a graph nobody is watching at three in the morning. It does not decide what is normal; thresholds and the hysteresis that stops a single blip from crying wolf are yours to tune. It does not watch itself — a Prometheus that dies stops recording the outage it should have caught. And it is not free: retention is disk, scraping is load, and a dashboard for every metric is a way to see nothing by drowning in everything. Observability is a practice, not an install.
The check that proves it
In order: a listening socket on 9100 for the exporter; /-/ready on 9090 for a Prometheus that is actually scraping; /api/health on 3000 for Grafana; a confirmed non-default admin password; ss -tlpnto prove that 9100 and 9090 are not facing the internet and only Grafana's door — proxied and authenticated — is; and, the real proof, one dashboard panel drawing a live value that travelled exporter → Prometheus → Grafana.
Where Servor fits
Servor ships each piece and the whole Compose stack as recipes — among 150+ vetted playbooks — and the recipe shape is what keeps “set up monitoring” from quietly leaving three unauthenticated ports open. A recipe is not a script fired blind: it starts with reconnaissance, checking whether 9100, 9090 and 3000 are free and whether any of it is already installed, so it adapts to the real machine. Each step carries its own verify — the listening socket, the /-/ready 200, the /api/health 200 — so the run proves the chain, not just the packages. The playbook is idempotent: an existing exporter or Prometheus is checked and reported, not stacked over. And the guardrails are explicit about the defaults that bite — change Grafana'sadmin/admin, keep the internal ports off 0.0.0.0, put a TLS proxy in front of anything public.
A self-hosted stack watches the inside of a machine in fine detail. It does not, by itself, tell you the machine is reachable from the outside, or that its certificate is about to expire, or that SSH stopped answering — that is what Servor's own monitors are for, from a different vantage point, with the configurable hysteresis that keeps one dropped packet from waking anyone. Running both is not redundant; they see different failures. Keeping that second layer quiet enough to be worth listening to is the subject of the piece on alert noise, and the wider operational picture lives in our monitoring overview.
Whichever recipe installs it, execution stays on the zero-knowledge path: every command is signed in your browser with a key derived from your vault key, relayed verbatim, and verified by the agent on the machine before it runs — the same discipline described in the Plan-Execute-Verify piece.