Installing and Configuring standard Prometheus Node Exporter metrics endpoints on Rocky Linux 9: Infrastructure audits

Prerequisites and System Preparation

Before proceeding with the installation of Prometheus Node Exporter on Rocky Linux 9, ensure your system is up-to-date and has the necessary tools installed. This section outlines the essential preparation steps for a robust infrastructure audit setup.

Verify that your Rocky Linux 9 system has internet connectivity and that the `dnf` package manager is functioning correctly. It’s also advisable to have `sudo` privileges for executing administrative commands.

Downloading and Installing Prometheus Node Exporter

Prometheus Node Exporter is typically distributed as a pre-compiled binary. We will download the latest stable release from the official Prometheus GitHub repository. This ensures you are using a version tested and supported by the Prometheus community.

First, identify the latest stable release version. You can usually find this on the Node Exporter releases page. For this guide, let’s assume version 1.7.0 is the latest. Adjust the URL if a newer version is available.

Create a dedicated directory for Prometheus binaries and navigate into it:

sudo mkdir -p /opt/prometheus
cd /opt/prometheus

Download the appropriate binary for your system architecture (amd64 in most server environments):

sudo wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz

Extract the downloaded archive:

sudo tar xvfz node_exporter-1.7.0.linux-amd64.tar.gz

The archive contains the `node_exporter` binary and a `textfile_collector` directory. We only need the binary for now. Move the binary to a directory in your system’s PATH, such as `/usr/local/bin`:

sudo mv node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/

Clean up the downloaded archive and extracted directory:

sudo rm -rf node_exporter-1.7.0.linux-amd64*

Configuring Node Exporter as a Systemd Service

To ensure Node Exporter runs reliably and starts automatically on boot, we will configure it as a systemd service. This is the standard and recommended approach on modern Linux distributions like Rocky Linux 9.

Create a dedicated user and group for Node Exporter to run under. This follows the principle of least privilege:

sudo groupadd --system node_exporter
sudo useradd --system --no-create-home --gid node_exporter node_exporter

Create the systemd service unit file:

[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
    --collector.textfile.directory=/var/lib/node_exporter/textfile_collector

[Install]
WantedBy=multi-user.target

Save this content to `/etc/systemd/system/node_exporter.service`. You can use `sudo nano /etc/systemd/system/node_exporter.service` or your preferred editor.

Create the directory for the textfile collector, which will be used for custom metrics:

sudo mkdir -p /var/lib/node_exporter/textfile_collector
sudo chown node_exporter:node_exporter /var/lib/node_exporter/textfile_collector

Reload the systemd daemon to recognize the new service file:

sudo systemctl daemon-reload

Start the Node Exporter service and enable it to start on boot:

sudo systemctl start node_exporter
sudo systemctl enable node_exporter

Verify the status of the Node Exporter service:

sudo systemctl status node_exporter

You should see output indicating that the service is active and running.

Configuring Firewall Rules

By default, Node Exporter listens on port 9100. You need to open this port in the system’s firewall to allow Prometheus to scrape metrics from this instance. Rocky Linux 9 typically uses `firewalld`.

Add a permanent rule to allow traffic on port 9100:

sudo firewall-cmd --permanent --add-port=9100/tcp

Reload the firewall to apply the changes:

sudo firewall-cmd --reload

Verify that the port is open:

sudo firewall-cmd --list-all

You should see `9100/tcp` listed under the `ports` section.

Verifying Node Exporter Metrics Endpoint

With Node Exporter running and the firewall configured, you can now verify that the metrics endpoint is accessible. The default endpoint is `http://:9100/metrics`.

From a machine that can reach your Rocky Linux 9 server (e.g., your local machine or another server in the same network), use `curl` to fetch the metrics:

curl http://localhost:9100/metrics

If you are running this command from a different machine, replace `localhost` with the IP address or hostname of your Rocky Linux 9 server. You should receive a large output of text-based metrics, similar to this (truncated for brevity):

# HELP go_goroutines Go current number of goroutines
# TYPE go_goroutines gauge
go_goroutines 15
# HELP go_memstats_alloc_bytes_total Go allocate stats
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 123456789
# HELP node_cpu_seconds_total Seconds the CPU spent in various modes.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 12345.67
node_cpu_seconds_total{cpu="0",mode="user"} 9876.54
# HELP node_disk_io_time_seconds_total Seconds spent doing I/O to disks.
# TYPE node_disk_io_time_seconds_total counter
node_disk_io_time_seconds_total{device="sda",read="true"} 123.45
# HELP node_load1 The average system load over the last minute.
# TYPE node_load1 gauge
node_load1 0.5
...

This output confirms that Node Exporter is running correctly and exposing system metrics.

Enabling and Configuring Specific Collectors

Node Exporter comes with a wide array of collectors that expose various system metrics. By default, many are enabled. For infrastructure audits, you might want to ensure specific collectors are active and potentially disable others to reduce overhead or focus on relevant data.

You can list all available collectors and their default status using the `–help` flag:

/usr/local/bin/node_exporter --help

This will output a long list of collectors. To enable or disable specific collectors, you modify the `ExecStart` line in your systemd service file (`/etc/systemd/system/node_exporter.service`).

For example, to explicitly enable only CPU, memory, disk I/O, network, and the textfile collector, you would modify the `ExecStart` line to:

ExecStart=/usr/local/bin/node_exporter \
    --collector.cpu \
    --collector.meminfo \
    --collector.diskstats \
    --collector.netdev \
    --collector.textfile.directory=/var/lib/node_exporter/textfile_collector \
    --no-collector.filesystem \
    --no-collector.hwmon \
    --no-collector.insecure_hardware

In this example, we explicitly enable `cpu`, `meminfo`, `diskstats`, and `netdev`. We also explicitly disable `filesystem`, `hwmon`, and `insecure_hardware` using the `–no-collector.` prefix. This is crucial for fine-tuning your monitoring for audit purposes, ensuring you only collect what’s necessary and relevant.

After modifying the service file, remember to reload the systemd daemon and restart the Node Exporter service:

sudo systemctl daemon-reload
sudo systemctl restart node_exporter

Utilizing the Textfile Collector for Custom Metrics

The `textfile_collector` is invaluable for adding custom metrics that Node Exporter doesn’t natively provide. This is particularly useful for auditing specific application states or custom hardware configurations.

Metrics are added by placing files with a `.prom` extension into the directory specified by `–collector.textfile.directory` (e.g., `/var/lib/node_exporter/textfile_collector`). These files must adhere to the Prometheus textfile format.

Let’s create a custom metric to track the number of running Docker containers:

# First, ensure docker is installed and running
# sudo dnf install docker-ce docker-ce-cli containerd.io -y
# sudo systemctl start docker
# sudo systemctl enable docker

# Create a script to generate the metrics file
sudo nano /usr/local/bin/docker_metrics.sh

#!/bin/bash

# Path to the textfile collector directory
TEXTFILE_DIR="/var/lib/node_exporter/textfile_collector"
METRIC_FILE="${TEXTFILE_DIR}/docker_containers.prom"

# Count running Docker containers
RUNNING_CONTAINERS=$(docker ps --format '{{.Names}}' | wc -l)

# Create or update the metrics file
cat << EOF > "${METRIC_FILE}"
# HELP docker_running_containers Number of currently running Docker containers
# TYPE docker_running_containers gauge
docker_running_containers ${RUNNING_CONTAINERS}
EOF

# Set correct ownership for the metrics file
chown node_exporter:node_exporter "${METRIC_FILE}"

Make the script executable:

sudo chmod +x /usr/local/bin/docker_metrics.sh

Now, create a systemd timer to run this script periodically (e.g., every minute). This ensures the metrics file is updated regularly.

[Unit]
Description=Update Docker container metrics for Node Exporter

[Timer]
OnBootSec=1min
OnUnitActiveSec=1min

[Install]
WantedBy=timers.target

Save this to `/etc/systemd/system/docker_metrics.timer`. Then, create the corresponding service file:

[Unit]
Description=Run Docker metrics script

[Service]
Type=oneshot
ExecStart=/usr/local/bin/docker_metrics.sh

Save this to `/etc/systemd/system/docker_metrics.service`. Reload systemd, enable and start the timer:

sudo systemctl daemon-reload
sudo systemctl enable docker_metrics.timer
sudo systemctl start docker_metrics.timer

After a minute, you should be able to fetch the custom metric:

curl http://localhost:9100/metrics | grep docker_running_containers

This demonstrates how to extend Node Exporter’s capabilities for detailed infrastructure auditing by integrating custom metrics.

Integrating with Prometheus Server

The final step is to configure your Prometheus server to scrape metrics from this Node Exporter instance. This is done within the Prometheus configuration file, typically located at `/etc/prometheus/prometheus.yml`.

Add a new scrape job to your `prometheus.yml` file:

scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: [':9100'] # Replace with your Rocky Linux server's IP
        labels:
          instance: 'rocky-server-01' # Or any descriptive label for this instance
          env: 'production' # Example environment label

Replace `` with the actual IP address of your Rocky Linux 9 server. The `labels` section is crucial for organizing and filtering metrics in Prometheus, especially when auditing multiple servers.

After saving the `prometheus.yml` file, reload the Prometheus configuration:

sudo systemctl reload prometheus

Navigate to your Prometheus web UI (usually `http://:9090`), go to the “Targets” page, and you should see your `node_exporter` job listed with a state of “UP”. This confirms that Prometheus is successfully scraping metrics from your Rocky Linux 9 instance, making its data available for dashboards and alerts relevant to infrastructure audits.