Troubleshooting Btrfs filesystem errors, corrupt metadata, and sector recovery pipelines on openSUSE Leap 15.5

Initial Assessment: Identifying Btrfs Corruption Symptoms

When a Btrfs filesystem begins exhibiting signs of corruption, it’s rarely a sudden, catastrophic event. More often, it manifests as subtle performance degradations, unexpected application crashes, or specific error messages logged by the kernel. For enterprise infrastructure, early detection is paramount. Common indicators include:

• Kernel log entries containing “BTRFS error”, “corrupt leaf”, “bad tree root”, “checksum mismatch”, or “IO error” related to Btrfs operations.
• Applications failing to read or write files, often with “Input/output error” or “Stale file handle” messages.
• Slowdowns during file operations, especially on directories with many files or large files.
• Inability to mount the filesystem or mounting it read-only.
• Unexpected system reboots or kernel panics originating from the filesystem layer.

The first step in any troubleshooting scenario is to consult the system logs. On openSUSE Leap 15.5, this typically involves examining journalctl. We’ll filter for Btrfs-related messages and look for patterns indicating specific issues.

Diagnostic Commands: Probing Btrfs Health

openSUSE Leap leverages Btrfs extensively, often as the default root filesystem. The primary tool for interacting with and diagnosing Btrfs is btrfs-progs. We’ll start with a read-only check to avoid exacerbating any existing issues.

Running a Read-Only Scrub

A Btrfs scrub is a crucial operation that reads all data and metadata blocks, verifies their checksums, and, if redundancy is available (e.g., RAID1, RAID10), attempts to repair any corrupted blocks using good copies. Running it in read-only mode first is a safe diagnostic step.

Command Syntax

To initiate a read-only scrub on a mounted Btrfs filesystem (e.g., mounted at /), use the following command. Replace / with the mount point of the affected filesystem if it’s not the root.

Example: Read-only scrub on root filesystem

sudo btrfs scrub start -B /

The -B flag indicates a background scrub, which is less intrusive. For immediate feedback, you can omit it, but be aware this can impact performance. To monitor the scrub’s progress, use:

sudo btrfs scrub status /

If the scrub reports errors, it indicates that corruption has been detected. The output will detail the types of errors (e.g., checksum errors, uncorrectable errors). If the scrub completes without errors, the issue might be intermittent or related to specific operations rather than persistent data corruption.

Checking the Btrfs Tree Structure

Btrfs uses a tree structure for its metadata. Corruption in these trees can lead to filesystem inconsistencies. The btrfs check command is a more in-depth diagnostic tool, but it must be run on an unmounted filesystem to prevent further damage.

Unmounting the Filesystem

This is a critical step. If the filesystem is your root filesystem (/), you will need to boot from a live environment (e.g., openSUSE Leap installation media, a rescue disk) to unmount it. For other mounted filesystems, use umount.

# If it's not the root filesystem
sudo umount /path/to/btrfs/mountpoint

# If it's the root filesystem, boot from live media and identify the device
# e.g., /dev/sda2
sudo umount /dev/sda2

Running btrfs check

Once unmounted, run btrfs check. The --readonly flag is highly recommended for the initial check.

sudo btrfs check --readonly /dev/sdXN

Replace /dev/sdXN with the actual device name of your Btrfs partition. If this command reports errors, it confirms structural issues within the Btrfs metadata. The output will often pinpoint specific tree roots or block numbers that are problematic.

Recovery Pipelines: From Corruption to Data Restoration

When corruption is confirmed, the recovery strategy depends on the severity and the available data. The goal is to extract as much intact data as possible and then rebuild the filesystem or restore from backups.

Leveraging Btrfs Snapshots

Btrfs’s snapshotting capability is its strongest defense against data loss. If you have recent, uncorrupted snapshots, this is your primary recovery mechanism.

Restoring from a Snapshot

The process involves mounting the snapshot and copying data back. This is best done from a live environment to avoid issues with the currently mounted (and potentially corrupt) filesystem.

# Boot from live media
# Identify your Btrfs device and mount it read-write
sudo mount /dev/sdXN /mnt/btrfs

# Mount a specific snapshot (assuming snapshots are in a .snapshots subvolume)
# Find your snapshot name, e.g., @/.snapshots/snapshot_name/snapshot
sudo mount -o subvol=@/.snapshots/snapshot_name/snapshot /mnt/snapshot

# Copy data back (example: restoring /home)
# Ensure the target directory exists and is empty or you intend to overwrite
sudo rsync -avh --progress /mnt/snapshot/home/ /mnt/btrfs/home/

# Unmount everything cleanly
sudo umount /mnt/snapshot
sudo umount /mnt/btrfs

Important: Always ensure the snapshot you are restoring from is known to be good. If the corruption occurred before the snapshot was taken, restoring from it will not help. If snapshots are also affected, the corruption might be at a lower level (e.g., hardware). If the corruption is widespread and affects snapshots, consider the next steps.

Data Extraction with `btrfs restore`

When the filesystem is severely damaged and cannot be mounted, or if snapshots are unavailable or also corrupted, btrfs restore is the last resort for data extraction. This tool attempts to scan the raw device for Btrfs metadata and extract files.

Prerequisites

You will need a separate, healthy filesystem with enough space to store the extracted data. This is typically a USB drive or a network share mounted in your live environment.

Running `btrfs restore`

The command requires the device path and a destination directory on a separate filesystem.

# Mount your destination filesystem (e.g., a USB drive)
sudo mount /dev/sdYN /mnt/recovery

# Run btrfs restore (replace /dev/sdXN with your corrupt Btrfs device)
sudo btrfs restore -v -i /dev/sdXN /mnt/recovery/

The -v flag provides verbose output, and -i attempts to reconstruct the filesystem tree. This process can be very time-consuming, especially on large drives. It will attempt to recover files and directories based on the metadata it can find. Files that are severely damaged or have missing metadata might be recovered with corrupted content or with generic names.

Sector Recovery and Hardware Considerations

If btrfs check reports uncorrectable errors or if btrfs scrub consistently fails with I/O errors, the problem often lies at the hardware level. This could be a failing hard drive, faulty SATA cables, or issues with the RAID controller (if applicable).

Identifying Failing Sectors

Use SMART tools to check the health of your drives.

sudo smartctl -a /dev/sdX

Look for attributes like “Reallocated_Sector_Ct”, “Current_Pending_Sector”, and “Offline_Uncorrectable”. Non-zero values in these fields, especially if increasing, are strong indicators of drive failure. Btrfs’s checksumming helps detect corruption caused by bad sectors, but it cannot fix hardware failures on its own without redundancy.

Rebuilding with Redundancy

If your Btrfs filesystem is configured with redundancy (e.g., RAID1, RAID10), Btrfs will attempt to use a good copy of a block if one is found during a scrub. If a drive is failing, you might see errors during scrub that Btrfs can correct by reading from another drive. The next step is to replace the failing drive and allow Btrfs to rebuild the data onto the new drive.

# Example: Replacing a drive in a RAID1 setup
# Identify the failing drive (e.g., /dev/sdX)
# Physically replace the drive with a new one of equal or larger size.
# Add the new drive to the Btrfs filesystem (assuming it's already part of a RAID profile)
sudo btrfs replace start /dev/sdX /dev/sdY  # sdX is the old drive, sdY is the new drive

# Monitor the rebuild process
sudo btrfs device stats /path/to/btrfs/mountpoint

If the corruption is severe and Btrfs cannot recover, or if the underlying hardware is compromised, the most reliable path forward is often to:

• Use btrfs restore to extract as much data as possible to a separate, healthy storage medium.
• Reformat the affected drive(s) with a known good filesystem (or a fresh Btrfs instance after ensuring hardware integrity).
• Restore the extracted data from the recovery medium.
• If available, verify against known good backups.

Proactive monitoring of SMART data and regular, verified Btrfs scrubs are essential for preventing such catastrophic data loss scenarios in production environments.