Troubleshooting Linux

Jump to the Examples

Disclaimer: Should you decide to act upon any information on this website, you do so at your own risk! A single wrong character in a command that runs as root, such as rm, dd, chmod -R, mkfs or fsck on the wrong device, can erase a disk or make a system unbootable, and Linux will not ask you whether you are sure. Back up first, check the device name twice, read what a command does before you press Enter, and never paste a line that you cannot explain, especially one that pipes a download straight into a shell.

Linux does not break less than other systems, it breaks differently. The good news is that it usually tells you what is wrong, in plain text, in a log, with a time stamp. The bad news is that it will not open a window about it, so you have to know where to read. This note is about that: a method that works on every distribution, a map of the boot process, the commands I would use first, and a few typical failures worked through step by step. The examples were written for Debian and its relatives (Ubuntu, Mint) and for the Red Hat family (Fedora, Rocky, Alma), and where the two differ, both versions are shown.

Other notes continue where this one stops. Choosing and installing a distribution, also on an old computer, is in Reviving an Old Computer. The terminal is in Communication with the Machines, the network in Computer Networks, a Debian server with SSH, updates and containers in Constructing a Cloud, and the dangers and the backups in Security Considerations. If you are coming from the other side, the same approach for the other big system is in Troubleshooting Windows.

A Method Before the Commands

On Linux it is especially tempting to skip straight to a command from a forum, because there is always one. Slow down for a few minutes first.

1. Read the message. Really read it, all of it, including the last line. An error such as No such file or directory, Permission denied or Connection refused already says which family of problems you are in. Copy the exact text into your notes, or photograph the screen if you cannot copy it.
2. Ask what changed. An update, a new package, an edited file in /etc, a new disk, a different kernel, a full disk. The package manager keeps a history (/var/log/apt/history.log on Debian, dnf history on Fedora), and it is often the shortest route to the cause.
3. Look at the logs before you guess. journalctl is the first place for almost everything (see the next sections).
4. Save your data first. You can reinstall Linux in twenty minutes, but you cannot reinstall your photographs. A live USB stick can reach the files even when the system is unusable.
5. Change one thing at a time, and write it down. Make a copy of a configuration file before you edit it (cp file file.bak), so that going back is one command.
6. Test the cheapest explanation first. A full disk, a stopped service, a typo in a configuration file, a wrong permission. These are the causes of most of what happens, and they take a minute to rule out.
7. Distrust copy and paste of anything with sudo in it, from a post that is five years old, for a distribution you do not use. If you cannot say what the command does, look it up first, with man or tldr.
8. Know when to stop. See the section on reinstalling. On Linux it costs less than it does elsewhere.

How Linux Starts, and Where It Breaks

The boot is a chain, and the symptom tells you which link is broken. It helps to know the names, because the error messages and the guides use them.

Power on
    |
[ Firmware (UEFI / BIOS) ]       hardware test, finds the boot device
    |
[ Boot loader (GRUB, systemd-boot) ]   shows the menu, loads the kernel
    |
[ Kernel + initramfs ]           drivers for the disk, finds and mounts the root file system
    |
[ systemd (PID 1) ]              mounts /etc/fstab, starts the services, reaches a "target"
    |
[ Login manager (GDM, SDDM, LightDM) ]   the graphical login, or a text login
    |
[ Desktop session ]

Where it stops, and where I would look first (my own rules of thumb, not a complete list):

The Tools Already on the Machine

As on every system, you do not need to install anything to begin. These are present on practically every distribution, with the exception of a few that are in separate packages, and those are named in the table. The basics of the terminal are in Communication with the Machines.

What there is and what it tells you:

One detail about the logs. On some distributions the journal is kept only in memory and is lost at the restart, which is exactly when you want it. If journalctl -b -1 says that there are no entries for the previous boot, make it permanent with sudo mkdir -p /var/log/journal and restart the machine (or set Storage=persistent in /etc/systemd/journald.conf). Do it today, before the problem.

Common Symptoms and the First Things to Check

The table is only a starting point. Each row says what I would look at first, and what comes next if the first look shows nothing. The numbered examples after it go through the steps in detail.

Symptoms, first checks and the next step:

A Note on SELinux and AppArmor

Besides the usual file permissions, many distributions add a second layer that decides what a program may do, whoever starts it. The Red Hat family (Fedora, RHEL, Rocky, Alma) uses SELinux, switched on by default. Ubuntu and Debian use AppArmor. When a service gets "Permission denied" while the ordinary permissions are perfect, and it started working when you moved a file somewhere else, this layer is the usual explanation.

getenforce                                 # Enforcing, Permissive or Disabled
sudo ausearch -m avc -ts recent            # the recent SELinux denials
sudo restorecon -Rv /path/to/directory     # put the default labels back on files
sudo aa-status                             # AppArmor: the profiles and their modes

The most common cause on SELinux is a file that was moved (not copied) from a home directory into the directory of a service, so that it keeps the old label. restorecon usually fixes that. sudo setenforce 0 switches SELinux to permissive until the next restart, which is a good test: if the problem goes away, you know that it is SELinux. Then switch it back on with setenforce 1 and fix the label or the rule. "Disable SELinux" in the first answer of a forum thread is not a fix, it is giving up the protection of the machine.

Examples: Step by Step

Each example is the order in which I would try things, not the only possible one. Stop as soon as the problem is solved, and write down what worked. Lines that begin with # in the boxes are comments.

Example 1: The Machine Has Become Slow

uptime                      # load average: compare it with the number of processor cores (nproc)
free -h                     # is "available" memory small, is swap in use?
top                         # press M to sort by memory, P by processor
journalctl -k | grep -i "out of memory"    # did the kernel have to kill something?
systemd-analyze blame | head -15           # if the problem is a slow start

1. Read the load against the cores. A load of 4 is nothing on a machine with 8 cores, and a disaster on one with 2. A high load with a low processor use usually means that processes are waiting for the disk or the network.
2. Memory. The column that matters in free -h is available, not free: Linux uses the unused memory for caching, and gives it back at once. When available is small and swap is busy, the machine is out of memory, and a browser with many tabs is the usual guilty party. If the kernel killed a process, the log says "Out of memory: Killed process".
3. Disk. A disk that is full, or dying, makes everything slow. Check df -h and the SMART data: sudo smartctl -a /dev/sda (for an NVMe disk the device is /dev/nvme0). Reallocated or pending sectors, or a status other than PASSED, mean that the first job is a backup, and the second is a new disk.
4. Heat. sensors shows the temperatures. A laptop at 95 degrees under a light load has a blocked cooler, and the processor slows itself down to survive.
5. The start-up. If it is only the boot that is slow, systemd-analyze blame shows the unit that takes the longest. A network share, or a disk that is no longer there, that waits for a time-out is typical.
6. If the system is on a mechanical hard disk, the fastest improvement is an SSD, with any software.

Example 2: The Disk Is Full

df -h                                      # which file system is full?
df -i                                      # or is it out of inodes?
sudo du -xh --max-depth=1 /var | sort -h   # where is the space? (repeat in the biggest directory)
journalctl --disk-usage                    # the size of the logs
sudo journalctl --vacuum-size=200M         # keep only 200 MB of them
sudo lsof +L1                              # deleted files that a program still keeps open

The -x option of du keeps it on one file system, so that it does not wander into other disks. If df -h shows space but programs still say "No space left on device", the inodes are the problem: a directory with millions of tiny files (a mail queue, session files, a cache) is the reason, and df -i proves it. If a big file was deleted and the space did not come back, a running program still has it open. lsof +L1 lists those, and a restart of that program releases the space.

The usual safe savings are the package cache and the packages that nobody needs any more:

sudo apt clean                  # Debian family: the downloaded packages
sudo apt autoremove             # packages that were installed as dependencies and are not needed now
sudo dnf clean all              # Red Hat family
sudo dnf autoremove

Old kernels also take up space, and so do the container images (docker system df shows them). Do not delete anything under /usr, /lib or /etc by hand to make room, and keep in mind that a completely full root file system can also mean that you cannot log in at all, because the login wants to write a small file. In that case the live USB stick is the way in.

Example 3: A Service Will Not Start

systemctl status nginx                   # replace nginx with your service
journalctl -u nginx -b --no-pager | tail -30
sudo nginx -t                            # most servers can check their own configuration
sudo ss -tulpn | grep :80                # is something else already on the port?

1. The status and the log. The last lines nearly always contain the sentence that explains it: a syntax error in line 14, an address that is already in use, a file that cannot be opened.
2. Test the configuration with the program's own check, if it has one: nginx -t, apachectl configtest, sshd -t, named-checkconf, postfix check. Never restart sshd over an SSH session before sshd -t has said that it is fine, and keep a second session open while you test.
3. The port. If another process holds it, ss -tulpn shows which one.
4. Permissions and labels. Does the service's user have access to the files and the directories on the way (namei -l)? On the Red Hat family, look at SELinux (above).
5. The unit file. If you edited a unit, run sudo systemctl daemon-reload first. The right way to change the unit of a package is sudo systemctl edit name, which creates a small override that survives updates, not the edit of the original file.
6. After the fix: sudo systemctl restart name, and systemctl status name once more. A service that is in the state failed repeatedly can be reset with sudo systemctl reset-failed name.

Example 4: The Network Does Not Work

The method is a ladder: go up one step at a time, and the first step that fails shows where the problem is. The background of addresses, gateways and DNS is in Computer Networks.

ip a                        # does the interface have an address? is it "UP"?
ip route                    # is there a "default via" line, the gateway?
ping -c 3 192.168.1.1       # the router (use the address from the line above)
ping -c 3 1.1.1.1           # the Internet by address, DNS is not needed
ping -c 3 example.org       # the Internet by name, DNS is needed
resolvectl status           # which DNS servers are in use (systemd-resolved)

• No address, or the interface is down: the cable or the Wi-Fi first. rfkill list shows whether the radio is blocked by software or by a key. For the Wi-Fi card that is not seen at all, lspci -k and sudo dmesg | grep -i firmware tell whether the driver wants a firmware file that is missing (the package is often called firmware-iwlwifi, linux-firmware or similar).
• An address, but nothing from the router: the wrong network, the wrong password, or the router is the problem. Try another device.
• The router answers, the address does not: the connection of the router itself to the Internet.
• The address answers, the name does not: DNS. Try resolvectl query example.org, then dig @1.1.1.1 example.org to see whether another DNS server answers. If it does, the one in use is the problem.
• Everything fine locally, and the one service is unreachable from outside: ss -tulpn (is it listening on the right address?), the firewall (sudo nft list ruleset, or sudo firewall-cmd --list-all on the Red Hat family, sudo ufw status on Ubuntu), and then the router's port forwarding.

On most desktop distributions the network is managed by NetworkManager, so nmcli shows the connections, and sudo systemctl restart NetworkManager is a mild first aid. Do not mix several network managers: a file edited in /etc/network/interfaces together with NetworkManager, or a netplan file together with an old configuration, is a reliable source of mysterious problems.

Example 5: Linux Will Not Start

Start with the cheap things: is the disk visible in the firmware setup, and does the GRUB menu appear? In the menu, Advanced options lists the older kernels, and starting one of them is the quickest test of whether the latest update is the problem. After that, it depends on where it stops (the table in the section on starting).

(a) Emergency mode, or a long wait at the start. The system started, but a mount failed. Log in with the root password in the emergency shell (or press Ctrl + D to try to continue), and read:

journalctl -xb | tail -50       # what failed?
sudo blkid                      # the real UUIDs of the disks
sudo nano /etc/fstab            # compare the UUIDs, add nofail to disks that are not essential
sudo mount -a                   # test: no error means that fstab is good

(b) A black screen, or a broken graphical start after an update. At the GRUB menu press e over the entry, add nomodeset at the end of the line that starts with linux, and continue with Ctrl + X. This is only for this boot and it makes the system use a basic graphics mode, so if it works, the graphics driver is the problem, and it is a matter of reinstalling or rolling back the proprietary driver. With a text console (Ctrl + Alt + F3) the same repairs are possible.

(c) GRUB or the initramfs is broken. Boot the live USB stick and repair from a chroot. First find out which partition is which, and double-check the names. They are different on every computer:

lsblk -f                                   # find the root partition and the EFI partition (small, vfat)
sudo mount /dev/nvme0n1p2 /mnt             # the root partition (an example name!)
sudo mount /dev/nvme0n1p1 /mnt/boot/efi    # the EFI partition (UEFI computers only)
for d in dev proc sys run; do sudo mount --rbind /$d /mnt/$d; done
sudo chroot /mnt

Now you are working inside the installed system, and the repair is the one of that family:

# Debian family, UEFI:
grub-install --target=x86_64-efi --efi-directory=/boot/efi
update-grub
update-initramfs -u -k all

# Red Hat family:
dnf reinstall grub2-efi-x64 shim-x64       # UEFI: puts the boot files back
grub2-mkconfig -o /boot/grub2/grub.cfg
dracut --force --regenerate-all            # regenerate the initramfs

The commands for an old computer with a BIOS and not UEFI differ: grub-install /dev/sda (the disk, not the partition). Exit with exit, unmount, restart. If the system is encrypted with LUKS, open it first with cryptsetup open, and have the passphrase ready. The details of every distribution are on its own wiki, and the ArchWiki is good even for those that are not Arch.

(d) A damaged file system. Run fsck only on a file system that is not mounted, so from the live system or from the rescue shell: sudo fsck -f /dev/sdXN. Doing it on a mounted file system can wreck it. First look at the SMART data: if the disk is failing, copy the data off before the long repairs, and do not run them on it twice.

Example 6: The Package Manager Is Broken

The usual reasons are an interrupted update (the power, a closed terminal, a full disk), a repository that does not match your version, or a package from outside (a PPA, a downloaded .deb or .rpm) that conflicts with the system. Check the free space first, then:

# Debian family
sudo dpkg --configure -a        # finish what was interrupted
sudo apt --fix-broken install
sudo apt update
sudo apt full-upgrade
apt-mark showhold               # packages that are held back on purpose, and so block others

# Red Hat family
sudo dnf clean all
sudo dnf history                # what was done last, and the numbers of the transactions
sudo dnf history undo 42        # undo the transaction number 42, if it was the cause

If the cause is a third-party repository, switch it off (remove the file under /etc/apt/sources.list.d/ or /etc/yum.repos.d/), and then try again. Mixing the repositories of different releases ("just for this one program") is how the system becomes a puzzle that nobody can solve, so avoid it. A package that is damaged on the disk can be reinstalled with sudo apt install --reinstall name or sudo dnf reinstall name. And never fix a conflict with --force or by deleting files from the package database, unless you are prepared to reinstall afterwards.

When to Stop Repairing

Here Linux has a real advantage. The settings are text files, the system and your data can live in separate places, and a reinstall that keeps your files takes about as long as one long troubleshooting session. There is no shame in it. Do it when you repeat yourself, when the fix of last month is needed again, when you cannot explain what is going on, or when the number of manual changes has made the machine something that nobody would be able to repeat.

The options, from the gentlest to the most thorough:

Before you wipe anything, back up the home directory, the /etc directory (the settings of every service), the databases (with their own dump tool, not by copying the files of a running server – see Constructing a Database), and the list of installed packages (apt-mark showmanual, or dnf repoquery --userinstalled). Then a reinstall is also a documentation exercise. A system that was damaged by an intrusion is not reinstalled over the old one with the same passwords: see Security Considerations.

Making It Less Likely Next Time

• Back up, and try the restore once. rsync, BorgBackup and restic are the usual tools. A backup that has never been restored is a hope. The 3-2-1 rule is in Security Considerations.
• Take a snapshot before a big update. Timeshift (for the system files, on ext4 with rsync or on Btrfs) or Snapper on Btrfs and openSUSE. Updates that go wrong then cost a restart, not an evening.
• Put /etc under version control with etckeeper. It commits the changes of the settings before and after each package operation, so that "what changed" has a precise answer.
• Keep a separate /home on a separate partition or disk if you can, and a fallback kernel in the boot menu (the package managers keep the last two or three, which is a good default).
• Stay with the repositories of the distribution. Every extra repository and every curl | sudo bash is a debt that is paid later, at the update of the next release.
• Use the release that fits your patience. A long-term release for the machine that has to work, and a rolling or a fast one for the machine that you like to play with. Read the release notes before the big upgrade, and do not do it the evening before you need the computer.
• Watch the disks and the space. smartd (from smartmontools) can send an e-mail when the disk begins to fail, and the same goes for a cron job that warns at 85 percent of full. The logs should not be able to fill the root file system.
• Keep the system updated, with automatic security updates on servers (see Constructing a Cloud), and a restart when a kernel or a library that is in use has been updated.
• Write down what you did, in a text file in the home directory or in a notes system. In six months this will be the best documentation of the machine that exists.

Some resources:

• ArchWiki – General troubleshooting (useful for every distribution)
• journalctl manual and systemctl manual
• Debian Reference
• Fedora Quick Docs and Red Hat documentation
• GNU GRUB manual
• SystemRescue, smartmontools
• The Linux man pages online

A Few More Thoughts

Linux repays the habit of reading. Almost everything is a text file and almost every failure leaves a line somewhere, so the skill is less a matter of knowing commands than of knowing where to look, and of staying calm enough to read the whole message. When you finally ask somebody else for help, a question that contains the distribution and its version, the exact error, the relevant lines from the journal, and what you have already tried gets an answer in an hour. A question that says "it does not work" gets a request for exactly that information, a day later.

I find it worth remembering that most of what is called "breaking" in Linux is something that we did ourselves, usually at the end of a long evening, with a command from a page that we did not fully read. That is also the good news: it means that the habits of this note, the copies, the notes, the snapshot and the one change at a time, remove most of the risk. If this machine is one of several that you run, and one of them is the mail server of Constructing E-mail or the cloud of Constructing a Cloud, these habits are no longer optional.