Linux server troubleshooting: CPU, memory, disk, and process investigation

A server under pressure does not need a random sequence of commands. It needs a bounded investigation that separates impact from cause and protects the service while evidence is collected.

Operational warning Avoid restarting a service just to make a graph look better. First establish what is failing, when it changed, and which host or dependency is involved. A restart can remove the evidence needed to make the next incident shorter.

Quick diagnostic flow

Symptom → Scope → CPU / Memory / Disk / Process → Logs → Correlate → Remediate → Verify

Start with the user-visible impact: is this latency, availability, resource exhaustion, or data persistence? Record the first observed time, affected service and host, and whether the problem is isolated or shared by a dependency. That framing determines which signal is meaningful.

Start with impact and scope

Before interpreting a metric, answer four questions: what is actually failing, when did it start, which host or service is affected, and what changed near that time? A high load average on an otherwise healthy batch host has a different meaning from the same value on a latency-sensitive API node.

Capture a small baseline before changing anything:

uptime
top -b -n 1 | head -40
free -h
df -h
df -i
ps -eo pid,ppid,stat,%cpu,%mem,rss,vsz,comm --sort=-%cpu | head -20
systemctl --failed
journalctl -p warning..alert --since '30 minutes ago'

Each command asks a different question. uptime gives recent load, top exposes CPU and process pressure, free -h distinguishes available memory from cache, df and df -i separate capacity from inode exhaustion, and journalctl anchors symptoms in time.

Investigate CPU and load

Load average is not CPU utilisation. It represents runnable work and, on Linux, work waiting in uninterruptible I/O. Compare load with CPU state before deciding that CPU is the bottleneck.

uptime
mpstat -P ALL 1 3
ps -eo pid,ppid,stat,%cpu,%mem,etime,comm --sort=-%cpu | head -15

Look for sustained user CPU when application work is hot, system CPU when kernel/network work dominates, and iowait when processes are waiting on storage. A high load with modest CPU can point to blocked I/O rather than a runaway compute task. Process-level CPU, elapsed time, parent PID, and state help distinguish a normal busy worker from a loop or a stuck child process.

Investigate memory pressure

Used memory alone is not a diagnosis. Linux intentionally uses spare memory for page cache; available is usually more useful than used. Check swap activity, OOM messages, and per-process resident memory before changing limits or restarting workloads.

free -h
vmstat 1 5
ps -eo pid,ppid,%mem,rss,vsz,comm --sort=-rss | head -20
journalctl -k --since '2 hours ago' | grep -Ei 'oom|killed process|out of memory'

RSS is resident memory currently backed by RAM. VSZ is virtual address space and can be much larger without proving pressure. Reclaimable cache is normal; growing RSS, active swap-in/out, allocation failures, or OOM-killed processes are not. Treat a memory symptom as a workload and lifecycle question, not just a capacity number.

Investigate disk and I/O

A filesystem can fail because it is full, because it has no free inodes, or because a deleted file is still held open. Storage latency can also create high load without high CPU.

df -h
df -i
du -xhd1 /var | sort -h
find /var -xdev -type f -size +1G -printf '%s %p\n' | sort -n
lsof +L1
 iostat -xz 1 3

Use du to narrow large directories, then inspect the owning application before removing data. lsof +L1 finds deleted-but-open files that still consume space until the owning process closes them. With iostat, look for sustained queueing, high utilisation, and latency that aligns with request errors or slow database calls.

Inspect processes, services, and logs

Process state explains how a resource signal is being produced. A D state process is waiting in uninterruptible sleep, often on I/O. Parent/child relationships show whether a supervisor is repeatedly spawning work or whether a worker is orphaned.

ps -eo pid,ppid,stat,etime,comm --forest
systemctl status your-service
journalctl -u your-service --since '30 minutes ago'
ss -ltnp

Check service status and logs before acting. If termination is required, use the service manager or a graceful signal first, then confirm the process and downstream health have recovered. SIGKILL is a last resort because it bypasses application cleanup.

Correlate before remediation

CPU, memory, disk, logs, and process state should be read together. A memory-heavy process may also be writing large temporary files. A full disk may create application errors that trigger retries and load. A failing dependency may leave workers blocked in I/O while CPU remains quiet.

Record command output, relevant log lines, timestamps, recent deployments or configuration changes, affected process IDs, and current resource state. The operating order is observe → isolate → verify → remediate → validate. Choose the least disruptive action that tests a clear hypothesis.

Verify after remediation

After a change, verify service health, resource normalisation, error rate, logs, and downstream dependencies. Check the original symptom rather than assuming a restarted process resolved it. Watch long enough to detect recurrence, especially after traffic, scheduled work, or a dependency call that previously triggered the issue.

Make the next incident easier

Recurring symptoms should improve the system: monitoring that detects the right failure mode, alerts with useful context, capacity planning based on observed growth, concise runbooks, incident notes, and automation for repetitive evidence collection. The goal is not more commands; it is removing recurring root causes and shortening safe diagnosis.

Conclusion

Production troubleshooting is disciplined correlation. Start with impact, collect evidence across CPU, memory, disk, process state, and logs, then remediate only after the most likely mechanism is clear. The result is safer change during an incident and a better operating model afterward.

Need help with production infrastructure? KRSTACK supports Linux, DevOps, reliability, and infrastructure engineering conversations where diagnosis needs to become a maintainable operating improvement. Start a conversation.