Advanced Linux: 200 Senior/SRE-Level Commands

Here Linux stops being "using the terminal" and becomes understanding what the system is doing: where the time goes, why the latency went up, who held on to the disk, what the kernel decided and how to recover a machine that no longer boots.

These are 200 commands for diagnosis, tuning and hardening — the ones you use during an incident, not in a tutorial.

A warning: a good number of these commands change the system's behaviour. Test in a staging environment, understand the effect and have a way back before running them in production.

Contents

• 1–30 · Performance and observability
• 31–50 · Kernel and system parameters
• 51–70 · Namespaces, cgroups and isolation
• 71–95 · Firewall and advanced security
• 96–115 · Advanced storage
• 116–135 · Advanced networking
• 136–160 · Process debugging
• 161–180 · Boot, initramfs and recovery
• 181–200 · Automation and operations

Performance and observability

The USE method: for each resource, measure Utilisation, Saturation and Errors.

1. The first 60 seconds

The sequence Netflix popularised for diagnosing any slow machine.

uptime
dmesg -T | tail
vmstat 1 5
mpstat -P ALL 1 3
pidstat 1 3
iostat -xz 1 3
free -m
sar -n DEV 1 3
top

2. Load average in practice

It is the queue of processes waiting for CPU and I/O. Always compare it with the number of cores.

uptime
nproc
# a load of 8 on 8 cores = saturated; on 32, relaxed

3. vmstat

The overview: processes, memory, swap, I/O and CPU. A high r = waiting for CPU; a high b = waiting for I/O.

vmstat 1

4. mpstat per core

Finds a single saturated CPU while the average looks low.

mpstat -P ALL 1

5. Interpreting %CPU

us user, sy kernel, wa I/O wait, st steal from the hypervisor.

top -bn1 | head -5
# a high %st in the cloud = a noisy neighbour

6. pidstat — per process

CPU, memory and I/O of each process over time.

pidstat 1
pidstat -d 1      # I/O
pidstat -r 1      # memory

7. sar — history

Data collected through the day: it lets you investigate an incident that is already over.

sar -u 1 3          # CPU now
sar -u -f /var/log/sysstat/sa10   # CPU on the 10th
sar -r -s 14:00 -e 15:00

8. iostat

Disk latency and utilisation. A high await with %util at 100 is an I/O bottleneck.

iostat -xz 1

9. iotop

Who is generating I/O right now.

sudo iotop -oPa

10. Memory for real

available is what matters; a low free with plenty of cache is normal and healthy.

free -m
cat /proc/meminfo | head -20

11. Consumption per process

Sorts by resident memory.

ps aux --sort=-%mem | head -10
ps -eo pid,comm,rss,vsz --sort=-rss | head

12. smem — proportional memory

PSS divides shared memory up — far more honest than RSS.

smem -rs pss | head -15

13. A process's memory map

What is mapped and how much it takes up.

pmap -x 12345 | tail -1
cat /proc/12345/status | grep -i vm

14. Cache and page cache

The kernel uses free RAM as cache — and gives it back when it needs to.

free -h
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches   # for testing only, never in production

15. File descriptors

An exceeded limit takes a service down with "too many open files".

ulimit -n
cat /proc/sys/fs/file-nr
ls /proc/12345/fd | wc -l

16. Raise the file limit

Under systemd, limits.conf does not apply to services.

# /etc/security/limits.conf
app soft nofile 65535
app hard nofile 65535

# systemd unit
[Service]
LimitNOFILE=65535

17. Context switching

Many context switches indicate excessive concurrency or thrashing.

vmstat 1 | awk '{print $12, $13}'
pidstat -w 1

18. perf top

Where the CPU is being spent, function by function, in real time.

sudo perf top
sudo perf top -p 12345

19. perf record and report

Records the profile and analyses it afterwards.

sudo perf record -F 99 -a -g -- sleep 30
sudo perf report --stdio

20. Flame graph

Turns the profile into a graph where the bottleneck jumps out at you.

sudo perf record -F 99 -a -g -- sleep 30
sudo perf script | stackcollapse-perf.pl | flamegraph.pl > perfil.svg

21. bpftop and biotop

eBPF tools that measure at almost no cost.

sudo biotop      # I/O per process
sudo execsnoop   # every exec on the system
sudo opensnoop   # every open

22. Disk latency with eBPF

The distribution of the latency, not just the average.

sudo biolatency 10 1

23. Trace slow syscalls

Shows system calls that went over a threshold.

sudo funcslower -u vfs_read 10000

24. stress-ng — generate load

Reproduces the problem to validate the hypothesis.

stress-ng --cpu 4 --timeout 60s
stress-ng --vm 2 --vm-bytes 1G --timeout 60s

25. Disk benchmark

fio measures real IOPS and latency.

fio --name=teste --rw=randread --bs=4k --size=1G --numjobs=4 --runtime=60 --group_reporting

26. Monitor everything with dstat

CPU, disk, network and memory on the same screen.

dstat -tcdngy 1

27. Processes in state D

Uninterruptible I/O wait — they do not die even with kill -9.

ps -eo pid,state,comm | awk '$2 == "D"'

28. Zombies

A dead process the parent has not reaped. The problem is in the parent.

ps -eo pid,ppid,state,comm | awk '$3 == "Z"'

29. CPU usage per container

For when the machine is shared.

systemd-cgtop
docker stats --no-stream

30. A baseline before the incident

Without history, every number looks abnormal. Collect metrics continuously.

# enable sysstat so you have history
sudo systemctl enable --now sysstat
sar -u -f /var/log/sysstat/sa$(date +%d)

Kernel and system parameters

Tuning the core's behaviour — knowing what each number does.

31. sysctl — read and adjust

The interface to the kernel's parameters at runtime.

sysctl -a | less
sysctl net.ipv4.ip_forward
sudo sysctl -w net.ipv4.ip_forward=1

32. Make it permanent

A file in /etc/sysctl.d/ survives the reboot.

# /etc/sysctl.d/99-tuning.conf
net.core.somaxconn = 4096
vm.swappiness = 10

sudo sysctl --system

33. The connection queue

A low somaxconn drops connections at a traffic peak.

sysctl net.core.somaxconn
sudo sysctl -w net.core.somaxconn=4096

34. TIME_WAIT and reuse

A server with many short connections exhausts the ephemeral ports.

sudo sysctl -w net.ipv4.tcp_tw_reuse=1
sudo sysctl -w net.ipv4.ip_local_port_range="10000 65535"

35. Network buffers

Increasing them helps on a high-latency, high-bandwidth link.

sudo sysctl -w net.core.rmem_max=16777216
sudo sysctl -w net.core.wmem_max=16777216

36. Congestion control

BBR usually yields more than cubic on a lossy link.

sysctl net.ipv4.tcp_congestion_control
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr

37. Memory overcommit

Defines whether the kernel promises more memory than it has.

sysctl vm.overcommit_memory
# 0 heuristic | 1 always allows | 2 genuinely limits

38. Tune the OOM killer

Protects (or sacrifices) a specific process.

echo -1000 | sudo tee /proc/12345/oom_score_adj   # nearly immune
cat /proc/12345/oom_score

39. Dirty pages

Controls how much can stay in memory before going to disk.

sysctl vm.dirty_ratio vm.dirty_background_ratio

40. Maximum processes and threads

The system's global limit.

sysctl kernel.pid_max
cat /proc/sys/kernel/threads-max

41. Kernel modules

Listing, loading and removing.

lsmod
sudo modprobe nf_conntrack
sudo rmmod nome_modulo
modinfo nf_conntrack

42. Blacklist a module

Prevents an unwanted driver from loading automatically.

# /etc/modprobe.d/blacklist-custom.conf
blacklist pcspkr

43. Boot parameters

See what was passed to the kernel at startup.

cat /proc/cmdline

44. Kernel version and features

Confirms support for a feature before enabling it.

uname -r
grep CONFIG_BPF /boot/config-$(uname -r)

45. Switch kernels

List the installed ones and choose GRUB's default.

dpkg --list | grep linux-image
sudo grub-set-default 1
sudo update-grub

46. Transparent Huge Pages

Databases usually ask for them to be turned off.

cat /sys/kernel/mm/transparent_hugepage/enabled
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled

47. CPU governor

performance removes the latency of frequency changes.

cpupower frequency-info
sudo cpupower frequency-set -g performance

48. NUMA

On a large server, memory on the wrong node costs latency.

numactl --hardware
numactl --cpunodebind=0 --membind=0 ./programa

49. Entropy

Low entropy stalls key generation on a VM.

cat /proc/sys/kernel/random/entropy_avail
sudo apt install haveged

50. The virtual filesystem panel

/proc and /sys are the source of nearly every metric.

cat /proc/loadavg
cat /proc/stat | head -3
ls /sys/class/net/

Namespaces, cgroups and isolation

What lies underneath every container.

51. See a process's namespaces

Each link shows which namespace it is in.

sudo ls -l /proc/12345/ns/

52. lsns — list namespaces

An overview of all the system's namespaces.

sudo lsns
sudo lsns -t net

53. unshare — create a namespace

Runs an isolated process, with no Docker at all.

sudo unshare --pid --fork --mount-proc bash
ps aux   # it only sees its own namespace

54. nsenter — enter the namespace

Enters a container's namespace to debug it.

sudo nsenter -t 12345 -n ip a       # the process's network
sudo nsenter -t 12345 -m -u -i -n -p bash

55. Network namespace

Creates a separate network stack.

sudo ip netns add teste
sudo ip netns list
sudo ip netns exec teste ip a

56. Link two namespaces

A veth pair: the virtual cable Docker uses.

sudo ip link add veth0 type veth peer name veth1
sudo ip link set veth1 netns teste
sudo ip addr add 10.0.0.1/24 dev veth0
sudo ip link set veth0 up

57. cgroups v2

Where the resource limits per group of processes live.

cat /sys/fs/cgroup/cgroup.controllers
systemd-cgls

58. Limit memory with a cgroup

A limit applied directly by the kernel.

sudo mkdir /sys/fs/cgroup/meugrupo
echo 512M | sudo tee /sys/fs/cgroup/meugrupo/memory.max
echo 12345 | sudo tee /sys/fs/cgroup/meugrupo/cgroup.procs

59. Limit CPU with a cgroup

The quota and the period define the CPU slice.

echo "50000 100000" | sudo tee /sys/fs/cgroup/meugrupo/cpu.max   # 50% of one core

60. systemd-run with limits

The practical way to apply a cgroup to a command.

sudo systemd-run --scope -p MemoryMax=512M -p CPUQuota=50% ./tarefa-pesada.sh

61. See a process's cgroup

Finds out which container/service it belongs to.

cat /proc/12345/cgroup

62. systemd-cgtop

Consumption per cgroup: it shows which service is eating the machine.

systemd-cgtop

63. capabilities

Granular privileges instead of the whole of root.

getcap /usr/bin/ping
sudo setcap cap_net_bind_service=+ep /usr/local/bin/minha-api
capsh --print

64. seccomp

Filters which syscalls the process may call — the basis of container isolation.

grep Seccomp /proc/12345/status
docker run --security-opt seccomp=perfil.json minha-app

65. chroot

Changes the root of the filesystem. Weak isolation, but useful in recovery.

sudo chroot /mnt/sistema /bin/bash

66. See a container's processes

From the host's side, a container is an ordinary process.

sudo systemd-cgls /system.slice/docker-*.scope
ps -eo pid,comm,cgroup | grep docker

67. Find the PID on the host

The bridge between the container and the host's tools.

docker inspect -f '{{.State.Pid}}' meu-container

68. Debug a container with no shell

Enter its namespaces with the host's tools.

pid=$(docker inspect -f '{{.State.Pid}}' minha-app)
sudo nsenter -t $pid -n ss -tulpn

69. overlayfs

The layered filesystem of container images.

mount | grep overlay
sudo mount -t overlay overlay -o lowerdir=/base,upperdir=/mudancas,workdir=/trabalho /mnt/final

70. PID limit

Prevents a fork bomb inside a service.

# systemd unit
[Service]
TasksMax=512

Firewall and advanced security

Reducing the attack surface and detecting when something got through.

71. nftables — rules

The successor to iptables. A single syntax for IPv4 and IPv6.

sudo nft list ruleset
sudo nft add table inet filtro
sudo nft add chain inet filtro entrada '{ type filter hook input priority 0; policy drop; }'

72. Open a port in nftables

The minimum rule set for a web server.

sudo nft add rule inet filtro entrada ct state established,related accept
sudo nft add rule inet filtro entrada iif lo accept
sudo nft add rule inet filtro entrada tcp dport { 22, 80, 443 } accept

73. iptables — classic rules

Still omnipresent on legacy systems.

sudo iptables -A INPUT -p tcp --dport 443 -j ACCEPT
sudo iptables -P INPUT DROP
sudo iptables-save > /etc/iptables/rules.v4

74. Rate-limit connections

Mitigates brute force and flooding on the SSH port.

sudo iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW \
  -m limit --limit 3/min --limit-burst 3 -j ACCEPT

75. conntrack

The table of tracked connections — it can overflow on a busy server.

sudo conntrack -L | wc -l
sysctl net.netfilter.nf_conntrack_max

76. Port knocking

The port only opens after a sequence of attempts.

sudo apt install knockd
# /etc/knockd.conf defines the sequence

77. SELinux — state

RHEL's mandatory access control.

getenforce
sudo setenforce 0        # permissive (temporary)
sestatus

78. SELinux — context

A file with the wrong context makes the service fail with no clear error.

ls -Z /var/www/html
sudo restorecon -Rv /var/www/html
sudo semanage fcontext -a -t httpd_sys_content_t "/dados(/.*)?"

79. SELinux — denials

Finds out what was blocked and why.

sudo ausearch -m avc -ts recent
sudo sealert -a /var/log/audit/audit.log

80. AppArmor

Ubuntu's equivalent, based on paths.

sudo aa-status
sudo aa-complain /etc/apparmor.d/usr.sbin.nginx
sudo aa-enforce /etc/apparmor.d/usr.sbin.nginx

81. auditd — kernel auditing

Records file access and command execution.

sudo auditctl -w /etc/passwd -p wa -k senhas
sudo ausearch -k senhas
sudo aureport --summary

82. Detect an unexpected SUID binary

The classic gateway for privilege escalation.

sudo find / -perm -4000 -type f 2>/dev/null | sort > suid-hoje.txt
diff suid-base.txt suid-hoje.txt

83. Check integrity with AIDE

Detects a change in a system file.

sudo aide --init
sudo aide --check

84. rkhunter and chkrootkit

A scan for known rootkits.

sudo rkhunter --update && sudo rkhunter --check
sudo chkrootkit

85. Suspicious connections

A process talking to a strange address.

sudo ss -tnp state established
sudo lsof -i -n -P | grep ESTABLISHED

86. A process with no binary on disk

A strong sign of in-memory malware.

sudo ls -l /proc/*/exe 2>/dev/null | grep deleted

87. Disk encryption (LUKS)

Encrypts the whole partition at rest.

sudo cryptsetup luksFormat /dev/sdb1
sudo cryptsetup open /dev/sdb1 dados
sudo mkfs.ext4 /dev/mapper/dados

88. Certificates with Let's Encrypt

Free TLS and automatic renewal.

sudo certbot --nginx -d exemplo.com -d www.exemplo.com
sudo certbot renew --dry-run

89. Check a certificate's validity

Avoids the outage from an expired certificate.

echo | openssl s_client -connect exemplo.com:443 2>/dev/null | openssl x509 -noout -enddate

90. Generate a key pair and a CSR

For an internal certificate or manual issuance.

openssl req -newkey rsa:4096 -nodes -keyout chave.pem -out pedido.csr

91. Harden the kernel's networking

Blocks spoofing and improper redirection.

# /etc/sysctl.d/99-hardening.conf
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.tcp_syncookies = 1

92. Disable an unnecessary service

What does not run cannot be exploited.

systemctl list-unit-files --state=enabled
sudo systemctl disable --now avahi-daemon

93. Hardened /tmp mount

Stops a binary being executed from a world-writable area.

# /etc/fstab
tmpfs /tmp tmpfs defaults,noexec,nosuid,nodev 0 0

94. Rotate and protect a secret

A credentials file needs restricted permissions and the right owner.

sudo chown root:app /etc/app/segredos.env
sudo chmod 640 /etc/app/segredos.env

95. Post-breach checklist

The order matters: preserve the evidence before cleaning up.

# 1. isolate the machine from the network (without shutting it down)
# 2. copy the logs and memory to another host
# 3. identify persistence: cron, systemd, authorized_keys, LD_PRELOAD
# 4. reinstall — a compromised machine is not fixed with a tidy-up

Advanced storage

RAID, LVM, filesystem tuning and what to do when the disk becomes the bottleneck.

96. Software RAID

mdadm assembles the array with no dedicated controller.

sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc
cat /proc/mdstat

97. RAID state

Following the sync and detecting a degraded disk.

sudo mdadm --detail /dev/md0
watch cat /proc/mdstat

98. Replace a disk in the RAID

Mark it as failed, remove it and add the new one.

sudo mdadm /dev/md0 --fail /dev/sdb --remove /dev/sdb
sudo mdadm /dev/md0 --add /dev/sdd

99. LVM: move data live

Migrates a volume to another disk without stopping the service.

sudo pvmove /dev/sdb1 /dev/sdc1

100. LVM: SSD cache

Uses an SSD as a cache for a volume on a spinning disk.

sudo lvcreate --type cache --cachevol cache-lv -L 100G vg/dados-lv

101. LVM: thin provisioning

Allocates on demand — take care not to overflow the pool.

sudo lvcreate -L 100G -T vg/pool
sudo lvcreate -V 500G -T vg/pool -n volume-fino

102. ext4 tuning

Reduces unnecessary writes and reserved space.

sudo tune2fs -m 1 /dev/sdb1        # reserves 1% instead of 5%
sudo tune2fs -l /dev/sdb1 | head -20

103. XFS: grow and repair

XFS only grows (it never shrinks) and repairs with its own tool.

sudo xfs_growfs /dados
sudo xfs_repair /dev/sdb1   # unmounted partition

104. Btrfs: subvolume and snapshot

An instant, cheap snapshot in the filesystem itself.

sudo btrfs subvolume create /dados/sub
sudo btrfs subvolume snapshot /dados/sub /dados/snap-$(date +%F)

105. ZFS: pool and dataset

Integrity by checksum, compression and native snapshots.

sudo zpool create dados mirror /dev/sdb /dev/sdc
sudo zfs create dados/postgres
sudo zfs set compression=lz4 dados/postgres

106. I/O scheduler

none/mq-deadline for NVMe; bfq for the desktop.

cat /sys/block/nvme0n1/queue/scheduler
echo none | sudo tee /sys/block/nvme0n1/queue/scheduler

107. TRIM on an SSD

Keeps the SSD's performance up over time.

sudo fstrim -av
sudo systemctl enable --now fstrim.timer

108. Align and measure latency

Latency per device, separated for reads and writes.

iostat -xz 1
sudo biolatency-bpfcc 10 1

109. The disk's write cache

Turning it off increases durability and reduces performance — a conscious decision on a database.

sudo hdparm -W /dev/sda
sudo hdparm -W0 /dev/sda

110. Duplicate a disk with dd

A bit-for-bit copy. status=progress shows how it is going.

sudo dd if=/dev/sda of=/dev/sdb bs=4M status=progress conv=fsync

111. Recover deleted data

Stop writing to the disk immediately and work on an image.

sudo ddrescue /dev/sdb imagem.img log.txt
sudo testdisk imagem.img
sudo photorec imagem.img

112. Erase securely

Overwrites to prevent recovery before discarding the media.

sudo shred -vfz -n 3 /dev/sdb
sudo blkdiscard /dev/nvme0n1   # SSD

113. Disk quotas

Limits the space per user or group.

sudo quotacheck -cug /home
sudo edquota -u maria
sudo repquota -a

114. Exhausted inodes

A disk with free space but no inodes: millions of small files.

df -i
find /var -xdev -type f | cut -d/ -f2-3 | sort | uniq -c | sort -nr | head

115. Emergency mount

Remounts read-write when the system goes into protected mode.

sudo mount -o remount,rw /

Advanced networking

Controlling traffic, simulating a problem and diagnosing what ping does not show.

116. Traffic control (tc)

Limits an interface's bandwidth.

sudo tc qdisc add dev eth0 root tbf rate 10mbit burst 32kbit latency 400ms
sudo tc qdisc show dev eth0

117. Simulate latency and loss

Tests how the application behaves on a bad network.

sudo tc qdisc add dev eth0 root netem delay 200ms loss 5%
sudo tc qdisc del dev eth0 root

118. Interface bonding

Aggregates links for redundancy or bandwidth.

sudo modprobe bonding
sudo ip link add bond0 type bond mode active-backup
sudo ip link set eth0 master bond0

119. VLAN

An interface tagged with 802.1Q.

sudo ip link add link eth0 name eth0.100 type vlan id 100
sudo ip link set eth0.100 up

120. Bridge

A virtual switch — the basis of container and VM networking.

sudo ip link add br0 type bridge
sudo ip link set eth0 master br0
bridge link show

121. NAT and masquerade

Shares the connection with an internal network.

sudo sysctl -w net.ipv4.ip_forward=1
sudo nft add rule ip nat postrouting oif eth0 masquerade

122. Policy routing

Chooses the route by the source, not just the destination.

sudo ip rule add from 192.168.2.0/24 table 100
sudo ip route add default via 10.0.0.1 table 100
ip rule show

123. Trace a packet through netfilter

Shows which rules the packet went through.

sudo nft add rule inet filtro entrada tcp dport 8080 meta nftrace set 1
sudo nft monitor trace

124. tcpdump with a fine filter

BPF at capture time cuts down noise and cost.

sudo tcpdump -i eth0 'tcp[tcpflags] & (tcp-syn) != 0 and not port 22' -n

125. Analyse a capture

Statistics without opening Wireshark.

tshark -r captura.pcap -q -z conv,tcp
tshark -r captura.pcap -Y "http.response.code >= 500"

126. TCP retransmissions

A clear sign of packet loss along the path.

netstat -s | grep -i retrans
ss -ti | grep -i retrans

127. Detailed connection state

RTT, window and congestion per socket.

ss -tin

128. A full listen queue

Connections refused even with the service up.

ss -ltn      # Send-Q is the configured backlog
nstat -az TcpExtListenOverflows

129. MTU and fragmentation

A wrong MTU causes an intermittent hang that is hard to diagnose.

ip link show eth0 | grep mtu
ping -M do -s 1472 destino   # tests MTU 1500

130. Find the real path

mtr combines ping and traceroute continuously.

mtr -rw -c 100 exemplo.com

131. DNS: debug the resolution

Follows the chain from the root servers.

dig +trace exemplo.com
dig +short TXT exemplo.com

132. Load balancing with nftables

Distributes connections between backends with no dedicated proxy.

sudo nft add rule ip nat prerouting tcp dport 80 dnat to numgen inc mod 2 map { 0 : 10.0.0.1, 1 : 10.0.0.2 }

133. WireGuard

A modern VPN: simple, fast and in the kernel.

wg genkey | tee chave-privada | wg pubkey > chave-publica
sudo wg-quick up wg0
sudo wg show

134. Multicast and IGMP

Diagnosing an application that uses multicast.

ip maddr show
netstat -g

135. NIC offloads

Turning offload off helps in diagnosing a strange capture.

ethtool -k eth0
sudo ethtool -K eth0 gro off tso off
ethtool -S eth0 | grep -i drop

Process debugging

When the log is not enough and you need to see what the process is really doing.

136. strace — syscalls

Shows each call to the kernel. The command that answers "where did it get stuck?".

sudo strace -p 12345
strace -f ./programa

137. Filtered strace

Only the calls that matter, with less noise.

sudo strace -p 12345 -e trace=openat,read,write
sudo strace -p 12345 -e trace=network

138. strace with timing

Finds out which call is taking the time.

sudo strace -p 12345 -T -tt
sudo strace -c -p 12345   # summary per syscall

139. A file the process cannot find

The classic: finding the ENOENT in strace.

strace -f -e trace=openat ./programa 2>&1 | grep ENOENT

140. ltrace — library calls

One level above strace.

ltrace ./programa

141. gdb on a live process

Attaches to the process and inspects the stack.

sudo gdb -p 12345
(gdb) bt
(gdb) thread apply all bt
(gdb) detach

142. Core dump

Enables and locates the memory dump of the process that crashed.

ulimit -c unlimited
cat /proc/sys/kernel/core_pattern
coredumpctl list
coredumpctl gdb 12345

143. Kernel stack

Where the process is stuck inside the kernel (state D).

sudo cat /proc/12345/stack
sudo cat /proc/12345/wchan

144. Everything about a process

The /proc/PID directory answers nearly any question.

ls -l /proc/12345/cwd /proc/12345/exe
cat /proc/12345/environ | tr '\0' '\n'
cat /proc/12345/limits

145. Files opened by a process

Finds the socket, log and library in use.

sudo lsof -p 12345
sudo ls -l /proc/12345/fd

146. Descriptor leak

A counter that only grows indicates an fd that is never closed.

watch -n 5 "ls /proc/12345/fd | wc -l"

147. Memory leak

RSS growing without stopping; valgrind confirms the origin.

while true; do ps -o rss= -p 12345; sleep 60; done
valgrind --leak-check=full ./programa

148. A process's threads

Which thread is consuming CPU.

top -H -p 12345
ps -T -p 12345

149. Signals

Sending a specific signal — many daemons reload on SIGHUP.

kill -l
kill -HUP 12345
kill -USR1 12345

150. Who killed my process

Auditing answers when the service dies with no explanation.

sudo auditctl -a always,exit -F arch=b64 -S kill -k mata
sudo ausearch -k mata

151. eBPF: trace execution

Every process started on the system, with its arguments.

sudo execsnoop-bpfcc
sudo exitsnoop-bpfcc

152. eBPF: opened files

Finds file access in real time.

sudo opensnoop-bpfcc -p 12345

153. eBPF: TCP connections

Every established connection, with the process and destination.

sudo tcpconnect-bpfcc
sudo tcpretrans-bpfcc

154. bpftrace — one-liners

Its own language for instrumenting the kernel on the spot.

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'

155. Function latency

A histogram of a syscall's duration.

sudo bpftrace -e 'kprobe:vfs_read { @inicio[tid] = nsecs; } kretprobe:vfs_read /@inicio[tid]/ { @us = hist((nsecs - @inicio[tid]) / 1000); delete(@inicio[tid]); }'

156. Debug a service that will not start

Run the ExecStart by hand, with the same user and environment.

systemctl cat minha-api
sudo -u app env $(cat /etc/minha-api/env | xargs) /usr/bin/node /opt/minha-api/server.js

157. Reproduce with a clean environment

Rules out an environment variable as the cause.

env -i /bin/bash --noprofile --norc

158. Dynamic libraries

A missing dependency is a common cause of "it will not run".

ldd /usr/local/bin/programa
LD_DEBUG=libs ./programa 2>&1 | head

159. Find which package a file belongs to

Traces the origin of a system binary.

dpkg -S /usr/bin/curl
rpm -qf /usr/bin/curl

160. Compare two environments

When it works on one server and not on another, the difference is in one of these.

diff <(ssh srv1 'env | sort') <(ssh srv2 'env | sort')
diff <(ssh srv1 'dpkg -l | awk "{print \$2,\$3}"') <(ssh srv2 'dpkg -l | awk "{print \$2,\$3}"')

Boot, initramfs and recovery

What to do when the machine will not come up — and all you have is the console.

161. The boot sequence

Understanding the order is half the diagnosis.

# BIOS/UEFI -> bootloader (GRUB) -> kernel -> initramfs -> systemd -> default target

162. Edit the boot in GRUB

e in the menu edits the kernel line for this startup.

# add at the end of the "linux" line:
systemd.unit=rescue.target
# or, in an extreme case:
init=/bin/bash

163. Rescue and emergency mode

They bring up the minimum needed to fix the system.

sudo systemctl isolate rescue.target
sudo systemctl isolate emergency.target

164. Reinstall GRUB

Fixing the bootloader from a live CD.

sudo mount /dev/sda2 /mnt
sudo mount /dev/sda1 /mnt/boot/efi
for d in dev proc sys run; do sudo mount --bind /$d /mnt/$d; done
sudo chroot /mnt grub-install /dev/sda
sudo chroot /mnt update-grub

165. Regenerate the initramfs

Necessary after changing a disk or encryption driver.

sudo update-initramfs -u -k all      # Debian/Ubuntu
sudo dracut -f --regenerate-all      # RHEL/Fedora

166. Inspect the initramfs

Checks whether the necessary module is in there.

lsinitramfs /boot/initrd.img-$(uname -r) | grep -i nvme

167. Stuck boot: where it stopped

The previous boot's log shows the last step.

journalctl -b -1 -p err
journalctl -b -1 | tail -50

168. Disable a problematic service at boot

Stops a broken service from hanging the startup.

sudo systemctl mask servico-problema
# or, in GRUB: systemd.mask=servico-problema.service

169. Reset the root password

Through GRUB, with the system mounted for writing.

# on the kernel line: rw init=/bin/bash
mount -o remount,rw /
passwd root
exec /sbin/init

170. fsck at boot

Forces a filesystem check at the next startup.

sudo touch /forcefsck
# or on the kernel line: fsck.mode=force

171. System mounted read-only

The kernel protects the disk when it detects an error. Check before remounting.

dmesg -T | grep -i "read-only\|ext4-fs error"
sudo mount -o remount,rw /

172. Space in /boot

Accumulated old kernels prevent an update.

df -h /boot
sudo apt autoremove --purge
dpkg -l | grep linux-image

173. UEFI and boot entries

Manages the startup order through the firmware.

efibootmgr -v
sudo efibootmgr -o 0002,0001

174. Snapshot before updating

With LVM or Btrfs, you can go back in minutes.

sudo lvcreate -L 10G -s -n antes-upgrade /dev/vg/root
# rollback: lvconvert --merge /dev/vg/antes-upgrade

175. Serial console

Access for when SSH and video do not respond.

# on the kernel line:
console=ttyS0,115200n8
sudo systemctl enable serial-getty@ttyS0.service

176. Kdump

Keeps the kernel dump after a panic for later analysis.

sudo systemctl status kdump
ls /var/crash/

177. Watchdog

Restarts the machine automatically if the kernel hangs.

cat /proc/sys/kernel/watchdog
sudo systemctl status watchdog

178. Deliberate read-only mode

An immutable root filesystem on an appliance or at the edge.

# /etc/fstab
/dev/sda2  /  ext4  ro,errors=remount-ro  0 1

179. Recover a container that will not start

Runs another command to inspect the filesystem.

docker run -it --entrypoint sh minha-app:1.0
docker create --name tmp minha-app:1.0 && docker cp tmp:/app ./app

180. Checklist before restarting production

A reboot fixes it — and also hides the cause. Collect the evidence first.

# 1. save dmesg, journalctl -b and ps aux
# 2. confirm the service starts at boot (systemctl is-enabled)
# 3. check fstab (mount -a) so it does not hang on the way back
# 4. warn whoever depends on the system

Automation and operations

Do it once, repeat it the same way every time — and leave a record.

181. Idempotent script

Running it twice has to give the same result. It is the principle behind all reliable automation.

# bad: echo "linha" >> /etc/hosts
# good:
grep -qxF "10.0.0.1 api" /etc/hosts || echo "10.0.0.1 api" | sudo tee -a /etc/hosts

182. Ansible: a one-off command

Runs on many machines at once, with no agent.

ansible todos -i inventario -m ping
ansible web -i inventario -a "systemctl status nginx" --become

183. Ansible: playbook

The desired state described in YAML.

- hosts: web
  become: true
  tasks:
    - name: nginx instalado
      apt: name=nginx state=present update_cache=yes
    - name: nginx ativo
      service: name=nginx state=started enabled=true

184. Ansible: simulate

--check shows what would change without applying it.

ansible-playbook site.yml --check --diff

185. Run on several machines

A simple loop when there is no configuration tool.

for host in web1 web2 web3; do
  ssh "$host" "sudo systemctl restart minha-api" || echo "falhou em $host" >&2
done

186. Parallelise with xargs

Much faster across dozens of servers.

cat hosts.txt | xargs -P 10 -I{} ssh {} "uptime"

187. tmux — a session that survives

The SSH connection drops and the process carries on. Indispensable during long maintenance.

tmux new -s deploy
# Ctrl+B D detaches
tmux attach -t deploy
tmux ls

188. screen

The classic alternative to tmux.

screen -S manutencao
# Ctrl+A D detaches
screen -r manutencao

189. Record the session

script records everything that happened in the terminal — great for a post-mortem.

script -a incidente-$(date +%F).log

190. Send an alert

Notifies you when the routine fails, instead of finding out later.

curl -s -X POST -H "Content-Type: application/json" \
  -d '{"text":"Backup falhou em '"$(hostname)"'"}' "$WEBHOOK_URL"

191. A health check in a script

Exits with a non-zero code so the monitoring understands.

if ! curl -sf http://localhost:8080/health > /dev/null; then
  logger -p user.err -t saude "API fora do ar"
  exit 1
fi

192. Retry with backoff

Networks fail; a good script tries again before giving up.

for tentativa in 1 2 3 4 5; do
  curl -sf "$URL" && break
  sleep $((2 ** tentativa))
done

193. Backup with verification

A backup that is not tested does not count.

tar -czf backup.tar.gz /dados && tar -tzf backup.tar.gz > /dev/null \
  && echo "backup íntegro" || echo "backup corrompido" >&2

194. Backup rotation

Keeps the N most recent and deletes the rest.

ls -1t /backup/*.tar.gz | tail -n +8 | xargs -r rm --

195. Sync to object storage

An external copy — the part of 3-2-1 that saves you from ransomware.

rclone sync /backup remoto:bucket/backup --progress
aws s3 sync /backup s3://meu-bucket/backup

196. Document the server

An automatic inventory is worth more than an out-of-date wiki.

{
  echo "== $(hostname) =="
  uname -a; lsb_release -d
  systemctl list-unit-files --state=enabled
  ss -tulpn
} > inventario-$(hostname).txt

197. Compare configuration between servers

Finding the difference that explains the different behaviour.

for h in web1 web2; do ssh $h "md5sum /etc/nginx/nginx.conf"; done

198. Version /etc

A history of everything that changed in the server's configuration.

cd /etc && sudo git init && sudo git add -A && sudo git commit -m "estado inicial"
# or use etckeeper

199. Maintenance window

Silence the alerts, warn people and bound the time of the operation.

# 1. silence the monitoring
# 2. warn the stakeholders in advance
# 3. have a written and tested rollback
# 4. set the abort time ("if it is not up by 4 a.m., we roll back")

200. A blameless post-mortem

The aim is to fix the system, not to find someone responsible. Record the timeline, cause, impact and actions.

# Template: what happened | when we detected it | how we mitigated it
# | root cause | why we did not catch it earlier | what changes from now on