The Complete Guide to Networking & Linux Internals — TCP/IP, DNS, iptables, systemd

[Application Data]
    ↓ Application Layer
[TCP Header | Application Data]
    ↓ Transport Layer
[IP Header | TCP Header | Application Data]
    ↓ Internet Layer
[Ethernet Header | IP Header | TCP Header | Application Data | Ethernet Trailer]
    ↓ Network Access Layer

# Capture packets and inspect layer headers
sudo tcpdump -i eth0 -vvv -c 10

Client                    Server
  |                         |
  |--- SYN (seq=x) ------->|   Step 1: Client sends SYN
  |                         |
  |<-- SYN-ACK (seq=y,      |   Step 2: Server responds with SYN-ACK
  |    ack=x+1) ------------|
  |                         |
  |--- ACK (ack=y+1) ----->|   Step 3: Client sends ACK
  |                         |
  |=== Connection Ready ====|

Client                    Server
  |                         |
  |--- FIN ----------------->|   Step 1: Client sends FIN
  |                         |
  |<-- ACK -----------------|   Step 2: Server sends ACK
  |                         |
  |<-- FIN -----------------|   Step 3: Server sends FIN
  |                         |
  |--- ACK ----------------->|   Step 4: Client sends ACK
  |                         |
  |== TIME_WAIT (2*MSL) ====|

# Check TCP connection states
ss -tan

# Count connections by state
ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn

# Check current TCP window sizes
ss -ti | grep -A 1 "ESTAB"

# Check TCP window scaling settings
sysctl net.ipv4.tcp_window_scaling

# Check current congestion control algorithm
sysctl net.ipv4.tcp_congestion_control

# List available algorithms
sysctl net.ipv4.tcp_available_congestion_control

# Switch to BBR (effective for high-bandwidth, high-latency networks)
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr

GET /api/users HTTP/1.1
Host: example.com
Connection: keep-alive
Accept: application/json

On a single TCP connection:

Stream 1: GET /index.html    ─────>  [HEADERS frame]  [DATA frame]
Stream 3: GET /style.css     ─────>  [HEADERS frame]  [DATA frame]
Stream 5: GET /script.js     ─────>  [HEADERS frame]  [DATA frame]

# HTTP/2 request with curl
curl -v --http2 https://example.com

# Inspect HTTP/2 frames with nghttp2
nghttp -v https://example.com

HTTP/1.1:  TCP + TLS (separate handshakes)
HTTP/2:    TCP + TLS (multiplexing, but TCP-level HOL blocking)
HTTP/3:    QUIC (UDP-based, independent flow control per stream)

# HTTP/3 request with curl
curl --http3 https://example.com

# Verify QUIC connection
curl -v --http3 https://cloudflare-quic.com

Browser
  |
  |-- 1. Check local cache
  |
  v
Local DNS Resolver (e.g., 192.168.1.1)
  |
  |-- 2. If not cached, start recursive query
  |
  v
Root DNS Server (.)
  |
  |-- 3. "The nameserver for .com is a.gtld-servers.net"
  |
  v
TLD DNS Server (.com)
  |
  |-- 4. "The nameserver for example.com is ns1.example.com"
  |
  v
Authoritative DNS Server (example.com)
  |
  |-- 5. "The IP for example.com is 93.184.216.34"
  |
  v
Local DNS Resolver (caches the result)
  |
  v
Browser (connects to IP address)

# Query A record
dig example.com A

# Query with a specific DNS server
dig @8.8.8.8 example.com

# Query MX record
dig example.com MX

# Query TXT record (check SPF)
dig example.com TXT

# Reverse DNS lookup
dig -x 93.184.216.34

# Full DNS trace
dig +trace example.com

# Short output
dig +short example.com

# DNSSEC validation
dig +dnssec example.com

# Check systemd-resolved cache statistics (Linux)
resolvectl statistics

# Flush DNS cache
sudo resolvectl flush-caches

# Inspect /etc/resolv.conf
cat /etc/resolv.conf

# Check DNS resolution order in nsswitch.conf
grep hosts /etc/nsswitch.conf

Client                                Server
  |                                     |
  |--- ClientHello ------------------>  |   Supported cipher suites, key share
  |    (+ key_share)                    |
  |                                     |
  |<-- ServerHello ------------------- |   Selected cipher suite, key share
  |    (+ key_share)                    |
  |<-- EncryptedExtensions ----------- |
  |<-- Certificate ------------------- |   Server certificate
  |<-- CertificateVerify ------------- |   Signature verification
  |<-- Finished ---------------------- |
  |                                     |
  |--- Finished --------------------->  |
  |                                     |
  |==== Encrypted Communication ======|

Root CA (self-signed)
  |
  |-- Intermediate CA (signed by Root CA)
        |
        |-- Server Certificate (signed by Intermediate CA)

# Inspect certificate chain
openssl s_client -connect example.com:443 -showcerts

# View certificate details
openssl x509 -in cert.pem -text -noout

# Check certificate expiration
echo | openssl s_client -connect example.com:443 2>/dev/null | \
  openssl x509 -noout -dates

# Verify TLS version and cipher suite
openssl s_client -connect example.com:443 -tls1_3

# Install certbot (Ubuntu)
sudo apt install certbot python3-certbot-nginx

# Issue certificate (Nginx)
sudo certbot --nginx -d example.com -d www.example.com

# Test certificate renewal
sudo certbot renew --dry-run

# Check auto-renewal timer
systemctl list-timers | grep certbot

# List open sockets
ss -tuln

# Show sockets with process info
ss -tulnp

# Socket statistics
ss -s

select/poll: O(n) - scans all fds to find ready ones
epoll:       O(1) - returns only the fds with events

How epoll works:
1. epoll_create() - create epoll instance
2. epoll_ctl()    - register/modify/delete fds to monitor
3. epoll_wait()   - wait for events (blocking or non-blocking)

                    PREROUTING
                        |
                  [Routing Decision]
                   /         \
              INPUT        FORWARD
                |              |
          [Local Process]   [Other Interface]
                |              |
             OUTPUT        POSTROUTING
                \            /
              POSTROUTING

# List current rules
sudo iptables -L -n -v

# Allow specific ports
sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 443 -j ACCEPT

# Block a specific IP
sudo iptables -A INPUT -s 10.0.0.100 -j DROP

# Rate-limit SSH (3 per minute)
sudo iptables -A INPUT -p tcp --dport 22 -m state --state NEW \
  -m recent --set --name SSH
sudo iptables -A INPUT -p tcp --dport 22 -m state --state NEW \
  -m recent --update --seconds 60 --hitcount 4 --name SSH -j DROP

# NAT configuration (masquerade)
sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# Save rules
sudo iptables-save > /etc/iptables/rules.v4

# List current nft rules
sudo nft list ruleset

# Create a table
sudo nft add table inet my_filter

# Create a chain
sudo nft add chain inet my_filter input \
  '{ type filter hook input priority 0; policy drop; }'

# Add rules
sudo nft add rule inet my_filter input tcp dport 80 accept
sudo nft add rule inet my_filter input tcp dport 443 accept

# Save rules
sudo nft list ruleset > /etc/nftables.conf

# Show routing table
ip route show

# Check default gateway
ip route | grep default

# Find route to a specific destination
ip route get 8.8.8.8

# Add a static route
sudo ip route add 10.10.0.0/24 via 192.168.1.1 dev eth0

# Policy-based routing
sudo ip rule add from 10.0.1.0/24 table 100
sudo ip route add default via 10.0.1.1 table 100

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application Server
After=network.target postgresql.service
Requires=postgresql.service
Documentation=https://example.com/docs

[Service]
Type=notify
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
Environment=NODE_ENV=production
EnvironmentFile=/etc/myapp/env
ExecStartPre=/opt/myapp/scripts/check-deps.sh
ExecStart=/usr/bin/node /opt/myapp/server.js
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
StartLimitBurst=3
StartLimitIntervalSec=60

# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/myapp /var/log/myapp
PrivateTmp=true

[Install]
WantedBy=multi-user.target

# Check service status
systemctl status myapp.service

# Start/stop/restart service
sudo systemctl start myapp.service
sudo systemctl stop myapp.service
sudo systemctl restart myapp.service

# Reload configuration without restarting (PID preserved)
sudo systemctl reload myapp.service

# Enable auto-start at boot
sudo systemctl enable myapp.service
sudo systemctl enable --now myapp.service  # Start immediately + auto-start

# Reload daemon after unit file changes
sudo systemctl daemon-reload

# List failed services
systemctl --failed

# Show service dependencies
systemctl list-dependencies myapp.service

# View logs for a specific service
journalctl -u myapp.service

# Follow logs in real-time
journalctl -u myapp.service -f

# Logs from the last hour
journalctl -u myapp.service --since "1 hour ago"

# Specific time range
journalctl --since "2026-04-12 10:00:00" --until "2026-04-12 12:00:00"

# Show only error level and above
journalctl -u myapp.service -p err

# Logs by boot
journalctl -b -1  # Previous boot
journalctl -b 0   # Current boot

# Check disk usage
journalctl --disk-usage

# Clean up old logs
sudo journalctl --vacuum-time=7d
sudo journalctl --vacuum-size=500M

# /etc/systemd/system/backup.timer
[Unit]
Description=Daily Backup Timer

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

# /etc/systemd/system/backup.service
[Unit]
Description=Daily Backup

[Service]
Type=oneshot
ExecStart=/opt/scripts/backup.sh

# Enable timer
sudo systemctl enable --now backup.timer

# List all timers
systemctl list-timers --all

# Check next timer execution
systemctl status backup.timer

Parent Process (PID 100)
    |
    |-- fork() --> Child Process (PID 101)
                       |
                       |-- exec("/bin/ls") --> ls program executes

# View process tree
pstree -p

# Detailed info for a specific process
cat /proc/PID/status

# File descriptors used by a process
ls -la /proc/PID/fd/

# Find zombie processes
ps aux | awk '$8 ~ /Z/'

# Orphan processes are automatically adopted by init (PID 1)
# Zombie processes require the parent to call wait() for cleanup

# Find the parent of zombie processes
ps -eo pid,ppid,stat,cmd | grep -w Z

# Verify cgroup v2
mount | grep cgroup2

# Check current cgroup
cat /proc/self/cgroup

# Create a CPU-limited cgroup (50%)
sudo mkdir /sys/fs/cgroup/my_limited_group
echo "50000 100000" | sudo tee /sys/fs/cgroup/my_limited_group/cpu.max

# Memory limit (512MB)
echo "536870912" | sudo tee /sys/fs/cgroup/my_limited_group/memory.max

# Assign a process to the cgroup
echo PID | sudo tee /sys/fs/cgroup/my_limited_group/cgroup.procs

# List namespaces
lsns

# Create a new network namespace
sudo ip netns add test_ns

# Run command inside namespace
sudo ip netns exec test_ns ip addr

# Create an isolated environment with unshare
sudo unshare --pid --fork --mount-proc bash

# List namespaces
ip netns list

User Process
    |
    |-- open(), read(), write() ...
    |
    v
VFS (Virtual File System)
    |
    ├── ext4
    ├── XFS
    ├── btrfs
    ├── NFS
    ├── procfs (/proc)
    └── sysfs (/sys)

# Check file inode number
ls -i file.txt

# Detailed inode information
stat file.txt

# Check filesystem inode usage
df -i

# Hard link vs symbolic link
# Hard link: shares the same inode
ln original.txt hardlink.txt

# Symbolic link: separate inode, points to a path
ln -s original.txt symlink.txt

# Check filesystem types
df -Th

# Filesystem details
sudo tune2fs -l /dev/sda1    # ext4
sudo xfs_info /dev/sda1      # XFS
sudo btrfs filesystem show    # btrfs

# CPU information
cat /proc/cpuinfo

# Memory information
cat /proc/meminfo

# Load average
cat /proc/loadavg

# Kernel version
cat /proc/version

# Network connection info
cat /proc/net/tcp

# Specific process information
cat /proc/PID/status    # Process status
cat /proc/PID/maps      # Memory map
cat /proc/PID/cmdline   # Command line

# Block device info
ls /sys/block/

# Network interface info
ls /sys/class/net/

# CPU frequency info
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq

# Disk scheduler
cat /sys/block/sda/queue/scheduler

# All TCP connections (including listening)
ss -tuln

# Only ESTABLISHED connections
ss -t state established

# Count connections to a specific port
ss -tn dport = :443 | wc -l

# Include process information
ss -tulnp

# Socket memory usage
ss -tm

# TIME_WAIT connections
ss -t state time-wait

# Capture on a specific interface
sudo tcpdump -i eth0

# Capture traffic to/from a specific host
sudo tcpdump -i eth0 host 10.0.0.1

# Capture on a specific port
sudo tcpdump -i eth0 port 80

# Capture DNS queries
sudo tcpdump -i eth0 port 53

# Save capture to file
sudo tcpdump -i eth0 -w capture.pcap

# Read saved file
tcpdump -r capture.pcap

# View HTTP request/response content
sudo tcpdump -i eth0 -A port 80

# Capture only SYN packets (new connection monitoring)
sudo tcpdump -i eth0 'tcp[tcpflags] & (tcp-syn) != 0'

# Trace system calls
strace -p PID

# Trace only network-related calls
strace -e trace=network -p PID

# Trace file-related calls
strace -e trace=file ls

# Include timing information
strace -T -p PID

# Follow child processes
strace -f -p PID

# Save output to file
strace -o output.txt -p PID

# Find process using a specific port
sudo lsof -i :80

# Files opened by a specific process
lsof -p PID

# Processes using a specific file
lsof /var/log/syslog

# Deleted but still open files (disk space not reclaimed)
lsof +L1

# Network connections
lsof -i -P -n

# Files opened by a specific user
lsof -u username

# CPU profiling (10 seconds)
sudo perf record -g -p PID -- sleep 10

# View results
sudo perf report

# Real-time top functions
sudo perf top -p PID

# System-wide statistics
sudo perf stat -a -- sleep 5

# Generate flame graph
sudo perf record -g -p PID -- sleep 30
sudo perf script > out.perf
# Convert with FlameGraph tools
./stackcollapse-perf.pl out.perf > out.folded
./flamegraph.pl out.folded > flamegraph.svg

# 1. Check interface status
ip addr show
ip link show

# 2. Verify routing
ip route show
traceroute TARGET_HOST

# 3. Check DNS
dig TARGET_DOMAIN
nslookup TARGET_DOMAIN

# 4. Test connectivity
ping TARGET_HOST
curl -v http://TARGET_HOST

# 5. Check ports
ss -tuln
sudo lsof -i :PORT

# 6. Inspect firewall
sudo iptables -L -n
sudo nft list ruleset

# 7. Capture packets
sudo tcpdump -i eth0 host TARGET_HOST

# 8. Check system resources
top
free -h
df -h

# 9. Review logs
journalctl -xe
dmesg | tail