Deep-Dive Linux Kernel Performance Tuning: Optimizing High-Concurrency Network Stacks and NVMe I/O Queues

핵심 내용

1. Raise network buffer sizes. net.core.rmem_max and wmem_max control socket buffer ceilings; defaults are too small for high throughput.
2. Tune TCP congestion control. Use BBR for high-latency or lossy paths, and tune tcp_notsent_lowat for interactive traffic.
3. Increase connection tracking limits. nf_conntrack_max must scale with concurrent connections to avoid dropped packets.
4. Optimize NVMe queue depth. Set nr_requests and the scheduler to none or mq-deadline for NVMe devices.
5. Pin interrupts to cores. Spread NIC and NVMe interrupts across cores to avoid a single bottleneck.
6. Enable busy polling. net.core.busy_poll and busy_read reduce latency for high-frequency small packets.
7. Tune the scheduler. EEVDF replaces CFS in recent kernels; adjust sched_latency_ns for your workload.
8. Measure with perf. Use perf stat and bpftrace to confirm bottlenecks before and after changes.

Architecture

The network path runs from the NIC through the kernel stack into the application, and the storage path runs from the application through the block layer to NVMe. Interrupts, queues, and buffers at each layer determine throughput and latency. Tuning aligns these layers with the workload.

Production configuration

# /etc/sysctl.d/99-network.conf
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.ipv4.tcp_congestion_control = bbr
net.core.netdev_max_backlog = 5000
net.core.somaxconn = 65535
net.netfilter.nf_conntrack_max = 1048576

# set scheduler to none for NVMe
echo none > /sys/block/nvme0n1/queue/scheduler
# increase queue depth
echo 1024 > /sys/block/nvme0n1/queue/nr_requests
# disable readahead for random I/O
echo 0 > /sys/block/nvme0n1/queue/read_ahead_kb
# verify
nvme list
iostat -x 1 5

# spread NIC interrupts across cores
for irq in $(grep eth0 /proc/interrupts | awk '{print $1}' | tr -d ':'); do
  echo $((1 << (RANDOM % 8))) > /proc/irq/$irq/smp_affinity

done
# verify distribution
cat /proc/interrupts | grep eth0
# profile
perf stat -e 'net:net_dev_xmit,block:block_rq_issue' -a sleep 10

Edge cases from production

• BBR can be unfair to CUBIC flows on shared links; test in your network before deploying.
• Increasing buffers raises memory usage; size them to your connection count.
• NVMe queue depth above device limits is ignored; check the drive's maximum.
• Interrupt pinning must be redone after a NIC driver reload or reboot.

자주 묻는 질문