High-performance Linux work begins with a question you can measure: which resource limits this workload, under these conditions? Adding cores or changing a kernel setting may do little when the application is waiting for memory, storage, synchronization, or another service.
This guide focuses on compute-heavy work on one Linux system, including scientific processing and data analysis. The method also helps with servers, but benchmark results must be interpreted in the context of the actual application and operating environment.
Define success and preserve the workload
Choose a measure that represents useful work: elapsed time for a fixed simulation, completed jobs per hour, or response latency under a stated load. Keep the input, output checks, application version, compiler options, and numerical precision fixed when comparing a change. A faster run that produces the wrong answer is a failed result.
Record the CPU model, topology, memory, storage, distribution, kernel, power conditions, and worker count. Note whether the machine is a virtual machine or shares resources with other jobs. Thermal throttling, background work, quotas, and different cache states can change a result without any code improvement.
Repeat trials and report the spread as well as a representative value. Separate startup from steady-state work when both matter. Do not compare a cold read from disk with a later cached read and attribute the whole difference to a tuning parameter.
Use several observations to narrow the cause
On a small screen, scroll the table sideways to read all columns.
| Observation | Possible explanation | Check before acting |
|---|---|---|
| One busy thread while other CPUs are idle | Serial work, a lock, or a single-threaded phase. | Which thread is busy, where time is spent, and whether parallelism is supported. |
| Many active CPUs but little scaling | Memory bandwidth, synchronization, oversubscription, or other shared limits. | Per-thread behavior, topology, memory access, and nested worker pools. |
| Memory pressure and repeated paging | The working set may exceed available memory or a container limit. | Application allocation, cgroup limits, and whether the pressure coincides with the slow phase. |
| High storage latency or a growing queue | The application may be waiting on storage, or another workload may be competing. | Read/write pattern, throughput, latency, device type, and the application’s own timing. |
On a Debian system with the relevant tools installed, these observation commands provide a starting point. lscpu comes from util-linux, vmstat from procps, and iostat from sysstat. Output fields vary by version and environment.
lscpu
vmstat 1 5
iostat -xz -y 1 5
cat /proc/pressure/cpu
cat /proc/pressure/memory
cat /proc/pressure/ioThe first vmstat report includes averages since boot; interpret the later interval reports for the current run. In iostat, -y suppresses its initial since-boot report. Device utilization and CPU I/O-wait values are clues, not universal saturation tests, particularly with modern parallel storage.
Linux pressure-stall information reports time that tasks are stalled by resource contention. Where supported and enabled, the /proc/pressure/ files add information that a utilization percentage alone cannot supply. Pressure is still an observation; it does not identify the exact line of application code that caused it.
Check memory locality and worker placement
On a NUMA system, memory access cost depends partly on where memory is located relative to the CPU doing the work. The kernel’s NUMA memory-policy documentation describes allocation policies and their scope. A large shared-memory workload can behave differently when its threads and data are spread across nodes.
Inspect topology and application allocation behavior before forcing affinity or memory placement. Pinning can help one workload and harm another, especially if it prevents the scheduler from balancing work or conflicts with a container’s allowed CPUs. Record the placement policy in benchmark results so that another person can reproduce the test.
Check for nested parallelism. An application can start several processes, each of which starts a threaded math library. The resulting runnable threads may greatly exceed the intended worker count. Adjust the application and library settings together, then repeat the same workload.
Worked example: faster completion versus efficient resource use
Invented results: the same fixed job produces an identical checked output in every run. The table shows the median of three trials per worker setting; workers are comparable execution slots, not a claim about any particular processor.
On a small screen, scroll the table sideways to read all columns.
| Workers | Median elapsed time | Speedup over one worker | Parallel efficiency |
|---|---|---|---|
| 1 | 40 seconds | 1.00× | 100.0% |
| 2 | 24 seconds | 1.67× | 83.3% |
| 4 | 14 seconds | 2.86× | 71.4% |
| 8 | 13 seconds | 3.08× | 38.5% |
Speedup is the one-worker time divided by the new time. Efficiency is speedup divided by worker count. At four workers: 40 ÷ 14 = 2.86×, and 2.86 ÷ 4 is approximately 71.4%. Calculations use unrounded values before display.
Moving from four workers to eight cuts elapsed time by only (14 − 13) ÷ 14 = 7.1%. Allocated worker-seconds rise from 4 × 14 = 56 to 8 × 13 = 104. Those are allocation-time units, not measured CPU time or energy. Eight workers may suit a deadline; four may suit a busy shared machine. The correct choice depends on the objective.
Download the experiment record and fictional trial data (plain text). It includes a place to record correctness checks, environmental changes, and the explanation being tested.
Change one thing and verify the explanation
Form a hypothesis from the observations, make a bounded change, repeat the workload, and retain the before-and-after configuration. Prefer a change that addresses the measured cause: a better algorithm, reduced allocation, batched I/O, or corrected worker settings may matter more than a global operating-system tweak.
For deeper profiling, use an appropriate profiler with authorized access. The kernel’s perf security documentation explains why performance-monitoring access is restricted. Do not disable system-wide protections merely to obtain a profile. Arrange the required access for the specific investigation.
Record regressions in memory use, latency, correctness, and other workloads. A benchmark win is useful when it survives representative inputs and repeated trials. Keep the change only when the evidence supports the explanation and the tradeoff fits the system’s purpose.