# Measure NVMe performance without misleading yourself

> Define the I/O pattern, control cache effects and inspect latency distributions before drawing an application conclusion from a disk test.

Canonical human page: https://swissoctet.com/insights/nvme-benchmark-methodology/

SwissOctet Editorial. Topic: storage. Level: Advanced. Reading time: 6 minutes.

## Direct answer
A useful storage test matches operation size, access pattern, concurrency and durability requirements to the application. Record the test conditions and latency distribution, then confirm the result with application measurements.

## Begin with a storage question, not a score
Decide what you need to know before running a benchmark. You may be investigating slow database commits, file-distribution throughput or a build process that creates many small files. Those workloads do not ask the disk to do the same thing. A large sequential copy can look impressive while saying little about the delay experienced by a service making small random reads.

NVMe was designed for non-volatile media and efficient parallel I/O, but its protocol name does not describe every part of a hosted storage path. Virtualisation, the file system, resource contention and the application still matter. Write a test hypothesis such as whether storage latency is slowing a particular operation, rather than trying to produce a number that proves the entire server is fast.

## Describe the workload in reproducible terms
Record the operation size, sequential or random pattern, read-to-write mix, number of jobs and queue depth. Include the dataset size and test duration. The fio documentation explains these workload controls and the available I/O engines. An intended depth may not be achieved by every engine or setup, so read the reported behaviour as well as the configuration you requested.

Keep a copy of the test description beside the result. Note the operating system, file system, mount options, service activity and available capacity. If you compare configurations, change one relevant condition at a time. A test with a different access pattern or concurrency cannot fairly be presented as if only the storage medium changed.

| Application pattern | Test dimension to describe | Result to examine |
| --- | --- | --- |
| Large file delivery | Sequential reads and concurrent clients | Sustained throughput and CPU use |
| Record lookup | Small random reads and representative concurrency | Latency distribution and completed operations |
| Durable transaction | Write and synchronisation requirements | Commit-related delay, not buffered write speed alone |
| Build or extraction | Mixed small-file operations | End-to-end job duration and I/O waits |

## Control caching without pretending it does not matter
A dataset that fits in memory may mostly measure cache behaviour after the first pass. That can be useful if the application normally benefits from the same cache, but it should not be labelled raw device performance. Choose deliberately whether the test examines a warm application path, a colder dataset or an I/O mode intended to reduce the operating system's buffering effect.

Direct-I/O support and alignment depend on the platform and engine. It also does not automatically express the application's durability requirements or eliminate every cache in the stack. State the chosen mode clearly, and avoid comparing a buffered test with an unbuffered test as if they represented the same work. Keep warm-up and measured intervals separate where that helps interpretation.

## Read latency tails alongside throughput
Average latency can conceal slow operations that affect real requests. Examine reported percentiles and the number of samples behind them. A high percentile describes the boundary below which a stated fraction of observed operations fell during that test; it is not a permanent service guarantee. Short or unrepresentative runs give limited evidence about busy production periods.

Higher queue depth can improve completed throughput while increasing waiting time. That trade-off may suit batch processing and harm an interactive service. Compare results at concurrency that resembles the application, and monitor CPU and I/O pressure at the same time. Stop before the test creates unnecessary disruption for another workload sharing the environment.

## Keep the test authorised and non-destructive
Use a dedicated test directory or a prepared disposable volume, with enough free space and an explicit cleanup plan. Never point a write benchmark at an unverified raw device or production dataset. Check the tool's target and write behaviour before starting, and begin with a limited run. Synthetic I/O can consume capacity and interfere with services even when it does not overwrite useful files.

For a production-like evaluation, agree a maintenance window and retain application monitoring. Use fabricated data when possible. Do not infer storage reliability from a speed test: integrity checks, backups and restoration are separate tasks. A device that completes a benchmark still needs a recovery strategy for deletion, software mistakes or loss of the machine.

## Close the loop with application evidence
Suppose a report service spends much of its time waiting for random reads. The team tests a representative read pattern, then repeats the actual report against comparable data. If the report improves while its other work remains stable, the result supports the storage hypothesis. If it does not, investigate query design, memory or another dependency instead of repeating broader synthetic tests.

Pulse NVMe and selected dedicated configurations list NVMe storage in the SwissOctet catalogue. Their labels are specifications, not published application benchmarks. Use a controlled method to evaluate the delivered environment and preserve the conditions with the findings. The decision becomes stronger when the storage measurement and the user-visible result answer the same question.

## Checklist
- Define the application question first.
- Record pattern, block size, concurrency and duration.
- State cache and I/O-mode assumptions.
- Inspect latency percentiles as well as throughput.
- Use a verified disposable target and limited initial load.
- Confirm the finding with an application test.

## Frequently asked questions
### Is a sequential copy enough to compare database storage?
Usually not. Database access may involve small random reads, writes and synchronisation. Model the relevant pattern and verify the actual application.

### Does direct I/O prove durable writes?
No. Buffering and durability are separate concerns. Match the synchronisation behaviour required by the application.

### Can I publish one test as a guaranteed plan speed?
No. Publish its conditions and scope. A measured result from one run is not a service-level or future performance guarantee.

## Primary sources
- [fio: Flexible I/O tester documentation](https://fio.readthedocs.io/en/latest/fio_doc.html)
- [NVM Express: Specification overview and FAQ](https://nvmexpress.org/faq_category/specifications/)
- [Linux kernel: Pressure Stall Information](https://docs.kernel.org/accounting/psi.html)

## Related infrastructure
- [Pulse NVMe](https://swissoctet.com/services/pulse/index.txt): Fast NVMe storage for applications where I/O is part of the performance equation.
- [Metal Hyper](https://swissoctet.com/services/metal-hyper/index.txt): High-capacity dedicated configurations for intensive applications and larger deployments.
- [Metal EPYC](https://swissoctet.com/services/metal-epyc/index.txt): AMD-powered dedicated hardware for parallel workloads and serious compute.

## Related guides
- [When NVMe makes a difference to your application](https://swissoctet.com/insights/nvme-storage-when-it-matters/index.txt): Distinguish capacity, latency and throughput so that a faster storage label answers a real application problem.
- [Snapshots and backups: choose the failure you need to survive](https://swissoctet.com/insights/snapshots-backups-consistency/index.txt): Compare recovery points, failure boundaries and application consistency instead of assuming a snapshot and an independent backup provide the same protection.
- [Object, file or block storage: choose the interface your application needs](https://swissoctet.com/insights/object-file-block-storage/index.txt): Distinguish storage interfaces, access semantics and operational responsibilities before turning a capacity requirement into an application architecture.