Linux NVMe-oF Initiator and StarWind NVMe-oF Initiator for Windows: Performance Comparison Part 2.1

Introduction

As you already know, NVMe-oF (Non-Volatile Memory Express over Fabrics) is a groundbreaking technology that extends the advantages of NVMe across a network. In short, NVMe-oF allows direct access to storage media over high-speed networks, capitalizing on the speedy and large-capacity architecture of NVMe.

What does it mean? NVMe-oF significantly ramps up data transfer speeds, cuts down latency, and enhances overall storage efficiency. Basically, this technology takes storage performance to new levels and potentially changes the approach to building storage solutions in its entirety.

Purpose

In our previous material, we already delved into comparing the storage performance of Linux NVMe-oF initiator for Linux and StarWind NVMe-oF Initiator for Windows over RDMA and TCP.
That setup is as close to an actual business environment as it gets. Results have shown that the performance of each initiator was on par with the other, whether you use RDMA or TCP.

Now, we’re going to try it again with the same setup but different NVMe-oF implementations. Our purpose here is to compare the performance of the Linux NVMe-oF Initiator with the StarWind NVMe-oF Initiator for Windows (versions 1.9.0.567 and 1.9.0.578) over RDMA and TCP. Let’s see how they fare this time!

Benchmarking Methodology, Details & Results

The benchmark was conducted with the help of the fio utility in Linux NVMe-oF Initiator scenarios and diskspd in StarWind NVMe-oF Initiator for Windows scenarios.

Testbed:

Hardware:

Software:

Hardware:

Software linux-cli:

Software windows-cli:

Test Patterns:

• random read 4k;
• random write 4k;
• random read 64K;
• random write 64K;
• sequential read 1M;
• sequential write 1M.

Linux – fio parameter examples

#Parameters “numjobs” and “iodepth” can change.

 Windows – diskspd parameter examples:

#Parameters “-t (threads)” and “-o (number of outstanding I/O requests)” can change.

Performance Comparison: Linux and Windows Storage Stack:

In order to have a clearer understanding of this setup, it would be best if we compare the local performance of 1 NVMe for Linux and Windows.

As you can see for yourself, based on random read/write 4k patterns, to achieve comparable performance in Windows, you need to double the ‘numjobs’ parameter. This will result in a latency increase of 100-150% and a CPU usage increase of 425-470%.

Similar results are seen in the random read 64K pattern. The performance is virtually the same for both operating systems in the random write 64K, read 1M, and write 1M patterns. In other words, the difference in the results acquired during the benchmarking of the two NVMe-oF initiators described below is no mystery when examining the difference in Linux and Windows storage stack performance outlined above.

Benchmarking Results:

Performance Comparison: Linux NVMe-oF Initiator and StarWind NVMe-oF Initiator for Windows (v1.9.0.567 and v1.9.0.578) over RDMA:

The optimizations in StarWind NVMe-oF Initiator for Windows v1.9.0.578 have played a vital role. Thanks to them, we managed to gain an 11% performance increase in the random read 4k pattern and 4% in the random write 4k pattern (compared to v1.9.0.567). As a result, the thread count (‘numjobs’) has been reduced from 24 in v1.9.0.567 to 16 or 18 in v1.9.0.578, leading to decreased latency and CPU usage values.

When compared with Linux NVMe-oF Initiator, StarWind NVMe-oF Initiator for Windows only shows a difference in the random read 4k pattern (14% lower performance). The results are largely comparable in all other patterns. Higher latency and CPU usage values are explained in the section “Performance Comparison: Linux and Windows Storage Stack.”

Performance Comparison: Linux NVMe-oF Initiator and StarWind NVMe-oF Initiator for Windows (v1.9.0.567 and v1.9.0.578) over TCP:

Conclusion

Thanks to the adjustments, StarWind NVMe-oF Initiator for Windows v1.9.0.578 could increase its performance by 24% in the random read 4k pattern, by 12% in the random write 4k pattern, and by 4% in the random read 64K pattern. Not bad at all! Compared with Linux NVMe-oF Initiator, StarWind NVMe-oF Initiator for Windows v1.9.0.578 only gives way in the random read 4k pattern (6%). As you can already tell, the performance is comparable or higher in all other patterns. I hope this information proves useful and assists you in the future!

This material has been prepared in collaboration with Viktor Kushnir, Technical Writer with almost 4 years of experience at StarWind.

Related materials:

• Linux NVMe-oF Initiator and StarWind NVMe-oF Initiator: Performance Comparison Part 1
• Linux NVMe-oF Initiator and StarWind NVMe-oF Initiator: Performance Comparison Part 2
• Real-World Production Scenario: Squeezing all possible performance out of Microsoft SQL Server AGs (Availability Groups). Local NVMe Vs. RDMA-mapped (NVMe-oF) remote NVMe storage. Does the juice worth a squeeze?