Linux NVMe-oF Initiator and StarWind NVMe-oF Initiator: Performance Comparison Part 2

Introduction

As you know from the previous material, we have already determined whether the NVMe-oF technology can be effectively used to share fast storage over a network and how does it affect performance. Practically, we have established that both Linux NVMe-oF Initiator and StarWind NVMe-oF Initiator can operate almost without any performance decrease. However, although that does answer the question about the prospects of further use of NVMe-oF Initiator in real-life circumstances, like any other benchmarking it still does not cover all of the points and leaves a lot of unanswered questions.

Today we are going to find answers to some of those.

Purpose

We are going to continue benchmarking Linux NVMe-oF Initiator and StarWind NVMe-oF Initiator on a more profound level, making it even more interesting to system admins. This time, the comparison of their performances over both TCP and RDMA using Intel® Optane™ SSD DC P4800X will take us as close to the actual business IT infrastructure as it gets.

Benchmarking Details & Results

Testbed:

Hardware:

Software:

Hardware:

Software linux-cli:

Software windows-cli:

Benchmarking Results:

over TCP:

over RDMA:

RDMA vs TCP:

Conclusion

Overall, both Initiators have shown impressive results, as were expected. The performance of each Initiator is on par with another, no matter if you’re using RDMA or TCP. However, as it always is, there are a couple of nuances.

In general, the results aren’t that different from each other. If we’re talking about the basic SPDK (Storage Performance Development Kit) advantages such as user space usage and polling-based asynchronous I/O model, both RDMA and TCP are good to go. Thanks to StarWind NVMe-oF Initiator, you can use both with equal efficiency.

In particular, if your NICs support RoCE, then RDMA is an obviously better choice. Why? The answer is simple: latency. Even though in both cases you can rip the benefits for your storage, RDMA essentially moves your data with less latency than the alternative. That difference isn’t all that critical, but if you’re stuck with the applications that are more demanding than usual latency-wise, you’ll be better off with RDMA.

This material has been prepared in collaboration with Viktor Kushnir, Technical Writer with almost 3 years of experience at StarWind.