If you’ve ever built a virtualization cluster, you’ve probably watched this happen: high-performance NVMe SSDs sitting inside individual hosts while neighboring servers can’t touch that capacity. NVMe over Fabrics fixes that. It turns stranded local flash into a shared storage pool every host can reach.
What you’ll find below: how NVMe-oF actually works, how the three transports stack up, and why HA matters once you put it in production.
What is NVMe over Fabrics?
NVMe over Fabrics (NVMe-oF) – is a storage protocol that extends NVMe’s high-speed, low-latency access beyond a single host, allowing servers to connect to remote NVMe storage across network fabrics such as Ethernet, Fibre Channel, or InfiniBand. It enables shared, scalable storage while retaining performance close to locally attached NVMe drives.
NVMe was built for flash and direct PCIe-attached storage. It cuts the overhead of legacy disk protocols, including the SCSI command translation layer that treated SSDs like spinning disks, and it uses the parallelism of modern SSDs. NVMe over Fabrics takes that same efficient NVMe command set and stretches it across a network fabric, so the fast flash drives don’t have to live inside the server that uses them.
Hosts can then reach remote NVMe storage with lower latency and lower protocol overhead than most older stacks. NVMe-oF supports three transports: TCP, RDMA, and Fibre Channel, all listed by the Storage Networking Industry Association as standard NVMe-oF fabrics. The whole point is consolidation. Instead of trapping fast NVMe drives inside individual servers, you present NVMe-class storage to many hosts at once.
The evolution of NVMe-oF
NVMe-oF didn’t appear overnight. It evolved alongside the NVMe standard as organizations wanted the benefits of NVMe beyond locally attached storage.
- 2011 – NVMe 1.0 introduces a protocol optimized for PCIe-attached flash storage.
- 2016 – The first NVMe over Fabrics specification is published as a standalone standard, extending NVMe across network fabrics.
- 2018 – NVMe/TCP is ratified, making NVMe-oF accessible over standard Ethernet without specialized networking hardware.
- 2021 – NVMe 2.0 folds NVMe-oF into the Base Specification, with dedicated NVMe/RDMA, NVMe/TCP, and NVMe/FC transport specs.
By 2021, NVMe-oF had moved from a separate extension to part of the core specification. Today it’s a default choice for shared high-performance storage in virtualization, database, AI, and enterprise environments.
Why legacy storage protocols hit a wall
iSCSI and Fibre Channel have powered enterprise storage for decades, and they still work fine for many workloads, but they weren’t designed for NVMe-class performance over the network. The bigger problem is that they were designed long before NVMe SSDs, PCIe 4.0 and 5.0 devices, high-core-count CPUs, and 100-400 GbE networking became common.
As storage hardware got faster, the bottleneck stopped being the drives. The protocols themselves didn’t keep up, and they’ve been playing catch-up ever since. The bottleneck moved into command queue handling, CPU utilization, and the networking stack, and it’s still there. NVMe was built to remove those bottlenecks for locally attached flash: deep command queues, parallel I/O, minimal protocol overhead. NVMe-oF carries those advantages across the network.
iSCSI can’t fully exploit modern NVMe SSDs because it adds protocol translation and processing overhead. That’s the gap NVMe-oF was built to close.
That doesn’t make iSCSI obsolete. It’s still a practical, well-understood, and cost-effective choice for many workloads, and it won’t be going away anytime soon. But if your infrastructure is built around NVMe SSDs and you want NVMe-class performance over the network, NVMe-oF is the protocol built for exactly that.
How NVMe-oF works?
Here’s the basic shape. NVMe-oF keeps the familiar NVMe model and stretches it across the network. The host, called the initiator, issues NVMe commands just like it would to a local drive, and the transport carries those commands across the fabric.
On the other side, a target exposes remote NVMe namespaces through an NVMe subsystem. The host connects to them and uses them as ordinary block storage. It’s the same model, just with extra hops. A discovery service helps hosts find available subsystems, and multipathing gives you several paths to storage for redundancy and load balancing. The goal is to add as little latency as possible compared to local NVMe, and transports like RDMA push this furthest by moving data directly between memory on each side, bypassing most CPU and OS overhead.
Core components of an NVMe-oF architecture
Here’s what makes up an NVMe-oF deployment. A typical setup is built from a handful of pieces. The host, called the initiator, is the server that connects to remote NVMe storage. A good Windows example is the StarWind NVMe-oF Initiator. It talks to an NVMe-oF target, the storage endpoint that exposes NVMe resources as an NVMe subsystem. Within that subsystem, each namespace is the actual block device exposed through NVMe. Connecting the two ends is the transport, running over a network fabric of Ethernet, an RDMA-capable network, or a Fibre Channel SAN. A discovery service helps hosts find available subsystems, and multipathing provides redundant paths and failover options.
What you actually get from NVMe-oF
NVMe over Fabrics (NVMe-oF) introduces a range of compelling benefits that significantly enhance data storage and application performance. Some of the key advantages include:
Ultra-low latency
One of the primary benefits of NVMe-oF is its ability to provide remote storage access with minimal additional latency. Because it preserves the native NVMe command set, it introduces significantly less protocol overhead than many legacy storage stacks.
The actual latency depends on the transport. NVMe/RDMA delivers the lowest latency by enabling direct memory-to-memory data transfers, while NVMe/TCP offers an excellent balance between performance and deployment simplicity by running over standard Ethernet networks.
More parallel requests
With NVMe-oF, hosts can simultaneously send multiple data requests, allowing for concurrent data transfers and processing. The gap over legacy protocols here is dramatic. SCSI-based transports like iSCSI and Fibre Channel typically rely on a single command queue holding around 256 commands, while NVMe-oF supports up to 65,535 I/O queues, each able to hold up to 65,535 commands. That is what delivers true parallelism and a real boost in overall data throughput.
Efficiency
NVMe-oF lets you fully utilize the NVMe storage subsystem instead of throttling it through an older protocol. By removing the iSCSI or Fibre Channel translation layer, and the bottlenecks that come with it, NVMe-oF uses the underlying flash more efficiently and gets more performance out of every device. Adding capacity as you grow is easy too, but the real win is doing more with the hardware you already have.
Higher performance
The protocol’s low latency and high-speed capabilities result in enhanced overall system performance, making it ideal for data-intensive applications. Just remember that the gain depends on the transport: NVMe/TCP runs over ordinary Ethernet yet still beats iSCSI on latency and throughput, while NVMe/RDMA is a genuine powerhouse for data-intensive workloads thanks to direct memory access.
Smaller OS storage stack
By allowing compute and storage resources to scale independently, NVMe-oF helps organizations consolidate storage infrastructure while maintaining NVMe-class performance. NVMe-oF streamlines storage stack operations on the server side, reducing the length of the data retrieval and processing path. You can pool resources and allocate storage where it’s needed most. This typically improves utilization, simplifies capacity planning, and reduces storage silos.
Better storage array performance
NVMe-oF’s efficiency and speed improvements extend to storage arrays, ensuring that data storage systems perform at their peak.
Easier move to NVMe SSDs
Many organizations are replacing SAS and SATA storage with NVMe SSDs, but modernizing storage shouldn’t mean sacrificing the flexibility of shared infrastructure. NVMe-oF allows organizations to adopt NVMe while continuing to build highly available virtualization clusters, software-defined storage platforms, and HCI environments that depend on shared storage.
Implementation flexibility
NVMe-oF offers a variety of implementation options, accommodating different scenarios and use cases, from real-time analytics to cloud services. This adaptability ensures that the protocol can meet the specific needs of your organization, regardless of the application or workload.
Lower total cost of ownership
NVMe-oF can help reduce the total cost of ownership by optimizing hardware utilization, so the same equipment does more work. Measured in cost per IOPS rather than cost per gigabyte, an efficient NVMe-oF setup often comes out ahead, and simply moving from HDDs to NVMe SSDs improves that ratio further.
Headroom for growth
As the demand for high-performance storage continues to grow, NVMe-oF positions organizations to meet future data processing requirements without the need for a complete infrastructure overhaul.
NVMe over Fabrics empowers organizations with low-latency, high-speed connectivity, efficient data handling, and the ability to consolidate diverse applications on shared networked storage. It not only matches the performance of direct-attached storage but also outperforms traditional storage protocols, resulting in more efficient and parallel data processing and data center consolidation.
Common NVMe-oF use cases
It isn’t just for HPC anymore. As NVMe SSDs have become the default for enterprise storage, organizations use NVMe-oF anywhere they need to share that performance across multiple servers without dragging in extra storage overhead. If your applications depend on low latency, high IOPS, or large amounts of concurrent I/O, NVMe-oF is usually worth a serious look.
Some of the most common deployment scenarios:
- Virtualization and HCI – multiple hypervisor hosts need fast, highly available shared storage.
- Databases and analytics – low storage latency directly affects transaction throughput and query performance.
- AI and ML – GPUs and inference servers continuously read large datasets.
- HPC – scientific simulations and engineering applications generate extremely parallel storage workloads.
- Cloud and service provider platforms – storage performance has to scale across many tenants.
- Media and entertainment – video editing, rendering, and content creation depend on sustained throughput.
- Financial services – algorithmic trading and risk analysis rely on predictable low-latency storage access.
- Healthcare and life sciences – medical imaging, genomics, and research platforms process enormous datasets.
- Telecommunications – real-time analytics and network services need responsive shared storage.
- Oil and gas exploration – NVMe-oF speeds up seismic data processing and improves exploration and drilling decisions.
These environments look different, but they all share one thing: storage performance directly drives application performance. As NVMe SSDs get faster, preserving that performance across the network gets more important.
NVMe vs NVMe-oF: what’s the difference?
The names are similar enough that people mix them up. Here’s the difference. NVMe defines how a server communicates with locally attached NVMe storage. NVMe-oF extends that same protocol across a network.
| Area | NVMe | NVMe-oF |
|---|---|---|
| Access model | Local storage | Remote / shared storage |
| Connection | PCIe | TCP, RDMA, or Fibre Channel |
| Typical scope | Single server | Multiple hosts |
| Main value | Maximum local SSD performance | Shared NVMe-class storage |
| Best fit | Local high-performance storage | Virtualization, HCI, SAN, cloud, clustered workloads |
| HA potential | Depends on local server design | Can use redundant paths, replicas, and failover |
If your applications only need local storage inside a single server, standard NVMe is sufficient. As soon as multiple hosts need access to the same high-performance storage, NVMe-oF becomes the natural extension.
NVMe-oF transports
One of NVMe-oF’s biggest advantages is flexibility. The protocol supports multiple transports, so you can pick the one that matches your infrastructure and performance goals.
The three standardized transports are NVMe/TCP, NVMe/RDMA, and NVMe/FC. All carry native NVMe commands, but each makes different trade-offs between performance, deployment complexity, and infrastructure requirements.
NVMe-oF over RDMA
NVMe-oF over RDMA, which stands for Remote Direct Memory Access, is the high-performance option: minimal CPU overhead and very low latency. RDMA is built for efficient data transfer between hosts and storage, which makes it well suited for demanding workloads. It uses RDMA protocols such as RoCE and iWARP for Ethernet, plus InfiniBand.
Memory-to-memory transport is the key trick. RDMA moves data directly between two computers’ memory, bypassing the OS and CPU. The result is minimal overhead and microsecond-level access.
Implementing NVMe-oF with RDMA pushes storage network performance as far as it can go. Because RDMA transfers data directly between memory on each side, it keeps latency extremely low. Lower latency translates directly into higher performance for demanding workloads.
NVMe over Fibre Channel
NVMe over Fibre Channel (NVMe/FC) combines FC’s speed and reliability with NVMe’s performance. In practice, few organizations build a new FC fabric specifically for NVMe-oF; most reach for RDMA or TCP instead. But teams that already run an FC SAN can reuse it for NVMe/FC and keep the deterministic performance FC is known for. NVMe/FC maps NVMe, SCSI, and IBM’s proprietary FICON onto the same fabric to move data and commands between host and target storage.
Coexistence with legacy FC is one of its strengths. NVMe/FC can coexist with Gen 5, 6, or 7 FC in the same infrastructure, so you can avoid disruptive upgrades, and that’s a real win for shops that don’t want forklift upgrades.
Firmware-based upgrade means you can refresh existing FC switches through firmware, but you need HBAs that support 16 Gbps or 32 Gbps FC, and your targets have to be NVMe-oF capable.
Backward compatibility is where things get interesting. The FC protocol supports access to shared NVMe flash. There’s a performance hit from interpreting and translating encapsulated SCSI commands to NVMe, and the Fibre Channel Industry Association is working on standards for backward-compatible NVMe/FC. This lets a single NVMe/FC adapter support SCSI disks, traditional SSDs, and PCIe-connected NVMe flash cards.
NVMe over TCP/IP
NVMe over TCP/IP extends NVMe storage access over standard Ethernet networks. It’s a cost-effective option for organizations that want to use their existing Ethernet infrastructure for NVMe-oF.
You can run NVMe-oF across a standard Ethernet network, which removes the equipment and configuration headaches of specialized transports. No special NICs, no FC switches, just plain Ethernet.
TCP/IP is the default standard for network communications, so it’s an easy fit for data exchange across most networks.
The TCP binding defines how data between a host and a non-volatile memory subsystem gets encapsulated and delivered. It also outlines how queues, capsules, and data are mapped to support TCP communications between NVMe-oF hosts and controllers through IP networks.
Transport comparison: How to choose the right NVMe-oF transport
There isn’t a single “best” NVMe-oF transport. The table below shows the pros and cons of each:
| Transport | Best for | Strengths | Trade-offs |
|---|---|---|---|
| NVMe/TCP | SMB, ROBO, edge, HCI, virtualization | Standard Ethernet, easy adoption, good cost profile | Higher latency than RDMA |
| NVMe/RDMA | Analytics, HPC, latency-sensitive workloads | Very low latency, lower CPU overhead | Needs RDMA-capable NICs and tuned networking |
| NVMe/FC | Existing enterprise Fibre Channel SANs | Fits the FC operational model, strong SAN continuity | Requires FC infrastructure and skills |
Pick NVMe/TCP when standard Ethernet and an easy rollout matter most, NVMe/RDMA when the lowest latency and CPU overhead justify the extra hardware and tuning, and NVMe/FC when you’ve already invested in a Fibre Channel SAN.
NVMe-oF high availability: Why it’s needed?
NVMe-oF solves remote NVMe access, but mission-critical production storage needs more than speed. It needs high availability. The moment VMs and applications depend on shared storage, a storage device, controller, or path failure can take everything offline.
HA is essential for virtualization, HCI, ROBO, edge, SMB, and clustered workloads. It removes single points of failure through active-active access to data, synchronous replication between nodes, and automatic failover. A cluster node can go down without interrupting service or losing data.
This is where StarWind Virtual SAN (VSAN) fits in. VSAN pools flash and disks within a cluster into a highly available shared storage pool that every host can use, with active-active NVMe/TCP HA starting from just two nodes. It ships as a software-defined solution and as turnkey hardware appliances, so you can deliver NVMe-class performance and high availability together without running a complex SAN.
StarWind stands out as a pragmatic choice for implementing NVMe-oF storage infrastructure. First and foremost, StarWind is constantly investing in the development of NVMe-oF-ready solutions, helping to bring about its mass adoption. For example, the StarWind NVMe-oF Initiator is a battle-tested, production-ready NVMe-oF initiator for Microsoft Windows.
Implementation considerations for NVMe-oF
A few practical points are worth planning before you deploy.
Design the network first. Keep storage traffic on its own dedicated or segmented network away from general VM and application traffic. The transport shapes those network requirements: NVMe/RDMA needs lossless Ethernet with PFC, ECN, and DCB tuned, plus RDMA-capable NICs. NVMe/TCP runs comfortably on standard Ethernet.
Make sure your NICs or HBAs, switch firmware, and storage targets all support the NVMe-oF level you need. Enable multipathing for redundancy, and keep security and isolation in mind. From day one, monitor latency, queue depth, and path health so you’ll see issues before they bite.
Best practices for NVMe-oF deployment
A few principles keep NVMe-oF deployments healthy.
Start from your workload requirements, including latency, IOPS, throughput, and availability, and pick the transport based on your infrastructure rather than the marketing. Design HA in from day one, separate storage traffic from other traffic, and validate multipathing and failover before going to production.
Watch queue depth, latency, path health, and target status. Document your rollback and maintenance procedures before you need them – not during an outage.
Where NVMe-oF is heading?
NVMe/TCP is becoming the mainstream fabric because it runs on standard Ethernet, and NVMe-oF is increasingly the default for highly available shared storage. Demand keeps rising as AI and ML workloads and all-flash data centers grow, so you can scale performance without replacing your whole infrastructure. The roadmap is more of the same: more transports, more features. The practical change you’ll see in current deployments is NVMe-oF becoming boring infrastructure, and that’s a good thing.
Wrapping up
NVMe-oF extends the NVMe command set across TCP, RDMA, or Fibre Channel so multiple hosts can share NVMe-class storage with low latency. The main thing you’ll need to pick is the transport, and that pick is mostly driven by what you already have. If you run Ethernet, start with NVMe/TCP. If you need the lowest possible latency and can afford RDMA NICs and tuned switches, NVMe/RDMA. If you already operate an FC SAN, NVMe/FC drops in.
For production, treat HA as part of the design, not a bolt-on. Multipathing, replication, and active-active failover are what keep NVMe-oF deployments from being one cable-pull away from an outage.
FAQ
What is NVMe over Fabrics?
NVMe over Fabrics (NVMe-oF) is a protocol that extends the NVMe command set across a network fabric, so hosts can access remote, shared NVMe storage with low latency over TCP, RDMA, or Fibre Channel.
Is NVMe-oF the same as NVMe?
No. NVMe is for local, PCIe-attached storage inside a single server. NVMe-oF extends NVMe over a network so multiple hosts can share NVMe-class storage.
What is the difference between NVMe and NVMe-oF?
NVMe connects drives locally over PCIe. NVMe-oF connects hosts to remote storage over TCP, RDMA, or Fibre Channel, which adds shared access, scalability, and HA potential.
What are the main NVMe-oF transports?
The three standard transports are NVMe/TCP, NVMe/RDMA (RoCE, iWARP, InfiniBand), and NVMe over Fibre Channel (FC-NVMe).
Is NVMe/TCP slower than NVMe/RDMA?
Generally, yes. NVMe/RDMA offers the lowest latency and CPU overhead, while NVMe/TCP trades a little latency for running on standard Ethernet with no special hardware.
When should I use NVMe/TCP?
Choose NVMe/TCP when you want simple, cost-effective adoption on existing Ethernet. It works well for SMB, ROBO, edge, HCI, and virtualization.
When should I use NVMe/RDMA?
Choose NVMe/RDMA for latency-sensitive workloads such as analytics and HPC, where the lowest latency justifies RDMA-capable NICs and tuned networking.
What is NVMe over Fibre Channel?
FC-NVMe runs NVMe-oF over a Fibre Channel SAN, fitting organizations that already operate FC and want NVMe performance with their existing operational model.
Does NVMe-oF support high availability?
Yes. With redundant paths, replication, and multipathing, NVMe-oF can deliver active-active HA, which is essential for production virtualization and clustered workloads.
How does StarWind provide NVMe-oF HA?
StarWind Virtual SAN pools flash and disks into a highly available shared storage layer for NVMe-oF/TCP, with HA starting from two nodes, available as software or as appliances.
