StarWind NVMe over Fabrics (NVMe-oF) Initiator

Introduction to StarWind NVMe-oF Initiator

NVMe over Fabrics (NVMe-oF) is a protocol that is set to be the standard for the shared NVMe storage.

This guide is intended for IT professionals and enthusiasts who would like to configure the StarWind NVMe over Fabrics Initiator solution and enable NVMe over Fabrics support. It provides detailed instructions on how to configure StarWind NVMe over Fabrics Initiator and connect shared storage on Windows Server.

For any technical inquiries, please, visit our online community, Frequently Asked Questions page, or use the support form to contact our technical support department.

VSAN from StarWind eliminates any need for physical shared storage just by mirroring internal flash and storage resources between hypervisor servers. Furthermore, the solution can be run on the off-the-shelf hardware. Such design allows VSAN from StarWind to not only achieve high performance and efficient hardware utilization but also reduce operational and capital expenses.
Find out more about ➡ VSAN from StarWind

Solution Diagram

Here is the network diagram for the configuration described in this guide.

StarWind NVMeOF Initiator

Preconfiguring the Servers

In this document, the first host with Mellanox ConnectX-5 adapter, SPDK NVMe over Fabrics target, and NVMe drive is running CentOS. The second host is running Windows Server 2019  and has Mellanox ConnectX-5 adapter installed accordingly. StarWind NVMe over Fabrics Initiator is deployed on the second Windows Server 2019 and connected to the first CentOS via 172.16.77.x subnet.

Windows Servers 2019 should have the latest Mellanox driver installed, which is available here:

https://www.mellanox.com/products/adapter-software/ethernet/windows/winof-2

Network Connections

To check the RDMA connectivity and bandwidth between the initiator server and target server, use the StarWind rPerf utility, which can be downloaded here: https://www.starwindsoftware.com/starwind-rperf

The article on how to install and configure SPDK NVMe over Fabrics target could be found here:  https://www.starwindsoftware.com/resource-library/starwind-nvme-of-initiator-creating-microsoft-failover-cluster-with-windows-server/

Installing StarWind NVMe over Fabrics Initiator

1. Download StarWind NVMeoF in thlink here:  https://www.starwindsoftware.com/starwind-nvme-of-initiator

2. Execute the starwind-nvmeof.exe  to install StarWind NVMe-oF Initiator and follow the steps in the wizard.

StarWInd EULA

3. Restart the server.

StarWind NVMeoF Initiator

4. Open Device Manager to check that StarWind NVMe-oF Initiator is installed on the system.

NVMe-oF Starwind

Working with StarWind NVMe over Fabrics Initiator

There are three ways to work with StarWind NVMe over Fabrics Initiator: using StarNVMeoF_Ctrl command-line utility (preferred), from Microsoft iSCSI initiator GUI and via PowerShell commandlets:

Using command-line utility

The StarNVMeoF_Ctrl.exe command-line utility is used to work with the NVMe-oF initiator.

Discovering targets

1. To discover the target, run the discovery command: StarNVMeoF_Ctrl.exe discovery <target_ip_addr:[port]> <local_ip_addr> by specifying the protocol, target host IP address and port number:

Example: StarNVMeoF_Ctrl.exe discovery 172.16.77.1:4420 172.16.77.2

Where:

<172.16.77.1:4420> — NVMe-oF target host IP and port;
<172.16.77.2> — initiator host IP.

NVMe-oF Starwind

Connecting targets

1. To connect the target, run the command: StarNVMeoF_Ctrl.exe insert <target_ip_addr[:port]> <local_ip_addr> <SubNQN> <HostNQN> [<num_io_queues> <io_queue_depth> <first_core>]

Example: StarNVMeoF_Ctrl.exe insert 172.16.77.1:4420 172.16.77.2 nqn.2008-08.com.starwindsoftware:NVMeofTarget SW_1 512 6 0

Where:

  • <172.16.77.1:4420 > — target host IP and port;
  • <172.16.77.2> — initiator host IP;
  • <nqn.2008-08.com.starwindsoftware:NVMeofTarget> — SubNQN of the target (may be copied from the discovery results screen);
  • <SW_1> — local HostNQN;
  • <512> — quantity of connections to the target;
  • <6> — queue depth;
  • <0> — number of the initial core.

NVMe-oF StarWind

2. After the “insert” command is executed, disk LUNs for the connected controller namespaces should appear in the Disk Management.

NVMe-oF StarWind

Getting information about targets

1. To show the list of connected NVMe-oF controllers, run the “StarNVMeoF_Ctrl.exe list” command.

NVMe-oF StarWind

Disconnecting targets

1. To disconnect LUNs from the system, run the controller disconnection command:  StarNVMeoF_Ctrl.exe remove <controllerId>
Example: StarNVMeoF_Ctrl.exe remove 1
Where:

  • <1> — controller Id

NVMe-oF StarWind

NOTE: Make sure that LUNs are not used by other applications at the moment of disconnection, as removing LUNs with active file operations may lead to data corruption.

Using MS iSCSI Initiator GUI

Discovering targets

1. Launch Microsoft iSCSI InitiatorStart -> Windows Administrative Tools -> iSCSI Initiator. Alternatively, launch it using the command below in the command line interface:

2. Navigate to the Discovery tab.
iscsi

3. In Discover Target Portal dialog, type in the IP address of the NVMe-oF target server that will be used to connect the provisioned targets and port number (4420).  Click Advanced.

StarWind NVMeoF initiator

4. Select StarWind NVMe over Fabrics Storage Controller as the Local adapter, select the Initiator IP in the same subnet as the IP address of the target server from the previous step. Confirm the actions to complete the Target Portal discovery.

StarWind NVMeoF initiator

5. Now, the target portal is added to the initiator server.

Connecting targets

1. Click the Targets tab. The previously created target is listed in the Discovered Targets section.

StarWind NVMeoF initiator

2. Press Connect and select StarWind NVMe over Fabrics Storage Controller in the Local adapter dropdown menu.
In the Initiator IP field select the IP address for the initiator.
In the Target portal IP, select the corresponding portal IP from the same subnet. Confirm the actions.
StarWind NVMeoF initiator

3. Target is connected

StarWind NVMeoF initiator

4. Open the Disk Management snap-in. The connected disk(s) will appear as unallocated and offline.

NVMe-oF StarWind

5. Bring the disks online by right-clicking on them and selecting the Online menu option.

Disconnecting targets

1. To disconnect the NVMe-oF target choose the specified target from the Discovered Targets list a press the Disconnect button.

NOTE: Make sure that LUNs are not used by other applications at the moment of disconnection, as removing LUNs with active file operations may lead to data corruption.

Using PowerShell

Discovering targets

1. To discover the target, run the discovery commandlet in Powershell: New-IscsiTargetPortal -TargetPortalAddress <target IP> -TargetPortalPortNumber <target port> -InitiatorPortalAddress <initiator IP> -InitiatorInstanceName <‘initiator controller‘> by specifying target host IP address and port number:

Example:

Where:

<172.16.77.1> -TargetPortalAddress;

<4420> -InitiatorPortalAddress

<172.16.77.2> -InitiatorPortalAddress

<‘ROOT\SCSIADAPTER\0000_0’> -InitiatorInstanceName – StarWind NVMe over Fabrics Storage Controller

StarWind Powershell

2. To get NVME targets name, run the commandlet Get-IscsiTarget:

StarWind Powershell

Connecting targets

1. To connect the target, run the commandlet: Connect-IscsiTarget -NodeAddress <NodeAddress>-IsPersistent $True

Example:

StarWind NVMeoF initiator Powershell

4. To get  information about the sessions, run the command:

StarWind NVMeoF initiator

Disconnecting targets

1. To disconnect the target, run the command: Disconnect-IscsiTarget -NodeAddress <NodeAddress>

Example:

StarWind NVMeoF initiator

NOTE: Make sure that LUNs are not used by other applications at the moment of disconnection, as removing LUNs with active file operations may lead to data corruption.

Save your time finding all the answers to your questions in one place!
Have a question? Doubt something? Or just want to know an independent opinion?
StarWind Forum exists for you to “trust and verify” any issue already discussed and solved