StarWind NVMe-oF Initiator:
Creating Microsoft failover cluster
with Windows Server

Prepare the new CentOS core with multipathing support

Preparation of the Storage Node involves multipath configuration. To configure it, it is necessary to update the OS core and install additional components. Perform the commands below as root on Storage Node.

To install the new CentOS core, it is necessary to run these commands. Find more details about CentOS core installation at https://wiki.centos.org/Documentation

To activate multipathing, it is necessary to find and delete the rows below in the .config file.

Use the make menuconfig tool to enable NVMe multipath support.

In Kernel Configuration, select Device Drivers -> NVME Support, choose NVMe multipath support  and press the Save button

Confirm the changes.

Compile the core. Use the lscpu command to learn the number of logical CPUs in the system; use that number as the parameter in the make -j command (make-j* where *- the count of logical CPU).

In this sample, Linux core was compiled with 8 CPUs.

Make the new system the first in the boot menu.

Restart the OS to boot from the new core.

Installing the necessary drivers and components and enabling RDMA

Run the following commands to install the Mellanox drivers and components.

Use the command below to check that NVMe multipathing support is enabled. The system should return “Y”.

Run the following commands to install the components for SPDK NVMe over Fabrics target.

 

Installing and starting the SPDK NVMe over Fabrics target

This section describes installing, initialization, and creation of SPDK NVMe over Fabrics target. Find more details at https://spdk.io/doc/getting_started.html

1. SPDK installation

Navigate to the root folder and get the list of available SPDK versions.

Select the SPDK version.

2. Starting SPDK NVMe over Fabrics target

In order to start SPDK NVMe over Fabrics target, it is necessary to initialize the NVMe drive. To initialize the NVMe drive, please run setup.sh and gen_nvme.sh from spdk/scripts folder

NOTE: gen_nvme.sh generates the NVMe namespace that is going to be reused to create an SPDK NVMe over Fabrics target. The generated namespace should look like Nvme0n1 1. Make sure to note this parameter.

SPDK NVMe over Fabrics target requires nvmf.conf file to get all the parameters for its functioning. The file must be created in spdk folder. Please, find the sample of nvmf.conf file below.

NOTE: Nvme0n1 1 is the namespace generated by gen_nvme.sh.172.16.77.1 is the IP address of the target NIC.NQN is the qualified name to identify a storage target.
Run the following command to create the SPDK NVMe over Fabrics target based on the previously created nvmf.conf

NOTE: After the server restarts, the target does not start automatically. The ./app/nvmf_tgt/nvmf_tgt -c nvmf.conf command should be added to autostart of the CentOS server.

Installing StarWind NVMe over Fabrics Initiator

1. Download StarWind NVMe–oF in the link here:  https://www.starwindsoftware.com/starwind-nvme-of-initiator

2. Execute the starwind-nvmeof.exe  to install StarWind NVMe-oF Initiator and follow the steps in the wizard.

3. Restart the server.

4. Open Device Manager to check that StarWind NVMe-oF Initiator is installed on the system.

Working with StarWind NVMe over Fabrics Initiator

The StarNVMeoF_Ctrl.exe command-line utility is used to work with the NVMe-oF initiator:

1. Find available NVMe-oF controllers and subsystems (targets). The command will show a list of names and properties of controllers for the address:

StarNVMeoF_Ctrl.exe discovery <target_ip_addr[:port]> <local_ip_addr>

2. Connect the NVMe-oF controller. The command will create a connection to the target.

StarNVMeoF_Ctrl.exe insert <target_ip_addr[:port]> <local_ip_addr> <SubNQN> <HostNQN> [<num_io_queues> <io_queue_depth>]

3. Show the list of connected NVMe-oF controllers. The command will show the list of connected controllers:

StarNVMeoF_Ctrl.exe list

4. Disconnect the NVMe-oF controller that is connected to the system. The command will disconnect the controller. The LUNs will be removed from the system:

StarNVMeoF_Ctrl.exe remove <controllerId>

Discovering and Connecting NVMe-over-Fabrics Targets on Windows Server

1. To discover the target, run the discovery command: StarNVMeoF_Ctrl.exe discovery <target_ip_addr:[port]> <local_ip_addr> by specifying the protocol, target host IP address and port number:

Example: StarNVMeoF_Ctrl.exe discovery 172.16.77.1:4420 172.16.77.2

Where:

<172.16.77.1:4420> — NVMe-oF target host IP and port;
<172.16.77.2> — local host IP.

2. To connect the target, run the command: StarNVMeoF_Ctrl.exe insert <target_ip_addr[:port]> <local_ip_addr> <SubNQN> <HostNQN> [<num_io_queues> <io_queue_depth> <first_core>]

Example: StarNVMeoF_Ctrl.exe insert 172.16.77.1:4420 172.16.77.2 nqn.2008-08.com.starwindsoftware:NVMeofTarget SW_1 512 6 0

Where:

• <172.16.77.1:4420 > — target host IP and port;
• <172.16.77.2> — local host IP;
• <nqn.2008-08.com.starwindsoftware:NVMeofTarget> — SubNQN of the target (may be copied from the discovery results screen);
• <SW_1> — local HostNQN;
• <512> — quantity of connections to the target;
• <6> — queue depth;
• <0> — number of the initial core.

3. After the “insert” command is executed, disk LUNs for the connected controller namespaces should appear in the system.

4. To show the list of connected NVMe-oF controllers, run the “StarNVMeoF_Ctrl.exe list” command.

5. To disconnect LUNs from the system, run the the controller disconnection command StarNVMeoF_Ctrl.exe remove <controllerId>
Example: StarNVMeoF_Ctrl.exe remove 1
Where:

• <1> — controller Id

NOTE: Make sure that LUNs are not used by other applications at the moment of disconnection, as removing LUNs with active file operations may lead to data corruption.