Search
StarWind is a hyperconverged (HCI) vendor with focus on Enterprise ROBO, SMB & Edge

VMFS Datastore on a USB drive

  • October 8, 2019
  • 25 min read
Cloud and Virtualization Architect. Kevin focuses on VMware technologies and has vast expertise in cloud solutions, virtualization, storage, networking, and IT infrastructure administration.
Cloud and Virtualization Architect. Kevin focuses on VMware technologies and has vast expertise in cloud solutions, virtualization, storage, networking, and IT infrastructure administration.

Introduction

As an admin, I often have to deal with the necessity to transfer large OVF and ISO files or even move virtual machines (VMs) between ESXi hosts that have poor network performance or disposed in different locations with no network connection whatsoever. If a case like this occurs, a USB flash drive or USB external drive is a way to go. More specifically, you can use a drive in two different ways (by the way, both authors have my most sincere regards and appreciation for presenting these methods in a most coherent and simultaneously simple way). The first one is to copy files from a USB device to a host directly through SSH session without changing drive format as the following article suggests. However, such an approach has some issues, the most tedious one being maximum file size limitations, which certainly limits its use for larger files. You cannot work with the Graphical user interface (GUI) as well. Instead, you gotta use SSH and dive deep into the host data storage system, so you can figure I am a much bigger fan of a second way: creating a VMFS datastore on a USB drive. You can connect it to ESXi hosts and transfer files from one to another (or just store a temporary copy on an external storage device).

As you’ve probably noticed, this approach implies connecting to host via SSH as well, but such datastore also allows using WebGui. That takes a load off your shoulders. Nevertheless, working with VMFS is like using any other technology. One mistake and it’s hit-or-miss. The author of the article on creating a datastore himself points out that carelessness will cost a lot. Even mastering partedUtil won’t save you from everything, so the only appropriate way to do this is to establish an algorithm. This article is dedicated to creating such a scenario. Now, I won’t be repeating the content of these articles; you don’t need me explaining it. What makes more sense, is getting more details on the order of actions you need to take to create a VMFS datastore on a USB drive.

Сreate a VMFS datastore on a USB drive

You must have two ESXi hosts to run this script (source host and destination host). I, for one, have tested it on VMware vCenter Server 6.7 Update 1, but it should be working with earlier versions as well. Also, you’ll need a Windows 10 (Windows Server 2016) PC or VM with a network connection to ESXi hosts. In case an external network option is out of the table, a virtual network connecting a VM and ESXi host will do. However, don’t forget that for the latter scenario to work, you must have the Posh-SSH module manually installed, so look it up here. USB 2.0 or USB 3.0. is also on the list of necessary requirements.

NOTE: You cannot create a datastore on an 8 GB (or less) USB flash drive! Not sure why, but it’s no go.

Science

The main trick to configure a USB device at the ESXi host level is stopping the Usbarbitrator service temporarily. Then you’ll just have to leave it to the ESXCLI command sequence listed up here. ESXCLI is the only option to create a VMFS datastore since it cannot be done with GUI. Naturally, since ESXCLI doesn’t support scenarios, here comes PowerShell, which is, basically, why you need to have the Posh-SSH module installed by now. You use SSH to connect to a remote server via the command line. In our case, you will execute ESXCLI commands to emulate an SHH host session successfully.

This script helps you to have all the necessary commands wholly automated with the following script parameters:

  • NewDS – create a new USB datastore, stop the Usbarbitrator service and disable it permanently until restarting it through UnmountDS or until rebooting the host;
  • UnmountDS – disconnect datastore from host, enable the Usbarbitrator service on the host;
  • MountDS – mount existing datastore to ESXi host, disable the Usbarbitrator service on the host.

Now it’s time to dig into the details and command sequences.

NewDS

That parameter is a driving force of the whole script since it is responsible for creating a new datastore on a USB drive. As a USB drive, USB flash or USB HDD pocket will do just fine. Pay attention to details: this command sequence will disrupt the work of the Usbarbitrator service! If your ESXi host is installed on a USB drive, the further actions can disrupt its work! As I have already stated, carelessness can cost you a lot, so choose your USB drive wisely!

The PowerShell command sequence:

  • ESXi host name parameter initialization.

  • Connection to ESXi host with a VM with virtualized ESXi host (password and login to connect to the ESXi host).

  • Get the status of the TSM-SSH service (if the service is started, the exit status of the commands will be “True”; in this case, skip the next paragraph).

  • Start the TSM-SSH service if it was disabled.

  • Start SSH host session (password and login to connect to the ESXi host).

  • Send an ESXCLI command to the ESXi host via SSH session to get the Usbarbitrator service status.

  • Send an ESXCLI command to the ESXi via SSH session to stop the Usbarbitrator service if it was enabled.

  • Send an ESXCLI command to the ESXi host via SSH session to update the list of devices attached.

  • Send an ESXCLI command to the ESXi host via SSH session to scan the file system structure on devices attached to the host.

  • Send an ESXCLI command to the ESXi host via SSH to get the list of available drives attached (need to choose the required drive).

  • USB datastore path parameter initialization ($DriveName). Choose path manually from the list you received with the previous command. In the name of the USB drive (Display Name line) should be “Local USB Direct-Access”.

  • Send an ESXCLI command to the ESXi host via SSH session to install gpt for the USB drive structure.

  • Send an ESXCLI command to the ESXi host via SSH session to find out the partition info.

  • ESXCLI write command parameter initialization.

  • Send an ESXCLI command to the ESXi host via SSH session to determine the partition end sector ($EndSector).

  • ESXCLI write command parameter initialization.

  • Send an ESXCLI command to the ESXi host via SSH session to create partition.

  • Datastore name parameter initialization ($DSName). You can choose any other name if not ‘USB-Ds’.

  • ESXCLI write command parameter initialization.

  • Send an ESXCLI command to the ESXi host via SSH session to create VMFS 6 datastore on the USB drive.

  • Disconnect from the ESXi host.

  • Remove SSH session from the ESXi host.

  • Remove ESXi host certificate from the trusted hosts list.

Now you have a new datastore on your host with name set in datastore name parameter ($DSName)!

You can invoke this command sequence with the Mode parameter (NewDS value).

To start the script, you need to save its listing – USB-DS.ps1.

USB-DS.ps1

 

Enter the host via WebGUI and make sure no existing datastore name matches the new USB datastore name.

0a98cb16f0061e24449a78313b6e6abe

Start previously saved script with PowerShell as admin. Run the script from the directory, and enter the following command:

Then you’ll have password and login screen to connect to the ESXi host; an account should have the administrative privileges on this host.

Password and login

As you can see, in my case there are two USB drives available for creating a datastore. Choose one of the attached USB drives and confirm your choice. At this stage, you can still dismiss datastore creation just by choosing “q”! Be attentive! After selecting a USB drive, all data will be erased instantly!

script

If all is performed correctly, you will receive a notification about new partition creation. After pressing Enter, the script work will be stopped, and the PowerShell console will be cleared.

script

You can check the presence and start working with the new USB datastore with the help of WebGUI. You need to restart the page to check the presence of the new datastore.

USB datastore

The new datastore is ready for work. After finishing the work, you’ll need the UnmountDS parameter, which will be explained further.

NOTE: Datastore will be available until you reboot the host.

UnmountDS parameter

UnmountDS

Before you get on with this parameter, please, make sure that you have finished all input and output operations in the datastore, unless you want to lose your data!

The PowerShell command sequence:

  • ESXi host name parameter initialization.

  • Connection to ESXi host with a VM with virtualized ESXi host (password and login to connect to the ESXi host).

  • Get the status of the TSM-SSH service (if the service is started, the exit status of the commands will be “True”; in this case, skip the next paragraph).

  • Start the TSM-SSH service if it was disabled.

  • Start SSH host session (password and login to connect to the ESXi host).

  • Send an ESXCLI command to the ESXi host via SSH session to get the Usbarbitrator service status.

  • Send an ESXCLI command to the ESXi via SSH session to start the Usbarbitrator service if it was disabled.

  • Disconnect from the ESXi host.

  • Remove SSH session from the ESXi host.

  • Remove ESXi host certificate from the trusted hosts list.

After completing this command sequence, all the USB drives attached to the host can be physically removed.

Start the script as admin, much like in the previous case. To start it, run the script from the directory, and enter the following command:

Unlike in the previous case, you don’t need to choose the datastore name. Then you’ll have password and login screen to connect to the ESXi host.

Password and login

Remove the USB drive from the host. Reboot the host to regain access to the USB drives from the VM. If you don’t need the access right now, you can do it later.

Remove the USB drive from the host

You can check if everything is correct with the help of WebGUI. You need to restart the page to see the changes.

Restart the page to see the changes

MountDS

This parameter is for stopping the Usbarbitrator service and updating the info about devices and their file systems. You can use this parameter on the source host to connect to the datastore after the reboot.

The PowerShell command sequence:

  • ESXi host name parameter initialization.

  • Connection to ESXi host with a VM with virtualized ESXi host (password and login to connect to the ESXi host).

  • Get the status of the TSM-SSH service (if the service is started, the exit status of the commands will be “True”; in this case, skip the next paragraph).

  • Start the TSM-SSH service if it was disabled.

  • Start SSH host session (password and login to connect to the ESXi host).

  • Send an ESXCLI command to the ESXi host via SSH session to get the Usbarbitrator service status.

  • Send an ESXCLI command to the ESXi via SSH session to stop the Usbarbitrator service if it was enabled.

  • Disconnect from the ESXi host.

  • Remove SSH session from the ESXi host.

  • Remove ESXi host certificate from the trusted hosts list.

After completing this command sequence, all the USB drives attached to the host need to be physically reattached.

Don’t attach the USB drive before you start this script! Start the script as admin, much like in both previous cases. To do so, run the script from the directory, and enter the following command:

Like in the previous case, you don’t need to choose the datastore name. Then you’ll have password and login screen to connect to the ESXi host.

Password and login

Connect the USB datastore and press Enter to continue.

Connect the USB datastore

You can check the presence and start working with the new USB datastore with the help of WebGUI. You need to restart the page to check the presence of the new datastore. However, if you started the script with the datastore disabled before rebooting the host, you won’t be able to check its presence with WebGUI!

You can check the presence the new USB datastore with WebGUI

USB drive size and structure haven’t changed, it is ready for work.

USB drive size and structure haven't changed

After you finished working with the datastore, use the UnmountDS parameter to save your data. All active input and output operations in the datastore should be finished by now. After the complete end of using the datastore, you can restore the USB drive to its original state by standard means. This is it for now! I do hope that my work will help to get rid of the especially tedious processes.

Found Kevin’s article helpful? Looking for a reliable, high-performance, and cost-effective shared storage solution for your production cluster?
Dmytro Malynka
Dmytro Malynka StarWind Virtual SAN Product Manager
We’ve got you covered! StarWind Virtual SAN (VSAN) is specifically designed to provide highly-available shared storage for Hyper-V, vSphere, and KVM clusters. With StarWind VSAN, simplicity is key: utilize the local disks of your hypervisor hosts and create shared HA storage for your VMs. Interested in learning more? Book a short StarWind VSAN demo now and see it in action!