ESXi 5 Performance Issues

[DarkDiamond]

Hello,

I'm experiencing some performance issues with a virtualized file server in my lab while using ESXi 5 and Starwind as the iSCSI target.

My Hardware:
SAN - Xeon X3440 w/ 16GB RAM. 3 10K Disks in a Raid 5 generating more than 230MB sequential read and 150MB sequential write speeds. This happened to be spare hardware I wasn't using.
ESXi - Core i3 with 16GB RAM
2 GB NICs dedicated to iSCSI traffic configured with multipath. I have the IOPS setting in VMware set to 1. Both kernels are on a vswitch.
I've unchecked Delayed ACK within ESXi.

Here's what appears to be happening...

I have a virtualized file server running Windows 2008 R2. If I run IOMETER within the vm, I can drive almost 180MB/s sequential reads using 16 outstanding IOs. If I try to copy a file (4GB ISO) from my PC to the virtualized file server, I can only drive approximately 35MB/s. I've run iperf from my station to the VM and I can get saturate the gigabit connection.

If I transfer that same 4GB iso file directly from my PC to the SAN, I can also saturate my gigabit ethernet connection (~110MB/s).

It seems to me that ESXi is responsible for the performance issue when copying to/from my PC to the virtalized file server. Is there something I need to configure within the VM to be able to get decent bandwidth? Is this normal behavior?

Does anyone have any ideas? Please let me know if there's more information I can provide...

Thanks!
DarkDiamond

[Max (staff)]

Hi DarkDiamond,
1. Make sure that the vmkernels are on different vSwitches, VMware claims that this gives more performance.
2. I'm wondering if you can make a comparative test and see if there is a difference between copying data from your PC to iSCSI based hdd inside a VM and PC - > vm hdd (physical disk in the ESX host)

[DarkDiamond]

Hi DarkDiamond,
1. Make sure that the vmkernels are on different vSwitches, VMware claims that this gives more performance.
2. I'm wondering if you can make a comparative test and see if there is a difference between copying data from your PC to iSCSI based hdd inside a VM and PC - > vm hdd (physical disk in the ESX host)

1. I tried using different vSwitches and using one vSwitch and didn't see any difference in performance.

2. I set up iSCSI within a VM and am still only seeing approximately 40MB/s on reads. On writes I was able to saturate the gigabit connection from my PC. I don't have jumbo frames set up anywhere.

I found something else to be very interesting as well. I installed Windows Server 2008 R2 and enabled the Hyper-V role on my second ESX host. I configured the iSCSI Initiator on the Hyper-V host and proceeded to install a test VM and shared out a folder on this test VM. Final result - I was able to transfer files to and from the VM (large 4GB ISOs) at > 95MB/s - substantially different from a VM within ESXi. This definitely points me in the direction that there's something going on with ESXi's iSCSI initiator and rules out the SAN as an issue.

Any thoughts?

[anton (staff)]

Lack of Jumbo frames is killing performance...

[DarkDiamond]

Lack of Jumbo frames is killing performance...

On the HyperV install, I'm not using jumbo frames and easily maximized the bandwidth. I've also tried tweaking my raid stripe sizes on both the SAN and formatting the disk within the VM with no luck

[Anatoly (staff)]

In the VM - are you using GPT formatting or MBR?

[DarkDiamond]

During my testing I was using small luns (< 50 gigs) with MBR formatting.

[Anatoly (staff)]

Yes, but using GPT not only allows to use >2TB volumes, but also includes disk allignes from the possible reasons of bottleneck.
If this wont help we will try to figure out what`s wrong via remote session, if you wish.

[DarkDiamond]

Yes, but using GPT not only allows to use >2TB volumes, but also includes disk allignes from the possible reasons of bottleneck.
If this wont help we will try to figure out what`s wrong via remote session, if you wish.

After some more testing, I reset my ESXi host to factory defaults, reworked the NIC configuration (I was using the Realtek NIC which is now autodetected by ESXI5) to use an Intel NIC and I'm getting transfers around 70-75 MB/s from a Win 2008 VM to my Win 7 machine. This I'm happy with as the ESXi 5 hosts are only Core i3's.

Within the VM using CrystalDiskMark I'm getting reads and writes averaging nearly 100MB/s. Keeping in mind that I have two iSCSI NICs round robin at a setting of 1 IOPS, does this seem like a reasonable speed?

[anton (staff)]

Much better but still no... With paired NICs in RR you should get something like 200+ MB/sec (in single direction). Try playing with IOPS settings, 2 or 5 maybe better then 1.

Yes, but using GPT not only allows to use >2TB volumes, but also includes disk allignes from the possible reasons of bottleneck.
If this wont help we will try to figure out what`s wrong via remote session, if you wish.

After some more testing, I reset my ESXi host to factory defaults, reworked the NIC configuration (I was using the Realtek NIC which is now autodetected by ESXI5) to use an Intel NIC and I'm getting transfers around 70-75 MB/s from a Win 2008 VM to my Win 7 machine. This I'm happy with as the ESXi 5 hosts are only Core i3's.

Within the VM using CrystalDiskMark I'm getting reads and writes averaging nearly 100MB/s. Keeping in mind that I have two iSCSI NICs round robin at a setting of 1 IOPS, does this seem like a reasonable speed?