Advice on maximizing the speed that vSphere can clone VMs

DavidMcKnight · Mon Jan 06, 2014 5:18 pm

I asked this over on the VMware Communities forums and I'm not getting a lot of useful help, so I thought I'd ask it over here too.

I have a cluster of VMware Hosts that have dual 10Gig NICs. I'm using Starwind 6.0.5713 as my iSCSI storage solution with dual 10gig nics. I'm doing iSCSI port binding on the two 10gig ports on the VMHost. So I end up with 4 paths to each iSCSI volume. Then I do round robin across the 4 paths. I have a PowerCLI script that will clone out 50 VMs one after another. When I clone a single VM I'm seeing 6 to 8 gigabit of traffic on my Starwind box. If I clone two VMs at once I see 14 to 15 gigabit of traffic.

My question is why don't I get 14 to 15 (or better yet 20) gigabit of traffic when cloning a single VM?

I have the bandwidth. If I run ATTO Disk Benchmark inside a single VM on one of my hosts, it can generate over 20 gigabit of traffic. I assume a lot of that is hitting the Starwind cache, but at least I know a single host and the Starwind service can saturate my network bandwidth, which does not happen when cloning.

I know I have more storage speed since I can clone two VMs to the same time and get 190% the performance not the full 200%.

Is there some advanced setting that tells vSphere not to use all the available network/storage capacity while cloning?

I'd appreciate any advice on maximizing the speed that vSphere can clone VMs,

Thanks,

David

Mon Jan 06, 2014 8:45 pm

1) With HA configs backbone performance (synchronization channel) is considered to be a limiting write factor so throwing more links between nodes (we don't ask for them to be switched, just an opposite - recommend direct cable connection, so pair of 10 GbE adapters can go $200 each, also look for dual port adapters as they are a real bargain) would increase write performance and cut down provisioning time. That's for V6 and upcoming V8.

2) We work on VAAI (you can try it with current V8 beta) that would dramatically offload hosts when doing copy operation. Should help to your task quite a lot. V8 only.

Hope this helped

DavidMcKnight · Mon Jan 06, 2014 9:31 pm

1) With HA configs

This particular iSCSI node is not HA.

2) We are working on VAAI

If I understand it right, in the case of cloning a VM, it would only help if the source VM was on the same iSCSI target as the destination VM. right?

Mon Jan 06, 2014 9:47 pm

1) In this case you can throw faster spindles into storage node (sorry, pretty much obvious).

2) Yes, LUNs should be the same. I don't think anybody does target-to-target VAAI (at least I'm not aware of such an implementations). Is it anything you need? I mean target-to-target offload?

DavidMcKnight wrote:1) With HA configs

This particular iSCSI node is not HA.

2) We are working on VAAI

If I understand it right, in the case of cloning a VM, it would only help if the source VM was on the same iSCSI target as the destination VM. right?

DavidMcKnight · Tue Jan 07, 2014 4:06 pm

1) In this case you can throw faster spindles into storage node (sorry, pretty much obvious).

These are SSDs (15 @ raid 6 + 1 hot spare)

2) Yes, LUNs should be the same. I don't think anybody does target-to-target VAAI (at least I'm not aware of such an implementations). Is it anything you need? I mean target-to-target offload?

Well, strictly from a performance point of view, would you not be better off reading from one set of drives and writing to another? When I clone I typically try to have my master VM on a complete different datastore than where I clone to. So, yea, not only target-to-target but between datastores as well.

Tue Jan 07, 2014 9:29 pm

1) We'll need to check block size VMFS is doing writes with. If it's not huge (2-4 MB) then parity RAID is slowing down copy process with "ready-modify-write" updates.

2) Sure it would be faster then using the same set of a drives for both operations @ the same time. However VAAI offloads HOST so data does not travel down Ethernet and
whole copy is handled as atomic "read-write" operation on StarWind (or any other VAAI-compatible) array. Basically you remove a) Ethernet and b) Host CPU-memory from copy sequence.

DavidMcKnight wrote:1) In this case you can throw faster spindles into storage node (sorry, pretty much obvious).

These are SSDs (15 @ raid 6 + 1 hot spare)

2) Yes, LUNs should be the same. I don't think anybody does target-to-target VAAI (at least I'm not aware of such an implementations). Is it anything you need? I mean target-to-target offload?

Well, strictly from a performance point of view, would you not be better off reading from one set of drives and writing to another? When I clone I typically try to have my master VM on a complete different datastore than where I clone to. So, yea, not only target-to-target but between datastores as well.