Nested Virtualization: VMware ESXi vs. Microsoft Hyper-V

Overview

Nested virtualization lets you run a hypervisor inside a virtual machine. It’s like Inception, but for servers – a VM inside a VM inside… well, you get the idea. This isn’t just a tech flex. It’s an essential tool for labs, testing, training, and development when there isn’t enough physical hardware to go around.

Both VMware ESXi and Microsoft Hyper-V support nested virtualization on modern CPUs. But the way they handle it, their quirks, limitations, and performance are quite different. This guide breaks it all down – warts and all.

Hypervisor Types

Firstly, let’s remember what the hypervisor types are:

• Type 1 (Bare-Metal): Runs directly on hardware. VMware ESXi and Microsoft Hyper-V (when installed as a role on Windows Server) fall into this category. They offer better performance and lower overhead.
• Type 2 (Hosted): Runs as an application inside an OS. Examples: VMware Workstation, Oracle VirtualBox. These are fine for lightweight tasks but not ideal for nested production-like environments.

What is Nested Virtualization?

Nested virtualization allows a VM to function as a hypervisor itself, hosting other VMs. In practice, it means spinning up a VMware ESXi or Hyper-V instance inside a VM running on another hypervisor.

This is widely used for:

• Simulating enterprise setups without additional hardware.
• Testing hypervisor-specific configurations.
• Practicing disaster recovery or failover scenarios.
• Developing and testing automation pipelines.

System Requirements

Proper hardware and firmware settings are critical for nested virtualization. Below is a comparison of base requirements for running ESXi 8.0 and Hyper-V on Windows Server 2022.

BIOS/UEFI must have virtualization extensions (Intel VT-x or AMD-V) and Data Execution Prevention (DEP/NX) enabled. Without these, nested virtualization will not work.

Software Requirements

Each platform has specific software-related prerequisites.

VMware ESXi:

• VMs must use hardware version 9 or newer.
• Nested virtualization works only if “Expose hardware-assisted virtualization to guest OS” is enabled.
• Cross-platform nesting (Hyper-V inside ESXi) is possible.
• Officially supported use case: vSAN Witness Appliance.

Microsoft Hyper-V:

• Host OS must be Windows Server 2016 or later. For AMD CPU support, Windows Server 2022 or newer is required.
• VM configuration version 8.0 or higher.
• Nested virtualization requires static memory. Dynamic Memory must be disabled.
• Features like Live Migration and VM Save/Restore are not supported in nested VMs.

Configuration Procedures

Enabling Nested Virtualization in ESXi via PowerCLI

1. Create a VM that will host the ESXi hypervisor.

2. Shut down the VM if it’s running.

3. Run PowerCLI commands:

Connect-VIServer -Server <ESXi IP>

Stop-VM -VM <VM Name> -Confirm:$false

$VM = Get-VM -Name <VM Name>

$Spec = New-Object VMware.Vim.VirtualMachineConfigSpec

$Spec.NestedHVEnabled = $true

$VM.ExtensionData.ReconfigVM($Spec)

Start-VM -VM <VM Name>

4. To allow Layer 2 VM traffic, enable promiscuous mode:

Get-VirtualSwitch -Server $VM.VMHost | Get-SecurityPolicy |

Set-SecurityPolicy -AllowPromiscuous $true -ForgedTransmits $true

Enabling Nested Virtualization in ESXi via Web GUI

1. Shut down the VM.
2. Edit the VM settings.
3. Under CPU options, enable “Expose hardware-assisted virtualization”.
4. Edit the security settings on the vSwitch or port group and set Promiscuous Mode and Forged Transmits to Accept.
5. Install ESXi within the VM.

Enabling Nested Virtualization in Hyper-V via PowerShell

1. Ensure the VM is off:

Stop-VM <VM Name> -Confirm:$false

2. Enable nested virtualization on the VM:

Set-VMProcessor -VMName <VM Name> -ExposeVirtualizationExtensions $true

3. Enable MAC address spoofing:

Get-VMNetworkAdapter -VMName <VM Name> | Set-VMNetworkAdapter -MacAddressSpoofing On

4. Disable dynamic memory:

Set-VMMemory -VMName <VM Name> -DynamicMemoryEnabled $false

5. Start the VM:

Start-VM <VM Name>

6. Inside the VM or via PowerShell Direct, install Hyper-V:

Install-WindowsFeature -Name Hyper-V -IncludeAllSubFeature -IncludeManagementTools -Restart

Enabling Nested Virtualization in Hyper-V via GUI

1. In Hyper-V Manager, open the VM settings.
2. Under “Memory”, disable Dynamic Memory.
3. Under Network Adapter > Advanced Features, enable MAC Address Spoofing.
4. Run PowerShell to enable virtualization extensions.
5. Install Hyper-V via Server Manager inside the guest VM.

Cross-Platform and Multi-Layer Support

• VMware ESXi can host a Windows Server VM with Hyper-V role enabled (nested Hyper-V).
• Hyper-V does not reliably support nested ESXi. Attempting it may lead to failures like PSOD.
• Two-layer nesting (L0 > L1 > L2) is stable and practical on both platforms. Three or more layers are possible but suffer from major performance drops.

Use Cases

Learning and Training

Nested virtualization enables simulation of enterprise infrastructure on a single physical host. IT professionals can train on vSphere or Hyper-V clusters without needing multiple servers.

Test Environments

DevOps and infrastructure teams can validate automation scripts and deployment pipelines on nested setups. This is especially useful when testing changes across multiple virtual hosts.

CI/CD Integration

Some CI/CD pipelines require dynamic provisioning of virtual infrastructure. Nested virtualization enables building and destroying environments on demand for integration and testing.

Network Function Virtualization (NFV)

Developers working with routers, firewalls, or SDN appliances can simulate multi-tiered virtual networks within nested environments.

Fault Tolerance and High Availability Testing

Nested clusters allow validating HA behaviors, such as failover and vMotion, without using production hardware.

Performance and Limitations

While nested setups provide flexibility, they come with overhead:

• Each virtualization layer adds CPU and memory load.
• I/O throughput is reduced compared to physical or first-level VMs.
• Not all features are available (e.g., DirectPath I/O, FT, Live Migration).

Careful resource planning is necessary. Assign static memory to nested hosts and allocate sufficient vCPUs. Enabling features like EPT (Intel) or RVI (AMD) improves performance in nested scenarios.

Automation Scripts

Both platforms support automation.

PowerCLI for ESXi – Automate nested VM configuration and ESXi deployments. PowerShell for Hyper-V – Automate VM provisioning, enable extensions, and install features.

Sample automation snippet for ESXi VM setup:

param (

[string]$ESXi, [string]$VMName

)

Connect-VIServer -Server $ESXi

$VM = Get-VM -Name $VMName

$Spec = New-Object VMware.Vim.VirtualMachineConfigSpec

$Spec.NestedHVEnabled = $true

$VM.ExtensionData.ReconfigVM($Spec)

Start-VM -VM $VMName

Disconnect-VIServer -Server $ESXi -Confirm:$false

Sample automation snippet for Hyper-V:

param (

[string]$VMName

)

Stop-VM $VMName -Confirm:$false

Set-VMProcessor -VMName $VMName -ExposeVirtualizationExtensions $true

Set-VMMemory -VMName $VMName -DynamicMemoryEnabled $false

Start-VM $VMName

Summary

VMware ESXi and Microsoft Hyper-V both support nested virtualization on current hardware, each with platform-specific constraints and setup methods. ESXi provides broader flexibility for running other hypervisors within VMs. Hyper-V is more limited in cross-platform support but integrates well within Windows environments.

Nested virtualization is ideal for labs, dev/test environments, and automation workflows. With appropriate resource planning and understanding of each hypervisor’s limitations, nested setups can replicate production-like infrastructure on a single host.

While not suited for production workloads, nested virtualization remains a powerful tool for anyone needing isolated environments to build, break, and rebuild as needed.