Draft note: this is a lab workaround for a specific Software Memory Tiering placement block. I am not treating it as a Broadcom-supported design recommendation.
This started as a lab exercise to test a 2-node vSAN ESA setup on my vcf-wld vSphere 8 workload domain. Because a 2-node vSAN cluster needs a witness, I deployed a vSAN ESA 8.0U3i witness appliance into my management vCenter 9.0.2.
The witness VM deployed cleanly, but it would not power on.
Environment
- Management vCenter: VMware vCenter Server 9.0.2, build 25148086
- Management ESXi hosts: VMware ESXi 9.0.2.0, build 25148076
- Witness appliance template: VMware vSAN ESA Witness 8.0U3i, build 25205845
Symptom
The first symptom was a generic vCenter compatibility failure:
No host is compatible with the virtual machine.
The event log showed:
Placing or Powering on a VM vcf-wld-vsan-witness.wynner.ie that does not support user level monitor on a host esx02.wynner.ie with Software Memory Tiering enabled is not allowed.
I tried upgrading VM compatibility to vmx-22, and setting the guest OS to "VMware ESXi 8" (vmkernel8Guest), but the Software Memory Tiering placement fault remained. That is what pointed to nested virtualization rather than the guest OS label or virtual hardware version as the actual blocker.
All management cluster hosts had Software Memory Tiering enabled through the boot option VMkernel.Boot.memoryTiering = true.
Support Position
Broadcom KB 398302 documents that some VM profiles and VMX options are not supported when Software Memory Tiering is enabled. The unsupported list includes nested virtualization through vhv.enable = "True".
Broadcom KB 313547 is also relevant. It says nested ESXi is generally unsupported in production environments, but it explicitly carves out the vSAN Witness Appliance as supported when run as a VM on a physical ESXi host.
The unsupported part here is not “vSAN witness as a VM” by itself. The unsupported edge is the combination of Software Memory Tiering and a VM configured for nested virtualization. My workaround changes appliance VM settings to get past that placement block, so I would not present it as a supported Broadcom design.
The likely reason is that Software Memory Tiering changes memory scheduling and page placement behavior, while nested virtualization requires exposing hardware virtualization features to the guest hypervisor. That combination adds monitor and scheduler complexity. Broadcom’s memory-tiering KB frames the current limitation as a product constraint and notes that these constraints are planned to be removed in a future release.
What I Changed
The deployed witness VM was originally configured as:
- Guest OS ID:
vmkernel65Guest - VM compatibility:
vmx-11 - Nested hardware virtualization: enabled
The important workaround was disabling nested hardware virtualization. A vSAN witness appliance is an ESXi-based virtual appliance, but for this use case it is not intended to run nested VMs. Disabling nested HV removed the Software Memory Tiering placement block without disabling memory tiering on the physical hosts.
After the changes, the VM configuration was:
- Guest OS ID:
vmkernel8Guest - VM compatibility:
vmx-22/ ESXi 9.0 and later - Nested hardware virtualization: disabled
Result
The witness VM powered on successfully after nested hardware virtualization was disabled.
This is useful lab evidence, but I would still treat it carefully. For a supportable design, the safer choices are to avoid Software Memory Tiering on hosts that need to run nested-virtualization VMs, place the witness on hosts without memory tiering enabled, or wait for Broadcom to document support for this exact combination.