Can we run (nested) KVM VMs on the top of IBM PowerVM Linux LPARs?

Updated! No longer a rumor but officially supported as of July 19, 2024 (see annoucement)

A brief history of nested virtualization on IBM Hardware

Nested virtualization enables a virtual machine (VM) to host other VMs, creating a layered virtualization environment. This capability is particularly beneficial in enterprise scenarios where flexibility, scalability, and efficient resource management (if we save on CPU we do on $$$ licenses) are critical.

While it can be used for testing purposes with KVM on x86 or VMware, the performance is often suboptimal due to multiple translations and modifications of hardware instructions before they reach the CPU or I/O subsystem. This issue is not unique to these platforms and can affect other virtualization technologies as well.

On platforms like Z, although the performance impact of nested virtualization exists, improvements and optimisations in the hypervisor can mitigate these effects, making it 100% viable for enterprise use.

Virtualization layers on IBM Mainframe

Before delving into nested KVM on PowerVM, it’s essential to understand similar technologies. If the mainframe is the grandfather of current server technology, then logical partitioning (LPARs) and virtualization technologies (zVM) are the grandmothers of hypervisor solutions.

In this picture (taken from this GREAT article from Anbarasan Sekar) you can see up to 4 layers

Level 1 Virtualization: Shows an LPAR running Linux natively

Level 2 Virtualization: Shows VMs running on z/VM or KVM Hypervisor

Level 3 Virtualization: Shows nesting of z/VM Virtual Machines

Level 4 Virtualization: Shows Linux containers that can either run as stand-alone containers or can be orchestrated with kubernetes

Now have a look to this old (2010) image from the IBM Power platform architecture. Can you see anything similar? :) Let’s move on!

Deploying VMs on the top of a PowerVM Linux LPAR

If we have LPARs on Power where we can run AIX, Linux, and IBM i, and in Linux, we can install KVM, can we run VMs inside an LPAR?

Not quite; it will fail at some point. Why? Because KVM is not zVM (for now), and we need some tweaks in the Linux kernel code to support nested virtualization not just with IBM Power9 or Power10 processors, but also with the Power memory subsystem and I/O.

By examining the kernel.org mailing lists, we can see promising developments. Successfully running multiple VMs with KVM on a PowerVM LPAR means porting some fantastic mainframe virtualization technology to IBM Power, allowing us to run VMs and Kubernetes/OpenShift Virtualization on ppc64le for production purposes. This would make a significant difference if the performance penalty is minimal.CPU virtualization on Power and Mainframe systems simply allocates processor time without mapping a full thread as KVM or VMware do. Therefore, it is technically possible to add a hypervisor on top without significantly affecting performance as IBM does with LinuxOne.

Latest news for KVM on IBM PowerVM LPARs  (May 2024)

1) Add a VM capability to enable nested virtualization
Summary: This message discusses the implementation of nested virtualization capabilities in KVM for PowerPC, including module configurations and support on POWER9 CPUs.

2) Nested PAPR API (KVM on PowerVM)
Summary: It details the extension of register state for the nested PAPR API, the management of multiple VCPUs, and the implementation of specific hypercalls.

3) KVM: PPC: Book3S HV: Nested HV virtualization
Summary: A series of patches improving nested virtualization in KVM for PowerPC, including the handling of hypercalls, page faults, and mapping tables in debugfs.

For more detailed information, you can consult the following links:

• PPC KVM paravirtual interface
• lore.kernel.org/kvm-ppc
• Gautam’s Patch

Will we be able to install Windows on Power Systems (for fun)?

Stay tuned!!