Virtualization Blog

Discussions and observations on virtualization.

The reality of a XenServer 64 bit dom0

One of the key improvements in XenServer Creedence is the introduction of a 64 bit control domain (dom0). This is something I've heard requests for over the past several years, and while there are many reasons to move to a 64 bit dom0, there are some equally good reasons for us to have waited this long to make the change. It's also important to distinguish between a 64 bit hypervisor, which XenServer has always used, and a 64 bit control domain which is the new bit.

Isn't XenServer 64 bit bare-metal virtualization?

The good news for people who think XenServer is 64 bit bare metal virtualization is they're not wrong. All versions of XenServer use the Xen Project hypervisor, and all versions of XenServer have configured that hypervisor to run in 64 bit mode. In a Xen Project based environment, the first thing the hypervisor does is load its control domain (dom0). For all versions of XenServer prior to Creedence, that control domain was 32 bit.

The goodness of 64 bit

A 64 bit Linux dom0 allows us to remove a number of bottlenecks which were arbitrarily present with a 32bit Linux dom0.

Goodbye low memory

One of the biggest limiting factors with a 32bit Linux dom0 is the concept of "low memory". On 32 bit computers, the maximum directly addressable memory is 4GB. There are a variety of extensions available to address beyond the 4GB limit, but they all come with a performance and complexity cost. Further, as 32bit Linux evolved, a concept of "Low" memory and "High" memory was created with most everything related to the kernel using low memory and userspace processes running in high memory. Since we're talking about kernel operations consuming low memory, that also means that any kernel memory maps and kernel drivers also consume low memory and you can see how low memory quickly becomes a precious commodity. It also can be increasingly consumed the more memory available to a 32bit Linux dom0. In a typical XenServer installation this value is only 752MB, and with the move to a 64bit dom0 will soon be a thing of the past.

Improved driver stability

In a 32bit BIOS, MMIO regions are always placed between the 1GB and 3GB physical address space, and with a 64bit BIOS they are always placed at the top of the physical memory. If an MMIO hole is created dom0 can choose to re-map that memory so that it is now shadowed in virtual address space. The kernel must map a roughly equal amount of memory to the size of the MMIO holes, which in a 32bit kernel must be done in low memory. Many drivers map kernel memory to MMIO holes on demand, resulting in boot time success but potential instability in the driver if there is insufficient low memory to satisfy the dynamic request for a MMIO hole remap. Additionally, while XenServer currently supports PAE, and can address more than 4GB, if the driver insists on having its memory allocated in 32bit physical address space, it will fail.

Support for more modern drivers

Hardware vendors want to ensure the broadest adoption for their devices, and with modern computers shipping with 64bit processors and memory configurations well in excess of 4GB, the majority of those drivers have been authored for 64bit operating systems, and 64bit Linux is no different. Moving to a 64bit dom0 gives us an opportunity to incorporate those newer devices and potentially have fewer restrictions on the quantity of devices in a system due to kernel memory usage.

Improved computational performance

During the initial wave of operating systems moving from 32bit to 64bit configurations, one of the often cited benefits was the computational improvements offered from such a migration. We fully intend for dom0 to take advantage of general purpose improvements offered from the processor no longer needing to operate in what is today more of a legacy configuration. One of the most immediate benefits is with a 64bit dom0, we can use 64bit compiler settings and take advantage of modern processor extensions. Additionally, there are several performance concessions (such as the quantity of event channels) and code paths which can be optimized when compiled for a 64bit system versus a 32bit system.

Challenges do exist, though

While there are clear benefits to a 64 bit migration, the move isn't without its set of issues as well. The most significant issue relates to XenServer pool communications. In a XenServer pool, all hosts have historically been required to be at the same version and patch level except during the process of upgrading the pool. This restriction serves to ensure that any data being passed between hosts, and configuration information being replicated in xenstore and all operations are consistent. With the introduction of Storage XenMotion in XenServer 6.1, this requirement was also extended to cross pool operations when a vdi is being migrated.

 

In essence you can think of the version requirement as being the insurance against bad things happening to your precious VM cargo. Unfortunately, that requirement has an assumption of dom0 having the same architecture and our migration from 32bit to 64bit complicates things. This is due to the various data structures used in pool operations having been designed with a consistent view of a world based on a 32bit operating system. This view of the world is directly challenged with a pool consisting of both 32bit and 64bit XenServer hosts as would be present during an upgrade. It's also challenged during cross pool storage live migration if one pool is 32bit while the second is 64bit. We are working to resolve this problem at least for the 32bit to 64bit upgrade, but it will be something we're going to want very active participation from our user community to test once we have completed our implementation efforts. After all we do subscribe to the notion that your virtual machines are pets and not cattle; regardless of the scale of your virtualized infrastructure.     

XenServer Creedence Alpha 2 Released
Validation of the Creedence Alpha

Related Posts

 

Comments 9

chaitanya on Friday, 06 June 2014 17:52

Hi,

Good concept!! so, will this be included in alpha.2?

Chaitanya.

0
Hi, Good concept!! so, will this be included in alpha.2? Chaitanya.
Tim Mackey on Tuesday, 10 June 2014 17:15

Chaitanya,

Alpha.1 already has a 64 bit dom0, and with Alpha.2 we're bringing more improvements online.

-tim

0
Chaitanya, Alpha.1 already has a 64 bit dom0, and with Alpha.2 we're bringing more improvements online. -tim
Tobias Kreidl on Tuesday, 10 June 2014 14:38

Great synopsis, Tim. This release is going to be the best ever.

0
Great synopsis, Tim. This release is going to be the best ever.
Guest - Glyn on Friday, 13 June 2014 20:31

What does this mean for the 2TB limit presently applied to NFS / LVM guest disk images?

( http://support.citrix.com/servlet/KbServlet/download/34966-102-704363/CTX137837_XenServer%206_2_0_Configuration%20Limits.pdf )

0
What does this mean for the 2TB limit presently applied to NFS / LVM guest disk images? ( http://support.citrix.com/servlet/KbServlet/download/34966-102-704363/CTX137837_XenServer%206_2_0_Configuration%20Limits.pdf )
Guest - james on Tuesday, 17 June 2014 06:39

the 2TB limit is based on the virtual disk format VHD - this remains unchanged. But, there have been requests from user community to switch to VHDX or similar formats - nothing has been committed in the XS roadmap yet.

0
the 2TB limit is based on the virtual disk format VHD - this remains unchanged. But, there have been requests from user community to switch to VHDX or similar formats - nothing has been committed in the XS roadmap yet.
Guest - james on Tuesday, 17 June 2014 14:01

@Tim,
could you please elaborate the gains made in modern processor extensions by switching to 64-bit Dom0?
How does it compare to the planned PVH Dom0 support?

0
@Tim, could you please elaborate the gains made in modern processor extensions by switching to 64-bit Dom0? How does it compare to the planned PVH Dom0 support?
Guest - Arvind on Tuesday, 29 December 2015 04:18

Why this one is still in pre-release .. its more than year now :(

0
Why this one is still in pre-release .. its more than year now :(
Nishit on Sunday, 12 June 2016 09:04

Hello,
We have Centos 7.2.1511 OS 64 bit, 64 GB Ram on server and have installed Xen on that.
We are planning to install email on host Dom0 and websites on VM DomU.
What is the recommended Ram size for Dom0? Can we assign 32 GB Ram to Dom0? Does it affect the server performance?
Please let me know.

Thanks,
Nishit

0
Hello, We have Centos 7.2.1511 OS 64 bit, 64 GB Ram on server and have installed Xen on that. We are planning to install email on host Dom0 and websites on VM DomU. What is the recommended Ram size for Dom0? Can we assign 32 GB Ram to Dom0? Does it affect the server performance? Please let me know. Thanks, Nishit
David Cottingham on Monday, 13 June 2016 15:07

Hi Nishit,

It looks like you’re asking about taking standard CentOS and using it with Xen, rather than installing XenServer.

XenServer is a distribution that packages up a Dom0 distribution (CentOS-based), Xen, xapi, and many other components as a complete virtualisation platform. Dom0 memory is configured for you automatically.

I’d therefore recommend looking at installing XenServer to your environment, particularly if you’re new to Xen-based virtualisation.

Best wishes,

David.

0
Hi Nishit, It looks like you’re asking about taking standard CentOS and using it with Xen, rather than installing XenServer. XenServer is a distribution that packages up a Dom0 distribution (CentOS-based), Xen, xapi, and many other components as a complete virtualisation platform. Dom0 memory is configured for you automatically. I’d therefore recommend looking at installing XenServer to your environment, particularly if you’re new to Xen-based virtualisation. Best wishes, David.

About XenServer

XenServer is the leading open source virtualization platform, powered by the Xen Project hypervisor and the XAPI toolstack. It is used in the world's largest clouds and enterprises.
 
Commercial support for XenServer is available from Citrix.