I will teach how to use XenServer.
“Unlimited” Event Channels
David Vrabel, Citrix
Event Channels are Xen's mechanism for paravirtualized interrupts. These were limited to only 4096 which then limits the number of guests that a host may support to around 300 to 500. This presentation will give a brief introduction to event channels, a detailed look at the new, innovative FIFO-based event channel ABI that increases the limit to over 100,000 as well as having several other useful features (e.g., multiple priorities). Some of the key performance measurements of the new ABI will be shown.
Test-as-a-Service and XenRT
Alex Brett, Citrix
In this presentation, Alex Brett will show how Citrix has constructed a Test-as-a-Service environment which is used by the wider XenServer engineering team, highlighting the benefits the approach provides, together with an introduction to the (recently open sourced) XenRT automation framework which powers it, and discuss how this could be applied within the Xen Project community.
Increasing XenServer’s VM Density
Jonathan Davies, Citrix
As the number of CPU cores in server-class hardware increases, the demand on a virtualisation platform increases for greater VM density. Most commercial virtualisation platforms now support several hundred VMs per host. This talk will describe the scalability challenges that were overcome in Citrix XenServer 6.2 to enable support for up to 500 fully virtualised or 650 paravirtualised VMs per host. These include limits with event channels, blktap, xenstored and consoled. It will also discuss how dom0 CPU utilisation was reduced in order to make a large number of VMs responsive and thus usable, and will present benchmark measurements quantifying these improvements.
Open Source Citrix Windows PV Drivers
Paul Durrant, Citrix
Citrix has recently spent several months making sure all the key parts of XenServer are open source. Part of this effort made the XenServer Windows Paravirtual (PV) drivers available in source form under a BSD 2 clause license on GitHub. Building these drivers outside of the internal Citrix XenServer build environment was quite hard and the resulting binaries would only run correctly in a XenServer host environment. I have recently spent many weeks modifying the drivers so that they should work on any recent upstream Xen host environment thus making it much easier for anyone outside of Citrix to build and deploy the drivers. I would therefore like to give a brief tour of all the drivers, their source, what each of them does, and how they all interact. I will also discuss plans for posting signed versions of these drivers onto Windows Update for general use by the community.
xenserver-core: What it is, How it is Built and How to get Involved
Euan Harris, Citrix
XenServer is open source and freely available, but it is packaged as an appliance image which must be installed on dedicated hardware. xenserver-core repackages the core components of XenServer so they can easily be built and installed on a standard Linux distribution. Its main goals are: * to make it easy to download, modify and build XenServer components, or just learn how they work; * to help upstream distributions to include up-to-date XenServer packages; * to provide an environment for experimentation. This talk will explain the motivations behind xenserver-core and how it relates to the open-sourcing of XenServer. For developers, it will cover how to get the code, how to build it and how to contribute back to the project. For packagers, it will explain the project's development and release processes and what an upstream maintainer can expect from it.
SecureServe: A Multi-level Secure Server Virtualization Platform on Xen
Jason Sonnek, Adventium Labs
Due to the rapid shift toward cloud computing, virtual desktop infrastructure (VDI) and thin client computing, many organizations in the government desire a high assurance, multi-level secure server virtualization platform that is low-cost, open and enterprise ready. In this presentation, Jason Sonnek will present SecureServe, a recently launched effort to develop such a platform by building on the open-source Citrix XenServer. The SecureServe project will draw upon research in a number of areas, including dom0 disaggregation, Xen Security Modules mandatory access controls and static/dynamic attestation. In this presentation, Jason will describe the project objectives and requirements, the project's relation to Citrix XenClient XT and XenServer Windsor, current development status and plans for moving forward.
Delivering Continuous Deployment of Xen-API at Cloud Scale
John Garbutt, Rackspace
Currently xen-api is really only installed today as part of XenServer. It has traditional enterprise style releases, with controlled upgrades and hotfixes when required. When deploying OpenStack Rackspace, with the help of the OpenStack community, have adopted an approach where any check-in could be deployed, and the system upgraded, from any other checkin from that last release, or earlier in the current release. It would be interesting to see if xen-api could move towards a model. At a minimum having more regular check points where an upgrade would be possible. When running a cloud, a very small amount of control plane downtime is possible, but ideally there should be zero downtime for user's virtual machines. We should explore the ability to only upgrade Xen as a last resort, but still be able to update as much the control and data plane as possible, while keeping VMs alive.
Multiple Device Emulators for HVM Guests
Paul Durrant, Citrix
Currently Xen only allows a single device emulator to be attached to each HVM guest in a system and, to date, this has been QEMU generally running as a process in the same domain as the toolstack, or in a stub domain. To enable the deployment of virtual GPUs to HVM guests in XenServer, patches were created to allow multiple device emulators to be attached to each HVM guest. QEMU continues to be used to emulate the majority of the devices, but a second process is spawned to handle the virtual GPU. This opens up the possibility of the GPU vendors supplying 'appliance' driver domains in future. I'd like to give an overview of the changes that we've made to Xen and QEMU to enable the use of multiple emulators, the potential benefits to driver domains, plus the knock on effect of emulator disaggregation on the 'unplug' protocol and what we could do about this.
Xen and XenServer Storage Performance
Felipe Franciosi, Citrix
The development of low latency storage media such as modern Solid State Drives (SSD) brings new challenges to virtualisation platforms. For the first time, we are witnessing storage back ends which are so fast that the CPU time spent in processing data significantly impacts the delivered throughput. This is aggravated by CPU speeds remaining largely constant while storage solutions get faster by orders of magnitude. To meet user demands and fully exploit SSD performance under Xen, new technologies are necessary. This talk will discuss the Xen storage virtualisation data path when using various back ends (e.g. blkback, tapdisk, qemu). It will explain why it is hard to exploit SSD performance with current technologies and present measurement data for a variety of workloads. Finally, it will show how techniques such as persistent grants and indirect I/O can help to mitigate the problem.
This past month saw some significant progress toward our objective of converting XenServer from a closed source product developed within Citrix to an open source project. This is a process which is considerably more difficult and detailed than simply announcing that we’re now open source, and I’m pleased to announce that in October we completed the publication of all sources making up XenServer. While there is considerable work left to be done, not only can interested parties view all of the code, but we have also posted nightly snapshots of the last thirty builds from trunk. For organizations looking to integrate with XenServer, these builds represent an ideal early access program from which to test integrations.
In addition to the code progress, we’ve also been busy building capabilities and supporting a vibrant ecosystem. Some of the highlights include:
Now no status report would be complete without some metrics, and we’ve got some pretty decent stats as well. Unique visitors to xenserver.org in October were up 12% to over 26,000. Downloads of the core XenServer installation ISO directly from xenserver.org were up by over 1000 downloads. Mailing list activity was up 50% and we had over 80 commits to the XenServer repositories. What’s even more impressive with these numbers is that XenServer is built from a number of other open source projects, so the real activity level within XenServer is considerably larger.
At the end of the day this is one month, but it is a turning point. I’ve been associated in one form or another with XenServer since 2008, and even way back then there were many who expected XenServer was unlikely to be around for long. Five years later there are more competing solutions, but the future for XenServer is as solid as ever. We’re working through some of the technical issues which have artificially limited XenServer in recent years, but we are making significant progress. If you are looking for a solid, high performance, open source virtualization platform; then XenServer needs to be on your list. If you are looking to contain the costs of delivering virtualized infrastructure, the same holds true.
More important than all these excellent steps forward is how XenServer can benefit the ecosystem of vendors and fellow open source projects which are required to fully deliver virtualized infrastructure at large scale. Over the next several months I’m going to be reaching out to various constituencies to see what we should be doing to make participating in the ecosystem more valuable. If you want to be included in that process, please let me know.
In a previous article, I described how dom0 event channels can cause a hard limitation on VM density scalability.
Event channels were just one hard limit the XenServer engineering team needed to overcome to allow XenServer 6.2 to support up to 500 Windows VMs or 650 Linux VMs on a single host.
In my talk at the 2013 Xen Developer Summit towards the end of October, I spoke about a further six hard limits and some soft limits that we overcame along the way to achieving this goal. This blog article summarises that journey.
Firstly, I'll explain what I mean by hard and soft VM density limits. A hard limit is where you can run a certain number of VMs without any trouble, but you are unable to run one more. Hard limits arise when there is some finite, unsharable resource that each VM consumes a bit of. On the other hand, a soft limit is where performance degrades with every additional VM you have running; there will be a point at which it's impractical to run more than a certain number of VMs because they will be unusable in some sense. Soft limits arise when there is a shared resource that all VMs must compete for, such as CPU time.
Here is a run-down of all seven hard limits, how we mitigated them in XenServer 6.2, and how we might be able to push them even further back in future:
dom0 event channels
blktap2 device minor numbers
aio requests in dom0
dom0 grant references
Connections to xenstored
Connections to consoled
dom0 low memory
Summary of limits
Okay, so what does this all mean in terms of how many VMs you can run on a host? Well, since some of the limits concern your VM configuration, it depends on the type of VM you have in mind.
Let's take the example of Windows VMs with PV drivers, each with 1 vCPU, 3 disks and 1 network interface. Here are the number of those VMs you'd have to run on a host in order to hit each limitation:
|Limitation||XS 6.1||XS 6.2||Future|
|dom0 event channels||150||570||no limit|
|blktap minor numbers||341||682||no limit|
|aio requests||368||869||no limit|
|dom0 grant references||372||no limit||no limit|
|xenstored connections||333||500||no limit|
|consoled connections||no limit||no limit||no limit|
|dom0 low memory||650||650||no limit|
The first limit you'd arrive at in each release is highlighted. So the overall limit is event channels in XenServer 6.1, limiting us to 150 of these VMs. In XenServer 6.2, it's the number of xenstore connections that limits us to 500 VMs per host. In the future, none of these limits will hit us, but there will surely be an eighth limit when running many more than 500 VMs on a host.
What about Linux guests? Here's where we stand for paravirtualised Linux VMs each with 1 vCPU, 1 disk and 1 network interface:
|Limitation||XS 6.1||XS 6.2||Future|
|dom0 event channels||225||1000||no limit|
|blktap minor numbers||1024||2048||no limit|
|aio requests||368||869||no limit|
|dom0 grant references||no limit||no limit||no limit|
|xenstored connections||no limit||no limit||no limit|
|consoled connections||341||no limit||no limit|
|dom0 low memory||650||650||no limit|
This explains why the supported limit for Linux guests can be as high as 650 in XenServer 6.2. Again, in the future, we'll likely be limited by something else above 650 VMs.
What about the soft limits?
After having pushed the hard limits such a long way out, we then needed to turn our attention towards ensuring that there weren't any soft limits that would make it infeasible to run a large number of VMs in practice.
Felipe Franciosi has already described how qemu's utilisation of dom0 CPUs can be reduced by avoiding the emulation of unneeded virtual devices. The other major change in XenServer 6.2 to reduce dom0 load was to reduce the amount of xenstore traffic. This was achieved by replacing code that polled xenstore with code that registers watches on xenstore and by removing some spurious xenstore accesses from the Windows guest agent.
These things combine to keep dom0 CPU load down to a very low level. This means that VMs can remain healthy and responsive, even when running a very large number of VMs.
For those community members, who could not attend the Xen Project User Summit in New Orleans, we now published all the videos on the Xen Project's video stream. The summit program included quite a few XenServer sessions, which you may want to check out.
For your convenience, we also created a portal page on XenProject.org that links to all recordings of the user summit. Again, a big Thank You to our speakers and to Russell Pavlicek for organizing the event.
If you are using XenServer, then you are using software developed by the Linux Foundation Xen Project. More specifically you will be using the Xen Hypervisor and XAPI. If you want to know about the latest trends, cutting edge features and innovations that are developed by the Xen Project and will be coming to Citrix products in future, the Xen Project Developer Summit may be for you!
The Xen Project Developer Summit is the Xen Project’s annual developer conference. It brings together the developers and power users that define the Xen Project. We will be sharing ideas and experiences, discuss the latest technical developments and innovations and plan the evolution of Xen and its sub projects for 2014.
This year's summit features sessions covering the latest developments and innovations in the Xen Hypervisor, Client Virtualization, Security, Graphics and Audio Virtualization and Cloud Computing. With the addition of ARM support to the Xen Hypervisor, we will also show and discuss new use-cases for virtualization on mobile devices, in automotive and middlebox appliances such as firewalls and NATs.
The summit also features many XenServer related sessions, such as:
While overcoming the architectural obstacles in order to allow XenServer 6.2.0 to run up to 500 HVM guests per host, we came across an interesting observation: even for apparently idle Windows 7 VMs, there is some amount of work done in dom0. With careful analysis and optimisations in the way we emulate virtual hardware for such VMs, we managed to eliminate most of this work for guests that are idle. This allows us to effectively run 500 Windows VMs and keep dom0 load average practically at zero (after the VMs have finished booting).
We started by observing the CPU utilisation of the QEMU process that is responsible for the hardware emulation of a Windows 7 guest. Even when the guest was just waiting for login and not running disk indexing services, Windows Update or other known I/O operations, the QEMU process would consistently consume between 1% and 3% of CPU time in dom0 (depending mostly on the host's characteristics). In this scenario, there is one QEMU process for each HVM guest running on a host.
When we succeeded at overcoming the hard limits that prevented us from starting 500 guests (see this post), we soon realised that the small amount of work done by each QEMU would quickly add up. With 3% of dom0 vCPU consumption per running Windows 7 VM, it takes just over 130 VMs to completely max out four dom0 vCPUs. At that point, dom0's load average quickly increases and the host's performance becomes compromised.
Without available CPU time, dom0 is unable to promptly serve network and storage I/O requests. Other control plane operations are also affected since the toolstack (e.g. xapi, xenopsd, xenstore) will struggle to run and respond to requests from XenCenter or other clients.
In order to investigate why those QEMUs were "spinning" so much, we created Project Pulsar (named after neutron stars that spin considerably). The project conducted a detailed analysis of every event that disturbed QEMU from its otherwise sleeping state.
With careful debugging, we learned that there are two categories of events that cause QEMU to wake up. Firstly, there are internal timers that occur several times a second to poll for certain events (e.g. buffered I/O). Secondly, there are actual interrupts generated by guests to check on certain virtual hardware (e.g. USB and the parallel port).
The following table shows the amount of events disturbing QEMU during the lifetime of a Windows 7 VM running for 5 minutes. The events are grouped in 30 seconds time intervals for simplicity. This facilitates the visualisation of the boot period (within the first 30 seconds interval), the time that the VM is allegedly idle and the shutdown phase at the end. We separated the columns into Timer, Read and Write Events. Timer events are internal to QEMU and Read or Write events come from the VM as interrupts. After the PV drivers are loaded (which happens during the boot), storage and network I/O are not handled by QEMU and therefore are not accounted anymore.
|30 Secs Interval||Timer Events||Read Events||Write Events|
While certain events only happen during the guest initialisation or shortly after the boot completed (e.g. parallel port scans), others remain constant throughout the life of the VM (e.g. USB protocols).
To address the first category of events that disturb QEMU (e.g. internal timers), we first studied why they were ever required. As it turned out, newer versions of QEMU were already patched to disable some of these. A good example is the case of buffered I/O which required polling. With the creation of a dedicated event channel for buffered I/O interrupts (see this patch) and a corresponding change to the device model (see this patch), QEMU no longer needs to poll.
The other timers we identified are necessary only when certain features are enabled. These are the QEMU monitor (that allows debugging tools to be hooked up to QEMU processes), serial port emulation and VNC connections. Considering XenServer does not support QEMU debugging via its monitor feature nor direct serial connections to HVM guests, these could be safely disabled. The last timer would only be active when a VNC connection is established to a guest (e.g. via XenCenter).
The second category of events happens due to the very nature of the hardware that is being emulated. If we present a Windows VM with an IDE DVD-ROM drive, the guest OS handles this (virtual) drive as a real drive. For example, it will poll the drive every so often to check whether the media has changed. Similar types of interrupts will be initiated by the guest to communicate with other emulated hardware.
In order to address these, we modified Xapi to allow the emulation of USB, parallel and serial ports to be turned off. Parallel port emulation fits in the same category as the serial port (i.e. there is no supported way to plug virtual devices to these ports in a guest) and the emulation of both are fairly safe and symptomless to be turned off.
Disabling USB, however, may have side effects. When using VNC to connect to a guest's console, a USB tablet device driver is used to allow for absolute coordinates of the mouse on the screen. When not using this USB driver, the VNC falls back to a PS/2 emulation which can only provide relative mouse positioning. The side effect is that, without USB, the mouse pointer of the VNC client will very likely be misaligned with the mouse pointer in the guest. This makes the console very hard to use.
The good news is that the Windows Remote Desktop Protocol (RDP) does not rely on the USB tablet driver. If the guest is configured to allow RDP connections, the USB emulation can be disabled without this side effect. When available, XenCenter already prefers RDP over VNC connections to Windows VMs by default.
The recommendations of Project Pulsar were adopted as defaults wherever possible and have been incorporated in XenServer 6.2.0. These included changes not visible to the VM (such as QEMU internal timers). However, we decided not to change the virtual hardware presented to VMs unless this is explicitly configured.
In order to configure Xapi to disable the Serial Port emulation, use the following command:
xe vm-param-set uuid=<vm-uuid> platform:hvm_serial=none
Similarly, the Parallel Port emulation can be disabled as follows:
xe vm-param-set uuid=<vm-uuid> platform:parallel=none
Finally, the USB emulation can be disabled as follows:
xe vm-param-set uuid=<vm-uuid> platform:usb=false
xe vm-param-set uuid=<vm-uuid> platform:usb_tablet=false
Note that two commands are necessary to completely disable the USB emulation. These disable both the virtual USB hub and the virtual tablet device used for positioning the mouse.
The best way to disable the emulation of the DVD-ROM drive is to delete the associated VBD. For information on how to do that, refer to Section A.4.23.3 of the XenServer 6.2.0 Administrator's Guide.
The following table shows the amount of events disturbing QEMU during the lifetime of a Windows 7 VM running for 5 minutes. This is the same VM used for the numbers in the table above, except that this time we incorporated the timer patches in QEMU and disabled the USB, DVD-ROM, Monitor, Parallel- and Serial- port emulation.
|30 Secs Interval||Timer Events||Read Events||Write Events|
With these figures, we are able to start 500 Windows 7 VMs on one host and keep dom0 load average practically at zero (after the VMs have booted).
In a previous post, Dave Scott showed how easy it now is to install the core components of XenServer on a standard installation of CentOS, using xenserver-core. Since then, we have been hard at work adding features and functionality, and making it even smoother and easier to install from our repositories. However, since developers are a major part of the audience for xenserver-core, we have also paid a lot of attention to making it as smooth and easy as possible to build the packages for yourself.
On RPM-based distributions, the packages are built using a tool called mock. Mock builds each new package in a fresh chroot environment, ensuring that your system isn't affected by the by-products of building packages, and that missing dependencies in the package specifications don't go unnoticed. To install it on a RHEL/CentOS system then you will need to add the EPEL repositories. Here is a useful article for CentOS.
After adding EPEL, install and set up mock:
yum install -y mock rpm-build
Mock will refuse to run as root - you can run it using your own user account, or a special account you create for this purpose. To add this account to the mock group, type the following as root:
useradd -G mock <username>
su - <username>
With the prerequisites installed, building xenserver-core is a 4-step process:
Clone the xenserver-core repository. If you are planning to make changes, you should fork the repository on GitHub and clone that instead. To clone the main xenserver-core repository, type:
git clone https://github.com/xapi-project/xenserver-core.git
This configures mock and initializes an RPM repository in the RPMS directory, where the built packages will be stored. Enter the directory created by git (by default, it will be called "xenserver-core") and type:
Generate the makefile which will actually build the xenserver-core packages:
./makemake.py > Makefile
Start the build:
Building all the xenserver-core packages takes around 90 minutes on a reasonably fast machine, and requires a couple of gigabytes of free disk space. When the build finishes, you will find source packages in the SRPMS directory, and binaries in RPMS.
The most flexible way to install your packages is to copy newly-built RPM repository to a web server and configure YUM on your target machine to download packages from there. Assuming that you already have a web server set up, copy the RPMS and SRPMS directories to your server's DOCROOT directory.
To configure the target machine to use your new repository, create /etc/yum.repos.d/xapi.conf, filling in your repository's baseurl as appropriate:
name=CentOS-$releasever - xenserver-core
name=CentOS-$releasever - xenserver-core
After this, type yum install xenserver-core to install the packages from your new repository. Once the installation is complete, run the xenserver-install-wizard command to configure xenserver-core. Please note that, unless you were already running under Xen, you will have to reboot to complete the installation.
When you cloned the xenserver-core repository you may have noticed that it doesn't actually contain any software. Instead, it has a collection of RPM SPEC files, along with the build scripts we ran above. When you run make, it downloads each package's source code from the upstream repository, builds a source RPM, then uses mock to compile the binary RPM, as you can see in this snippet of build output:
If you run into any problems or bugs in xenserver-core's packaging or build scripts, please raise an issue on our GitHub issues page.
Following the instructions above will build xenserver-core for an RPM-based distribution such as Fedora or CentOS. What if you prefer Ubuntu or another Debian-based distribution? Try cloning the repository onto your Ubuntu machine and running the build there. The results may surprise you!
More updates coming soon...
As a follow up activity to the open sourcing of XenServer, Citrix is pleased to announce the open sourcing of its automated test platform, XenRT.
XenRT ("Xen Regression Test") is a test automation framework, written in Python, providing abstractions for the various components under test (pool, host, VM, storage, network etc). The library code which makes up these abstractions simplifies the process of writing tests, allowing quite complex operations to be performed in a single method call.
In a full deployment, XenRT handles all aspects of the testing process - it will schedule a test job onto a host, bootstrap it (via DHCP/PXE), install the build to be tested, carry out the testing, and collect all necessary logs for troubleshooting, without any user interaction required.
In addition to basic functional, regression, and stress testing, XenRT has suites of tests that are used for testing performance, scalability, and interoperability.
Within Citrix, XenRT is used with a distributed lab comprised of an extremely wide range of hardware, and is developed and maintained by a team of some 25 developers. Tests are also written and executed directly by the wider XenServer engineering team, in a true "Test-as-a-Service" platform - see this post on the Citrix blog for more information.
XenRT has been open sourced to leverage Citrix's experience and resources in test automation to help improve the quality of open source Xen and XenServer releases, to benefit the entire community.
To get started with XenRT, follow the links below to the code and a README document (which contains getting started instructions - further documentation will follow in the near future). For discussion a mailing list has been created - information about this can be found at https://lists.xenserver.org/sympa/info/xenrt-users.
Citrix XenServer is undergoing a transition to an open and transparent development model. This model will enable users, customers, partners, vendors, consultants and pretty much anyone with an interest in the future and direction of XenServer become active participants. The transition from a closed development model to an open one doesn’t take place overnight, and it doesn’t come without its share of challenges. As we work towards completing this transition and fulfilling the promise of a vibrant open source project, this page will contain the current status of our efforts, and the next major milestones.
In traditional product management there is always discussion about “roadmaps”. Since we’re transitioning from closed development practices there will undoubtedly be those who wish to see a roadmap for XenServer. I hate to disappoint, but we don’t have one. Instead we have a development and project plan. The plan shows the current prioritization of feature development, and the current development status. Work is actively underway to make that public, but it will take a bit more time. Of course, that then leads people to conclude that an underlying roadmap with specific feature deliveries forms part of the plan. While true in theory, in open source development things are just a little bit different. With an open source project, it’s not the features that matter but rather support of those features.
Long time users of XenServer will be very familiar with the “Supported”, “Experimental” and “Unsupported” labels given to features, and that model will continue as we move forward. In fact what distinguishes a release is the date of supportability and the support labels given to features. For those unfamiliar with these labels, and to refresh for those who are, here is how we define them for those people who have software maintenance with Citrix:
In practice this means that until something has official support, the support burden will be borne by the community. Once a release occurs, then certain features will potentially become supported. So what community members can expect is that when we release we will clearly state which features are Supported or Experimental and anything which has no specification can be assumed Unsupported with the support burden remaining with the community itself. Further with each release will also come a date when support or maintenance won’t be available. As with all shipping and supported Citrix products, this is listed in the product lifecycle matrix.
So with that as the backdrop, here are the next items on our agenda:
Major project activity: Publish project plan through next release cycle
Next release cycle: Q1 2014
The Xen Project has announced the sessions for the 2013 Xen Project User Summit to be held in conjunction with Linuxcon North America next month in New Orleans. This is the first time that the the Xen hypervisor team will be having a conference focused entirely on the users of the Xen hypervisor. The Xen Project develops the Xen Hypervisor and XAPI toolstack, both core components of XenServer.
Jay Williams, VP of Product Management for CA Technologies, will deliver the keynote address. He will explain why Xen's strengths are critical for powering CA AppLogic and platforms like OpenStack.
Greg Kroah-Hartman, maintainer of the stable branch of the Linux kernel (among a mass of other things), will discuss how using kexec in a paravirtualized user domain, with no changes to the control Domain or Xen itself, can allow you to boot your own kernel, no matter what the hosting provider is forcing you to run.
Attendees will also learn about:
Folks new to Xen will benefit from an introductory session, as well as a short session to explain the differences between Xen Project, XenServer, and XAPI. Everyone, from long-time Xen users to those just kicking the tires, will receive plenty of useful information.
The schedule is subject to change due to speaker availability. As the User Summit is co-located with LinuxCon North America and CloudOpen North America, you can find the full schedule with times and descriptions on the LinuxCon NA Schedule page.We hope to see you in New Orleans in September!
So XenServer is now open-source, what does that mean? I look at XenServer as two things: firstly a set of components selected and engineered to work together as a system; and secondly a Linux distribution to provide the base platform to host and execute those components. Of course these two things are tightly coupled because the choice of base Linux distro and the set of packages installed will be part of the story when engineering the system as a whole. However we want to make it such that XenServer's core components, the stuff that does all the virtualization, management, monitoring and so on, can be used on a variety of Linux distros. This means we need to cleanly separate the components from the base distro, e.g. making XenServer components work with any reasonable distro and avoiding making assumptions about particular versions etc.
Let's start with the core components and refer to them collectively as "xenserver-core" (e.g. that could be the name of the meta-package to install them all to a distro "yum install xenserver-core"/"apt-get install xenserver-core" as used in Dave Scott's recent tech preview). These components include the xapi tool stack, storage manager, network daemon and related tools, HA daemon, etc. A second group of core components includes Xen, the Linux kernel, libvirt and qemu; although considered core components it is desirable to be able to use existing distro versions where possible. With suitable package dependencies it should be possible to manage all of the above.
It's important to remember that many of the core components are derived from upstream projects. For example the xapi tool stack is part of the Linux Foundation's Xen Project but is consumed by XenServer, you could think of the XenServer version of xapi being a short term fork of the upstream code. In practice I don't expect to see much divergence between XenServer's xapi and the upstream xapi because it's the same people working on both and XenServer is the primary consumer of the project. For other components that are more widely used, such as qemu, libvirt and Xen I would expect short term divergence as critical features and fixes are ported to the version used by XenServer (just like Linux distros do) but with a rule that all required code is upstreamed to the relevant project to avoid long term divergence.
OK, we have xenserver-core which can now be installed on top of any reasonable Linux distribution. So when I talk about "XenServer" what do I mean? In general I mean the end result of both aspects of XenServer, the components and the base distro, all wrapped up in an ISO with an installer of some kind. This "appliance" model is how XenServer has been for years and provides a turn-key virtualization platform that does not require Linux sysadmin experience to install. This means we start with xenserver-core packages, choose a particular base distribution and set of packages from it and glue the whole lot together somehow. In a sense this is a distro-customization exercise. XenServer's build system has been doing this since day one albeit in a rather more complex way than described above. As part of the open-sourcing of XenServer we need to clean up this packaging and assembly phase by using standard tools and methods to take the core components and an off-the-shelf Linux distro and put the two together. This tooling, and all the configuration management (which versions of which packages etc.) will become part of the xenserver.org project.
What does Citrix actually release? Citrix XenServer is a particular instance of the XenServer appliance built, packaged, assembled, tested, warranted and certified by Citrix. It is only Citrix XenServer that can be supported by Citrix (remember that Citrix XenServer is free, anyone can use it without paying, but to get support and maintenance a package can be purchased from Citrix).
If you want XenServer you have some choices of how to get it (once all the necessary pieces are published of course):
One of the most noteworthy improvements in XenServer 6.2 is the support for a significantly increased number of VMs running on a host: now up to 500 Windows VMs or 650 Linux VMs.
We needed to remove several obstacles in order to achieve this huge step up. Perhaps the most important of the technical changes that led to this was to increase in the number of event channels available to dom0 (the control domain) from 1024 to 4096. This blog post is an attempt to shed some light on what these event channels are, and why they play a key role in VM density limits.
What is an event channel?
It's a channel for communications between a pair of VMs. An event channel is typically used by one VM to notify another VM about something. For example, a VM's paravirtualised disk driver would use an event channel to notify dom0 of the presence of newly written data in a region of memory shared with dom0.
Here are the various things that a VM requires an event channel for:
Therefore VMs will typically require at least four dom0 event channels depending on the configuration of the VM. Requiring more than ten is not an uncommon configuration.
Why can event channels cause scalability problems when trying to run lots of VMs?
The total number of event channels any domain can use is part of a shared structure in the interface between a paravirtualised VM and the hypervisor; it is fixed at 1024 for 32-bit domains such as XenServer's dom0. Moreover, there are normally around 50--100 event channels used for other purposes, such as physical interrupts. This is normally related to the number of physical devices you have in your host. This overhead means that in practice there might be not too many more than 900--950 event channels available for VM use. So the number of available event channels becomes a limited resource that can cause you to experience a hard limit on the number of VMs you can run on a host.
To take an example: Before XenServer 6.2, if each of your VMs requires 6 dom0 event channels (e.g. an HVM guest with 3 virtual disks, 1 virtual network interface and 1 virtual CPU) then you'll probably find yourself running out of dom0 event channels if you go much over 150 VMs.
In XenServer 6.2, we have made a special case for our dom0 to allow it to behave differently to other 32-bit domains to allow it to use up to four times the normal number of event channels. Hence there are now a total of 4096 event channels available.
So, on XenServer 6.2 in the same scenario as the example above, even though each VM of this type would now use 7 dom0 event channels, the increased total number of dom0 event channels means you'd have to run over 570 of them before running out.
What happens when I run out of event channels?
On VM startup, the XenServer toolstack will try to plumb all the event channels through from dom0 to the nascent VM. If there are no spare slots, the connection will fail. The exact failure mode depends on which subsystem the event channel was intended for use in, but you may see error messages like these when the toolstack tries to connect up the next event channel after having run out:
error 28 mapping ring-refs and evtchn
message: xenopsd internal error: Device.Ioemu_failed("qemu-dm exited unexpectedly")
In other words, it's not pretty. The VM either won't boot or will run with reduced functionality.
That sounds scary. How can I tell whether there's sufficient spare event channels to start another VM?
XenServer has a utility called "lsevtchn" that allows you to inspect the event channel plumbing.
In dom0, run the following command to see what event channels are connected to a particular domain.
For example, here is the output from a PV domain with domid 36:
[root@xs62 ~]# /usr/lib/xen/bin/lsevtchn 36
1: VCPU 0: Interdomain (Connected) - Remote Domain 0, Port 51
2: VCPU 0: Interdomain (Connected) - Remote Domain 0, Port 52
3: VCPU 0: Virtual IRQ 0
4: VCPU 0: IPI
5: VCPU 0: IPI
6: VCPU 0: Virtual IRQ 1
7: VCPU 0: IPI
8: VCPU 0: Interdomain (Connected) - Remote Domain 0, Port 55
9: VCPU 0: Interdomain (Connected) - Remote Domain 0, Port 53
10: VCPU 0: Interdomain (Connected) - Remote Domain 0, Port 54
11: VCPU 0: Interdomain (Connected) - Remote Domain 0, Port 56
You can see that six of this VM's event channels are connected to dom0.
But the domain we are most interested in is dom0. The total number of event channels connected to dom0 can be determined by running
/usr/lib/xen/bin/lsevtchn 0 | wc -l
Before XenServer 6.2, if that number is close to 1024 then your host is on the verge of not being able to run an additional VM. On XenServer 6.2, the number to watch out for is 4096. However, before you'd be able to get enough VMs up and running to approach that limit, there are various other things you might run into depending on configuration and workload. Watch out for further blog posts describing how we have cleared more of these hurdles in XenServer 6.2.
One of the benefits of making XenServer fully open-source is that it’s easier to integrate directly with other great open-source projects. A project that I’m particularly interested in is Ceph: a distributed storage system which is particularly suitable for storing VM disk images in the cloud. The Ceph community has already integrated support directly into two other open-source projects:
All that’s needed to make Ceph work with XenServer is therefore:
After much coffee-fuelled late-night hacking^Wsoftware development, I’m pleased to announce the availability of a “technology preview” of XenServer integrated with Ceph via libvirt. To give it a go, first install yourself a normal CentOS 6.4 x86_64 system. Second login as root and type:
rpm -ihv http://xenbits.xen.org/djs/xenserver-tech-preview-release-0.3.0-0.x86_64.rpm
This will add the experimental preview software repository based on the Xen code already in CentOS. Next type:
yum install xenserver-core
-- notice how easy it is to install the core packages of XenServer using the normal CentOS distro tools. Now that XenServer is fully open-source, expect to see more like this in future!
It’s now necessary to set up a basic XenServer configuration. The easiest way is to use a simple install “wizard” by typing:
After the wizard has done its magic and you’ve rebooted, you should be able to connect XenCenter like you would with a regular XenServer.
Assuming you have already configured a Ceph cluster, you need to configure your machine to be a Ceph client, as you would with a regular CentOS host. I recommend having a read of the excellent Ceph documentation. Once complete you should be able to list the currently available Ceph storage pools:
ceph osd lspools
Next you should create a XenServer “Storage Repository” to allow VM virtual disks to be stored on Ceph-- this is where libvirt comes in. In dom0, create a libvirt “storage pool” XML file, as if you were going to issue a “virsh pool-create”. I created a file “ceph.xml” which looks like the following:
<pool type='rbd'> <name>ceph</name> <source> <name>rbd</name> <host name='10.80.237.208' port='6789'/> </source> </pool>
xe sr-create type=libvirt name-label=ceph device-config:xml-filename=ceph.xml
You should now have a functioning XenServer Storage Repository which can be managed via the XenAPI and from XenCenter. At this point you should be able to install VMs (both PV and HVM should work) and run them from the Ceph storage.
For more in-depth info about how it all works, known issues and ways you could get involved, have a look at the:
Today I presented at the CloudStack Collaboration Conference 2013 in Santa Clara on evolving XenServer to better meet the needs of large-scale cloud deployments. XenServer started life as "XenEnterprise" in 2006 and was aimed at SMBs and Enterprises and targeted Windows IT pros who may not have Linux experience. Therefore a lot of the Linux structure of XenServer was hidden. With many cloud shops being Linux shops this hiding is a barrier; for example XenServer's non-standard installation model doesn't fit well with large scale server deployment and management tools like Chef and Puppet - XenServer is built on a standard Linux OS so why can't we deploy lit like a standard Linux OS?
I outlined some of the initiatives currently underway within Citrix that will continue under the xenserver.org project:
Hyperspace: fixing up the build and packaging to allow individual components to be easily built without complex build system requirements. Making XenServer be a "Linux distro+packages" where we take an off-the-shelf Linux distro, add the XenServer components as packages, and bundle the whole thing as XenServer.
Fusion (I called it "Project Upstream" in the presentation to avoid conflict with a commercial product name): getting libvirt and upstream qemu into XenServer.
Windsor: making XenServer more modular, exploiting dom0 disaggregation.
As we move more of XenServer project planning and development to xenserver.org I look forward to more discussion here on these initiatives and the wider topic of making XenServer the best platform for the cloud.
You can find a copy of my slides at CCC13_EvolvingXenServerForTheCloud_JamesBulpin_20130625.pdf
A video of the talk should be up on buildacloud.org soon.
This is a big day for the XenServer product as version 6.2 is now available, it is now not only a product supported by Citrix but also becomes a fully open source project that invites participation from the user community. Citrix will still provide a XenServer product which includes support for those customers who chose to purchase a support license.
Much of XenServer already was open source, leveraging packages from the Xen Project, Linux kernel and the XAPI Project. We believe that open source plays a strategic role in the future of virtualization and cloud technology and that only open source offers the opportunities for collaborative, open innovation and the economies of scale that these markets demand. By open sourcing XenServer, customers, partners and developers gain full public visibility into the ongoing development and future of XenServer and can directly engage with us to contribute new XenServer functionality, build deeper integrations and steer the architectural direction of the platform. In summary, an open source strategy was chosen because:
With the change to open source, XenServer is now available for free to everyone on the new XenServer.org community. Citrix will provide support for a distribution of XenServer. This is very similar to leading open source enterprise products like Red Hat Enterprise Linux verses say CentOS Linux. By purchasing Citrix XenServer support licenses you get:
So this is just the beginning of XenServer.org. We'll begin by releasing the source code and providing binaries for users to download and install. For now we'll provide mailing lists and still use the existing forums provided by Citrix. Over time we'll transition them to this site. Additionally, we'll add all the things that you would expect from an open source project including a way to track bugs, request features and interact with the developers to collaborate on the code. We also include a number of ways to ask questions and get support via the mailing lists, forums and our Q&A system.