Virtualization Blog

Discussions and observations on virtualization.

A New Year, A New Way to Build for XenServer

Building bits of XenServer outside of Citrix has in the past been a bit of a challenging task, requiring careful construction of the build environment to replicate what 'XenBuilder', our internal build system, puts together. This has meant using custom DDK VMs or carefully installing by hand a set of packages taken from one of the XenServer ISOs. With XenServer Dundee, this will be a pain of the past, and making a build environment will be just a 'docker run' away.

Part of the work that's being done for XenServer Dundee has been moving things over to using standard build tools and packaging. In previous releases there have been a mix of RPMs, tarballs and patches for existing files, but for the Dundee project everything installed into dom0 is now packaged into an RPM. Taking inspiration and knowledge gained while working on xenserver/buildroot, we're building most of these dom0 packages now using mock. Mock is a standard tool for building RPM packages from source RPMs (SRPMS), and it works by constructing a completely clean chroot with only the dependencies defined by the SRPM. This means that everything needed to build a package must be in an RPM, and the dependencies defined by the SRPM must be correct too.

From the point of view of making reliably reproducible builds, using mock means there is very little possibility of having the build dependent upon the the environment. But there is also a side benefit of this work: If you actually want to rebuild a bit of XenServer you just need to have a yum repository with the XenServer RPMs in, and use 'yum-builddep' to put in place all of the build dependencies, and then building should be as simple as cloning the repository and typing 'make'.

The simplest place to do this would be in the dom0 environment itself, particularly now that the partition size has been bumped up to 20 gigs or so. However, that may well not be the most convenient. In fact, for a use case like this, the mighty Docker provides a perfect solution. Docker can quickly pull down a standard CentOS environment and then put in the reference to the XenServer yum repository, install gcc, OCaml, git, emacs and generally prepare the perfect build environment for development.

In fact, even better, Docker will actually do all of these bits for you! The docker hub has a facility for automatically building a Docker image provided everything required is in repository on Github. So we've prepared a repository containing a Dockerfile and associated gubbins that sets things up as above, and then the docker hub builds and hosts the resulting docker image.

Let's dive in with an example on how to use this. Say you have a desire to change some aspect of how networking works on XenServer, something that requires a change to the networking daemon itself, 'xcp-networkd'. We'll start by rebuilding that from the source RPM. Start the docker container and install the build dependencies:

$ docker run -i -t xenserver/xenserver-build-env
[root@15729a23550b /]# yum-builddep -y xcp-networkd

this will now download and install everything required to be able to build the network daemon. Next, let's just download and build the SRPM:

[root@15729a23550b /]# yumdownloader --source xcp-networkd

At time of writing, this downloads the SRPM "xcp-networkd-0.9.6-1+s0+0.10.0+8+g96c3fcc.el7.centos.src.rpm". This will build correctly in our environment:

[root@15729a23550b /]# rpmbuild --rebuild xcp-networkd-*
[root@15729a23550b /]# ls -l ~/rpmbuild/RPMS/x86_64/
total 2488
-rw-r--r-- 1 root root 1938536 Jan  7 11:15 xcp-networkd-0.9.6-1+s0+0.10.0+8+g96c3fcc.el7.centos.x86_64.rpm
-rw-r--r-- 1 root root  604440 Jan  7 11:15 xcp-networkd-debuginfo-0.9.6-1+s0+0.10.0+8+g96c3fcc.el7.centos.x86_64.rpm

To patch this, it's just the same as for CentOS, Fedora, and any other RPM based distro, so follow one of the many guides available.

Alternatively, you can compile straight from the source. Most of our software is hosted on github, either under the xapi-project or xenserver organisations. xcp-networkd is a xapi-project repository, so we can clone it from there:

[root@15729a23550b /]# cd ~
[root@15729a23550b ~]# git clone git://

Most of our RPMs have version numbers constructed automatically containing useful information about the source, and where the source is from git repositories the version information comes from 'git describe'.

[root@15729a23550b ~]# cd xcp-networkd
[root@15729a23550b xcp-networkd]# git describe --tags

The important part here is the hash, in this case '96c3fcc'. Comparing with the SRPM version, we can see these are identical. We can now just type 'make' to build the binaries:

[root@15729a23550b xcp-networkd]# make

this networkd binary can then be put onto your XenServer and run.

The yum repository used by the container is being created directly from the snapshot ISOs uploaded to, using a simple bash script named available on github. The container default will be to use the most recently available release, but the script can be used by anyone to generate a repository from the daily snapshots too, if this is required. There’s still a way to go before Dundee is released, and some aspect of this workflow are in flux – for example, the RPMs aren’t currently signed. However, by the time Dundee is out the door we hope to make many improvements in this area. Certainly here in Citrix, many of us have switched to using this for our day-to-day build needs, because it's simply far more convenient than our old custom chroot generation mechanism.

Recent comment in this post
Shawn Edwards is down, so this method of developing for xenserver currently doesn't work. Who do I need to bug to get this... Read More
Thursday, 02 June 2016 22:07
Continue reading
8653 Hits
1 Comment

XenServer Dundee Alpha.3 Available

The XenServer team is pleased to announce the availability of the third alpha release in the Dundee release train. This release includes a number of performance oriented items and includes three new functional areas.

  • Microsoft Windows 10 driver support is now present in the XenServer tools. The tools have yet to be WHQL certified and are not yet working for GPU use cases, but users can safely use them to validate Windows 10 support.
  • FCoE storage support has been enabled for the Linux Bridge network stack. Note that the default network stack is OVS, so users wishing to test FCoE will need to convert the network stack to Bridge and will need to be aware of the feature limitations in Bridge relative to OVS.
  • Docker support present in XenServer 6.5 SP1 is now also present in Dundee

Considerable work has been performed to improve overall I/O throughput on larger systems and improve system responsiveness under heavy load. As part of this work, the number of vCPUs available to dom0 have been increased on systems with more than 8 pCPUs. Early results indicate a significant improvement in throughput compared to Creedence. We are particularly interested in hearing from users who have previously experienced responsiveness or I/O bottlenecks to look at Alpha.3 and provide their observations.

Dundee alpha.3 can be downloaded from the pre-release download page.     

Recent Comments
Sam McLeod
Hi Tim, Great to hear about the IO! As you know, we're very IO intensive and have very high speed flash-only iSCSI storage. I'll ... Read More
Thursday, 20 August 2015 01:50
Tim Mackey
Excellent, Sam. I look forward to hearing if you feel we've moved in the right direction and by how much.
Thursday, 20 August 2015 02:08
Chris Apsey
Has any testing been done with 40gbps NICS? We are considering upgrading to Chelsio T-580-LP-CRs, and if the ~24gbps barrier has ... Read More
Thursday, 20 August 2015 19:25
Continue reading
17816 Hits

XenServer 6.5 SP1 Released

Wait, another XenServer release? Yes folks, there is no question we've been very busy improving upon XenServer over the past year, and the pace is quite fast. In case you missed it, we released XenServer 6.5 in January (formerly known as Creedence). Just a few weeks ago I announced and made available pre-release binaries for Dundee, and now we've just announced availability at Citrix Synergy of the first service pack for XenServer 6.5. Exciting times indeed.

What's in XenServer 6.5 SP1

I could bury the lead talk about hot fixes and roll-ups (more on that later), but the real value for SP1 is in the increased capabilities. Here are the lead items for this service pack:

  1. The Docker work we previewed in January at FOSDEM and later on is now available. If you've been using xscontainer in preview form, it should upgrade fine, but you should back up any VMs first. Completion of the Docker work also implies that CoreOS 633.1.0 is also an officially supported operating system with SP1. Containers deployed in Unbuntu 14.04 and RHEL, CentOS, and Oracle Enterprise Linux 7 and higher are supported.
  2. Adoption of LTS (long term support) guest support. XenServer guest support has historically required users of guest operating system to wait for XenServer to adopt official support for point releases in order to remain in a supported configuration. Starting with SP1, all supported operating systems can be upgraded within their major version and still retain "supported" status from Citrix support. For example, if a CentOS 6.6 VM is deployed, and the CentOS project subsequently releases CentOS 6.7, then upgrading that VM to CentOS 6.7 requires no changes to XenServer in order to remain a supported configuration.
  3. Intel GVT-d support for GPU pass through for Windows guests. This allows users of Xeon E3 Haswell processors to use the embedded GPU in those processors within a Windows guest using standard Intel graphics drivers.
  4. NVIDIA GPU pass though to Linux VMs allowing OpenGL and CUDA support for these operating systems.
  5. Installation of supplemental packs can now be performed through XenCenter Update. Note that since driver disks are only a special case of a supplemental pack, driver updates or installation of drivers not required for host installation can now also be performed using this mechanism
  6. Virtual machine density has been increased to 1000. What this means is that if you have a server which can reasonably be expected to run 1000 VMs of a given operating system, then using XenServer you can do so. No changes were made to the supported hardware configuration to accommodate this change.

Hotfix process

As with all XenServer service packs, XenServer 6.5 SP1 contains a rollup of all existing hot fixes for XenServer 6.5. This means that when provisioning a new host, your first post-installation step should be to apply SP1. It's also important to call out that when a service pack is released, hotfixes for the prior service pack level will no longer be created within six months. In this case, hotfixes for XenServer 6.5 will only be created through November 12th and following that point hotfixes will only be created for XenServer 6.5 SP1. In order for the development teams to streamline that transition, any defects raised for XenServer 6.5 in should be raised against 6.5 SP1 and not base 6.5.

Where to get XenServer 6.5 SP1


Downloading XenServer 6.5 SP1 is very easy, simply go to and download it!     

Recent Comments
Tobias Kreidl
Very nice! One quick question: It we want to work with both XS 6.5 SP1 and the technical pre-release Dundee version, is there or w... Read More
Tuesday, 12 May 2015 18:28
Tobias Kreidl
Found out these are 99% compatible (thanks, Andy!), so either should work fine. The Dundee XenCenter release shows a slightly newe... Read More
Wednesday, 13 May 2015 09:39
Stephen Turner
Hi, Tobias. Dundee XenCenter already includes all the changes in 6.5SP1, so you can just use that one to manage everything (everyt... Read More
Wednesday, 13 May 2015 08:23
Continue reading
32134 Hits

Is it really “containers vs. VMs”?

There are some in the Docker and container world that believe that there is some kind of competition between Docker and hypervisors; they would have us believe that containers render VMs, and therefore hypervisors, redundant. Is that really true? I think not. I believe that containers and VMs perform complementary roles and add value to each other.

Let's look at what VMs and containers are really all about. Firstly let's consider what they have in common: they can both be used to encapsulate an application and therefore both use images containing the application, its libraries and other runtime dependencies (in fact you could argue that a Docker image is conceptually just a VM image without a kernel and init scripts). Hypervisor vendors have been telling us for years to have just one application per OS instance, that's the normal model with AWS AMIs too – again, this looks just like a Docker image.

But that's a top down, application-centric view. Let's now look at it from the infrastructure perspective. The boundary of a container is the boundary of an application, the separation between the internal workings of the applications and its external interface. The boundary of a VM is the boundary of the resource allocation, ownership, trust and availability of a piece of abstracted infrastructure.

By separating these application and infrastructure boundaries we get more flexibility than if a single entity tries to implement both:

  • I can put multiple application containers within one VM where they share the same level of trust and only have to worry about protecting the trust boundary once, rather than multiple times for individual applications. VMs' trust and ownership boundaries have long been used to provide multi-tenancy – this isn't just important in public clouds but matters for enterprises that increasingly see applications being provided by individual departments or individual employees.
  • Applications often work together with other applications, that's why Docker has inter-container communication mechanisms such as "links". I can use the application container to keep each app nicely encapsulated and I can use the VM boundary to put a hard shell around the set of cooperating applications. I can also use this VM boundary to define the unit of resource accounting and reporting.
  • I can put cooperating application containers in a VM to share a common availability boundary; if they're working together then I probably want them to fail and succeed together. Resource isolation boundaries are good for containing faults – I'd rather have the "blast radius" of a faulty container being the VM which contains that container and its collaborating applications rather than an entire server.

So am I arguing that VMs are better than containers? Absolutely not. I believe that both mechanisms have a valuable part to play in the deployment of scalable, efficient, secure and flexible systems. That's why we're looking at ways to enhance XenServer to make it a great platform for running containers within VMs. Our recent preview of Docker integration is just the start. As well as requests to support other Docker-optimized Linux distributions (the preview supports CoreOS) we heard that you want to see infrastructure level information made available to higher level management tools for audit and reporting. Stay tuned for more.

Recent Comments
Tobias Kreidl
What about Microsoft's embracing of Docker and its container deployment "Hyper-V Containers" plus its "Nano Server" -- how will th... Read More
Friday, 10 April 2015 15:50
James Bulpin
Should mesh well. One of XenServer's strengths is it's equally happy with both Windows and Linux workloads - should extend nicely ... Read More
Friday, 10 April 2015 16:10
Continue reading
13126 Hits

Preview of XenServer support for Docker and Container Management

I'm excited to be able to share with you a preview of our new XenServer support for Docker and Container Management. Downloads can be found on the preview page, read on for installation instructions and more details.

Today many Docker applications run in containers within VMs hosted on hypervisors such as XenServer and other distributions of Xen. The synergy between containers as an application isolation mechanism and hypervisors as a secure physical infrastructure virtualization mechanism is something that I'll be blogging more about in the future. I firmly believe that these two technologies add value to each other, especially if they are aware of each other and designed to work together for an even better result.

That's why we've been looking at how we can enhance XenServer to be a great platform for Docker applications and how we can contribute to the Docker ecosystem to best leverage the capabilities and services available from the hypervisor. As a first step in this initiative I'm pleased to announce a preview of our new XenServer support for Docker applications. Those who attended Citrix Summit in January or FOSDEM in February may have seen an earlier version of this support being demo'd.

The preview is designed to work on top of XenServer 6.5 and comes in two parts: a supplemental pack for the servers and a build of XenCenter with the UI changes. XenCenter is installed in the normal Windows manner. The supplemental pack is installed in the same way as other XenServer supp-packs by copying the ISO file to each server in the pool and executing the following command in domain 0:

xe-install-supplemental-pack xscontainer-6.5.0-100205c.iso
mount: xscontainer-6.5.0-100205c.iso is write-protected, mounting read-only
Installing 'XenServer Container Management'...

Preparing...                ########################################### [100%]
   1:guest-templates        ########################################### [ 50%]
Waiting for xapi to signal init complete
Removing any existing built-in templates
Regenerating built-in templates
   2:xscontainer            ########################################### [100%]
Pack installation successful.

So what do you get with this preview? First off you get support for running CoreOS Linux VMs - CoreOS is a minimal Linux distribution popular for hosting Docker apps. The XenCenter VM installation wizard now includes a template for CoreOS and additional dialogs for setting the VM up (that's setting up a cloud config drive under the hood). This process also prepares the VM to be managed, to enable the main part of the preview's functionality to interact with it.


Secondly, and most importantly, XenServer becomes aware of “Container managed” VMs running Docker containers. It queries the VMs to enumerate the application containers running on each and then displays these within XenCenter's infrastructure view. XenCenter also allows interaction with the containers to start, stop and pause them. We want XenServer to be a platform for Docker and complement, not replace, the core part of the Docker application ecosystem, and therefore we expect that the individual Docker Engine instances in the VMs will be managed by one of the many Docker management tools such as Kubernetes, Docker Compose or ShipYard.


So what can you do with this preview?

Monitoring and visibility - knowing which VMs are in use for Docker hosting and which containers on them are actually running. Today's interface is more of a "pets" than "cattle" one but we've got experience  in showing what's going on at greater scale.

Diagnostics - easy access to basic container information such as forwarded network ports and originating Docker image name. This can help accelerate investigations into problems where either or both of the infrastructure and application layers may be implicated. Going forward we’d like to also provide easy access to the container-console.

Performance - spotted a VM that's using a lot of resource? This functionality allows you to see which containers are running on that VM, what processes run inside, how much CPU time each consumed, to help identify the one consuming the resource. In the future we'd like to add per-container resource usage reporting for correlation with the VM level metrics.

Control applications - using XenCenter you can start, stop and pause application containers. This feature has a number of use cases in both evaluation and deployment scenarios including rapidly terminating problematic applications.

We'd love to hear your feedback on this preview: what was useful, what wasn't? What would you like to see that wasn't there? Did you encounter problems or bugs? Please can share your feedback using our normal preview feedback mechanism by creating a ticket in the "XenServer Org" (XSO) project at

This preview is a first step towards a much richer Docker-XenServer mutual awareness and optimization to help bridge the gap between the worlds of the infrastructure administrator and the application developer/administrator. This is just the beginning, we expect to be improving, extending and enhancing the overall XenServer-Docker experience beyond that. Look out for more blog posts one this topic...

For a detailed guide to using this preview please see this article.

Recent Comments
Thomas Subotitsch
Great news. Hope that API will also get commands for docker management.
Tuesday, 17 March 2015 07:41
Get this error when I try to start the Core-OS vm: Only 1 LUN may be used with shared OCFS I tried iSCSI SR and Local storage, s... Read More
Friday, 24 April 2015 19:23
James Bulpin
Slava: You need to use XenServer 6.5 "Creedence" for this preview. As the error message "Only 1 LUN may be used with shared OCFS" ... Read More
Monday, 27 April 2015 12:26
Continue reading
38796 Hits

About XenServer

XenServer is the leading open source virtualization platform, powered by the Xen Project hypervisor and the XAPI toolstack. It is used in the world's largest clouds and enterprises.
Commercial support for XenServer is available from Citrix.