Virtualization Blog

Discussions and observations on virtualization.

XenServer Dundee Beta 3 Available

I am pleased to announce that Dundee beta 3 has been released, and for those of you who monitor Citrix Tech Previews, beta 3 corresponds to the core XenServer platform used for Dundee TP3. This third beta marks a major development milestone representing the proverbial "feature complete" stage. Normally when announcing pre-release builds, I highlight major functional advances but this time I need to start with a feature which was removed.

Thin Provisioned Block Storage Removed

While its never great to start with a negative, I felt anything related to removal of a storage option takes priority over new and shiny. I'm going to keep this section short, and also highlight that only the new thin provisioned block feature was removed and existing thin provisioned NFS and file based storage repositories will function as they've always done.

What should I do before upgrading to beta 3?

While we don't actively encourage upgrades to pre-release software, we do recognize you're likely to do it at least once. If you have built out infrastructure using thin provisioned iSCSI or HBA storage using a previous pre-release of Dundee, please ensure you migrate any critical VMs to either local storage, NFS or thick provisioned block storage prior to performing an upgrade to beta 3.

So what happened?

As is occasionally the case with pre-release software not all features which are previewed will make it to the final release; for any of a variety of reasons. That is of course one reason we provide pre-release access. In the case of the thin provisioned block storage implementation present in earlier Dundee betas, we ultimately found that it had issues under performance stress. As a result, we've made the difficult decision to remove it from Dundee at this time. Investigation into alternative implementations are underway, and the team is preparing a more detailed blog on future directions.

Beta 3 Overview

Much of difference between beta 2 and beta 3 can be found in some of the details. dom0 has been updated to a CentOS 7.2 userspace, the Xen project hypervisor is now 4.6.1 and the kernel is 3.10.96. Support for xsave and xrestor floating point instructions has been added, enabling guest VMs to utilize AVX instructions available on newer Intel processors. We've also added experimental support for the Microsoft Windows Server 2016 Tech Preview and the Ubuntu 16.04 beta.

Beta 3 Bug Fixes

Earlier pre-releases of Dundee had an issue wherein performing a storage migration of a VM with snapshots and in particular orphaned snapshots would result in migration errors. Work has been done to resolve this, and it would be beneficial for anyone taking beta 3 to exercise storage motion to validate if the fix is complete.

One of the focus areas for Dundee is to improve scalability, and as part of that we've uncovered some situations where overall reliability wasn't what we wanted. An example of such a situation, which we've resolved, occurs when a VM with a very large number of VBDs is running on a host, and a XenServer admin requests the host to shutdown. Prior to the fix, such a host would become unresponsive.

The default logrotate for xensource.log has been changed to rotate at a 100MB in addition to daily. This change was done as on very active systems excessive disk consumption could result in the prior configuration.

Download Information

You can download Dundee beta.3 from the Preview Download page (, and any issues found can be reported in our defect database (     

Recent Comments
Tobias Kreidl
Given the importance of thin provisioning for block-based storage devices, I'm sure that Citrix will find a solution. It takes mak... Read More
Wednesday, 16 March 2016 03:08
Tim Mackey
@Tobias Thanks. We haven't given up on thin provisioned block at all. We're already down a different path which is looking promis... Read More
Wednesday, 16 March 2016 13:32
Gianni D
Not sure if this was by design but the Beta 3 XenCenter won't allow vm migration on older versions of XenServer. Option doesn't e... Read More
Wednesday, 16 March 2016 04:26
Continue reading
12239 Hits


2015 刚刚结束, 2016新年开始,正是我们向XenServer社区发节日礼物的好时节。我们发布了下一代XenServer产品Dundee预览版2。我们花了大量精力解决已上报的各种问题,也导致该测试版本和九月份的预览版1相比有些延迟。我们确信该预览版和Steve Wilson博客 ( 中对思杰在XenServer项目贡献的肯定是很好的新年礼物。此外,我们已开始在xenserver.org网站提供一系列博客把主要改进点做深度介绍。对于那些更关注该预览版亮点的朋友,现在就让我在下面为您做一一介绍。




当代的服务器致力于增强其能力,我们不仅要与时俱进,更要确保用户能创建真正反应物理服务器能力的虚拟机。Dundee预览版2 目前支持512个物理CPU(pCPU),可创建高达1.5TB内存的用户虚拟机。您也许会问是否考虑增加虚拟CPU上限,我们已经把上限扩大到32个。在Xen项目的管理程序中,我们已经默认支持PML(Page Modification Logging)。 对PML设计的详细信息已经发布在Xen项目归档中等待审核。最后,XenServer的Dom0内核版本已经升级到3.10.93。


SUSE为其企业服务器版SLES(SUSE Linux Enterprise Server)和企业桌面版SLED(SUSE Linux Enterprise Desktop)发布了version 12 SP1,这两者在Dundee中都已得到支持。


自从Dundee预览版1在九月下旬发布后,若干安全相关的热补丁已经在XenServer 6.5 SP1中发布。 同样的补丁也应用到了Dundee并且已经包含在预览版2中。


您可以在预览页 下载Dundee预览版2,任何发现的问题都欢迎上报到我们的故障库。

Recent Comments
Tim Mackey
Evgeny, The original English version can be found here: Read More
Saturday, 09 January 2016 13:29
Sunday, 10 January 2016 14:22
Continue reading
6933 Hits

XenServer Dundee Beta.2 Available

With 2015 quickly coming to a close, and 2016 beckoning, it's time to deliver a holiday present to the XenServer community. Today, we've released beta 2 of project Dundee. While the lag between beta 1 in September and today has been a bit longer than many would've liked, part of that lag was due to the effort involved in resolving many of the issues reported. The team is confident you'll find both beta 2 and Steve Wilson's blog affirming Citrix's commitment to XenServer to be a nice gift. As part of that gift, we're planning to have a series of blogs covering a few of the major improvements in depth, but for those of you who like the highlights - let's jump right in!

CPU leveling

XenServer has supported for many years the ability to create resource pools with processors from different CPU generations, but a few years back a change was made with Intel CPUs which impacted our ability mix the newest CPUs with much older ones. The good news is that with Dundee beta.2, that situation should be fully resolved, and may indeed offer some performance improvements. Since this is an area where we really need to get things absolutely correct, we'd appreciate anyone running Dundee to try this out if you can and report back on successes and issues.

Increased scalability

Modern servers keep increasing their capacity, and not only do we need to keep pace, but we need to ensure users can create VMs which mirror the capacity of a physical machines. Dundee beta.2 now supports up to 512 physical cores (pCPUs), and can create guest VMs with up to 1.5 TB RAM. Some of you might ask about increasing vCPU limits, and we've bumped those up to 32 as well. We've also enabled Page Modification Logging (PML) in the Xen Project hypervisor as a default. The full design details for PML are posted in the Xen Project Archives for review if you'd like to get into the weeds of why this is valuable. Lastly we've bumped the kernel version to 3.10.93.

New SUSE templates

SUSE have released version 12 SP1 for both (SUSE Linux Enterprise Server) SLES and (SUSE Linux Enterprise Desktop) SLED, both of which are now supported templates in Dundee.

Security updates

Since Dundee beta.1 was made available in late September, a number of security hotfixes for XenServer 6.5 SP1 have been released. Where valid, those same security patches have been applied to Dundee and are included in beta.2.

Download Information

You can download Dundee beta.2 from the Preview Download page (, and any issues found can be reported in our defect database (     

Recent Comments
Tobias Kreidl
Great news, Tim! Is it correct the 32 VCPUs are now supported for both Linux and Windows guests (where of course supported by the ... Read More
Tuesday, 22 December 2015 04:43
Tim Mackey
Thanks, Tobias. This would be 32 vCPUS for both PV and HVM. I *think* we're at 254 VDIs in Dundee, but with the caveat things mi... Read More
Thursday, 24 December 2015 01:59
Tim Mackey
N3ST, We never commit to a stable upgrade path from any preview version to a final release. It's been known to work in previous ... Read More
Thursday, 24 December 2015 02:02
Continue reading
20024 Hits

XenServer Dundee Beta 1 Available

We are very pleased to make the first beta of XenServer Dundee available to the community. As with all pre-release downloads, this can be found on the XenServer Preview page. This release does include some potential commercial features, and if you are an existing Citrix customer you can access those features using the XenServer Tech Preview. It's also important to note that a XenServer host installed from the installer obtained from either source will have identical version number and identical functionality. Application of a Tech Preview license unlocks the potential commercial functionality. So with the "where do I get Dundee beta 1" out of the way, I bet you're all interested in what the cool bits are, and what things might be worth paying attention to. With that in mind, here are some of the important platform differences between XenServer 6.5 SP1 and Dundee beta 1.

Updated dom0

The control domain, dom0, has undergone some significant changes. Last year we moved to a 64 bit control domain with a 3.10 kernel as part of our effort to increase overall performance and scalability. That change allowed us to increase VM density to 1000 VMs per host while making some significant gains in both storage and network performance. The dom0 improvements continue, and could have a direct impact on how you manage a XenServer.

CentOS 7

dom0 now uses CentOS 7 as it's core operating system, and along with that change is a significant change in how "agents" and some scripts run. CentOS 7 has adopted systemd, and by extension so too has XenServer. This means that shell scripts started at system initialization time will need to change to follow the unit and service definition model specified for systemd.

cgroups for Control Isolation

Certain xapi processes have been isolated into Linux control groups. This allows for greater system stability under extreme resource pressure which has created a considerably more deterministic model for VM operations. The most notable area where this can be observed is under bootstorm conditions. In XenServer 6.5 and prior, starting large numbers of VMs could result in start operations being blocked due to resource contention which could result in some VMs taking significantly longer to start than others. With xapi isolation into cgroups, VM start operations no longer block as before resulting in VM start times being much more equitable. This same optimization can be seen in other VM operations such as when large quantities of VMs are shutdown.

RBAC Provider Changes

XenServer 6.5 and prior used an older version of Likewise to provide Active Directory. Likewise is now known as Power Broker, and XenServer is using the Power Broker Identity Services to provide authentication for RBAC. This has improved performance, scale and reliability, especially for complex or nested directory structures. Since RBAC is core to delegated management of a XenServer environment, we are particularly interested in feedback on any issues users might have with RBAC in Dundee beta 1.

dom0 Disk Space Usage

In XenServer 6.5 and prior, dom0 disk space was limited to 4GB. While this size was sufficient for many configurations, it was limiting for more than a few of you. As a result we've split dom0 disk into three core partitions; system, log and swap. The system partition is now 18GB which should provide sufficient for some time to come. This also means that the overall install space required for XenServer increases from 8GB to 46GB. As you can imagine, given the importance of this major change, we are very interested to learn of any situations where this change prevents XenServer from installing or upgrading properly.

Storage Improvements

Having flexible storage options is very important to efficient operation of any virtualization solution. To that end, we've added in support for three highly requested storage improvements; thin provisioned block storage, NFSv4 and FCoE.

Thin Provisioned Block Storage

iSCSI and HBA block storage can now be configured to be thinly provisioned. This is of particular value to those users who provision guest storage with a high water mark expecting that some allocated storage won't be used. With XenServer 6.5 and prior, the storage provider would allocate the entire disk space which could result in a significant reduction in storage utilization which in turn would increase the cost of virtualization. Now block storage repositories can be configured with an initial size and an increment value. Since storage is critical in any virtualization solution, we are very interested in feedback on this functional change.


Fibre Channel over Ethernet is protocol which allows Fibre Channel traffic to travel over standard ethernet networks. XenServer now is able to communicate with FCoE enabled storage solutions, and can be configured at install time to allow boot from SAN with FCoE. If you are using FCoE in your environment, we are very interested in learning both any issues as well as learning what CNA you used during your tests.

Operational Improvements

Many additional system level improvements have been made for Dundee beta 1, but the following highlight some of the operational improvements which have been made.


XenServer 6.5 and prior required legacy BIOS mode to be enabled on UEFI based servers. With Dundee beta 1, servers with native UEFI mode enabled should now be able to install and run XenServer. If you encounter a server which fails to install or boot XenServer in UEFI mode, please provide server details when reporting the incident.

Automatic Health Check

XenServer can now optionally generate a server status report on a schedule and automatically upload it to Citrix Insight Services (formerly known as TaaS). CIS is a free service which will then perform the analysis and report on any health issues associated with the XenServer installation. This automatic health check is in addition to the manual server status report facility which has been in XenServer for some time.

Improved Patch Management in XenCenter

Application of XenServer patches through XenCenter has become easier. The XenCenter updates wizard has been rewritten to find all patches available on Citrix’s support website, rather than ones that have been installed on other servers. This avoids missing updates, and allows automatic clean-up of patches files at the end of the installation.

Why Participate in the Beta Program

These platform highlights speak to how significant the engineering effort has been to get us to beta 1. They also overshadow some other arguably core items like a move to the Xen Project Hypervisor 4.6, host support for up to 5TB of host RAM or even Windows guest support for up to 1TB RAM. What they do show is our commitment to the install base and their continued adoption of XenServer at scale. Last year we ran an incredibly successful prerelease program for XenServer Creedence, and its partly through that program that XenServer 6.5 is as solid as it is. We're building on that solid base in the hopes that Dundee will better those accomplishments, and we're once again requesting your help. Download Dundee. Test it in your environment. Push it, and let us know how it goes. Just please be aware that this is prerelease code which shouldn't be placed in production and that we're not guaranteeing you'll ever be able to upgrade from it.

Download location:

Defect database:

Recent Comments
Ezequiel Mc Govern
Ceph SR suppport is On the roadmap?
Tuesday, 22 September 2015 19:15
Tim Mackey
Dan, I'll need to check the local SR file system, but think its still ext3. Local storage is an important use case, so I'm curiou... Read More
Monday, 28 September 2015 15:31
Senol Colak
Hi Tim, simple, ext3 is not supporting SSD drives with TRIM. You need ext4 to get TRIM support. After the successfull installatio... Read More
Thursday, 01 October 2015 13:19
Continue reading
45870 Hits

XenServer Dundee Alpha.3 Available

The XenServer team is pleased to announce the availability of the third alpha release in the Dundee release train. This release includes a number of performance oriented items and includes three new functional areas.

  • Microsoft Windows 10 driver support is now present in the XenServer tools. The tools have yet to be WHQL certified and are not yet working for GPU use cases, but users can safely use them to validate Windows 10 support.
  • FCoE storage support has been enabled for the Linux Bridge network stack. Note that the default network stack is OVS, so users wishing to test FCoE will need to convert the network stack to Bridge and will need to be aware of the feature limitations in Bridge relative to OVS.
  • Docker support present in XenServer 6.5 SP1 is now also present in Dundee

Considerable work has been performed to improve overall I/O throughput on larger systems and improve system responsiveness under heavy load. As part of this work, the number of vCPUs available to dom0 have been increased on systems with more than 8 pCPUs. Early results indicate a significant improvement in throughput compared to Creedence. We are particularly interested in hearing from users who have previously experienced responsiveness or I/O bottlenecks to look at Alpha.3 and provide their observations.

Dundee alpha.3 can be downloaded from the pre-release download page.     

Recent Comments
Sam McLeod
Hi Tim, Great to hear about the IO! As you know, we're very IO intensive and have very high speed flash-only iSCSI storage. I'll ... Read More
Thursday, 20 August 2015 01:50
Tim Mackey
Excellent, Sam. I look forward to hearing if you feel we've moved in the right direction and by how much.
Thursday, 20 August 2015 02:08
Chris Apsey
Has any testing been done with 40gbps NICS? We are considering upgrading to Chelsio T-580-LP-CRs, and if the ~24gbps barrier has ... Read More
Thursday, 20 August 2015 19:25
Continue reading
21681 Hits

Storage XenMotion in Dundee

Before we dive into the enhancements brought in the Storage XenMotion(SXM) feature in the XenServer Dundee Alpha 2 release; here is the refresher of various VM migrations supported in XenServer and how users can leverage them for different use cases.

XenMotion refers to the live migration of a VM(VM's disk residing on a shared storage) from one host to another within a pool with minimal downtime. Hence, with XenMotion we are moving the VM without touching its disks. E.g. XenMotion feature is very helpful during host and pool maintenance and is used with XenServer features such as Work Load Balancing(WLB)and Rolling Pool Upgrade(RPU) where the VM's residing on shared ­­­­storage can be moved to other host within a pool.

Storage XenMotion is the marketing name given to two distinct XenServer features, live storage migration and VDI move. Both features refer to the movement of a VM's disk (vdi) from one storage repository to another. Live storage migration also includes the movement of the running VM from one host to another host. In the initial state, the VM’s disk can reside either on local storage of the host or shared storage. It can then be motioned to either local storage of another host or shared storage of a pool (or standalone host). The following classifications exist for SXM:

  • When source and destination hosts happen to be part of the same pool we refer to it as IntraPool SXM. You can choose to migrate the VM's vdis to local storage of the destination host or another shared storage of the pool. E.g. Leverage it when you need to live migrate VMs residing on a slave host to the master of a pool.
  • When source and destination hosts happen to be part of different pools we refer to it as CrossPool SXM. VM migration between two standalone hosts can also be regarded as CrossPool SXM. You can choose to migrate the VM's vdis to local storage of the destination host or shared storage of the destination pool. E.g. I often leverage CrossPool SXM when I need to migrate VM’s residing on pool having old version of XenServer to the pool having latest XenServer
  • When the source and destination hosts are the same, we refer to it as LiveVDI Move. E.g. Leverage LiveVDI move when you want to upgrade your storage arrays but you don’t want to move the VMs to some another host. In such cases, you can LiveVDI move VM's vdi say from the shared storage (which is planned to be upgraded) to another shared storage in a pool without taking down your VM’s.

When SXM was first rolled out in XenServer 6.1 ,there were some restrictions on VMs before they can be motioned such as maximum number of snapshots a VM can have while undergoing SXM, VM having checkpoints cannot be motioned, the VM has to be running (otherwise it’s not live migration) etc. For the XenServer DundeeAlpha 2 release the XenServer team has removed some of those constraints .Thus below is the list of enhancements brought to SXM.

1. VM can be motioned regardless of its power status. Therefore I can successfully migrate a suspended VM or move a halted VM within a pool (intra pool) or across pools (cross pool)



2. VM can have more than one snapshot and checkpoint during SXM operation. Thus VM can have a mix of snapshots and checkpoints and it can still be successfully motioned.


3. A Halted VM can be copied from say pool A to pool B (cross-pool copy) .Note that VM’s that are not in halted state cannot be cross pool copied.


4. User created templates can also be copied from say pool A to pool B (cross-pool copy).Note that system templates cannot be cross pool copied.


Well this is not the end of the SXM improvements! Stay tuned with XenServer where in the upcoming release we aim to reduce VM downtime further during migration operations, and do download the Dundee preview and try it out yourself.     

Recent Comments
Tobias Kreidl
Nice post, Mayur! An important point to note in Dundee is that there is now also the option to create thin provisioned storage usi... Read More
Wednesday, 05 August 2015 21:38
Tobias Kreidl
Mayur, The images currently posted here are thumbnails and very hard to read (at least for some of us). Could you either post the ... Read More
Saturday, 08 August 2015 17:55
Mayur Vadhar
Thanks for the comments ,Tobias ! I will edit the post so the images are clearly visible. And I have mentioned cross pool copy opt... Read More
Monday, 10 August 2015 15:26
Continue reading
23956 Hits

Dundee Alpha 2 Released

I am pleased to announce that today we have made available the second alpha build for XenServer Dundee. For those of you who missed the first alpha, it was focused entirely on the move to CentOS 7 for dom0. This important operational change is one long time XenServer users and those who have written management tooling for XenServer should be aware of throughout the Dundee development cycle. At the time of Alpha 1, no mention was made for feature changes, and with Alpha 2 we're going to talk about some features. So here are some of the important items to be aware of.

Thin Provisioning on block storage

For those who aren't aware, when a XenServer SR is using iSCSI or an HBA, the virtual disks have always consumed their entire allocated space regardless of how utilized the actual virtual disk was. With Dundee we now have full thin provisioning for all block storage independent of storage vendor. In order to take advantage of this, you will need to indicate during SR creation that thin provisioning is required. You will also be given the opportunity to specify the default vdi allocation which allows users to optimize vdi utilization against storage performance. We do know about a number of areas still needing attention, but are providing early access such that the community can further identify issues our testing hasn't yet encountered.

NFS version 4

While a simple enhancement, this was identified as a priority item during the Creedence previews last year. We didn't really have the time then to fully implement it, but as of Dundee Alpha 2 you can specify NFS 4 for SR creation in XenCenter.

Intel GVT-d

XenServer 6.5 SP1 introduced support for Intel GVT-d graphics in Haswell and Broadwell chips. This support has been ported to Dundee and is now present in Alpha 2. At this point GPU operations in Dundee should have feature parity to XenServer 6.5 SP1.

CIFS for virtual disk storage

For some time we've had CIFS as an option for ISO storage, but lacked it for virtual disk storage. That has been remedied and if you are running CIFS you can now use it for all your XenServer storage needs.

Changed dom0 disk size

During installation of XenServer 6.5 and prior, a 4GB partition is created for dom0 with an additional 4GB partition created as a backup. For some users, the 4GB partition was too limiting, particularly if remote SYSLOG wasn't used or when third party monitoring tools were installed in dom0. With Dundee we've completely changed the local storage layout for dom0, and this has significant implications for all users wishing to upgrade to Dundee.

New layout

The new partition layout will consume 46GB from local storage. If there is less than 46 GB available, then a fresh install will fail. The new partition layout will be as follows:

  • 512 MB UEFI boot partition
  • 18 GB dom0 partition
  • 18 GB backup partition
  • 4 GB logs partition
  • 1 GB SWAP partition

As you can see from this new partition layout that we've separated logs and SWAP out of the main operating partition, and that we're now supporting UEFI boot.


During upgrade, if there is at least 46 GB available, we will recreate the partition layout to match that of a fresh install. In the event 46GB isn't available, we will shrink the existing dom0 partition from 4 GB to 3.5 GB and create the 512 MB UEFI boot partition.

Downloading Alpha 2


Dundee Alpha 2 is available for download from

Recent Comments
Tim Stephenson
46Gb+ seems very large for Dom0 :/ We're running XenServer on a set of Dell Poweredge R630's s equipped with dual 16Gb SD card bo... Read More
Tuesday, 14 July 2015 20:17
Tim Mackey
It's not 46GB for dom0, but 46GB for all storage dedicated to XenServer itself (dom0 is only 18GB). I hear you about the SD card ... Read More
Tuesday, 14 July 2015 20:26
Tobias Kreidl
First off, this is fantastic news, regarding what's mentioned (NFSv4, thin provisioning on LVM, bigger partition space, CIFS stora... Read More
Tuesday, 14 July 2015 20:45
Continue reading
30211 Hits

Preview of XenServer Administrators Handbook

Administering any technology can be both fun and challenging at times. For many, the fun part is designing a new deployment while for others the hardware selection process, system configuration and tuning and actual deployment can be a rewarding part of being an SRE. Then the challenging stuff hits where the design and deployment become a real part of the everyday inner workings of your company and with it come upgrades, failures, and fixes. For example, you might need to figure out how to scale beyond the original design, deal with failed hardware or find ways to update an entire data center without user downtime. No matter how long you've been working with a technology, the original paradigms often do change, and there is always an opportunity to learn how to do something more efficiently.

That's where a project JK Benedict and I have been working on with the good people of O'Reilly Media comes in. The idea is a simple one. We wanted a reference guide which would contain valuable information for anyone using XenServer - period. If you are just starting out, there would be information to help you make that first deployment a successful one. If you are looking at redesigning an existing deployment, there are valuable time-saving nuggets of info, too. If you are a longtime administrator, you would find some helpful recipes to solve real problems that you may not have tried yet. We didn't focus on long theoretical discussions, and we've made sure all content is relevant in a XenServer 6.2 or 6.5 environment. Oh, and we kept it concise because your time matters.

I am pleased to announce that attendees of OSCON will be able to get their hands on a preview edition of the upcoming XenServer Administrators Handbook. Not only will you be able to thumb through a copy of the preview book, but I'll have a signing at the O'Reilly booth on Wednesday July 22nd at 3:10 PM. I'm also told the first 25 people will get free copies, so be sure to camp out ;)

Now of course everyone always wants to know what animal which gets featured for the book cover. As you can see below, we have a bird. Not just any bird mind you, but a xenops. Now I didn't do anything to steer O'Reilly towards this, but find it very cool that we have an animal which also represents a very core component in XenServer; the xenopsd. For me, that's a clear indication we've created the appropriate content, and I hope you'll agree.



Recent Comments
prashant sreedharan
cool ! cant wait to get my hands on the book :-)
Tuesday, 07 July 2015 19:32
Tobias Kreidl
Congratulations, Tim and Jesse, as an update in this area is long overdue and in very good hands with you two. The XenServer commu... Read More
Tuesday, 07 July 2015 19:42
JK Benedict
Ah, Herr Tobias -- Danke freund. Danke fur ihre unterstutzung! Guten abent!
Thursday, 23 July 2015 09:26
Continue reading
13827 Hits

Introducing XenServer Dundee

It's spring time and after a particularly brutal winter here in Boston, I for one am happy to see the signs of spring. Grass and greenery, flowers budding, and warmer days all speak to good things coming. It's also time to unveil the next major XenServer project, code named Dundee. As with Creedence last year, we're going to be giving early access to a major new version of XenServer well in advance of its release. This project will have its share of functional improvements, and a few new features, but just like last year we're going to start with the platform and progress slowly.

CentOS 7 dom0

During the Creedence pre-release program, many commented on "Why CentOS 5.x? - CentOS 6 has been out for a while, and 7 is fresh". The answer to that question was pretty simple. We knew what userspace looked like with CentOS 5.x, and our users understood how to manage a CentOS 5.x system. CentOS 5 was being supported upstream until 2017, so there was no risk of us having something unsupported. Moving to CentOS 6.5 would've been a valid option if we didn't already plan on moving to CentOS 7, but we didn't want to change dom0 just to change it again in a years time. Plus if you recall, we took on quite a bit with Creedence in 2014.

So we're now a year later, and CentOS 7 makes perfect sense for dom0. Not only are there a few more upstream patches available, but Linux admins are now more comfortable with the changes in management paradigm. It's also those changes in paradigm which may present issues for you our users, and why this first alpha is all about validation. If you manage XenServer from a tool which uses the xapi SDK, then you shouldn't really experience too many problems. On the other hand, if you've favorite scripts, or tweaks you've made to configuration files, then you could be in for some extra work.

Now is also a perfect time to remind everyone that when you "upgrade" a XenServer, it's not an in place upgrade. We preserve the configuration files we know about, and then dom0 is reimaged. Any third party packages you have installed, custom scripts, and manual configuration changes have a good chance of being lost unless you've backed them up. In this case, with a move to CentOS 7, it's also possible that those items will need to be reworked to some degree.

Understanding the pre-release process

All pre-release downloads will be on our pre-release download page. We'll be providing new tagged builds every few weeks, and generally as we achieve internal milestones. With each build, we'll call out something which you as an interested participant in XenServer Dundee should be looking at. Issues encountered can be logged in the incident database at Since we've more than one version of XenServer covered in the incident database, please make certain you report Dundee issues under the "Dundee" version. Of course there is no guarantee we'll be able to resolve what you find, but we do want to know about it. With this first alpha, we’re interested in the “big issues” you may hit, i.e. areas which would block usage of features or functionality or cases where there is a major impact. These are really useful as the product develops and matures during the alpha stage. If you are developing something for XenServer, we invite you to ask your questions on the development mailing list, but do remember it's not a product support list.

Lastly, while we're in a pre-release period, its also likely you may eventually encounter functionality which may form part of a commercial edition. At this point we're not committing to what functionality will actually ship, when it might ship, or if it'll require a commercial license. I understand that might be concerning, but it shouldn't be. If something is destined for a commercial edition, you'll see it "commercialized" in a Citrix Tech Preview before we release. Historically we're many months away from when a Tech Preview might happen, so right now the most important thing is to focus on the changes we're interested in your feedback on today - everything to do with a CentOS 7 dom0.

Download Dundee alpha.1:

Recent Comments
Tobias Kreidl
Keep the XenServer evolution rolling! Great to see this initiative so soon after Creedence was released.
Tuesday, 28 April 2015 23:37
Tobias Kreidl
Is there a separate document available with just the release notes?
Wednesday, 29 April 2015 16:09
Martin Kralicek
Yes, the document/page containing roadmap or list of new features would be so good to have I am looking forward for next release.... Read More
Thursday, 07 May 2015 13:09
Continue reading
23199 Hits

Preview of XenServer support for Docker and Container Management

I'm excited to be able to share with you a preview of our new XenServer support for Docker and Container Management. Downloads can be found on the preview page, read on for installation instructions and more details.

Today many Docker applications run in containers within VMs hosted on hypervisors such as XenServer and other distributions of Xen. The synergy between containers as an application isolation mechanism and hypervisors as a secure physical infrastructure virtualization mechanism is something that I'll be blogging more about in the future. I firmly believe that these two technologies add value to each other, especially if they are aware of each other and designed to work together for an even better result.

That's why we've been looking at how we can enhance XenServer to be a great platform for Docker applications and how we can contribute to the Docker ecosystem to best leverage the capabilities and services available from the hypervisor. As a first step in this initiative I'm pleased to announce a preview of our new XenServer support for Docker applications. Those who attended Citrix Summit in January or FOSDEM in February may have seen an earlier version of this support being demo'd.

The preview is designed to work on top of XenServer 6.5 and comes in two parts: a supplemental pack for the servers and a build of XenCenter with the UI changes. XenCenter is installed in the normal Windows manner. The supplemental pack is installed in the same way as other XenServer supp-packs by copying the ISO file to each server in the pool and executing the following command in domain 0:

xe-install-supplemental-pack xscontainer-6.5.0-100205c.iso
mount: xscontainer-6.5.0-100205c.iso is write-protected, mounting read-only
Installing 'XenServer Container Management'...

Preparing...                ########################################### [100%]
   1:guest-templates        ########################################### [ 50%]
Waiting for xapi to signal init complete
Removing any existing built-in templates
Regenerating built-in templates
   2:xscontainer            ########################################### [100%]
Pack installation successful.

So what do you get with this preview? First off you get support for running CoreOS Linux VMs - CoreOS is a minimal Linux distribution popular for hosting Docker apps. The XenCenter VM installation wizard now includes a template for CoreOS and additional dialogs for setting the VM up (that's setting up a cloud config drive under the hood). This process also prepares the VM to be managed, to enable the main part of the preview's functionality to interact with it.


Secondly, and most importantly, XenServer becomes aware of “Container managed” VMs running Docker containers. It queries the VMs to enumerate the application containers running on each and then displays these within XenCenter's infrastructure view. XenCenter also allows interaction with the containers to start, stop and pause them. We want XenServer to be a platform for Docker and complement, not replace, the core part of the Docker application ecosystem, and therefore we expect that the individual Docker Engine instances in the VMs will be managed by one of the many Docker management tools such as Kubernetes, Docker Compose or ShipYard.


So what can you do with this preview?

Monitoring and visibility - knowing which VMs are in use for Docker hosting and which containers on them are actually running. Today's interface is more of a "pets" than "cattle" one but we've got experience  in showing what's going on at greater scale.

Diagnostics - easy access to basic container information such as forwarded network ports and originating Docker image name. This can help accelerate investigations into problems where either or both of the infrastructure and application layers may be implicated. Going forward we’d like to also provide easy access to the container-console.

Performance - spotted a VM that's using a lot of resource? This functionality allows you to see which containers are running on that VM, what processes run inside, how much CPU time each consumed, to help identify the one consuming the resource. In the future we'd like to add per-container resource usage reporting for correlation with the VM level metrics.

Control applications - using XenCenter you can start, stop and pause application containers. This feature has a number of use cases in both evaluation and deployment scenarios including rapidly terminating problematic applications.

We'd love to hear your feedback on this preview: what was useful, what wasn't? What would you like to see that wasn't there? Did you encounter problems or bugs? Please can share your feedback using our normal preview feedback mechanism by creating a ticket in the "XenServer Org" (XSO) project at

This preview is a first step towards a much richer Docker-XenServer mutual awareness and optimization to help bridge the gap between the worlds of the infrastructure administrator and the application developer/administrator. This is just the beginning, we expect to be improving, extending and enhancing the overall XenServer-Docker experience beyond that. Look out for more blog posts one this topic...

For a detailed guide to using this preview please see this article.

Recent Comments
Thomas Subotitsch
Great news. Hope that API will also get commands for docker management.
Tuesday, 17 March 2015 07:41
Get this error when I try to start the Core-OS vm: Only 1 LUN may be used with shared OCFS I tried iSCSI SR and Local storage, s... Read More
Friday, 24 April 2015 19:23
James Bulpin
Slava: You need to use XenServer 6.5 "Creedence" for this preview. As the error message "Only 1 LUN may be used with shared OCFS" ... Read More
Monday, 27 April 2015 12:26
Continue reading
50730 Hits

Understanding why certain Creedence builds don't work with certain features

Over the year end break, there were a couple of posts to the list which asked a very important question: "Does the DVSC work with the Release Candidate?" The answer was a resounding "maybe", and this post is intended to help clarify some of the distinction between what you get from, what you get from, and how everything is related.

At this point most of us are already familiar with XenServer virtualization being "open source", and that with XenServer 6.2 there was no functional difference between the binary you could download from and that from Logically, when we started the Creedence pre-release program, many assumed that the same download would exist in both locations, and that everything which might be part of a "XenServer" would also always be open source. That would be really cool for many people, and potentially problematic for others.

The astute follower of XenServer technology might also have noticed that several things commonly associated with the XenServer product never had their source released. StorageLink is a perfect example of this. Others will have noticed that the XenServer Tech Preview run on included a number of items which weren't present in any of the pre-release builds, and for which the sources aren't listed on There is of course an easy explanation for this, but it goes to the heart of what we're trying to do with is first and foremost about the XenServer platform. Everyone associated with, and by extension the entire team, would love for the data centers of the world to standardize on this platform. The core platform deliverable is called main.iso, and that's the thing from which you install a XenServer host. The source for main.iso is readily available, and other than EULA differences, the XenServer host will look and behave identically regardless of whether main.iso came from or The beauty of this model is that when you grow your XenServer based data center to the point where commercial support makes sense, the software platform you'd want supported is the same.

All of which gets me back to the DVSC (and other similar components). DVSC, StorageLink and certain other "features" include source code which Citrix has access to under license. Citrix provides early access to these feature components to those with a commercial relationship. Because there is no concept of a commercial relationship with, we can't provide early access to anything which isn't part of the core platform. Now of course we do very much want everyone to obtain the same XenServer software from both locations, so when an official release occurs, we mirror it for your convenience.

I hope this longish explanation helps clarify why when questions arise about "features" not present in main.iso that the response isn't as detailed as some might like. It should also help explain why components obtained from prior "Tech Preview" releases might not work with newer platform builds obtained as part of a public pre-release program.

Recent Comments
Tim Mackey
@xiao, It went live this morning, and you can download it from
Tuesday, 13 January 2015 19:40
Tim Mackey
@Nick I think a better way of thinking about this is that the hypervisor is free, and the platform features and functions are fre... Read More
Tuesday, 27 January 2015 23:43
Continue reading
15106 Hits

Status of Creedence

Over the past few weeks, and particularly as part of the Creedence World Tour, I've been getting questions about precisely when Creedence will be released. To the best of my ability, I've tried to take those questions head on, but the reality is we haven't been transparent about what happens when we release XenServer, and that's part of the problem. I'm going to try and address some of that in this post.

Now before I get into too much detail, it's important to note that XenServer is a packaged product which Citrix sells, and which is also available freely as open source. Citrix is a public company, so there is often a ton more detail I have, but which isn't appropriate for public disclosure. A perfect case in point is the release date. Since conceivably someone could change a PO based on this information, disclosing that can impact revenue and, well, I like my pay-cheque so I hope you understand when I'm quiet on some things.

So back to the question of what happens during finalization of a release, and how that can create a void. The first thing we do is take a look at all the defects coming in from all sources; with being one of many. We look at the nature of any open issues and determine what the potential for us to have a bad release from them. Next we create internal training to be delivered to the product support teams. These two tasks typically occur with either a final beta, or first release candidate. Concurrent to much of this work is finalization of documentation, and defining the performance envelope of the release. With each release, we have a "configuration limits" document, and the contents of that document represent both what Citrix is willing to deliver support on and what constitutes the limits of a stable XenServer configuration. For practical purposes, many of you have pushed Creedence in some way beyond what we might be comfortable defining as a "long term stable configuration", so its entirely possible the final performance could differ from what you've experienced so far.

Those are the technical bits behind the release, but this is also something which needs to be sold and that means we need to prepare for that as well. In the context of XenServer, selling means both why XenServer is great with XenDesktop, but also why it's great for anyone who is tired of paying more for their core virtualization requirements than really necessary. Given how many areas of data center operations XenServer touches, and the magnitude of the changes in Creedence, getting this right is critical. Then of course there is all the marketing collateral, and you get a sense of how much work is involved in getting XenServer out the door.

Of course, it can be argued that much of this "readiness" stuff could be handled in parallel, and for another project you'd be right. The reality is XenServer has had its share of releases which should've had a bit more bake time. I hope you agree with me that Creedence is better because we haven't rushed it, and that with Creedence we have a solid platform upon which to build. So with that in mind, I'll leave you with it hopefully being obvious that we intend to make a big splash with Creedence. Such a splash can't occur if we release during a typical IT lockdown period, and will need a bit larger stage than the one I'm currently on.


So stay tuned, my friends.  Good things are coming ;)

Recent Comments
Tim Mackey
Thanks @M.N..
Tuesday, 16 December 2014 14:42
tl;dr -- So what you're basically saying is that there is no release date yet.
Tuesday, 16 December 2014 12:03
Tim Mackey
@Christian It would be more precise to say that I know when it is intended to be released, but due to a variety of disclosure and... Read More
Tuesday, 16 December 2014 14:40
Continue reading
10436 Hits

Creedence Final Beta Available

As we move steadily towards a release of XenServer Creedence, I'm pleased to announce that we're ending the beta phase of development with the release of Creedence beta.3. Beta.3 sees us as functionally complete, and with the majority of known performance issues resolved. The performance issues resolved range from a dom0 memory leak when VIFs are disabled, through to resolution of a workaround with Mellanox 40Gbps NICs, and some are resolved with both an updated driver bundle and a bump of the ovs version from 2.1.2 to 2.1.3. Functionally, beta.3 differs from beta.2 in having PVHVM support for Ubuntu 14.04, RedHat Enterprise Linux 7 and CentOS 7. Since these are new operating systems for us, the team is really interested in learning what you see for performance and stability for them.

As with all previous pre-release builds, we'd like the community to help ensure Creedence is a rock solid release. This time we're a bit less interested in Creedence itself, and more about the operating and support environment. One of the less known "features" of Citrix support is the free "Insight Services" or "TaaS". TaaS was originally designed to be "Tools-as-a-Service", and deliver on demand insight into the operation of Citrix technologies. With XenServer, Citrix Insight Services consumes a server status report from your XenServer host or pool, and then provides detailed guidance on how to potentially avoid an issue (say due to outdated BIOS or firmware), or resolve an issue you might be having (say by applying a hotfix). Honestly, it's not a bad practice in general to upload a server status report post XenServer install to ensure there aren't any items which could be latent in the deployment; rather like a health check.

How does this relate to Creedence? Well, Insight Services uses a series of plugins to ensure the data is processed properly. The support team has recently updated TaaS to support Creedence, and I'd like to ensure two things. First I'd like to ensure the processing logic is capturing everything it should, and secondly I'd like to ensure that those of you who have been successfully running Creedence don't have any hidden errors. Since this is a free service offered by Citrix, I'd also like the open source XenServer install base to know about it as a way to ensure XenServer hosts are deployed in a manner which will allow for Citrix to support you if the need arises.

Here's how you can help.

  1. Install either beta.2 or beta.3 (beta.3 preferred) from the pre-release downloads:
  2. From either XenCenter or the CLI take a server status report.
    • XenCenter: Server status reports can be run from "Tools->Server Status Report ..."
    • CLI: xen-bugtool --yestoall --output zip
  3. Log into TaaS (create a free account if required):
  4. Upload your server status report and see if anything interesting is found. If anything unexpected is found, we'd like to know about it.  The best way to let us know would be to submit an incident to which contains the TaaS information.

Thanks again to everyone who has contributed to the success we're seeing with Creedence.

Recent Comments
JK Benedict
Great post, Tim! I will definitely be leveraging this tomorrow after I check in my previous posts for PVHVM OSes I have used from... Read More
Thursday, 25 September 2014 00:42
hmm.. ? should yum install work ? Loaded plugins: fastestmirror Determining fastest mirrors Could not retrieve mirrorlist http://... Read More
Friday, 26 September 2014 15:14
Tim Mackey
@Sebastian, XenServer is a packaged and tuned system, so we intentionally disable yum install. The biggest reason we do that is ... Read More
Friday, 26 September 2014 15:47
Continue reading
18516 Hits

Pushing XenServer limits with Creedence beta.2

Well folks, its that time once again; we've another XenServer build ripe for your inspection, and this one is a critical one for a number of reasons. Today we've released XenServer Creedence beta.2, and this is binary compatible with a Citrix Tech Preview refresh. The build number is 87850 and it presents itself to the outside world as 6.4.95. Over the past few announcements I've hinted at pushing the boundaries of XenServer and wanting the community at large to "have at it", but I've not put out too many details on the overall performance we're seeing internally. The most important attribute of this build is that internally, its going to form part of a series of long term stability tests. Yes folks, we're that confident in what we're seeing and I wanted to thank everyone who has participated in our pre-release activities by sharing a few performance tidbits:

  • Creedence can start and run 1000 PV VMs with only 8GB dom0 memory. That's up from the 650 we have in XenServer 6.2.
  • Booting 125 Windows 7 VMs on a single host takes only 350 seconds in a bootstorm scenario. That's down from 850 seconds in XenServer 6.2
  • Aggregate disk throughput has been measured to improve by as much as 100% when compared to XenServer 6.2
  • Aggregate intrahost network throughput has been measured to improve by as much as 200% when compared to XenServer 6.2
  • The number of virtual disks per host has been raised by a factor of four when compared to XenServer 6.2

When compared to beta.1, the team has been looking at a number of performance and scalability system aspects, with a primary focus on dom0 idle state behavior at scale. This is a very important aspect of system operation as overall system responsiveness is directly tied to the overhead of managing a large number of VMs. We did see two distinct areas for investigation, and are inviting the community to look into these and provide us with others. Those two areas are:

  • When using 40Gb NICs outbound (transmit) performance is below expectations. We have some internal fixes, but are encouraging anyone with such NICs to test and report their findings
  • When large numbers of hosts are pooled we're seeing VM start times appear to slow unexpectedly under large pool VM densities.


As always we're actively encouraging you to test the beta and provide your feedback (both positive and negative) in an incident report. You can download beta.2 from here:, and enter your feedback at     

Recent Comments
Tim Mackey
@bpbp If you're comparing the ovs in Creedence to previous Creedence builds, then they are identical. If you're comparing it to ... Read More
Sunday, 24 August 2014 23:37
Tim Mackey
@vati, If I understand correctly, you're looking for when the DVSC will appear? The DVSC is being handled via the Citrix Tech Pr... Read More
Monday, 25 August 2014 14:32
Tim Mackey
@bpbp, We're hoping that the upgrade to ovs 2.1.2 will address most of the issues seen with the older ovs. If you're in a positi... Read More
Monday, 25 August 2014 14:33
Continue reading
18676 Hits

XenServer Creedence Tech Preview and Creedence Alpha

This morning astute followers of XenServer activity noticed that Citrix had made available the previously announced Tech Preview for Creedence. The natural follow on question is how this relates to the alpha program we've been running on The easy answer is that the Citrix Tech Preview of XenServer Creedence is binary compatible with the alpha.4 pre-release binary you can get from From the perspective of the core platform (i.e. XenServer virtualization bits), the only difference is in the EULA.

So why run a Tech Preview if you have a successful alpha program already?

That's where the differences between a pure open source effort and a commercial effort begin. While the XenServer platform components are binary compatible, Citrix customers have expectations for features which couldn't be made open source, or implementations directly supporting Citrix commercial products. Perfect examples of these features and implementations can be seen on the Tech Preview download page, such as Microsoft System Center Integration, expanded vGPU support for XenDesktop, and the return of both the DVSC and WLB. While there is no guarantee any of those features or specific implementations will be present in the final Citrix release, or for that matter under what license, Citrix is seeking your input on them and a Tech Preview program is how that is accomplished.

I can't access the Tech Preview site; what's wrong?

If you can't login to the Tech Preview site, that likely means your account isn't associated with a Citrix commercial contract for XenServer. Since the alpha.4 pre-release is binary compatible, you can experience all the platform improvements yourself by downloading XenServer from the pre-release download page.

How do I provide feedback?

If you are able to participate in the Tech Preview program, you'll find the options for feedback listed on the Tech Preview page. Of course, even if you can participate in the Tech Preview program we're always accepting XenServer feedback through our public feedback options:

The XenServer incident database:

Development feedback (xs-devel):

What cool things are in alpha.4?

This is the best for last! We've already got in Creedence a 64 bit dom0, updated Linux 3.10 kernel, updated open virtual switch 2.1.2, improved boot storm handling, read caching for file based SRs; so what goodies are in here for the core platform people? Let's start with TRIM and UNMAP to better reclaim storage, then add in 32 bit to 64 bit VM migration to support upgrade scenarios and storage migration from XenServer 6.2 and prior, all with additional operating system validation with SLES 11 SP3 and Ubuntu 14.04 LTS.

What testing would you like us to do?

Don't let the alpha label fool you, alpha.4 of XenServer Creedence has been through quite a bit of testing, and is very much ready for you to try and stress it. The one thing we can say is that we're still working on the performance tuning so if you push things really hard, dom0 may run out of memory and you might need to follow CTX134951 to increase it (valid values are 8192, 16384 and 32768). This is particularly true if you're running more than 200 VMs per host, or need to attach more than 1200 virtual disks to VMs.     

Recent Comments
Tim Mackey
James, We now have the code in place to do a 32 bit to 64 bit migration, so upgrades are now on the table for testing, but I woul... Read More
Thursday, 10 July 2014 20:44
valentin radu
Hello, You can tell me when it will be released new stable version? Thx,
Tuesday, 15 July 2014 16:29
Hong Lae Cho
Hey Tim, Would you be able to provide a bit more clarification regarding "UNMAP" functionality in Creedence Alpha 4 & Tech Previ... Read More
Monday, 21 July 2014 05:10
Continue reading
26693 Hits

XenServer Creedence Alpha 2 Released

We're pleased to announce that XenServer Creedence Alpha 2 has been released. Alpha 2 builds on the capabilities seen in Alpha 1, and we're interested your feedback on this release. With Alpha 1, we were primarily interested in receiving basic feedback on the stability of the code, with Alpha 2 we're interested in feedback not only on basic operations, but also storage performance.

The following functional enhancements are contained in Alpha 2.

  • Storage read caching. Boot storm conditions in environments using common templates can create unnecessary IO on shared storage systems. Storage read caching uses free dom0 memory to cache common read IO and reduce the impact of boot storms on storage networks and NAS devices.
  • DM Multipath storage support. For users of legacy MPP-RDAC, this functionality has been deprecated in XenServer Creedence following storage industry practices. If you are still using MPP-RDAC with XenServer 6.2 or prior, please enter an incident in to record your usage such that we can develop appropriate guidance.
  • Support for Ubuntu 14.04 and CentOS 5.10 as guest operating systems

The following performance improvements were observed with Alpha 2 compared to Alpha 1, but we'd like to hear your experiences.

  • GRO enabled physical network to guest network performance improved by 65%
  • Aggregate network throughput improved by 50%
  • Disk IO throughput improved by 100%

While these improvements are rather impressive, we do need to be aware this is alpha code. What this means in practice is that when we start looking at overall scalability the true performance numbers could go down a bit to ensure stable operations. That being said, if you have performance issues with this alpha we want to hear about them. Please also look to this blog space for updates from our performance engineering team detailing how some of these improvements were measured.


Please do download XenServer Creedence Alpha 2, and provide your feedback in our incident database.     

Recent Comments
Tobias Kreidl
In Creedence Alpha 1, we did not see any discernible storage performance difference compared to XS 6.2 SP1, so it will definitely ... Read More
Tuesday, 10 June 2014 14:42
James Bulpin
We'd very much like to move to CentOS 7 when it becomes available. However the timing of this, and the need to integrate and stabi... Read More
Friday, 13 June 2014 15:44
Bruno de Paula Larini
But there will be plans to support RHEL7/CentOS 7 guests?
Thursday, 26 June 2014 12:27
Continue reading
17978 Hits

The reality of a XenServer 64 bit dom0

One of the key improvements in XenServer Creedence is the introduction of a 64 bit control domain (dom0). This is something I've heard requests for over the past several years, and while there are many reasons to move to a 64 bit dom0, there are some equally good reasons for us to have waited this long to make the change. It's also important to distinguish between a 64 bit hypervisor, which XenServer has always used, and a 64 bit control domain which is the new bit.

Isn't XenServer 64 bit bare-metal virtualization?

The good news for people who think XenServer is 64 bit bare metal virtualization is they're not wrong. All versions of XenServer use the Xen Project hypervisor, and all versions of XenServer have configured that hypervisor to run in 64 bit mode. In a Xen Project based environment, the first thing the hypervisor does is load its control domain (dom0). For all versions of XenServer prior to Creedence, that control domain was 32 bit.

The goodness of 64 bit

A 64 bit Linux dom0 allows us to remove a number of bottlenecks which were arbitrarily present with a 32bit Linux dom0.

Goodbye low memory

One of the biggest limiting factors with a 32bit Linux dom0 is the concept of "low memory". On 32 bit computers, the maximum directly addressable memory is 4GB. There are a variety of extensions available to address beyond the 4GB limit, but they all come with a performance and complexity cost. Further, as 32bit Linux evolved, a concept of "Low" memory and "High" memory was created with most everything related to the kernel using low memory and userspace processes running in high memory. Since we're talking about kernel operations consuming low memory, that also means that any kernel memory maps and kernel drivers also consume low memory and you can see how low memory quickly becomes a precious commodity. It also can be increasingly consumed the more memory available to a 32bit Linux dom0. In a typical XenServer installation this value is only 752MB, and with the move to a 64bit dom0 will soon be a thing of the past.

Improved driver stability

In a 32bit BIOS, MMIO regions are always placed between the 1GB and 3GB physical address space, and with a 64bit BIOS they are always placed at the top of the physical memory. If an MMIO hole is created dom0 can choose to re-map that memory so that it is now shadowed in virtual address space. The kernel must map a roughly equal amount of memory to the size of the MMIO holes, which in a 32bit kernel must be done in low memory. Many drivers map kernel memory to MMIO holes on demand, resulting in boot time success but potential instability in the driver if there is insufficient low memory to satisfy the dynamic request for a MMIO hole remap. Additionally, while XenServer currently supports PAE, and can address more than 4GB, if the driver insists on having its memory allocated in 32bit physical address space, it will fail.

Support for more modern drivers

Hardware vendors want to ensure the broadest adoption for their devices, and with modern computers shipping with 64bit processors and memory configurations well in excess of 4GB, the majority of those drivers have been authored for 64bit operating systems, and 64bit Linux is no different. Moving to a 64bit dom0 gives us an opportunity to incorporate those newer devices and potentially have fewer restrictions on the quantity of devices in a system due to kernel memory usage.

Improved computational performance

During the initial wave of operating systems moving from 32bit to 64bit configurations, one of the often cited benefits was the computational improvements offered from such a migration. We fully intend for dom0 to take advantage of general purpose improvements offered from the processor no longer needing to operate in what is today more of a legacy configuration. One of the most immediate benefits is with a 64bit dom0, we can use 64bit compiler settings and take advantage of modern processor extensions. Additionally, there are several performance concessions (such as the quantity of event channels) and code paths which can be optimized when compiled for a 64bit system versus a 32bit system.

Challenges do exist, though

While there are clear benefits to a 64 bit migration, the move isn't without its set of issues as well. The most significant issue relates to XenServer pool communications. In a XenServer pool, all hosts have historically been required to be at the same version and patch level except during the process of upgrading the pool. This restriction serves to ensure that any data being passed between hosts, and configuration information being replicated in xenstore and all operations are consistent. With the introduction of Storage XenMotion in XenServer 6.1, this requirement was also extended to cross pool operations when a vdi is being migrated.


In essence you can think of the version requirement as being the insurance against bad things happening to your precious VM cargo. Unfortunately, that requirement has an assumption of dom0 having the same architecture and our migration from 32bit to 64bit complicates things. This is due to the various data structures used in pool operations having been designed with a consistent view of a world based on a 32bit operating system. This view of the world is directly challenged with a pool consisting of both 32bit and 64bit XenServer hosts as would be present during an upgrade. It's also challenged during cross pool storage live migration if one pool is 32bit while the second is 64bit. We are working to resolve this problem at least for the 32bit to 64bit upgrade, but it will be something we're going to want very active participation from our user community to test once we have completed our implementation efforts. After all we do subscribe to the notion that your virtual machines are pets and not cattle; regardless of the scale of your virtualized infrastructure.     

Recent Comments
Hi, Good concept!! so, will this be included in alpha.2? Chaitanya.
Friday, 06 June 2014 17:52
Tim Mackey
Chaitanya, Alpha.1 already has a 64 bit dom0, and with Alpha.2 we're bringing more improvements online. -tim... Read More
Tuesday, 10 June 2014 17:15
Tobias Kreidl
Great synopsis, Tim. This release is going to be the best ever.
Tuesday, 10 June 2014 14:38
Continue reading
33696 Hits

Validation of the Creedence Alpha

On Monday May 19th early access to XenServer Creedence builds started from  The community has access to XenServer pre-release installation media of an alpha quality and is invited to provide feedback on it.

This blog describes the validation and system testing performed on the first alpha build.

Test Inventory

The XenServer development process incorporates daily automated regression testing complemented by various additional layers of testing, both automated and manual, that are executed less frequently.

In outline, these are the test suites and test cycles executed during XenServer development.

Automated short-cycle regression testing (“BVT”) for fast feedback to developers – on every build on every branch.

Automated medium-cycle regression testing (“BST”) to maintain quality on team branches

Automated long-cycle system regression test (“Nightly”) – on select builds on select branches, aimed at providing wide regression coverage on a daily basis

Automated performance regression test, measures several hundred key performance indicators – run on select builds on select branches on a regular basis

Automated stress test (huge numbers of lifecycle operations on single hosts) – run once per week on average

Automated pool stress test (huge numbers of lifecycle and storage operations on XS pools) - run once per week on average

Automated long-cycle system regression test (“Full regression”) – on select builds on select branches, aimed at providing extensive wide test coverage, this comprises a huge number of tests and takes several days to run, frequency of run is therefore one every two weeks on average

Automated large scale stability test (huge numbers of VMs on large XS pools, boot storm and other key ‘scale’ use-cases – run on demand, usually several times in the run up to a product release and ahead of key internal milestone including deliveries to other Citrix product groups

Automated soak test – run on demand, usually several times in the run up to a product release and ahead of key internal milestone including deliveries to other Citrix product groups, this comprises long-running tests aimed at validating XS over an extended time period

Automated upgrade test – run ahead of key milestones and deliveries to validate upgrade procedures for new releases.

Manual test – exploratory testing using XenCenter, aimed principally at testing edge cases and scenarios that are not well covered by automation, cycles of manual testing are carried out on a regular basis, and ahead of key milestones and deliveries.

Exit Criteria

Each stage of a XenServer release project requires different test suites to have been run “successfully” (usually meaning a particular pass-rate has been achieved and/or failures are understood and deemed acceptable).

However test pass rates are only a barometer of quality – if one test out of a hundred fails then that may not matter, but on the other what if that one test case failure represents a high impact problem affecting a common use-case? For this reason we also use defect counts and impact analyses as part of the exit criteria.

XenServer engineering maintains a high quality bar throughout the release cycle – the “Nightly” automated regression suite comprising several thousand test cases must always achieve over 95%. If it does not, then new feature development stops while bugs are fixed and code reverted until a high pass rate is restored.

The release is a drop from the Creedence project branch that has achieved the following pass rates on the following test suites.:

  • ·         Nightly regression  – 96.5%
  • ·         Stress –  passed (no pass rate for this suite)
  • ·         Pool stress –  passed (no pass rate for this suite)
  • ·         General regression –  91.3%

Drops later in the project lifecycle (e.g. Tech Preview) will be subjected to more testing and with more stringent exit criteria.

More Info

For more information on the automation framework used for these tests, please read my blog about XenRT!

Recent comment in this post
Rob Gilson
We downloaded and installed it earlier this month. Outwardly no changes but a lot of under-the-covers fixes and upgrades that were... Read More
Sunday, 22 June 2014 22:33
Continue reading
10578 Hits
1 Comment Alpha Available for Download Alpha Available

The XenServer engineering team is pleased to announce the availability an alpha of the next release of XenServer, code named “Creedence”. XenServer Creedence is intended to represent the latest capabilities in XenServer with a target release date determined by feature completeness. Several key areas have been improved over XenServer 6.2, and singificantly we have also introduced a 64 bit control domain architecture and updated the Xen Project hypervisor to version 4.4. Due to these changes, we are requesting tests using this alpha be limited to core functionality such as the installation process and basic operations like VM creation, start and stop. Performance and scalability tests should be deferred until a later build is nominated to alpha or beta status.

This is pre-release code and as such isn’t appropriate for production use, and is unlikely to function properly with provisioning solutions such as Citrix XenDesktop and Citrix CloudPlatform. It is expected that users of Citrix XenDesktop and Citrix CloudPlatform will be able to begin testing Creedence within the XenServer Tech Preview time-frame announced at Citrix Synergy. In preparation for the Tech Preview, all XenServer users, including those running XenDesktop, are encouraged to validate if Creedence is able to successfully install on their chosen hardware.

Key Questions

When does the alpha period start?

The alpha period starts on May 19th 2014

When does the alpha period end?

There is no pre-defined end to the alpha period. Instead, we’re providing access to nightly builds and from those nightly builds we’ll periodically promote builds to “alpha.x” status. The promotion will occur as key features are incorporated and stability targets are reached. As we progress the alpha period will naturally transition into a beta or Tech Preview stage ultimately ending with a XenServer release. Announcements will be made on when a new build is promoted.

Where do I get the build?

The build can be downloaded from at:

If I encounter a defect, how do I enter it?

Defects and incidents are expected with this alpha, and they can be entered at Users wishing to submit or report issues are advised to review our submission guidelines to ensure they are collecting enough information for us to resolve any issues.

Where can I find more information on Creedence?

We are pleased to announce a public wiki has been created at to contain key architectural information about XenServer; including details about Creedence.

How do I report compatibility information?

The defect system offers Hardware and Vendor compatibility projects to collect information about your environment. Please report both successes and failures for our review.

What about upgrades?

The alpha will not upgrade any previous version of XenServer, including nightly builds from trunk, and there should be no expectation the alpha can be upgraded.

Do I need a new XenCenter?

Yes, XenCenter has been updated to work with the alpha and can be installed from the installation ISO.

Will I need a new SDK?

If you are integrating with XenServer, the SDK has also been updated. Please obtain the SDK for the alpha from the download page.

Where can I ask questions?

Since the Creedence alpha is being posted to and managed by the team, questions asked on Citrix Support Forums are likely to go unanswered. Those forums are intended for released and supported versions of XenServer. Instead we are inviting questions on the xs-devel mailing list, and via twitter to @XenServerArmy. In order to post questions, you will need to subscribe to the mailing list which can be done here: Please note that the xs-devel mailing list is monitored by the engineering team, but really isn’t intended as a general support mechanism. If your question is more general purpose and would apply to any XenServer version, please validate if the issue being experienced is also present with XenServer 6.2 and if so ask the question on the Citrix support forums.  We've also created some guidelines for submitting incidents.

Recent Comments
Tim Mackey
James, This first release (alpha.1) is really about core functionality. With a new Xen Project hypervisor and 64bit dom0 there i... Read More
Monday, 19 May 2014 22:31
Tobias Kreidl
Tim, Awesome! The user community is collectively excited about this next evolutionary step for XenServer. It would be great to hav... Read More
Monday, 19 May 2014 18:04
Andrew Halley
Hi Tobias, there's a summary of the alpha(.1) content available on the wiki here: Read More
Tuesday, 20 May 2014 16:15
Continue reading
32677 Hits

Building xenserver-core

In a previous post, Dave Scott showed how easy it now is to install the core components of XenServer on a standard installation of CentOS, using xenserver-core. Since then, we have been hard at work adding features and functionality, and making it even smoother and easier to install from our repositories. However, since developers are a major part of the audience for xenserver-core, we have also paid a lot of attention to making it as smooth and easy as possible to build the packages for yourself.


On RPM-based distributions, the packages are built using a tool called mock.   Mock builds each new package in a fresh chroot environment, ensuring that your system isn't affected by the by-products of building packages, and that missing dependencies in the package specifications don't go unnoticed.  To install it on a RHEL/CentOS system then you will need to add the EPEL repositories. Here is a useful article for CentOS.

After adding EPEL, install and set up mock:

yum install -y mock rpm-build

Mock will refuse to run as root - you can run it using your own user account, or a special account you create for this purpose. To add this account to the mock group, type the following as root:

useradd  -G mock <username>
passwd <username>

su - <username>

Building the packages

With the prerequisites installed, building xenserver-core is a 4-step process:

  1. Clone the xenserver-core repository.   If you are planning to make changes, you should fork the repository on GitHub and clone that instead.   To clone the main xenserver-core repository, type:

    git clone
  2. This configures mock and initializes an RPM repository in the RPMS directory, where the built packages will be stored.   Enter the directory created by git (by default, it will be called "xenserver-core") and type:

  3. Generate the makefile which will actually build the xenserver-core packages:

     ./ > Makefile
  4. Start the build:


Building all the xenserver-core packages takes around 90 minutes on a reasonably fast machine, and requires a couple of gigabytes of free disk space.   When the build finishes, you will find source packages in the SRPMS directory, and binaries in RPMS.

Installing your new packages

The most flexible way to install your packages is to copy newly-built RPM repository to a web server and configure YUM on your target machine to download packages from there.  Assuming that you already have a web server set up,  copy the RPMS and SRPMS directories to your server's DOCROOT directory.

To configure the target machine to use your new repository, create /etc/yum.repos.d/xapi.conf, filling in your repository's baseurl as appropriate:

name=CentOS-$releasever - xenserver-core

name=CentOS-$releasever - xenserver-core

After this, type yum install xenserver-core to install the packages from your new repository. Once the installation is complete, run the xenserver-install-wizard command to configure xenserver-core.   Please note that, unless you were already running under Xen, you will have to reboot to complete the installation.

What's going on here?

When you cloned the xenserver-core repository you may have noticed that it doesn't actually contain any software.   Instead, it has a collection of RPM SPEC files, along with the build scripts we ran above.   When you run make, it downloads each package's source code from the upstream repository, builds a source RPM, then uses mock to compile the binary RPM, as you can see in this snippet of build output:

[CURL] SOURCES/libvhd-0.9.1.tar.gz
[RPMBUILD] SRPMS/ocaml-libvhd-0.9.1-1.src.rpm
[MOCK] RPMS/x86_64/ocaml-libvhd-0.9.1-1.x86_64.rpm
[CREATEREPO] RPMS/x86_64/ocaml-libvhd-0.9.1-1.x86_64.rpm

Please give us feedback

We want to make xenserver-core as easy as possible to install, build and use. We would love to hear what you think of our progress so far.   You can contact us through comments on this blog.  If you have development questions about XenServer and its components, try the This email address is being protected from spambots. You need JavaScript enabled to view it. mailing list.

If you run into any problems or bugs in xenserver-core's packaging or build scripts, please raise an issue on our GitHub issues page.

Prefer Ubuntu?

Following the instructions above will build xenserver-core for an RPM-based distribution such as Fedora or CentOS.   What if you prefer Ubuntu or another Debian-based distribution?   Try cloning the repository onto your Ubuntu machine and running the build there.   The results may surprise you!

More updates coming soon...

Continue reading
24497 Hits

About XenServer

XenServer is the leading open source virtualization platform, powered by the Xen Project hypervisor and the XAPI toolstack. It is used in the world's largest clouds and enterprises.
Technical support for XenServer is available from Citrix.