Virtualization Blog

Discussions and observations on virtualization.

Content related to the operation of XenServer

XenServer Administrators Handbook Published

Last year, I announced that we were working on a XenServer Administrators Handbook, and I'm very pleased to announce that it's been published. Not only have we been published, but based on the Amazon reviews to date we've done a pretty decent job. In part, I suspect that has a ton to do with the book being focused on what information you, XenServer administrators, need to be successful when running a XenServer environment regardless of scale or workload.

XenServer Administrators HandbookThe handbook is formatted following a simple premise; first you need to plan your deployment and second you need to run it. With that in mind, we start with exactly what a XenServer is, define how it works and what expectations it has on infrastructure. After all, it's critical to understand how a product like XenServer interfaces with the real world, and how its virtual objects relate to each other. We even cover some of the misunderstandings those new to XenServer might have.

While it might be tempting to go deep on some of this stuff, Jesse and I both recognized that virtualization SREs have a job to do and that's to run virtual infrastructure. As interesting as it might be to dig into how the product is implemented, that's not the role of an administrators handbook. That's why the second half of the book provides some real world scenarios, and how to go about solving them.

We had an almost limitless list of scenarios to choose from, and what you see in the book represents real world situations which most SREs will face at some point. The goal of this format being to have a handbook which can be actively used, not something which is read once and placed on some shelf (virtual or physical). During the technical review phase, we sent copies out to actual XenServer admins, all of whom stated that we'd presented some piece of information they hadn't previously known. I for one consider that to be a fantastic compliment.

Lastly, I want to finish off by saying that like all good works, this is very much a "we" effort. Jesse did a top notch job as co-author and brings the experience of someone who's job it is to help solve customer problems. Our technical reviewers added tremendously to the polish you'll find in the book. The O'Reilly Media team was a pleasure to work with, pushing when we needed to be pushed but understanding that day jobs and family take precedence.

So whether you're looking at XenServer out of personal interest, have been tasked with designing a XenServer installation to support Citrix workloads, clouds, or for general purpose virtualization, or have a XenServer environment to call your own, there is something in here for you. On behalf of Jesse, we hope that everyone who gets a copy finds it valuable. The XenServer Administrator's handbook is available from book sellers everywhere including:


Barnes and Noble:

O'Reilly Media:

If you need a copy of XenServer to work with, you can obtain that for free from:

Recent Comments
Tobias Kreidl
A timely publication, given all the major recent enhancements to XenServer. It's packed with a lot of hands-on, practical advice a... Read More
Tuesday, 03 May 2016 03:37
Eric Hosmer
Been looking forward to getting this book, just purchased it on Amazon. Now I just need to find that mythical free time to read ... Read More
Friday, 06 May 2016 22:41
Continue reading
7726 Hits

XenServer 6.5 Can Do True UEFI Boot


We were interested in getting XenServer 6.5 to boot via UEFI.  Leaving servers in Legacy/BIOS boot was not an option in our target environment.  We still have to do the initial install with the server in Legacy BIOS mode; however, I managed to compile Xen as an EFI bootable binary using the source and patches distributed by Citrix.  With that I am able to change the servers boot mode back to UEFI and boot XenServer.  Here are the steps I used to compile it.


  1. Prepare a DDK
  2. Prepare a build environment
  3. Build some prerequisites
  4. Unpack the SRPM
  5. Compile Xen

DDK Preparation

Development will be done inside a 6.5 DDK. This is a CentOS 5.4-based Linux that has the same kernel as Dom0 and some of the required development tools. 

Import the VM template per Citrix DDK developer documentation.

After importing, set the following VM options:

  • 2 vCPUs
  • Increase memory to 2048MB
  • Resize disk image to 10GB
  • Add a network interface for SSH

Start the VM, set a root password, and then finalize resizing the disk by running:

# fdisk /dev/xvda 
cmd: d  -del
cmd:  n  -new
cmd:  p  -primary
cmd:  1  -num 1
	use default size
cmd:  w  -save

Preparing the Build Environment

Install rpmdevtools:

# yum --disablerepo citrix install rpmdevtools
# rpmdev-setuptree

I also needed several packages many of which were provided on the binpkg ISO from Citrix. I made them available by inserting XenServer-6.5-binpkg.iso and running the following:

# mount /dev/xvdb /mnt
# mkdir /opt/binpkg
# cp -a /mnt/domain0/RPMS/* /opt/binpkg
# cd /opt/binpkg
# createrepo /opt/binpkg
# cat >/etc/yum.repos.d/binpkg.repo

I also added the epel repository:

rpm -Uvh

and I added the following packages:


Building the Prerequisites

This page: says that it is required to use gcc 4.5 or better and that that binutils must be compiled with --enable-targets=x86_64-pep. I could not satisfy this with packages in the repositories, so I compiled and installed some requirements for gcc:

and made sure the required files could be found:

# export LD_LIBRARY_PATH=:/usr/lib:/usr/local/lib:/usr/local/lib64:/usr/lib64
# ln -s /usr/lib64/

then I compiled binutils using:

# tar xzf binutils-2.22.tar.gz
# mkdir binutils-build
# cd binutils-build
# ../binutils-2.22/configure --disable-werror --enable-targets=x86_64-pep
# make
# make install

then I compiled gcc I using: I adapted compilation instructions from:

# cd gcc-4.6.2
# sed -i 's/install_to_$(INSTALL_DEST) //' libiberty/
# case `uname -m` in
i?86) sed -i 's/^T_CFLAGS =$/& -fomit-frame-pointer/' 
gcc/ ;;
# sed -i 's@./ true@' gcc/
# mkdir -v ../gcc-build
# cd ../gcc-build
# ../gcc-4.6.2/configure --prefix=/usr 
--libexecdir=/usr/lib --enable-shared 
--enable-threads=posix --enable-__cxa_atexit 
--enable-clocale=gnu --enable-languages=c,c++ 
--disable-multilib --disable-bootstrap --with-system-zlib
# make -j3
# ulimit -s 16384
# make -k check
# ../gcc-4.6.2/contrib/test_summary
# make install

Unpack the SRPM

At this point everything is ready to compile Xen as an EFI bootable binary.  The source code for Xen with Citrix's patches is available here: so download XenServer-6.5.0-source-main-1.iso and mount it at /mnt/cdrom:

# mount -r /dev/xvdb /mnt/cdrom

Now install the SRPM:

# rpm -ivh /mnt/cdrom/xen/xen-4.4.1- 

Compile Xen

The source code and required scripts are now all under /root/rpmbuild/, so just run:

# cd ~/rpmbuild
# QA_RPATHS=$[ 0x0020 ] rpmbuild -bc SPECS/xen.spec

The -bc flag causes the process to follow the spec file and patch the source, but then stop just before running the make commands. The make commands would fail to compile with warnings about uninitialized variables being treated as errors. Fix this by changing line 45 of ~/rpmbuild/BUILD/xen-4.4.1/xen/ to read:

CFLAGS += -Werror -Wno-error=uninitialized -Wredundant-decls

and line 39 of ~/rpmbuild/BUILD/xen-4.4.1/ to read:

HOSTCFLAGS = -Wall -Werror -Wno-error=uninitialized -Wstrict-prototypes -O2 -fomit-frame-pointer

after making those changes run:

# cd ~/rpmbuild/BUILD/xen-4.4.1/
# make clean
# make max_phys_cpus=256 XEN_TARGET_ARCH=x86_64 -C xen
XEN_VENDORVERSION=-xs90192 debug=n build

and the compile will finish successfully.  I probably could have just made a quick patch to add the -Wno-error flag and allowed the rpmbuild to run the full spec file, but I didn't actually need to compile xen-tools etc, those are already compiled and installed on the XenServer installation. The only file needed is ~/rpmbuild/BUILD/xen-4.4.1/xen/xen.efi. With that in hand I created a xen.cfg file like this:


options=console=vga,com1,com2 com1=115200,8n1,0x3F8,4
com2=115200,8n1,0x2F8,3 loglvl=all noreboot
kernel=vmlinuz-3.10-xen root=UUID=b4ee0ace-b587-41df-a66b-16f89731b2a8
rw ignore_loglevel acpi_rsdp=0x7B7FE014

where the root UUID is the boot disk created during the XenServer install and the RSDP number came from running:

# dmesg | grep RSDP

I ran that in an EFI booted live Linux environment. I found that some
vendors' UEFI implementations were able to provide the RSDP during boot
and some were not, so without specifying it in the xen.cfg I had trouble
with things like usb peripherals.

Boot XenServer

With the xen.efi and xen.cfg I was able to boot XenServer in UEFI boot mode using refind.  We have done extensive testing on several different servers and found no problems.  I was also able to repeat the process with the source code provided by the service packs up to and including Service Pack 1.  I haven't tried any further than that yet.

Editors Note

For those of you wishing to retain Citrix commercial support status, the above procedure will convert the XenServer 6.5 host into an "unsupported configuration".


Continue reading
6937 Hits

Integrating XenServer, RDO and Neutron

XenServer is a great choice of hypervisor for OpenStack based clouds, but there is no native integration between it and RedHat's RDO packages. This means that setting up an integrated environment using XenServer and RDO is more difficult than it should be. This blog post aims to resolve that, giving a method where CentOS can be set up easily to use XenServer as the hypervisor.


  • Hypervisor: XenServer: 6.5
  • Guest: CentOS 7.0
  • OpenStack: Liberty
  • Network: Neutron, ML2 plugin, OVS, VLAN

Install XenServer

The XenServer integration with OpenStack has some optimizations which means that only EXT3 storage is supported. Make sure when installing your XenServer you select Optimized for XenDesktop when prompted. Use XenCenter to check that the SR type is EXT3 as fixing it after creating the VMs will require deleting the VMs and starting again.

Install OpenStack VM

With XenServer, the Nova Compute service must run in a virtual machine on the hypervisor that they will be controlling. As we're using CentOS 7.0 for this environment, create a VM using the CentOS 7.0 template in XenCenter. If you want to copy and paste the scripts from the rest of the blog, use the name "CentOS_RDO" for this VM. Install the CentOS 7.0 VM but shut it down before installing RDO.

Create network for OpenStack VM

In single box environment, we need three networks, "Integration network", "External network", "VM network". If you have appropriate networks for the above (e.g. a network that gives you external access) then rename the existing network to have the appropriate name-label. Note that a helper script is provided for some of the later steps in this blog rely on these specific name labels, so if you choose not to use them then please also update the helper script.

You can do this via XenCenter or run the following commands in dom0:

    xe network-create name-label=openstack-int-network
    xe network-create name-label=openstack-ext-network
    xe network-create name-label=openstack-vm-network

Create virtual network interfaces for OpenStack VM

This step requires the VM to be shut down, as it's modifying the network setup and the PV tools have not been installed in the guest.

    vm_uuid=$(xe vm-list name-label=CentOS_RDO minimal=true)
    vm_net_uuid=$(xe network-list name-label=openstack-vm-network minimal=true)
    next_device=$(xe vm-param-get uuid=$vm_uuid param-name=allowed-VIF-devices | cut -d';' -f1)
    vm_vif_uuid=$(xe vif-create device=$next_device network-uuid=$vm_net_uuid vm-uuid=$vm_uuid)
    xe vif-plug uuid=$vm_vif_uuid
    ext_net_uuid=$(xe network-list name-label=openstack-ext-network minimal=true)
    next_device=$(xe vm-param-get uuid=$vm_uuid param-name=allowed-VIF-devices | cut -d';' -f1)
    ext_vif_uuid=$(xe vif-create device=$next_device network-uuid=$ext_net_uuid vm-uuid=$vm_uuid)
    xe vif-plug uuid=$ext_vif_uuid

You can also choose use helper script to do these in dom0.


Configure OpenStackVM/Hypervisor communications

Use HIMN tool (plugin for XenCenter) to add internal management network to OpenStack VMs. This effectively performs the following operations, which could also be performed manually in dom0 or use


Note: If using the commands manually, they should be run when the OpenStack VM is shut down.

Set up DHCP on the HIMN network for OpenStack VM, allowing OpenStack VM to access its own hypervisor on the static address Run helper script in domU.


Install RDO

Using the RDO Quickstart detailed installation guide, please follow the instructions step by step. This manual only points out the steps that you must pay attention to during installation.

Run Packstack to install OpenStack

Rather than running packstack immediately, we need to generate an answerfile so that we can tweak the configuration.

Generate answer file:

    packstack --gen-answer-file=

Install OpenStack services:

    packstack --answer-file=

These items in should be changed as below:


These items in should be changed according to your environment:



: physnet1 is physical network name for VLAN provider and tenant networks. 1000:1050 is VLAN tag ranges on each physical network for allocation to tenant networks.
: br-eth1 is OVS bridge for VM network. br-ex is OVS bridge for External network, neutron L3 agent use it for external traffic.
: eth1 is OpenStack VM's NIC which connected to VM network. eth2 is OpenStack VM's NIC which connected to External network.

Configure Nova and Neutron

Copy Nova and Neutron plugins to XenServer host.


Edit /etc/nova/nova.conf, switch compute driver to XenServer.




The integration_bridge above can be found from dom0:
    xe network-list name-label=openstack-int-network params=bridge is hypervisor dom0's address which OpenStack VM can reach via HIMN.

Install XenAPI Python XML RPC lightweight bindings.

    yum install -y python-pip 
pip install xenapi

Configure Neutron

Edit /etc/neutron/rootwrap.conf to support uing XenServer remotely.

# XenAPI configuration is only required by the L2 agent if it is to
# target a XenServer/XCP compute host's dom0.

Restart Nova and Neutron Services

    for svc in api cert conductor compute scheduler; do 
service openstack-nova-$svc restart;
service neutron-openvswitch-agent restart

Launch another neutron-openvswitch-agent to talk with dom0

XenServer has a seperation of dom0 and domU and all instances' VIFs are actually managed by dom0. Their corresponding OVS ports are created in dom0. Thus, we should manually start the other ovs agent which is in charge of these ports and is talking to dom0, refer xenserver_neutron picture.

Create ovs configuration file

    cp /etc/neutron/plugins/ml2/openvswitch_agent.ini etc/neutron/plugins/ml2/openvswitch_agent.ini.dom0
integration_bridge = xapi3
bridge_mappings = physnet1:xapi2

root_helper = neutron-rootwrap-xen-dom0 /etc/neutron/rootwrap.conf
root_helper_daemon =
minimize_polling = False

firewall_driver = neutron.agent.firewall.NoopFirewallDriver


xapi3 the integration bridge is xapX in the graph. xapi2 is vm network bridge, it's xapiY in the graph.
    xe network-list name-label=openstack-int-network params=bridgexe network-list name-label=openstack-vm-network params=bridge

Launch neutron-openvswitch-agent

Replace cirros guest with one setup to work for XenServer

    /usr/bin/python2 /usr/bin/neutron-openvswitch-agent \
--config-file /usr/share/neutron/neutron-dist.conf \
--config-file /etc/neutron/neutron.conf --config-file \
/etc/neutron/plugins/ml2/openvswitch_agent.ini.dom0 \
--config-dir /etc/neutron/conf.d/neutron-openvswitch-agent \
--log-file /var/log/neutron/openvswitch-agent.log.dom0 &
    nova image-delete cirros

glance image-create --name cirros --container-format ovf \
--disk-format vhd --property vm_mode=xen --visibility public \
--file cirros-0.3.4-x86_64-disk.vhd.tgz

Launching instance and test its connectivity

    source keystonerc_demo

[root@localhost ~(keystone_demo)]# glance image-list
| ID | Name |
| 5c227c8e-3cfa-4368-963c-6ebc2f846ee1 | cirros |


    [root@localhost ~(keystone_demo)]# neutron net-list
| id | name | subnets |
| 91c0f6ac-36f2-46fc-b075-6213a241fc2b | private | 3a4eebdc-6727-43e3-b5fe-8760d64c00fb |
| 7ccf5c93-ca20-4962-b8bb-bff655e29788 | public | 4e023f19-dfdd-4d00-94cc-dbea59b31698 |
    nova boot --flavor m1.tiny --image cirros --nic \
net-id=91c0f6ac-36f2-46fc-b075-6213a241fc2b demo-instance
    [root@localhost ~(keystone_demo)]# neutron floatingip-create public
    Created a new floatingip:
| Field | Value |
| fixed_ip_address | |
| floating_ip_address | |
| floating_network_id | 7ccf5c93-ca20-4962-b8bb-bff655e29788 |
| id | 2f0e7c1e-07dc-4c7e-b9a6-64f312e7f693 |
| port_id | |
| router_id | |
| status | DOWN |
| tenant_id | 838ec33967ff4f659b808e4a593e7085 |
    nova add-floating-ip demo-instance

After these above steps, we have succefully booted an instance with floating ip, use "nova list" will output the instances

    [root@localhost ~(keystone_demo)]# nova list
| ID | Name | Status | Task State | Power State | Networks |
| ac82fcc8-1609-4d34-a4a7-80e5985433f7 | demo-inst1 | ACTIVE | - | Running | private=, |
| f302a03f-3761-48e6-a786-45b324182545 | demo-instance | ACTIVE | - | Running | private=, |

Test the connectivity via floating ip, "ping" at the OpenStack VM, will properbly get outputs like:

    [root@localhost ~(keystone_demo)]# ping
    PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=63 time=1.76 ms
64 bytes from icmp_seq=2 ttl=63 time=0.666 ms
64 bytes from icmp_seq=3 ttl=63 time=0.284 ms
Recent Comments
security group is not supported with the already released OpenStack version. But we have fixed this and it is now waiting for more... Read More
Friday, 08 January 2016 01:40
thanks... Read More
Thursday, 21 January 2016 01:25
Continue reading
13045 Hits

iSCSI and Jumbo Frames

So, you either just setup iSCSI or are having performance issues with your current iSCSI device. Here are some pointers to ensure "networking" is not the limiting factor:

1. Are my packets even making it to the iSCSI target?
Always check in XenCenter that your NICS responsible for storage are pointing to the correct target IPS. If they are, ensure you can ping these targets from within XenServer's command line:

ping x.x.x.x

If you cannot ping the target, that may be the issue.

Use the 'route' command to show if XenServer has a device and target to hit on the iSCSI target's subnet. If route shows nothing related to your iSCSI target IPs or takes a long time to show the target's IP/Route information, revisit your network configuration: working from the iSCSI device config, switch ports, and all the way up to the storage interface defined for your XenServer(s).

Odds are the packets are trying to route out via another interface or there is a cable mismatch/VLAN tag mismatch.  Or, at worse, the network cable is bad!

2. Is your network really setup for Jumbo Frames?
If you can ping our iSCSI targets, but Re having performance issues with Jumbo Frames (9000 or 4500 Mtu size, based on vendor) ensure your storage interface on XenServer is configured to leverage this Mtu size.

One can also execute a ping command to see if there is fragmentation or support enabled for the larger MTUs:

ping x.x.x.x -M do -s 8972

This tells XenServer to ping, without fragmenting frames, your iSCSI target with an Mtu of 9000 (the rest comes from the ping and other overhead, so use 8972).

If this return fragments or other errors, check the cabling from XenServer along with the switch settings AND iSCSI setup. Sometimes these attributes can be powered after firmware updates to the iSCSI enabled, managed storage devicd

3. Always make sure your network firmware and drivers are up to date!

And these are but three simple ways to isolate issues with iSCSI connectivity/performance.  The rest, well, more to come...

--jkbs | @xenfomation | Blog

Recent Comments
Tobias Kreidl
Thanks for posting this, Jesse. There are of course numerous tweaks possible to improve stock network settings, published by Citri... Read More
Saturday, 18 April 2015 05:23
JK Benedict
Quite welcome, Tobias and it is always great to hear from you! The article you sent is, well, quite amazing. I have seen in trad... Read More
Monday, 20 April 2015 14:20
Continue reading
16072 Hits

Before Electing a New Pool Master


The following is a reminder of specific steps to take before electing a new pool master - especially in High Availability-enabled deployments.  Albeit, there are circumstances where this will happen automatically due to High Availability (by design) or in an emergency situation, but never-the-less, the following steps should be taken when electing a new pool master where High Availability is enabled.

Disable High Availability

Before electing a new master one must disable High Availability.  The reason is quite simple:

If a new host is designated as master with HA enabled, the subsequent processes and transition time can lead to HA see that a pool member is down.  It is doing what it is supposed to do from the "mathematical" sense, but from "reality" it is actually confused.

The end result is that HA could either recover with some time or fence as it attempts to apply fault tolerance in contradiction to the desire to "simply elect a new master".

It is also worth noting that upon recovery - if any Guests which had a mounted ISO are rebooted on another host - that "VDI not found" errors can appear although this is not the case.  The ISO image that is mounted is seen as a VDI and if that resource is not available on another host, the Guest VM will fail to resume: presenting the generic VDI error.

Steps to Take

HA must be disabled and for safe practice, I always recommend ejecting all mounted ISO images.  The latter can be accomplished by executing the following from the pool master:

xe vm-cd-eject --multiple

As for HA it can be disabled in two ways: via the command-line or from XenCenter.

From the command line of the current pool master, execute:

xe pool-ha-disable
xe pool-sync

If desired - just for safe guarding one's work - those commands can be executed on every other pool member.

As for XenCenter one can select the Pool/Pool Master icon in question and from the "HA" tab, select the option to disable HA for the pool.

Workload Balancing

For versions of XenServer utilizing Workload Balancing it is not necessary to halt this.

Now that HA is disabled, switch Pool Masters and when all servers are in an active state: re-enable HA from XenCenter or from the command line:

xe pool-recover-slaves
xe pool-ha-enable

I hope this is helpful and as always: questions and comments are welcomed!


--jkbs | @xenfomation

Continue reading
12276 Hits

About XenServer

XenServer is the leading open source virtualization platform, powered by the Xen Project hypervisor and the XAPI toolstack. It is used in the world's largest clouds and enterprises.
Commercial support for XenServer is available from Citrix.