Virtualization Blog

Discussions and observations on virtualization.
Research Scientist at Adventium Labs in Minneapolis, MN. Recent projects have focused on securing physical resources used by virtual machines in MagranaServer which is built on XenServer, and applications of parallel processing.

Enable XSM on XenServer 6.5 SP1

1 Introduction

Certain virtualization environments require the extra security provided by XSM and FLASK ( XenServer 7 benefits from its upgrade of the control domain to CentOS 7, which includes support for enabling XSM and FLASK. But what about legacy XenServer 6.5 installations that also require the added security? XSM and FLASK may be enabled on XenServer 6.5 as well, but it requires a bit more work.

Note that XSM is not currently a user-visible feature in XenServer, or a supported technology.

This article describes how to enable XSM and FLASK in XenServer 6.5 SP1. It makes the assumption that the reader is familiar with accessing, building, and deploying XenServer's Xen RPMs from source. While this article pertains to resources from SP1 source RPMs (XS65ESP1-src-pkgs.tar.bz2 included with SP1,, a similar approach can be followed for other XenServer 6.5 hotfixes.

2 Patching Xen and xen.spec

XenServer issues some hypercalls not handled by Xen's XSM hooks. The following patch shows one possible way to handle these operations and commands, which is to always permit them.

diff --git a/xs6.5sp1/xen/xen-4.4.1/xen/xsm/flask/hooks.c b/xs6.5sp1/xen/xen-4.4.1/xen/xsm/flask/hooks.c
index 0cf7daf..a41fcc4 100644
--- a/xs6.5sp1/xen/xen-4.4.1/xen/xsm/flask/hooks.c
+++ b/xs6.5sp1/xen/xen-4.4.1/xen/xsm/flask/hooks.c
@@ -727,6 +727,12 @@ static int flask_domctl(struct domain *d, int cmd)
     case XEN_DOMCTL_cacheflush:
         return current_has_perm(d, SECCLASS_DOMAIN2, DOMAIN2__CACHEFLUSH);

+    case XEN_DOMCTL_get_runstate_info:
+        return 0;
+    case XEN_DOMCTL_setcorespersocket:
+        return 0;
         printk("flask_domctl: Unknown op %dn", cmd);
         return -EPERM;
@@ -782,6 +788,9 @@ static int flask_sysctl(int cmd)
     case XEN_SYSCTL_numainfo:
         return domain_has_xen(current->domain, XEN__PHYSINFO);

+    case XEN_SYSCTL_consoleringsize:
+        return 0;
         printk("flask_sysctl: Unknown op %dn", cmd);
         return -EPERM;
@@ -1299,6 +1308,9 @@ static int flask_platform_op(uint32_t op)
     case XENPF_get_cpuinfo:
         return domain_has_xen(current->domain, XEN__GETCPUINFO);

+    case XENPF_get_cpu_features:
+        return 0;
         printk("flask_platform_op: Unknown op %dn", op);
         return -EPERM;

The only other file that needs patching is Xen's RPM spec file, xen.spec. Modify HV_COMMON_OPTIONS as shown below.  Change this line:

% define HV_COMMON_OPTIONS max_phys_cpus=256


% define HV_COMMON_OPTIONS max_phys_cpus=256 XSM_ENABLE=y FLASK_ENABLE=y

3 Compiling and Loading a Policy

To build a security policy, navigate to tools/flask/policy in Xen's source tree. Run make to compile the default security policy. It will have a name like xenpolicy.24, depending on your version of checkpolicy.

Copy xenpolicy.24 over to Dom0's /boot directory. Open /boot/extlinux.conf and modify the default section's append /boot/xen.gz ... line so it has --- /boot/xenpolicy.24 at the end. For example:

append /boot/xen.gz dom0_mem=752M,max:752M [.. snip ..] splash --- /boot/initrd-3.10-xen.img --- /boot/xenpolicy.24

After making this change, reboot.

While booting (or afterwards, via xl dmesg), you should see messages indicating XSM and FLASK initialized, read the security policy, and started in permissive mode. For example:

(XEN) XSM Framework v1.0.0 initialized
(XEN) Policy len  0x1320, start at ffff830117ffe000.
(XEN) Flask:  Initializing.
(XEN) Flask:  Starting in permissive mode.

4 Exercises for the Reader

  1. Create a more sophisticated implementation for handling XenServer hypercalls in xen/xsm/flask/hooks.c.
  2. Write (and load) a custom policy.
  3. Boot with flask_enforcing=1 set, and study any violations that occur (see xl dmesg output).
Recent comment in this post
anshul makkar
Good Work. Once the user builds the default policy, it should work for most of the scenario except for the specialized once like p... Read More
Friday, 28 October 2016 11:22
Continue reading
987 Hits
1 Comment

XenServer 6.5 Can Do True UEFI Boot


We were interested in getting XenServer 6.5 to boot via UEFI.  Leaving servers in Legacy/BIOS boot was not an option in our target environment.  We still have to do the initial install with the server in Legacy BIOS mode; however, I managed to compile Xen as an EFI bootable binary using the source and patches distributed by Citrix.  With that I am able to change the servers boot mode back to UEFI and boot XenServer.  Here are the steps I used to compile it.


  1. Prepare a DDK
  2. Prepare a build environment
  3. Build some prerequisites
  4. Unpack the SRPM
  5. Compile Xen

DDK Preparation

Development will be done inside a 6.5 DDK. This is a CentOS 5.4-based Linux that has the same kernel as Dom0 and some of the required development tools. 

Import the VM template per Citrix DDK developer documentation.

After importing, set the following VM options:

  • 2 vCPUs
  • Increase memory to 2048MB
  • Resize disk image to 10GB
  • Add a network interface for SSH

Start the VM, set a root password, and then finalize resizing the disk by running:

# fdisk /dev/xvda 
cmd: d  -del
cmd:  n  -new
cmd:  p  -primary
cmd:  1  -num 1
	use default size
cmd:  w  -save

Preparing the Build Environment

Install rpmdevtools:

# yum --disablerepo citrix install rpmdevtools
# rpmdev-setuptree

I also needed several packages many of which were provided on the binpkg ISO from Citrix. I made them available by inserting XenServer-6.5-binpkg.iso and running the following:

# mount /dev/xvdb /mnt
# mkdir /opt/binpkg
# cp -a /mnt/domain0/RPMS/* /opt/binpkg
# cd /opt/binpkg
# createrepo /opt/binpkg
# cat >/etc/yum.repos.d/binpkg.repo

I also added the epel repository:

rpm -Uvh

and I added the following packages:


Building the Prerequisites

This page: says that it is required to use gcc 4.5 or better and that that binutils must be compiled with --enable-targets=x86_64-pep. I could not satisfy this with packages in the repositories, so I compiled and installed some requirements for gcc:

and made sure the required files could be found:

# export LD_LIBRARY_PATH=:/usr/lib:/usr/local/lib:/usr/local/lib64:/usr/lib64
# ln -s /usr/lib64/

then I compiled binutils using:

# tar xzf binutils-2.22.tar.gz
# mkdir binutils-build
# cd binutils-build
# ../binutils-2.22/configure --disable-werror --enable-targets=x86_64-pep
# make
# make install

then I compiled gcc I using: I adapted compilation instructions from:

# cd gcc-4.6.2
# sed -i 's/install_to_$(INSTALL_DEST) //' libiberty/
# case `uname -m` in
i?86) sed -i 's/^T_CFLAGS =$/& -fomit-frame-pointer/' 
gcc/ ;;
# sed -i 's@./ true@' gcc/
# mkdir -v ../gcc-build
# cd ../gcc-build
# ../gcc-4.6.2/configure --prefix=/usr 
--libexecdir=/usr/lib --enable-shared 
--enable-threads=posix --enable-__cxa_atexit 
--enable-clocale=gnu --enable-languages=c,c++ 
--disable-multilib --disable-bootstrap --with-system-zlib
# make -j3
# ulimit -s 16384
# make -k check
# ../gcc-4.6.2/contrib/test_summary
# make install

Unpack the SRPM

At this point everything is ready to compile Xen as an EFI bootable binary.  The source code for Xen with Citrix's patches is available here: so download XenServer-6.5.0-source-main-1.iso and mount it at /mnt/cdrom:

# mount -r /dev/xvdb /mnt/cdrom

Now install the SRPM:

# rpm -ivh /mnt/cdrom/xen/xen-4.4.1- 

Compile Xen

The source code and required scripts are now all under /root/rpmbuild/, so just run:

# cd ~/rpmbuild
# QA_RPATHS=$[ 0x0020 ] rpmbuild -bc SPECS/xen.spec

The -bc flag causes the process to follow the spec file and patch the source, but then stop just before running the make commands. The make commands would fail to compile with warnings about uninitialized variables being treated as errors. Fix this by changing line 45 of ~/rpmbuild/BUILD/xen-4.4.1/xen/ to read:

CFLAGS += -Werror -Wno-error=uninitialized -Wredundant-decls

and line 39 of ~/rpmbuild/BUILD/xen-4.4.1/ to read:

HOSTCFLAGS = -Wall -Werror -Wno-error=uninitialized -Wstrict-prototypes -O2 -fomit-frame-pointer

after making those changes run:

# cd ~/rpmbuild/BUILD/xen-4.4.1/
# make clean
# make max_phys_cpus=256 XEN_TARGET_ARCH=x86_64 -C xen
XEN_VENDORVERSION=-xs90192 debug=n build

and the compile will finish successfully.  I probably could have just made a quick patch to add the -Wno-error flag and allowed the rpmbuild to run the full spec file, but I didn't actually need to compile xen-tools etc, those are already compiled and installed on the XenServer installation. The only file needed is ~/rpmbuild/BUILD/xen-4.4.1/xen/xen.efi. With that in hand I created a xen.cfg file like this:


options=console=vga,com1,com2 com1=115200,8n1,0x3F8,4
com2=115200,8n1,0x2F8,3 loglvl=all noreboot
kernel=vmlinuz-3.10-xen root=UUID=b4ee0ace-b587-41df-a66b-16f89731b2a8
rw ignore_loglevel acpi_rsdp=0x7B7FE014

where the root UUID is the boot disk created during the XenServer install and the RSDP number came from running:

# dmesg | grep RSDP

I ran that in an EFI booted live Linux environment. I found that some
vendors' UEFI implementations were able to provide the RSDP during boot
and some were not, so without specifying it in the xen.cfg I had trouble
with things like usb peripherals.

Boot XenServer

With the xen.efi and xen.cfg I was able to boot XenServer in UEFI boot mode using refind.  We have done extensive testing on several different servers and found no problems.  I was also able to repeat the process with the source code provided by the service packs up to and including Service Pack 1.  I haven't tried any further than that yet.

Editors Note

For those of you wishing to retain Citrix commercial support status, the above procedure will convert the XenServer 6.5 host into an "unsupported configuration".


Continue reading
6937 Hits

About XenServer

XenServer is the leading open source virtualization platform, powered by the Xen Project hypervisor and the XAPI toolstack. It is used in the world's largest clouds and enterprises.
Commercial support for XenServer is available from Citrix.