Virtualization Blog

Discussions and observations on virtualization.

Implementing VDI-per-LUN storage

With storage providers adding better functionality to provide features like QoS, fast snapshot & clone and with the advent of storage-as-a-service, we are interested in the ability to utilize these features from XenServer. VMware’s VVols offering already allows integration of vendor provided storage features into their hypervisor. Since most storage allows operations at the granularity of a LUN, the idea is to have a one-to-one mapping between a LUN on the backend and a virtual disk (VDI) on the hypervisor. In this post we are going to talk about the supplemental pack that we have developed in order to enable VDI-per-LUN.

Xenserver Storage

To understand the supplemental pack, it is useful to first review how XenServer storage works. In XenServer, a storage repository (SR) is a top-level entity which acts as a pool for storing VDIs which appear to the VMs as virtual disks. XenServer provides different types of SRs (File, NFS, Local, iSCSI). In this post we will be looking at iSCSI based SRs as iSCSI is the most popular protocol for remote storage and the supplemental pack we developed is targeted towards iSCSI based SRs. An iSCSI SR uses LVM to store VDIs over logical volumes (hence the type is lvmoiscsi). For instance:

[root@coe-hq-xen08 ~]# xe sr-list type=lvmoiscsi
uuid ( RO)                : c67132ec-0b1f-3a69-0305-6450bfccd790
          name-label ( RW): syed-sr
    name-description ( RW): iSCSI SR [ (; LUN 0: 6090A028408B4FC2846E4536000000B6: 10 GB (EQLOGIC))]
                host ( RO): coe-hq-xen08
                type ( RO): lvmoiscsi
        content-type ( RO):

The above SR is created from a LUN on a Dell EqualLogic. The VDIs belonging to this SR can be listed by:

[root@coe-hq-xen08 ~]# xe vdi-list sr-uuid=c67132ec-0b1f-3a69-0305-6450bfccd790 params=uuid
uuid ( RO)    : ef5633d2-2ad0-4996-8635-2fc10e05de9a

uuid ( RO)    : b7d0973f-3983-486f-8bc0-7e0b6317bfc4

uuid ( RO)    : bee039ed-c7d1-4971-8165-913946130d11

uuid ( RO)    : efd5285a-3788-4226-9c6a-0192ff2c1c5e

uuid ( RO)    : 568634f9-5784-4e6c-85d9-f747ceeada23

[root@coe-hq-xen08 ~]#

This SR has 5 VDI. From LVM’s perspective, an SR is a volume group (VG) and each VDI is a logical volume(LV) inside that volume group. This can be seen via the following commands:

[root@coe-hq-xen08 ~]# vgs | grep c67132ec-0b1f-3a69-0305-6450bfccd790
  VG_XenStorage-c67132ec-0b1f-3a69-0305-6450bfccd790   1   6   0 wz--n-   9.99G 5.03G
[root@coe-hq-xen08 ~]# lvs VG_XenStorage-c67132ec-0b1f-3a69-0305-6450bfccd790
  LV                                       VG                                                 Attr   LSize 
  MGT                                      VG_XenStorage-c67132ec-0b1f-3a69-0305-6450bfccd790 -wi-a-   4.00M                                 
  VHD-568634f9-5784-4e6c-85d9-f747ceeada23 VG_XenStorage-c67132ec-0b1f-3a69-0305-6450bfccd790 -wi-ao   8.00M                               
  VHD-b7d0973f-3983-486f-8bc0-7e0b6317bfc4 VG_XenStorage-c67132ec-0b1f-3a69-0305-6450bfccd790 -wi-ao   2.45G                               
  VHD-bee039ed-c7d1-4971-8165-913946130d11 VG_XenStorage-c67132ec-0b1f-3a69-0305-6450bfccd790 -wi---   8.00M                                
  VHD-ef5633d2-2ad0-4996-8635-2fc10e05de9a VG_XenStorage-c67132ec-0b1f-3a69-0305-6450bfccd790 -ri-ao   2.45G
VHD-efd5285a-3788-4226-9c6a-0192ff2c1c5e VG_XenStorage-c67132ec-0b1f-3a69-0305-6450bfccd790 -ri-ao  36.00M

Here c67132ec-0b1f-3a69-0305-6450bfccd790 is the UUID of the SR. Each VDI is represented by a corresponding LV which is of the format VHD-. Some of the LVs have a small size of 8MB. These are snapshots taken on XenServer. There is also a LV named MGT which holds metadata about the SR and the VDIs present in it. Note that all of this is present in an SR which is a LUN on the backend storage.

Now XenServer can attach a LUN at the level of an SR but we want to map a LUN to a single VDI. In order to do that, we restrict an SR to contain a single VDI. Our new SR has the following LVs:

[root@coe-hq-xen09 ~]# lvs VG_XenStorage-1fe527a4-7e96-cdd9-f347-a15c240f26e9
LV                                       VG                                                 Attr   LSize
MGT                                      VG_XenStorage-1fe527a4-7e96-cdd9-f347-a15c240f26e9 -wi-a- 4.00M
VHD-09b14a1b-9c0a-489e-979c-fd61606375de VG_XenStorage-1fe527a4-7e96-cdd9-f347-a15c240f26e9 -wi--- 8.02G
[root@coe-hq-xen09 ~]#


If a snapshot or clone of the LUN is taken on the backend, all the unique identifiers associated with the different entities in the LUN also get cloned and any attempt to attach the LUN back to XenServer will result in an error because of conflicts of unique IDs.

Resignature and supplemental pack

In order for the cloned LUN to be re-attached, we need to resignature the unique IDs present in the LUN. The following IDs need to be resignatured

  • LVM UUIDs (PV, VG, LV)
  • SR metadata in the MGT Logical volume

We at CloudOps have developed an open-source supplemental pack which solves the resignature problem. You can find it here. The supplemental pack adds a new type of SR (relvmoiscsi) and you can use it to resignature your lvmoiscsi SRs. After installing the supplemental pack, you can resignature a clone using the following command

[root@coe-hq-xen08 ~]# xe sr-create name-label=syed-single-clone type=relvmoiscsi 
Error parameters: , Error reporting error, unknown key The SR has been successfully resigned. Use the lvmoiscsi type to attach it,
[root@coe-hq-xen08 ~]#

Here, instead of creating a new SR, the supplemental pack re-signatures the provided LUN and detaches it (the error is expected as we don’t actually create an SR). You can see from the error message that the SR has been re-signed successfully. Now the cloned SR can be introduced back to XenServer without any conflicts using the following commands:

[root@coe-hq-xen09 ~]# xe sr-probe type=lvmoiscsi device-config:target= device-config:targetIQN=$IQN device-config:SCSIid=$SCSIid


[root@coe-hq-xen09 ~]# xe sr-introduce name-label=vdi-test-resign type=lvmoiscsi 

This supplemental pack can be used in conjunction with an external orchestrator like CloudStack or OpenStack which can manage both the storage and compute. Working with SolidFire we have implemented this functionality, available in the next release of Apache CloudStack. You can check out a preview of this feature in a screencast here.

Recent Comments
If I am reading this correctly, this is just basically setting up XS to use 1 SR per VM, this isn't scalable as the limits for LUN... Read More
Tuesday, 26 April 2016 14:57
Syed Ahmed
Hi Nick, The limit of 256 SRs is when using Multipating. If no multipath is used, the number of SRs that can be created are well... Read More
Tuesday, 26 April 2016 17:19
Syed Ahmed
There is an initial overhead when creating SRs. However, we did not find any performance degradation in our tests once the SR is s... Read More
Wednesday, 27 April 2016 09:21
Continue reading
4265 Hits

Citrix Joins OpenStack Foundation

Some of you might have noticed that Citrix joined the OpenStack Foundation yesterday and may be wondering what this means for two key technologies I've been closely involved with; Apache CloudStack and XenServer. The first, and arguably most important thing to note is that as Steve Wilson has stated, we're embracing both OpenStack and CloudStack to help further innovation. Nand Mulchandani also highlights that a culture of “anyness” is a core part of Citrix. With all the noise in the market about the various IaaS cloud solutions, supporting user choice is an important point to be clear on. So with that as backdrop, what does this really mean?

The XenServer Perspective on OpenStack

As I mentioned in my blog about OpenStack Summit, I really want XenServer to be a first class citizen within OpenStack. I tried to further that objective through submission of presentations to OpenStack Summit, but if you look at the schedule you'll note that no XenServer related talks were accepted. That's unfortunate, and really speaks to the challenge we face within a community when we're not the obvious or default choice. Obviously we can raise our profile through contributions and simply showing up at OpenStack events, but there is also a pretty important and easy thing we can change.

When a vendor evaluates a technology, they look at the ecosystem around it. OpenStack technology has a ton of buzz. If you look on job boards, you'll see many postings for OpenStack positions. If you search for cloud technologies, key supporters of OpenStack will be listed. Importantly, when selecting a technology suite, you'll look at who supports their technology with the suite and use them in your short list. Until today, it was unclear if Citrix actively supported the use of XenServer within OpenStack. Our joining the OpenStack Foundation is one way of signaling to those who prefer OpenStack that Citrix is supportive of their efforts. So if you've been quietly using XenServer in an OpenStack environment, I want to learn more about it. I want to learn what works, and where the pain points are so they might be addressed. If you've ever questioned if production support for XenServer when used with OpenStack could be supported, the answer is yes, and here's a link to buy support (hard sell over)!

The XenServer Perspective on CloudStack

For those of you who have adopted XenServer for your CloudStack clouds, nothing has changed and you should feel nothing change. XenServer will remain a first class citizen in CloudStack, and we'll continue to improve all aspects of XenServer operation within CloudStack such that XenServer remains an obvious choice. You'll continue to see XenServer content proposed to CloudStack events, and I hope you'll continue to accept those talks. I promise to continue to work on cool things like the Packer work I presented at CloudStack Day Austin which showed a method to migrate legacy infrastructure running on XenServer to a CloudStack cloud powered by XenServer; all without the users even noticing the migration happened. My hope is that the OpenStack community will want some of those same cool things, but that will take time and can't be forced.

So in the end this really isn't a commentary about which cloud solution is better, but a case of allowing customer choice. OpenStack has mindshare, and it only makes sense for Citrix and its technology suite to have a seat at the table. With Citrix openly supporting its technologies when deployed with OpenStack, everyone has the freedom to choose which solution works best.     

Recent Comments
I would like to see XenServer in OpenStack. At the moment we use XenServer on all our servers but we are looking for a solution li... Read More
Tuesday, 28 April 2015 08:07
Tim Mackey
Sebastian, XenServer is supported through the use of the "xapi" Nova driver in OpenStack, and also within CloudStack. Both OpenS... Read More
Tuesday, 28 April 2015 13:29
Continue reading
11720 Hits

About XenServer

XenServer is the leading open source virtualization platform, powered by the Xen Project hypervisor and the XAPI toolstack. It is used in the world's largest clouds and enterprises.
Commercial support for XenServer is available from Citrix.