Virtualization Blog

Discussions and observations on virtualization.

Implementing VDI-per-LUN storage

With storage providers adding better functionality to provide features like QoS, fast snapshot & clone and with the advent of storage-as-a-service, we are interested in the ability to utilize these features from XenServer. VMware’s VVols offering already allows integration of vendor provided storage features into their hypervisor. Since most storage allows operations at the granularity of a LUN, the idea is to have a one-to-one mapping between a LUN on the backend and a virtual disk (VDI) on the hypervisor. In this post we are going to talk about the supplemental pack that we have developed in order to enable VDI-per-LUN.

Xenserver Storage

To understand the supplemental pack, it is useful to first review how XenServer storage works. In XenServer, a storage repository (SR) is a top-level entity which acts as a pool for storing VDIs which appear to the VMs as virtual disks. XenServer provides different types of SRs (File, NFS, Local, iSCSI). In this post we will be looking at iSCSI based SRs as iSCSI is the most popular protocol for remote storage and the supplemental pack we developed is targeted towards iSCSI based SRs. An iSCSI SR uses LVM to store VDIs over logical volumes (hence the type is lvmoiscsi). For instance:

[root@coe-hq-xen08 ~]# xe sr-list type=lvmoiscsi
uuid ( RO)                : c67132ec-0b1f-3a69-0305-6450bfccd790
          name-label ( RW): syed-sr
    name-description ( RW): iSCSI SR [172.31.255.200 (iqn.2001-05.com.equallogic:0-8a0906-c24f8b402-b600000036456e84-syed-iscsi-opt-test; LUN 0: 6090A028408B4FC2846E4536000000B6: 10 GB (EQLOGIC))]
                host ( RO): coe-hq-xen08
                type ( RO): lvmoiscsi
        content-type ( RO):

The above SR is created from a LUN on a Dell EqualLogic. The VDIs belonging to this SR can be listed by:

[root@coe-hq-xen08 ~]# xe vdi-list sr-uuid=c67132ec-0b1f-3a69-0305-6450bfccd790 params=uuid
uuid ( RO)    : ef5633d2-2ad0-4996-8635-2fc10e05de9a

uuid ( RO)    : b7d0973f-3983-486f-8bc0-7e0b6317bfc4

uuid ( RO)    : bee039ed-c7d1-4971-8165-913946130d11

uuid ( RO)    : efd5285a-3788-4226-9c6a-0192ff2c1c5e

uuid ( RO)    : 568634f9-5784-4e6c-85d9-f747ceeada23

[root@coe-hq-xen08 ~]#

This SR has 5 VDI. From LVM’s perspective, an SR is a volume group (VG) and each VDI is a logical volume(LV) inside that volume group. This can be seen via the following commands:

[root@coe-hq-xen08 ~]# vgs | grep c67132ec-0b1f-3a69-0305-6450bfccd790
  VG_XenStorage-c67132ec-0b1f-3a69-0305-6450bfccd790   1   6   0 wz--n-   9.99G 5.03G
[root@coe-hq-xen08 ~]# lvs VG_XenStorage-c67132ec-0b1f-3a69-0305-6450bfccd790
  LV                                       VG                                                 Attr   LSize 
  MGT                                      VG_XenStorage-c67132ec-0b1f-3a69-0305-6450bfccd790 -wi-a-   4.00M                                 
  VHD-568634f9-5784-4e6c-85d9-f747ceeada23 VG_XenStorage-c67132ec-0b1f-3a69-0305-6450bfccd790 -wi-ao   8.00M                               
  VHD-b7d0973f-3983-486f-8bc0-7e0b6317bfc4 VG_XenStorage-c67132ec-0b1f-3a69-0305-6450bfccd790 -wi-ao   2.45G                               
  VHD-bee039ed-c7d1-4971-8165-913946130d11 VG_XenStorage-c67132ec-0b1f-3a69-0305-6450bfccd790 -wi---   8.00M                                
  VHD-ef5633d2-2ad0-4996-8635-2fc10e05de9a VG_XenStorage-c67132ec-0b1f-3a69-0305-6450bfccd790 -ri-ao   2.45G
VHD-efd5285a-3788-4226-9c6a-0192ff2c1c5e VG_XenStorage-c67132ec-0b1f-3a69-0305-6450bfccd790 -ri-ao  36.00M

Here c67132ec-0b1f-3a69-0305-6450bfccd790 is the UUID of the SR. Each VDI is represented by a corresponding LV which is of the format VHD-. Some of the LVs have a small size of 8MB. These are snapshots taken on XenServer. There is also a LV named MGT which holds metadata about the SR and the VDIs present in it. Note that all of this is present in an SR which is a LUN on the backend storage.

Now XenServer can attach a LUN at the level of an SR but we want to map a LUN to a single VDI. In order to do that, we restrict an SR to contain a single VDI. Our new SR has the following LVs:

[root@coe-hq-xen09 ~]# lvs VG_XenStorage-1fe527a4-7e96-cdd9-f347-a15c240f26e9
LV                                       VG                                                 Attr   LSize
MGT                                      VG_XenStorage-1fe527a4-7e96-cdd9-f347-a15c240f26e9 -wi-a- 4.00M
VHD-09b14a1b-9c0a-489e-979c-fd61606375de VG_XenStorage-1fe527a4-7e96-cdd9-f347-a15c240f26e9 -wi--- 8.02G
[root@coe-hq-xen09 ~]#

b2ap3_thumbnail_vdi-lun.png

If a snapshot or clone of the LUN is taken on the backend, all the unique identifiers associated with the different entities in the LUN also get cloned and any attempt to attach the LUN back to XenServer will result in an error because of conflicts of unique IDs.

Resignature and supplemental pack

In order for the cloned LUN to be re-attached, we need to resignature the unique IDs present in the LUN. The following IDs need to be resignatured

  • LVM UUIDs (PV, VG, LV)
  • VDI UUID
  • SR metadata in the MGT Logical volume

We at CloudOps have developed an open-source supplemental pack which solves the resignature problem. You can find it here. The supplemental pack adds a new type of SR (relvmoiscsi) and you can use it to resignature your lvmoiscsi SRs. After installing the supplemental pack, you can resignature a clone using the following command

[root@coe-hq-xen08 ~]# xe sr-create name-label=syed-single-clone type=relvmoiscsi 
device-config:target=172.31.255.200
device-config:targetIQN=$IQN
device-config:SCSIid=$SCSIid
device-config:resign=true
shared=true
Error code: SR_BACKEND_FAILURE_1
Error parameters: , Error reporting error, unknown key The SR has been successfully resigned. Use the lvmoiscsi type to attach it,
[root@coe-hq-xen08 ~]#

Here, instead of creating a new SR, the supplemental pack re-signatures the provided LUN and detaches it (the error is expected as we don’t actually create an SR). You can see from the error message that the SR has been re-signed successfully. Now the cloned SR can be introduced back to XenServer without any conflicts using the following commands:

[root@coe-hq-xen09 ~]# xe sr-probe type=lvmoiscsi device-config:target=172.31.255.200 device-config:targetIQN=$IQN device-config:SCSIid=$SCSIid

   		 5f616adb-6a53-7fa2-8181-429f95bff0e7
   		 /dev/disk/by-id/scsi-36090a028408b3feba66af52e0000a0e6
   		 5364514816

[root@coe-hq-xen09 ~]# xe sr-introduce name-label=vdi-test-resign type=lvmoiscsi 
uuid=5f616adb-6a53-7fa2-8181-429f95bff0e7
5f616adb-6a53-7fa2-8181-429f95bff0e7

This supplemental pack can be used in conjunction with an external orchestrator like CloudStack or OpenStack which can manage both the storage and compute. Working with SolidFire we have implemented this functionality, available in the next release of Apache CloudStack. You can check out a preview of this feature in a screencast here.

XenServer Administrators Handbook Published
Running XenServer... without a server

Related Posts

 

Comments 7

Nick on Tuesday, 26 April 2016 14:57

If I am reading this correctly, this is just basically setting up XS to use 1 SR per VM, this isn't scalable as the limits for LUNs per host is 256 in XS 6.5, so that would limit you to 256 VM's as well.

0
If I am reading this correctly, this is just basically setting up XS to use 1 SR per VM, this isn't scalable as the limits for LUNs per host is 256 in XS 6.5, so that would limit you to 256 VM's as well.
Syed Ahmed on Tuesday, 26 April 2016 17:19

Hi Nick,

The limit of 256 SRs is when using Multipating. If no multipath is used, the number of SRs that can be created are well above 1000

0
Hi Nick, The limit of 256 SRs is when using Multipating. If no multipath is used, the number of SRs that can be created are well above 1000
Guest - zheng.chai@citrix.com on Wednesday, 27 April 2016 02:16

good idea, will this cause too much overhead for Pool to manage too much SRs?

0
good idea, will this cause too much overhead for Pool to manage too much SRs?
Syed Ahmed on Wednesday, 27 April 2016 09:21

There is an initial overhead when creating SRs. However, we did not find any performance degradation in our tests once the SR is set up.

0
There is an initial overhead when creating SRs. However, we did not find any performance degradation in our tests once the SR is set up.
Guest - Davide on Wednesday, 27 April 2016 09:47

AFAIK it's something like the old good way storagelink mange the vm disk on equallogic.
Reading your doc I've notice that to fully use your pack it is required to use some sort or orchestrator which should have built in support for your software. To me sounds like a limitation to not use your good stuff with just plain vanilla XenServer

0
AFAIK it's something like the old good way storagelink mange the vm disk on equallogic. Reading your doc I've notice that to fully use your pack it is required to use some sort or orchestrator which should have built in support for your software. To me sounds like a limitation to not use your good stuff with just plain vanilla XenServer
Syed Ahmed on Wednesday, 27 April 2016 17:33

I really liked StorageLink and honestly I don't know why Citrix moved away from it. This is the next best thing to offer to achive StorageLink like capabilities and because of the way current storage is architectured in Xenserver, we have to use an external orchestrator for enabling this functionality. Xenserver 7 has a better storage architecture though.

0
I really liked StorageLink and honestly I don't know why Citrix moved away from it. This is the next best thing to offer to achive StorageLink like capabilities and because of the way current storage is architectured in Xenserver, we have to use an external orchestrator for enabling this functionality. Xenserver 7 has a better storage architecture though.
Tobias Kreidl on Thursday, 28 April 2016 02:31

I'm curious if the storage I/O rates approach those of directly attached storage to the VMs, that is to say, using iSCSi or NFS to bypass the SR mechanism altogether and creating the connection directly from the local VM but while using XenServer's networking infrastructure. Advantages are lower overhead and multipath capabilities, ability to migrate VMs to other servers or even pools without requiring storage Xenmotion and overall faster I/O than seen on SRs. Disadvantages are the exclusion of standard features associated with SRs, such as vm-export, XenCenter I/O graphs, inability to create copies or clones, and of course, being officially unsupported.

One concern might be the overhead created by having too many storage I/O queues, so a question of how well things really scale when you get into hundreds of VDIs.

0
I'm curious if the storage I/O rates approach those of directly attached storage to the VMs, that is to say, using iSCSi or NFS to bypass the SR mechanism altogether and creating the connection directly from the local VM but while using XenServer's networking infrastructure. Advantages are lower overhead and multipath capabilities, ability to migrate VMs to other servers or even pools without requiring storage Xenmotion and overall faster I/O than seen on SRs. Disadvantages are the exclusion of standard features associated with SRs, such as vm-export, XenCenter I/O graphs, inability to create copies or clones, and of course, being officially unsupported. One concern might be the overhead created by having too many storage I/O queues, so a question of how well things really scale when you get into hundreds of VDIs.

About XenServer

XenServer is the leading open source virtualization platform, powered by the Xen Project hypervisor and the XAPI toolstack. It is used in the world's largest clouds and enterprises.
 
Commercial support for XenServer is available from Citrix.