Import and Export Virtual Disk Images

Overview

When attempting to perform import and export operations on disks, such as backup and restore functions, which directly access the virtual disk image; XenServer provides the transfer plugin XAPI plugin.  This plugin is automatically installed in dom0 and presents the calls with an API exposing the disks associated with a given VM.  The transfer plugin interacts with the transfer VM supplied by XenServer to upload or download the virtual disk in a number of different formats.  

During XenServer host installation, or upgrade, a hidden template is created for the transfer VM.  On any given host, there will only ever exist a single instance of this template.  In the event the template isn't found, or that multiple instances are found, the transfer plugin will raise an exception.  All exception information during execution is raised in syslog. During execution, multiple instances of the transfer VM may be running to support concurrent operations.  

The transfer VM is natively used within XenCenter for import/export operations to facilitate bulk conversions of VMDK disks from vSphere environments to XenServer.  Conversion of files from OVF/OVA, disk images based on VHD or VMDK, and XenServer XVA formats are supported.  Details on importing from these various formats is covered in the XenCenter documentation under VM Import and Export.

Source Code

The source code for the transfer VM can be found on GitHub at: https://github.com/xenserver/transfervm, and the API is also on GitHub in the project wiki: https://github.com/xenserver/transfervm/wiki.

Plugin Details

Name

The transfer plugin is named transfer. 

Function

The plugin provides an API for launching, configuring, tearing down, and interrogating transfer VM instances, thus exposing VDIs as requested.

Required Ports

Access to the plugin requires HTTP/BITS (TCP 80/443) or iSCSI (TCP 3260/3261).

Usage Overview

All methods are synchronous, and block until they have successfully completed or failed. To call the methods asynchronously, use the call_plugin_async message to the XenServer host. All methods take a string-to-string map of arguments. The keys are described in the plugin API, and the values must contain appropriate data (for example integers or IP addresses). The values are restricted to characters a-zA-Z0-9, and may not be empty strings.

The following sequence is used to transfer a single VDI associated with a VM. 

1. Client calls the expose method in the transfer plugin on the host that owns the VDI. This returns a record handle for use with the subsequent calls.
2. Client calls the get_record method in the transfer plugin to get the new network address, authentication details and other metadata of the exposed VDI.
3. Client connects to this address using a HTTP, BITS or iSCSI client and transfers data.
4. Client calls the unexpose method in the transfer plugin to shut down the network access and release locks on the VDI.

Exceptions

In addition to the method specific exceptions ones, all methods MAY raise

  • ArgumentError if a required argument is missing, argument values are invalid, or incompatible arguments are given.
  • ConfigurationError if the server state is inconsistent, and cannot be safely fixed. (For example: the utility VM template is missing, or the cloned Transfer VM fails to start.)
  • TransferPluginError if there are any other failures.

The server-side plugin exceptions are raised as XenAPI.Failure in the client code. The server-side exception name and message are given in the client-side exception details.

Transfering Snapshot Trees

The method covered in the usage overview is applicable when transferring a single VDI. There is an additional API that allows you to transfer an entire tree of virtual machines (the primary VM, and all its snapshots). This will preserve sparseness in the snapshot tree.

This transfer necessarily uses HTTP GET for download, and BITS for upload. It is not supported for any other transfer mode.

Given a VM, there are a number of VDI trees, formed from the chains of snapshot deltas. Each VDI attached to the VM requires a base disk, plus a tree rooted at that base disk for each of the snapshots that was taken. There is also a VDI containing the VM memory image, if the snapshot included one.

The collection of trees of VDIs (plus all the associated metadata) is called the "forest".

The complete metadata representing the VM, all its snapshots and all the VDI trees is exported from xapi as a blob. For import, this blob is given back to XenServer as the contents of yet another VDI.

Export 

  1. Download the metadata for the VM, using the /export_metadata xapi URL, and include_vhd_parents=true. Save this as metadata-<VM UUID>.raw.
  2. Retrieve the list of all snapshots belonging to this VM, using the Xen-API call VM.get_snapshots().
  3. Use transfer.expose_forest to expose the entire VDI forest. One Transfer VM will be created per VDI tree, and each VM will have a number of VDIs attached, one for each of the leaves of the tree. A comma-separated list of handles will be returned.
  4. For each handle:
    1. Use transfer.get_record to get the appropriate metadata. This will include a number of leaf VDI UUIDs (in record[XSAG:'vdi_uuid']) and a number of non-leaf VDI UUIDs (in record[XSAG:'non_leaf_vdi_uuids']). These are both comma-separated lists of VDI UUIDs.
    2. For each VDI UUID (whether leaf or non-leaf):
      1. Use HTTP GET to download from record[XSAG:'url_full_<VDI UUID>'] + '.vhd' into <VDI UUID>.vhd.
  5. Unexpose the forest: for each handle:
    1. Use transfer.unexpose

A full example is available on GitHub: https://github.com/xenserver/transfervm/blob/master/transfertests/test_scvmm_get.py

Import

  1. Select a destination SR for the import.
  2. Upload the metadata for the VM from metadata-<VM UUID>.raw:
    1. Create a VDI big enough to contain metadata-<VM UUID>.raw. This needs to be rounded up to the nearest 512 byte
      boundary, because the Transfer VM cannot write to the final block if it's smaller than 512 bytes.
    2. Use transfer.expose to expose the new VDI.
    3. Use transfer.get_record to get the expose metadata.
    4. Upload metadata-<VM UUID>.raw to record[XSAG:'url_full'].
    5. Use transfer.unexpose to close down.
  3. Get the import instructions: use transfer.get_import_instructions(vm_metadata_vdi_uuid=<metadata VDI UUID>). This returns a newline-separated list of instructions to execute next.
  4. Execute the instructions. Each instruction starts with a command, and is followed by a number of parameters, all space-separated. These must be executed in order, and a map of old VDI UUID -> new VDI UUID is maintained across this process.
    1. create(old_vdi_uuid, virtual_size): new_vdi_uuid = VDI.create(size=virtual_size). BITS upload old_vdi_uuid.vhd to new_vdi_uuid. Set vdi_map[XSAG:old_vdi_uuid] = new_vdi_uuid.
    2. clone(child_uuid, parent_uuid): new_vdi_uuid = VDI.clone(vdi_map[XSAG:parent_uuid]). BITS upload child_uuid.vhd to new_vdi_uuid. Set vdi_map[XSAG:child_uuid] = new_vdi_uuid.
    3. reuse(child_uuid, parent_uuid): BITS upload child_uuid.vhd to vdi_map[XSAG:parent_uuid]. Set vdi_map[XSAG:child_uuid] = vdi_map[XSAG:parent_uuid]. Delete vdi_map[XSAG:parent_uuid].
    4. snap(vdi_uuid): dest_uuid = vdi_map[XSAG:vdi_uuid]; new_dest_uuid = VDI.snapshot(dest_uuid); VDI.destroy(dest_uuid); VDI.set_name_label(new_dest_uuid, 'Copy of %s' % vdi_uuid); vdi_map[XSAG:vdi_uuid] = new_dest_uuid.
    5. leaf(vdi_uuid): dest_uuid = vdi_map[XSAG:vdi_uuid]; VDI.set_name_label(dest_uuid, 'Copy of %s' % vdi_uuid); vdi_map[XSAG:vdi_uuid] = dest_uuid.
    6. pass: Do nothing. Skip on to the next instruction.
  5. Once all instructions have been executed, vdi_map will contain the mapping between the old UUIDs and the new ones, for all VDIs that are being imported. This then needs to be passed to plugin for the final metadata import. Call{{transfer.remap_vm(vm_metadata_vdi_uuid=<metadata VDI UUID>, vdi_map=<string form of vdi_map>, sr_uuid=<destination SR UUID>}}.

CLI Example

A complete example of the transfer plugin usage in CLI is on GitHub: https://github.com/xenserver/transfervm/wiki/Usage-Examples 

Debugging Using XenCenter

XenCenter natively leverages the transfer VM to import VMDK or OVF files, and to export OVF files.  When debugging the transfer VM or any process associated with it, localization of the problem can be accomplished by first determining if the transfer VM is functioning correctly.  Complete documentaion on the import and export of VMs from XenCenter is covered in the XenCenter documentation.  The implementation of import/export within XenCenter is located in the XenOvfTransport module of xenadmin.

About XenServer

XenServer is the leading open source virtualization platform, powered by the Xen Project hypervisor and the XAPI toolstack. It is used in the world's largest clouds and enterprises.
 
Commercial support for XenServer is available from Citrix.