Virtualization Blog

Discussions and observations on virtualization.

NAU VMbackup 3.0 for XenServer

NAU VMbackup 3.0 for XenServer

By Tobias Kreidl and Duane Booher

Northern Arizona University, Information Technology Services

Over eight years ago, back in the days of XenServer 5, not a lot of backup and restore options were available, either as commercial products or as freeware, and we quickly came to the realization that data recovery was a vital component to a production environment and hence we needed an affordable and flexible solution. The conclusion at the time was that we might as well build our own, and though the availability of options has grown significantly over the last number of years, we’ve stuck with our own home-grown solution which leverages Citrix XenServer SDK and XenAPI (http://xenserver.org/partners/developing-products-for-xenserver.html). Early versions were created from the contributions of Douglas Pace, Tobias Kreidl and David McArthur. During the last several years, the lion’s share of development has been performed by Duane Booher. This article discusses the latest 3.0 release.

A Bit of History

With the many alternatives now available, one might ask why we have stuck with this rather un-flashy script and CLI-based mechanism. There are clearly numerous reasons. For one, in-house products allow total control over all aspects of their development and support. The financial outlay is all people’s time and since there are no contracts or support fees, it’s very controllable and predictable. We also found from time-to-time that various features were not readily available in other sources we looked at. We also felt early on as an educational institution that we could give back to the community by freely providing the product along with its source code; the most recent version is available via GitHub at https://github.com/NAUbackup/VmBackup for free under the terms of the GNU General Public License. There was a write-up at https://www.citrix.com/blogs/2014/06/03/another-successful-community-xenserver-sdk-project-free-backup-tools-and-scripts-naubackup-restore-v2-0-released/ when the first GitHub version was published. Earlier versions were made available via the Citrix community site (Citrix Developer Network), sometimes referred to as the Citrix Code Share, where community contributions were published for a number of products. When that site was discontinued in 2013, we relocated the distribution to GitHub.

Because we “eat our own dog food,” VMbackup gets extensive and constant testing because we rely on it ourselves as the means to create backups and provide for restores for cases of accidental deletion, unexpected data corruption, or in the event that disaster recovery might be needed. The mechanisms are carefully tested before going into production and we perform frequent tests to ensure the integrity of the backups and that restores really do work. A number of times, we have relied on resorting to recovering from our backups and it has been very reassuring that these have been successful.

What VMbackup Does

Very simply, VMbackup provides a framework for backing up virtual machines (VMs) hosted on XenServer to an external storage device, as well as the means to recover such VMs for whatever reason that might have resulted in loss, be it disaster recovery, restoring an accidentally deleted VM, recovering from data corruption, etc.

The VMbackup distribution consists of a script written in Python and a configuration file. Other than a README document file, that’s it other than the XenServer SDK components which one needs to download separately; see http://xenserver.org/partners/developing-products-for-xenserver.html for details. There is no fancy GUI to become familiar with, and instead, just a few simple things that need to be configured, plus a destination for the backups needs to be made accessible (this is generally an NFS share, though SMB/CIFS will work, as well). Using cron job entries, a single host or an entire pool can be set up to perform periodic backups. Configurations on individual hosts in a pool are needed in that the pool master performs the majority of the work and it can readily change to a different XenServer, while individual host-based instances are also needed when local storage is also made use of, since access to any local SRs can only be performed from each individual XenServer. A cron entry and numerous configuration examples are given in the documentation.

To avoid interruptions of any running VMs, the process of backing up a VM follows these basic steps:

  1. A snapshot of the VM and its storage is made
  2. Using the xe utility vm-export, that snapshot is exported to the target external storage
  3. The snapshot is deleted, freeing up that space

In addition, some VM metadata are collected and saved, which can be very useful in the event a VM needs to be restored. The metadata include:

  • vm.cfg - includes name_label, name_description, memory_dynamic_max, VCPUs_max, VCPUs_at_startup, os_version, orig_uuid
  • DISK-xvda (for each attached disk)
    • vbd.cfg - includes userdevice, bootable, mode, type, unplugable, empty, orig_uuid
    • vdi.cfg - includes name_label, name_description, virtual_size, type, sharable, read_only, orig_uuid, orig_sr_uuid
  • VIFs (for each attached VIF)
    • vif-0.cfg - includes device, network_name_label, MTU, MAC, other_config, orig_uuid

An additional option is to create a backup of the entire XenServer pool metadata, which is essential in dealing with the aftermath of a major disaster that affects the entire pool. This is accomplished via the “xe pool-dump-database” command.

In the event of errors, there are automatic clean-up procedures in place that will remove any remnants plus make sure that earlier successful backups are not purged beyond the specified number of “good” copies to retain.

There are numerous configuration options that allow to specify which VMs get backed up, how many backup versions are to be retained, whether the backups should be compressed (1) as part of the process, as well as optional report generation and notification setups.

New Features in VMbackup 3.0

A number of additional features have been added to this latest 3.0 release, adding flexibility and functionality. Some of these came about because of the sheer number of VMs that needed to be dealt with, SR space issues as well as with changes coming to the next XenServer release. These additions include:

  • VM “preview” option: To be able to look for syntax errors and ensure parameters are being defined properly, a VM can have a syntax check performed on it and if necessary, adjustments can then be made based on the diagnosis to achieve the desired configuration.
  • Support for VMs containing spaces: By surrounding VM names in the configuration file with double quotes, VM names containing spaces can now be processed. 
  • Wildcard suffixes: This very versatile option permits groups of VMs to be configured to be handled similarly, eliminating the need to create individual settings for every desired VM. Examples include “PRD-*”, “SQL*” and in fact, if all VMs in the pool should be backed up, even “*”. There are however, a number of restrictions on wildcard usage (2).
  • Exclude VMs: Along with the wildcard option to select which VMs to back up, clearly a need arises to provide the means to exclude certain VMs (in addition to the other alternative, which is simply to rename them such that they do not match a certain backup set). Currently, each excluded VM must be named separately and any such VMs should de defined at the end of the configuration file. 
  • Export the OS disk VDI, only: In some cases, a VM may contain multiple storage devices (VDIs) that are so large that it is impractical or impossible to take a snapshot of the entire VM and its storage. Hence, we have introduced the means to backup and restore only the operating system device (assumed to be Disk 0). In addition to space limitations, some storage, such as DB data, may not be able to be reliably backed up using a full VM snapshot. Furthermore, the next XenServer release (Dundee) will likely support up to as many as perhaps 255 storage devices per VM, making a vm-export even more involved under such circumstances. Another big advantage here is that currently, this process is much more efficient and faster than a VM export by a factor of three or more!
  • Root password obfuscation: So that clear-text passwords associated with the XenServer pool are not embedded in the scripts themselves, the password can be basically encoded into a file.

The mechanism for a running VM from which only the primary disk is to be backed up is similar to the full VM backup. The process of backing up such a VM follows these basic steps:

  1. A snapshot of just the VM's Disk 0 storage is made
  2. Using the xe utility vdi-export, that snapshot is exported to the target external storage
  3. The snapshot is deleted, freeing up that space

As with the full VM export, some metadata for the VM are also collected and saved for this VDI export option.

These added features are of course subject to change in future releases, though typically later editions generally encompass the support of previous versions to preserve backwards compatibility.

Examples

Let’s look at the configuration file weekend.cfg:

# Weekend VMs
max_backups=4
backup_dir=/snapshots/BACKUPS
#
vdi-export=PROD-CentOS7-large-user-disks
vm-export=PROD*
vm-export=DEV-RH*:3
exclude=PROD-ubuntu12-benchmark
exclude=PRODtestVM

Comment lines start with a hash mark and may be contained anywhere with the file. The hash mark must appear as the first character in the line. Note that the default number of retained backups is set here to four. The destination directory is set next, indicating where the backups will be written to. We then see a case where only the OS disk is being backed up for the specific VM "PROD-CentOS7-large-user-disks" and below that, all VMs beginning with “PROD” are backed up using the default settings. Just below that, a definition is created for all VMs starting with "DEV-RH" and the default number of backups is reduced for all of these from the global default of four down to three. Finally, we see two excludes for specific VMs that fall into the “PROD*” group that should not be backed up at all.

To launch the script manually, you would issue from the command line:

./VmBackup.py password weekend.cfg

To launch the script via a cron job, you would create a single-line entry like this:

10 0 * * 6 /usr/bin/python /snapshots/NAUbackup/VmBackup.py password
/snapshots/NAUbackup/weekend.cfg >> /snapshots/NAUbackup/logs/VmBackup.log 2>&1

This would run the task at ten minutes past midnight on Saturday and create a log entry called VmBackup.log. This cron entry would need to be installed on each host of a XenServer pool.

Additional Notes

It can be helpful to break up when backups are run so that they don’t all have to be done at once, which may be impractical, take so long as to possibly impact performance during the day, or need to be coordinated with when is best for specific VMs (such as before or after patches are applied). These situations are best dealt with by creating separate cron jobs for each subset.

There is a fair load on the server, comparable to any vm-export, and hence the queue is processed linearly with only one active snapshot and export sequence for a VM being run at a time. This is also why we suggest you perform the backups and then asynchronously perform any compression on the files on the external storage host itself to alleviate the CPU load on the XenServer host end.

For even more redundancy, you can readily duplicate or mirror the backup area to another storage location, perhaps in another building or even somewhere off-site. This can readily be accomplished using various copy or mirroring utilities, such as rcp, sftp, wget, nsync, rsync, etc.

This latest release has been tested on XenServer 6.5 (SP1) and various beta and technical preview versions of the Dundee release. In particular, note that the vdi-export utility, while dating back a while, is not well documented and we strongly recommend not trying to use it on any XenServer release before XS 6.5. Doing so is clearly at your own risk.

The NAU VMbackup distribution can be found at: https://github.com/NAUbackup/VmBackup

In Conclusion

This is a misleading heading, as there is not really a conclusion in the sense that this project continues to be active and as long as there is a perceived need for it, we plan to continue working on keeping it running on future XenServer releases and adding functionality as needs and resources dictate. Our hope is naturally that the community can make at least as good use of it as we have ourselves.

Footnotes:

  1. Alternatively, to save time and resources, the compression can potentially be handled asynchronously by the host onto which the backups are written, hence reducing overhead and resource utilization on the XenServer hosts, themselves.
  2. Certain limitations exist currently with how wildcards can be utilized. Leading wildcards are not allowed, nor are multiple wildcards within a string. This may be enhanced at a later date to provide even more flexibility.

This article was written by Tobias Kreidl and Duane Booher, both of Northern Arizona University, Information Technology Services. Tobias' biography is available at this site, and Duane's LinkedIn profile is at https://www.linkedin.com/in/duane-booher-a068a03 while both can also be found on http://discussions.citrix.com primarily in the XenServer forum.     

Running XenServer... without a server
XenServer + Docker + CEPH/RBDSR for shared local s...

Related Posts

 

Comments 11

Lorscheider Santiago on Saturday, 09 April 2016 13:28

Tobias Kreidl and Duane Booher, Greart Article!

you have thought of a plugin for XenCenter?

0
Tobias Kreidl and Duane Booher, Greart Article! you have thought of a plugin for XenCenter?
Tobias Kreidl on Thursday, 14 April 2016 01:34

Thank you, Lorscheider, for your comment. Our thoughts have long been that others could take this to another level by developing a plugin or standalone Web interface that could leverage VMbackup. Ostensibly, various commercial packages already provide GUIs. However, we really never intended this to be competitive on that sort of level with some of the other more sophisticated user interfaces. Given how XenCenter also keeps evolving, I would think it would take a fair amount of effort to also keep up plugin code and we really don't have the resources to do that.

0
Thank you, Lorscheider, for your comment. Our thoughts have long been that others could take this to another level by developing a plugin or standalone Web interface that could leverage VMbackup. Ostensibly, various commercial packages already provide GUIs. However, we really never intended this to be competitive on that sort of level with some of the other more sophisticated user interfaces. Given how XenCenter also keeps evolving, I would think it would take a fair amount of effort to also keep up plugin code and we really don't have the resources to do that.
Niklas Ahden on Sunday, 17 April 2016 19:14

Hi,

First of all I want to thank you for this great article and NAUBackup.
I am wondering about the export-performance while using the "xe utility vdi-export" compared to "xe vm-export" - Is the export speed still limited to ~20% of the physical interface link-speed?
If not this is great news and it would improve my export-performance.

The second question is regarding incremental exports like Xen Orchestra has - Will this be a feature of your script aswell in the future?
//Niklas

0
Hi, First of all I want to thank you for this great article and NAUBackup. I am wondering about the export-performance while using the "xe utility vdi-export" compared to "xe vm-export" - Is the export speed still limited to ~20% of the physical interface link-speed? If not this is great news and it would improve my export-performance. The second question is regarding incremental exports like Xen Orchestra has - Will this be a feature of your script aswell in the future? //Niklas
Tobias Kreidl on Monday, 18 April 2016 01:28

Thanks, Niklas, first of all for your kind comments.

As to the VDI export, the mechanism utilized is different from VM export. Tests have shown it to be about 3.5x faster using the same primary network interface. Based on this improvement, the hope is that the VM export as directly provided and support by Citrix will eventually be able to leverage the same mechanism and therefore greatly speed up the export process, but there are currently some issues that still need to be resolved. As to a bandwidth limitation as a fraction of the total primary management interface, I have not seen any documentation that addresses this. The overall bandwidth consumption does not seem that different at first look comparing that of a VDI export to that of a VM export, so my guess is that there are more efficient transport mechanisms being utilized. In any case, I am optimistic these issues will be overcome, which would mean a huge benefit overall for being able to back up full VMs from XenServer. We would be very interested to hear what others' experiences are with VM vs. VDI export times. We most certainly did not expect there to be such a huge difference ourselves.

As to incremental backups, we have no immediate needs ourselves for these. The VMs we back up do not change that rapidly and for the ones that do, we use a commercial backup product that is suited to individual file and directory recovery. Frankly, it would be not that difficult to implement incremental VDI backups and a reasonable python programmer with some XenServer know-how (or vice versa) should be able to do this fairly easily. The much harder part is how to keep track of all the various incremental backups and manage them all. I think XenOrchestra has done a super job in that respect, as you would expect from a commercial product. Of course, we are more than willing to look at code contributions others may wish to make and take them into consideration for eventual incorporation in the main routine or perhaps the creation of a separate branch. For incremental backups, we are currently looking instead into possible storage device driven snapshot mechanisms that are handled internally within the capabilities of software defined storage (SDS) devices for SRs that are leveraged that way by XenServer hosts.

More likely, I would think that enhancements to the wildcard structure, perhaps incorporating general REGEXP, might be eventually forthcoming. Another one on the list might be adding on support for more than just the primary boot disk for the VDI backups. A lot obviously depends on what time is available and where the greatest needs are seen. This is, after all, still a basic utility and designed to be very easy to use, so we intentionally do not wish to make it overly complicated. The user base out there clearly has a need for a range of products with different levels of capabilities and features.

0
Thanks, Niklas, first of all for your kind comments. As to the VDI export, the mechanism utilized is different from VM export. Tests have shown it to be about 3.5x faster using the same primary network interface. Based on this improvement, the hope is that the VM export as directly provided and support by Citrix will eventually be able to leverage the same mechanism and therefore greatly speed up the export process, but there are currently some issues that still need to be resolved. As to a bandwidth limitation as a fraction of the total primary management interface, I have not seen any documentation that addresses this. The overall bandwidth consumption does not seem that different at first look comparing that of a VDI export to that of a VM export, so my guess is that there are more efficient transport mechanisms being utilized. In any case, I am optimistic these issues will be overcome, which would mean a huge benefit overall for being able to back up full VMs from XenServer. We would be very interested to hear what others' experiences are with VM vs. VDI export times. We most certainly did not expect there to be such a huge difference ourselves. As to incremental backups, we have no immediate needs ourselves for these. The VMs we back up do not change that rapidly and for the ones that do, we use a commercial backup product that is suited to individual file and directory recovery. Frankly, it would be not that difficult to implement incremental VDI backups and a reasonable python programmer with some XenServer know-how (or vice versa) should be able to do this fairly easily. The much harder part is how to keep track of all the various incremental backups and manage them all. I think XenOrchestra has done a super job in that respect, as you would expect from a commercial product. Of course, we are more than willing to look at code contributions others may wish to make and take them into consideration for eventual incorporation in the main routine or perhaps the creation of a separate branch. For incremental backups, we are currently looking instead into possible storage device driven snapshot mechanisms that are handled internally within the capabilities of software defined storage (SDS) devices for SRs that are leveraged that way by XenServer hosts. More likely, I would think that enhancements to the wildcard structure, perhaps incorporating general REGEXP, might be eventually forthcoming. Another one on the list might be adding on support for more than just the primary boot disk for the VDI backups. A lot obviously depends on what time is available and where the greatest needs are seen. This is, after all, still a basic utility and designed to be very easy to use, so we intentionally do not wish to make it overly complicated. The user base out there clearly has a need for a range of products with different levels of capabilities and features.
Steve on Friday, 29 April 2016 08:24

Hey Tobias,

Great work, great article. I've spotted a small oversight though :D

Line316: cmd = '%s/xe vm-list name-label=%s params=power-state | /bin/grep running' % (xe_path, vm_name)

You need to have: name-label='%s'
... for those of us that love spaces. Otherwise it reports:

Syntax error: Unknown switch: -
For usage run: 'xe help'
vm is NOT running

Backup still runs just fine though. Have you any tips for how to put spaces in a wildcard config entry? This doesn't seem to work:

vm-export=Client Name*
vm-export="Client Name*"
vm-export="Client Name"*

Cheers!

0
Hey Tobias, Great work, great article. I've spotted a small oversight though :D Line316: cmd = '%s/xe vm-list name-label=%s params=power-state | /bin/grep running' % (xe_path, vm_name) You need to have: name-label='%s' ... for those of us that love spaces. Otherwise it reports: Syntax error: Unknown switch: - For usage run: 'xe help' vm is NOT running Backup still runs just fine though. Have you any tips for how to put spaces in a wildcard config entry? This doesn't seem to work: vm-export=Client Name* vm-export="Client Name*" vm-export="Client Name"* Cheers!
Tobias Kreidl on Friday, 06 May 2016 16:55

Hi, Steve:
This has now been fixed and tested. Thanks for reporting the issue. The proper syntax should be

vm-export="Client Name*"

but I expect the unquoted version may work, as well. If you use the CLI, the quoted version above will definitely be necessary. The Github site has been updated with the corrected code.
Regards,
--Tobias

0
Hi, Steve: This has now been fixed and tested. Thanks for reporting the issue. The proper syntax should be [quote]vm-export="Client Name*"[/quote] but I expect the unquoted version may work, as well. If you use the CLI, the quoted version above will definitely be necessary. The Github site has been updated with the corrected code. Regards, --Tobias
Guest - Andreas Brauckmann on Sunday, 24 July 2016 06:17

Exporting only block changes/deltas

i think it is a good idea: is the possibility :-)

SNAPSHOT=$(xe vdi-snapshot uuid=$VDI)
xe vdi-export uuid=$VDI base=$SNAPSHOT filename=delta.vhd format=vhd --progress

Has anyone already started the code rewrite: Exporting only block changes/deltas?
I test xo-orchestra, but 15 days for a test is too short.
The free xo-orchestra version does not work well with nfs backup shares, there's a mount problem.

I will start next week, it would be nice to get help ...

0
Exporting only block changes/deltas i think it is a good idea: is the possibility :-) SNAPSHOT=$(xe vdi-snapshot uuid=$VDI) xe vdi-export uuid=$VDI base=$SNAPSHOT filename=delta.vhd format=vhd --progress Has anyone already started the code rewrite: Exporting only block changes/deltas? I test xo-orchestra, but 15 days for a test is too short. The free xo-orchestra version does not work well with nfs backup shares, there's a mount problem. I will start next week, it would be nice to get help ...
Tobias Kreidl on Tuesday, 26 July 2016 15:39

Hi, Andreas, and thank you for your feedback. Adding this in would be of course possible as the syntax for the actual export is pretty simple. We thought about this and decided that the tracking of these as yet another type of backup would make things more complicated and that it would also require maintaining a level zero backup, which takes up space and has to be left as a reference for incremental backups. None of our VMs are so large that doing a weekly backup was a problem and because a snapshot is only generated one at the time, we do not ever use up more space than the largest VM generates, and it's only temporary. Finally, we are planning on looking into an in-house means of doing whole volume snapshots based on ZFS which lends itself well to also incremental snapshots and decided if we were to invest time, that this might be a better option at least for our internal needs.

If you decide to push ahead with this, do let us know as we would be pleased to review such efforts and contributions!

0
Hi, Andreas, and thank you for your feedback. Adding this in would be of course possible as the syntax for the actual export is pretty simple. We thought about this and decided that the tracking of these as yet another type of backup would make things more complicated and that it would also require maintaining a level zero backup, which takes up space and has to be left as a reference for incremental backups. None of our VMs are so large that doing a weekly backup was a problem and because a snapshot is only generated one at the time, we do not ever use up more space than the largest VM generates, and it's only temporary. Finally, we are planning on looking into an in-house means of doing whole volume snapshots based on ZFS which lends itself well to also incremental snapshots and decided if we were to invest time, that this might be a better option at least for our internal needs. If you decide to push ahead with this, do let us know as we would be pleased to review such efforts and contributions!
Matthias Koterski on Monday, 22 August 2016 09:57

Great article. Thank you for this.

0
Great article. Thank you for this.
Zhang on Sunday, 25 September 2016 10:24

Hi Tobias,
Everytime when I run the script I got errors as below:

# ./VmBackup.py password weekend.cfg
./VmBackup.py: line 5: syntax error near unexpected token `newline'
./VmBackup.py: line 5: `'

Any hint for this?

0
Hi Tobias, Everytime when I run the script I got errors as below: # ./VmBackup.py password weekend.cfg ./VmBackup.py: line 5: syntax error near unexpected token `newline' ./VmBackup.py: line 5: `' Any hint for this?
Tobias Kreidl on Monday, 17 October 2016 02:45

Zhang, I apologize first off for not checking back sooner if there were any additional comments. There is probably a syntax error in the configuration file or possibly, the script got corrupted when downloading or editing it. I'm guessing it is probably a line break when it was copied and pasted!

Also, did you try adding the "preview" option to the end of the command, in other words:
./VmBackup.py password weekend.cfg preview
in your case to see if it generates a report regarding the configuration file syntax?

0
Zhang, I apologize first off for not checking back sooner if there were any additional comments. There is probably a syntax error in the configuration file or possibly, the script got corrupted when downloading or editing it. I'm guessing it is probably a line break when it was copied and pasted! Also, did you try adding the "preview" option to the end of the command, in other words: ./VmBackup.py password weekend.cfg preview in your case to see if it generates a report regarding the configuration file syntax?

About XenServer

XenServer is the leading open source virtualization platform, powered by the Xen Project hypervisor and the XAPI toolstack. It is used in the world's largest clouds and enterprises.
 
Commercial support for XenServer is available from Citrix.