Virtualization Blog

Discussions and observations on virtualization.

Creating backups with XenServer

Backup is an essential part of the business workflow for many of our customers - be it SMB, Enterprise Server Virtualisation or Virtual Desktop Infrastructure. Making the backup experience smoother is high up on our wishlist at XenServer Engineering and the delivery of improved VM import/export performance in XS 7.1 shows our commitment to that end. To continue improving our services supporting the backup ecosystem, we would like to better understand how you use backup with XenServer

 

  • How often do you backup? Do you have multiple jobs for monthly, weekly, daily backups?

  • How do you create your backups?

    • Use VM Export to backup VM metadata + disks

    • Snapshot at the VM level and use transfer/service VM to read off the snapshots

    • Use vdi-export to create differential disks (.vhd)

  • Do you use a third-party vendor for handling your backups?

  • Would support for incremental backups be useful for your use case?

Please leave a comment with your answers and any issues you may have with your backup experience today. We look forward to hearing from you!

Thank you,

Chandrika

 

XenServer 7.2 now available!
XenCenter 7.1 update now available!

Related Posts

 

Comments 24

Beck on Friday, 07 April 2017 02:30

I use storage replication for sr and bacula for vm data.
I still afraid of coalesce issue for using snapshot export.

0
I use storage replication for sr and bacula for vm data. I still afraid of coalesce issue for using snapshot export.
Chandrika Srinivasan on Tuesday, 02 May 2017 14:02

Hi Beck,

Is there a specific issue you are facing with coalesce? Which version of XenServer are you using?

-Chandrika

0
Hi Beck, Is there a specific issue you are facing with coalesce? Which version of XenServer are you using? -Chandrika
Olivier Lambert on Friday, 12 May 2017 19:23

Check this: https://xen-orchestra.com/blog/xenserver-coalesce-detection-in-xen-orchestra/

Should be integrated soon in our backup system :)

0
Check this: https://xen-orchestra.com/blog/xenserver-coalesce-detection-in-xen-orchestra/ Should be integrated soon in our backup system :)
Albert on Friday, 07 April 2017 09:56

Hi Chandrika,

We make a backup on the XenServer everyday. On the beginig we used import/export from XenServer, function vm protection. The next steps it a PHD Virtual Backup wich has a great function "replication", we can replicated VMs to the DRC.
After acquisition PHD by Unitrends we use UVBl,but we can't change to the newer version for UEB,because they remove replication options.
Maybe Citrix create a own backup software which I can run as virtual machine, with support storage: nfs,cifs,iscsi, scheduler, full backup, incrementals, replication options as I had in PHD ....


0
Hi Chandrika, We make a backup on the XenServer everyday. On the beginig we used import/export from XenServer, function vm protection. The next steps it a PHD Virtual Backup wich has a great function "replication", we can replicated VMs to the DRC. After acquisition PHD by Unitrends we use UVBl,but we can't change to the newer version for UEB,because they remove replication options. Maybe Citrix create a own backup software which I can run as virtual machine, with support storage: nfs,cifs,iscsi, scheduler, full backup, incrementals, replication options as I had in PHD ....
Olivier Lambert on Friday, 12 May 2017 17:22

Why not using Xen Orchestra? Support all storage, scheduler, full backup, incremental, replication etc.

0
Why not using Xen Orchestra? Support all storage, scheduler, full backup, incremental, replication etc.
Tobias Kreidl on Monday, 10 April 2017 05:04

How often do you backup? Do you have multiple jobs for monthly, weekly, daily backups?
=> We perform weekly backups. These are staggered so that the NFS server to which backups are written to is not overwhelmed by multiple backup operations that overlap. They are also conducted such that they are performed sequentially, allowing only one VM at the time to be backed up so as to not overwhelm any given XenServer. Currently, there is only this weekly backup mechanism in place and it is done using in-house scripts we have maintained for some six years or so now and make available to the public via Github.

How do you create your backups?
=> We use in-house developed scripts (see https://github.com/NAUbackup/VmBackup for details). There are options for determining how many backup copies are created and retained, where and how backups are stored, and scheduling options.

Use VM Export to backup VM metadata + disks
=> Yes, this is the basis for the backup mechanism we use. VM metadata can be backed up though our internals scripts as well as using the configurations options available in xsconsole for automating the metadata backup process independently.

Snapshot at the VM level and use transfer/service VM to read off the snapshots
=> For selected VMs, we also make use of this means so as to avoid dealing with either VMs that have multiple disks and are too large as a whole to be able to snapshot (not enough free space on the SR), or which would take too long because of being large, or which have data already backed up through other means. Currently, we have this set up such that only the root disk can be chosen instead of the entire VM.

Use vdi-export to create differential disks (.vhd)
=> Currently, no. This is primarily because of the extra amount of time it would take to develop a goo tracking mechanism for the large numbers of backups. It's at present good enough in most cases to be able to just create full backups.

Do you use a third-party vendor for handling your backups?
=> No. Everything we use at present is developed and deployed from in-house code.

Would support for incremental backups be useful for your use case?
=> Sure. If a good tracking and at least semi-automated means for creating and recovering using incremental backups were present as part of the XenServer package, we would certainly consider it.

0
How often do you backup? Do you have multiple jobs for monthly, weekly, daily backups? => We perform weekly backups. These are staggered so that the NFS server to which backups are written to is not overwhelmed by multiple backup operations that overlap. They are also conducted such that they are performed sequentially, allowing only one VM at the time to be backed up so as to not overwhelm any given XenServer. Currently, there is only this weekly backup mechanism in place and it is done using in-house scripts we have maintained for some six years or so now and make available to the public via Github. How do you create your backups? => We use in-house developed scripts (see https://github.com/NAUbackup/VmBackup for details). There are options for determining how many backup copies are created and retained, where and how backups are stored, and scheduling options. Use VM Export to backup VM metadata + disks => Yes, this is the basis for the backup mechanism we use. VM metadata can be backed up though our internals scripts as well as using the configurations options available in xsconsole for automating the metadata backup process independently. Snapshot at the VM level and use transfer/service VM to read off the snapshots => For selected VMs, we also make use of this means so as to avoid dealing with either VMs that have multiple disks and are too large as a whole to be able to snapshot (not enough free space on the SR), or which would take too long because of being large, or which have data already backed up through other means. Currently, we have this set up such that only the root disk can be chosen instead of the entire VM. Use vdi-export to create differential disks (.vhd) => Currently, no. This is primarily because of the extra amount of time it would take to develop a goo tracking mechanism for the large numbers of backups. It's at present good enough in most cases to be able to just create full backups. Do you use a third-party vendor for handling your backups? => No. Everything we use at present is developed and deployed from in-house code. Would support for incremental backups be useful for your use case? => Sure. If a good tracking and at least semi-automated means for creating and recovering using incremental backups were present as part of the XenServer package, we would certainly consider it.
Dorin Marcoci on Monday, 10 April 2017 17:34

I'd like to have the built-in ability to do scheduled vm-export backups (daily/weekly) on an a locally mounted or network connected ssh/sftp/ftp/smb/nfs storage.
Some options how many backups to retain will be great too, with a flexible graphic (1-7days: daily, up to 4 weeks: weekly, monthly).
Thanks!

0
I'd like to have the built-in ability to do scheduled vm-export backups (daily/weekly) on an a locally mounted or network connected ssh/sftp/ftp/smb/nfs storage. Some options how many backups to retain will be great too, with a flexible graphic (1-7days: daily, up to 4 weeks: weekly, monthly). Thanks!
Ronald Warsaski on Monday, 10 April 2017 21:39

We use Quadric Software's Alike, but what I would love to see is the ability to have the instant restore function that the other hypervisors have. The ability to start the VM from the backup files. This would be a huge step forward IMHO. I am unaware of any such option within XS, even products like StorageCraft can start their backups instantly on VMware, Hyper-V, and Virtual Box, but you can't do it on XenServer. I look forward to this ability someday, I hope.

0
We use Quadric Software's Alike, but what I would love to see is the ability to have the instant restore function that the other hypervisors have. The ability to start the VM from the backup files. This would be a huge step forward IMHO. I am unaware of any such option within XS, even products like StorageCraft can start their backups instantly on VMware, Hyper-V, and Virtual Box, but you can't do it on XenServer. I look forward to this ability someday, I hope.
Olivier Lambert on Friday, 12 May 2017 17:22

You can do that with DR or Continuous Replication of Xen Orchestra!

0
You can do that with DR or Continuous Replication of Xen Orchestra!
Andreas Becker on Wednesday, 12 April 2017 12:03

Hi Chandrika,

i´m very happy to see that there is someone at CITRIX who ´takes care of backup. In my opinion, there is currently no good backup solution at the market for xenserver. We use Commvault Simpana Backup right now to do the backups.
Please move xenserver in the right direction for the future.
Thanks a lot!

0
Hi Chandrika, i´m very happy to see that there is someone at CITRIX who ´takes care of backup. In my opinion, there is currently no good backup solution at the market for xenserver. We use Commvault Simpana Backup right now to do the backups. Please move xenserver in the right direction for the future. Thanks a lot!
RIZA on Monday, 24 April 2017 08:18

Hello,
We use xen-orchestra for backup jobs. It's very useful. It can take diff and full backup. We use free community edition. It's free to use from source without support.

0
Hello, We use xen-orchestra for backup jobs. It's very useful. It can take diff and full backup. We use free community edition. It's free to use from source without support.
Joze Volf on Friday, 28 April 2017 07:24

We also use Alike from Quadric. Restores can be pretty slow if you don't have a quite fast (random io) SR. You should also have a fast backup storage because of the deduplication.
For Windows guest (Vista/2008+) if you are not into buying something like Alike the best sollution IMHO is to do backups from within guest using Windows Image Backup (wbadmin) to a network share which is snapshot capable (e.g. Supermicro/OmniOS/ZFS). You get VSS, ZFS snapshots/retrospective, fast incremental backups. ZFS compression. You can also mount backup VHD directly from network share (even from the snapshots - hidden .zfs folder inside share). The problem is the "bare metal" recovery of VM if you use Windows installation media PE as boot because there are no Xen drivers and the network performance sucks. But you can have a small service VM ready for that purpose...

0
We also use Alike from Quadric. Restores can be pretty slow if you don't have a quite fast (random io) SR. You should also have a fast backup storage because of the deduplication. For Windows guest (Vista/2008+) if you are not into buying something like Alike the best sollution IMHO is to do backups from within guest using Windows Image Backup (wbadmin) to a network share which is snapshot capable (e.g. Supermicro/OmniOS/ZFS). You get VSS, ZFS snapshots/retrospective, fast incremental backups. ZFS compression. You can also mount backup VHD directly from network share (even from the snapshots - hidden .zfs folder inside share). The problem is the "bare metal" recovery of VM if you use Windows installation media PE as boot because there are no Xen drivers and the network performance sucks. But you can have a small service VM ready for that purpose...
Jordan on Monday, 01 May 2017 06:45

How often do you backup?
- Usually weekly backups. Some exceptions go for daily or monthly.
Do you have multiple jobs for monthly, weekly, daily backups?
- Yes.
How do you create your backups?
Use VM Export to backup VM metadata + disks
- This is our primary method. We use our own scripts that use cronjob for frequency and tags to filter out vms for different backups..
This is very convinient as VM admins just go in and mark the VMs whithout having access to the backend!
Snapshot at the VM level and use transfer/service VM to read off the snapshots
- Doing it for a very few selected VMs.
Use vdi-export to create differential disks (.vhd)
- No we dont do it.
Do you use a third-party vendor for handling your backups?
- After backups are transported to NFS with inhouse scripts, there is a HP Data Collector that goes once a week to migrate the backups on a tape.
Would support for incremental backups be useful for your use case?
- Yes it will be useful but it should be easy to automate.

0
[b]How often do you backup?[/b] - Usually weekly backups. Some exceptions go for daily or monthly. [b]Do you have multiple jobs for monthly, weekly, daily backups?[/b] - Yes. [b]How do you create your backups?[/b] [i]Use VM Export to backup VM metadata + disks[/i] - This is our primary method. We use our own scripts that use cronjob for frequency and tags to filter out vms for different backups.. This is very convinient as VM admins just go in and mark the VMs whithout having access to the backend! [i]Snapshot at the VM level and use transfer/service VM to read off the snapshots[/i] - Doing it for a very few selected VMs. Use vdi-export to create differential disks (.vhd) - No we dont do it. Do you use a third-party vendor for handling your backups? - After backups are transported to NFS with inhouse scripts, there is a HP Data Collector that goes once a week to migrate the backups on a tape. Would support for incremental backups be useful for your use case? - Yes it will be useful but it should be easy to automate.
Daniel on Wednesday, 03 May 2017 22:08

The big problem in XenServer is the process of coalesce that never works properly. I have a problem since the version 6.02, version 6.5 SP1, version 7 and in 7.1.

The process simply does not work, I managed to improve by changing the value LIVE_LEAF_COALESCE_TIMEOUT = 300 (/opt/xensource/sm/cleanup.py), but it is not yet 100%.

0
The big problem in XenServer is the process of coalesce that never works properly. I have a problem since the version 6.02, version 6.5 SP1, version 7 and in 7.1. The process simply does not work, I managed to improve by changing the value LIVE_LEAF_COALESCE_TIMEOUT = 300 (/opt/xensource/sm/cleanup.py), but it is not yet 100%.
Daniel on Thursday, 04 May 2017 11:48

Errors like below are always common in Coalesce operations. Whenever I turn on support, you are prompted to shut down and make a FULL COPY of the VM to consolidate the disk ... In the Citrix documentation itself, the feature will be improved soon, but this should be treated as a priority, since Disk are a bottleneck in XenServer.

May 4 08:31:40 SECH55 SM: [23604] ['/usr/bin/vhd-util', 'coalesce', '--debug', '-n', '/dev/VG_XenStorage-8ca84ba9-2f84-7070-ec0b-da0d6cb3a110/VHD-6d6a73cd-3a8f-4f66-887c-2ed40cc4321b']
May 4 08:33:21 SECH55 SM: [23604] FAILED in util.pread: (rc 5) stdout: 'error coalescing: -5
May 4 08:33:21 SECH55 SM: [23604] ', stderr: ''

0
Errors like below are always common in Coalesce operations. Whenever I turn on support, you are prompted to shut down and make a FULL COPY of the VM to consolidate the disk ... In the Citrix documentation itself, the feature will be improved soon, but this should be treated as a priority, since Disk are a bottleneck in XenServer. May 4 08:31:40 SECH55 SM: [23604] ['/usr/bin/vhd-util', 'coalesce', '--debug', '-n', '/dev/VG_XenStorage-8ca84ba9-2f84-7070-ec0b-da0d6cb3a110/VHD-6d6a73cd-3a8f-4f66-887c-2ed40cc4321b'] May 4 08:33:21 SECH55 SM: [23604] FAILED in util.pread: (rc 5) stdout: 'error coalescing: -5 May 4 08:33:21 SECH55 SM: [23604] ', stderr: ''
Ivan Grynenko on Thursday, 11 May 2017 01:17

We use an in-house maintained script, which works in a way similar to Tobias Kreidl described with few nicer additions (requiring external storage server running ZFS):

The backup script uses a ZFS storage server that is mounted to the XS pool via NFS. ZFS is configured to support compression and XS backups do not compress the data. We did not want to overload server's CPU having XS compressing the backups.

The backups we do support instant restoration of data, since the backup is made as a XA Template. This allows to instantly clone the template into a VM and start it running off the backup storage repository.. We then use storage migration to migrate VM's disks to the designated storage repositories.

More to your questions:

How often do you backup? Do you have multiple jobs for monthly, weekly, daily backups?

We backup daily (critical VMs), weekly (less critical VMs) and monthly (the rest). The frequency is defined by setting values for custom fields and used by our script to skip some VMs that have no value or set to be backed up less frequently.
An additional custom field allows to define how many copies of backups to retain.

How do you create your backups?

Daily, Weekly, Monthly

Use VM Export to backup VM metadata + disks

No, we use Export as a Template, to facilitate instant restoration of a VM that failed.

Snapshot at the VM level and use transfer/service VM to read off the snapshots

We use snapshots on a predefined schedule (4-hourly) and our script ensures retention and max snapshot policy (less than 30). Snapshots are stored on the same repository as the VM's disk.

Use vdi-export to create differential disks (.vhd)

No. ZFS reduplication (requires insane amount of RAM) handles this for us. We always export full VM template with all disks.

Do you use a third-party vendor for handling your backups?

No, They are either incredibly expensive or do not fit our requirements.

Would support for incremental backups be useful for your use case?

Probably not, however we see value in having this option available for the community. In our case incremental backups will increase VM restoration time, which is not appropriate in some cases - where VM disks are too large.
We currently use ZFS storage snapshotting, allowing us to restore the entire dataset - but this has to be configured as a standalone SR per VM. In this case shutting VM down and reverting ZFS snapshot reverts the state of the VM. This is a powerful feature, since it allows to take snapshots as often as minutely (we use 15-minutely), and we apply it to critical VMs only.

0
We use an in-house maintained script, which works in a way similar to Tobias Kreidl described with few nicer additions (requiring external storage server running ZFS): The backup script uses a ZFS storage server that is mounted to the XS pool via NFS. ZFS is configured to support compression and XS backups do not compress the data. We did not want to overload server's CPU having XS compressing the backups. The backups we do support instant restoration of data, since the backup is made as a XA Template. This allows to instantly clone the template into a VM and start it running off the backup storage repository.. We then use storage migration to migrate VM's disks to the designated storage repositories. More to your questions: [quote]How often do you backup? Do you have multiple jobs for monthly, weekly, daily backups?[/quote] We backup daily (critical VMs), weekly (less critical VMs) and monthly (the rest). The frequency is defined by setting values for custom fields and used by our script to skip some VMs that have no value or set to be backed up less frequently. An additional custom field allows to define how many copies of backups to retain. [quote]How do you create your backups?[/quote] Daily, Weekly, Monthly [quote]Use VM Export to backup VM metadata + disks[/quote] No, we use Export as a Template, to facilitate instant restoration of a VM that failed. [quote]Snapshot at the VM level and use transfer/service VM to read off the snapshots[/quote] We use snapshots on a predefined schedule (4-hourly) and our script ensures retention and max snapshot policy (less than 30). Snapshots are stored on the same repository as the VM's disk. [quote]Use vdi-export to create differential disks (.vhd)[/quote] No. ZFS reduplication (requires insane amount of RAM) handles this for us. We always export full VM template with all disks. [quote]Do you use a third-party vendor for handling your backups?[/quote] No, They are either incredibly expensive or do not fit our requirements. [quote]Would support for incremental backups be useful for your use case?[/quote] Probably not, however we see value in having this option available for the community. In our case incremental backups will increase VM restoration time, which is not appropriate in some cases - where VM disks are too large. We currently use ZFS storage snapshotting, allowing us to restore the entire dataset - but this has to be configured as a standalone SR per VM. In this case shutting VM down and reverting ZFS snapshot reverts the state of the VM. This is a powerful feature, since it allows to take snapshots as often as minutely (we use 15-minutely), and we apply it to critical VMs only.
Julien on Friday, 12 May 2017 10:00

Hi, see below my answers.

How often do you backup? Do you have multiple jobs for monthly, weekly, daily backups?
Weekly for our management servers. Daily for our NFS used by our VDI, it is done at the storage level.

How do you create your backups? Do you use a third-party vendor for handling your backups?
We use a third party product Fungusware Xackup to backup our ~40 management/ancillary VMs. It takes a snapshot, compress it and export it to a shared network + retention policy, all is done automatically with alert in case of any issue, very simple process. We have been using this tool for ~3 years and never had any issue related to the tool, the only "issue" we have is when running out of space for the snapshot on the storage.

Would support for incremental backups be useful for your use case? Not for us.

0
Hi, see below my answers. How often do you backup? Do you have multiple jobs for monthly, weekly, daily backups? Weekly for our management servers. Daily for our NFS used by our VDI, it is done at the storage level. How do you create your backups? Do you use a third-party vendor for handling your backups? We use a third party product Fungusware Xackup to backup our ~40 management/ancillary VMs. It takes a snapshot, compress it and export it to a shared network + retention policy, all is done automatically with alert in case of any issue, very simple process. We have been using this tool for ~3 years and never had any issue related to the tool, the only "issue" we have is when running out of space for the snapshot on the storage. Would support for incremental backups be useful for your use case? Not for us.
Cyrill on Monday, 22 May 2017 10:57

How often do you backup?
Daily for mission critical VMs, on-demand for the rest.
How do you create your backups?
We use Tobias Creidl's NAUbackup script for daily backups and "snapshot->VM Export" chain for the rest of VMs.
Do you use a third-party vendor for handling your backups?
No. We could not find any suitable free software and commercial is not an option for us.
Would support for incremental backups be useful for your use case?
Absolutely, we even thought about developing some incremental backup scripts ourselves.

NAUbackup is great but we're still missing VMPR for it's GUI integrated into XenCenter.

As active users of snapshotting feature we often face coalesce problem - on some big snapshots it fails with timeout exception and snapshot remains in VHD chain. Healing VM requires shutting it down for a remarkable period of time, it's difficult to find big enough time window.

Also it would be good if Xenserver got an upgraded HA solution providing complete VM replication (memory and disks) from one node to another, enabling hot-standby backup for running VMs like Remus or COLO does.

0
How often do you backup? Daily for mission critical VMs, on-demand for the rest. How do you create your backups? We use Tobias Creidl's NAUbackup script for daily backups and "snapshot->VM Export" chain for the rest of VMs. Do you use a third-party vendor for handling your backups? No. We could not find any suitable free software and commercial is not an option for us. Would support for incremental backups be useful for your use case? Absolutely, we even thought about developing some incremental backup scripts ourselves. NAUbackup is great but we're still missing VMPR for it's GUI integrated into XenCenter. As active users of snapshotting feature we often face coalesce problem - on some big snapshots it fails with timeout exception and snapshot remains in VHD chain. Healing VM requires shutting it down for a remarkable period of time, it's difficult to find big enough time window. Also it would be good if Xenserver got an upgraded HA solution providing complete VM replication (memory and disks) from one node to another, enabling hot-standby backup for running VMs like Remus or COLO does.
Rene on Tuesday, 23 May 2017 19:02

Hi!

I have been using script called "snapbak" for backups. I have modified it to better suit our needs.
It exports xva backups to NFS mount, doing daily and weeklys. It has been very stable and running for years.
Also very easy to add new VM's to backup through XenCenter custom fields. I have put my setup to github:
https://github.com/renx123/snapback


Incremental backups would be very helpful. Main problem is that full backups take huge amounts of storage space.

0
Hi! I have been using script called "snapbak" for backups. I have modified it to better suit our needs. It exports xva backups to NFS mount, doing daily and weeklys. It has been very stable and running for years. Also very easy to add new VM's to backup through XenCenter custom fields. I have put my setup to github: https://github.com/renx123/snapback Incremental backups would be very helpful. Main problem is that full backups take huge amounts of storage space.
Olivier Lambert on Saturday, 10 June 2017 11:49

Xen Orchestra supports incremental continuous backup to any NFS store ;) Take a look, it's also Open Source!

0
Xen Orchestra supports incremental continuous backup to any NFS store ;) Take a look, it's also Open Source!

About XenServer

XenServer is the leading open source virtualization platform, powered by the Xen Project hypervisor and the XAPI toolstack. It is used in the world's largest clouds and enterprises.
 
Commercial support for XenServer is available from Citrix.