Virtualization Blog

Discussions and observations on virtualization.

Creedence Final Beta Available

As we move steadily towards a release of XenServer Creedence, I'm pleased to announce that we're ending the beta phase of development with the release of Creedence beta.3. Beta.3 sees us as functionally complete, and with the majority of known performance issues resolved. The performance issues resolved range from a dom0 memory leak when VIFs are disabled, through to resolution of a workaround with Mellanox 40Gbps NICs, and some are resolved with both an updated driver bundle and a bump of the ovs version from 2.1.2 to 2.1.3. Functionally, beta.3 differs from beta.2 in having PVHVM support for Ubuntu 14.04, RedHat Enterprise Linux 7 and CentOS 7. Since these are new operating systems for us, the team is really interested in learning what you see for performance and stability for them.

As with all previous pre-release builds, we'd like the community to help ensure Creedence is a rock solid release. This time we're a bit less interested in Creedence itself, and more about the operating and support environment. One of the less known "features" of Citrix support is the free "Insight Services" or "TaaS". TaaS was originally designed to be "Tools-as-a-Service", and deliver on demand insight into the operation of Citrix technologies. With XenServer, Citrix Insight Services consumes a server status report from your XenServer host or pool, and then provides detailed guidance on how to potentially avoid an issue (say due to outdated BIOS or firmware), or resolve an issue you might be having (say by applying a hotfix). Honestly, it's not a bad practice in general to upload a server status report post XenServer install to ensure there aren't any items which could be latent in the deployment; rather like a health check.

How does this relate to Creedence? Well, Insight Services uses a series of plugins to ensure the data is processed properly. The support team has recently updated TaaS to support Creedence, and I'd like to ensure two things. First I'd like to ensure the processing logic is capturing everything it should, and secondly I'd like to ensure that those of you who have been successfully running Creedence don't have any hidden errors. Since this is a free service offered by Citrix, I'd also like the open source XenServer install base to know about it as a way to ensure XenServer hosts are deployed in a manner which will allow for Citrix to support you if the need arises.

Here's how you can help.

  1. Install either beta.2 or beta.3 (beta.3 preferred) from the pre-release downloads: http://xenserver.org/overview-xenserver-open-source-virtualization/prerelease.html
  2. From either XenCenter or the CLI take a server status report.
    • XenCenter: Server status reports can be run from "Tools->Server Status Report ..."
    • CLI: xen-bugtool --yestoall --output zip
  3. Log into TaaS (create a free account if required): https://taas.citrix.com/AutoSupport/
  4. Upload your server status report and see if anything interesting is found. If anything unexpected is found, we'd like to know about it.  The best way to let us know would be to submit an incident to https://bugs.xenserver.org which contains the TaaS information.

Thanks again to everyone who has contributed to the success we're seeing with Creedence.

PowerShell SDK examples
Log Rotation and Syslog Forwarding

Related Posts

 

Comments 20

Guest - Martins on Wednesday, 24 September 2014 14:18

Hello!
Thanks, for creating such great hypervizor. Will you in future only create PVHVM templates, PV will slowly depracted?

0
Hello! Thanks, for creating such great hypervizor. Will you in future only create PVHVM templates, PV will slowly depracted?
Guest - james on Wednesday, 24 September 2014 17:11

@Tim,
does this include the recently updated Xen 4.4.1 hypervisor? If not in beta, plans for final release?

0
@Tim, does this include the recently updated Xen 4.4.1 hypervisor? If not in beta, plans for final release?
JK Benedict on Thursday, 25 September 2014 00:42

Great post, Tim! I will definitely be leveraging this tomorrow after I check in my previous posts for PVHVM OSes I have used from Alpha to Beta!

0
Great post, Tim! I will definitely be leveraging this tomorrow after I check in my previous posts for PVHVM OSes I have used from Alpha to Beta!
Sebastian on Friday, 26 September 2014 15:14

hmm.. ? should yum install work ?

Loaded plugins: fastestmirror
Determining fastest mirrors
Could not retrieve mirrorlist http://updates.vmd.citrix.com/XenServer/6.4.96/domain0/mirrorlist error was
[Errno 14] HTTP Error 404: Not Found
Error: Cannot find a valid baseurl for repo: citrix

0
hmm.. ? should yum install work ? Loaded plugins: fastestmirror Determining fastest mirrors Could not retrieve mirrorlist http://updates.vmd.citrix.com/XenServer/6.4.96/domain0/mirrorlist error was [Errno 14] HTTP Error 404: Not Found Error: Cannot find a valid baseurl for repo: citrix
Tim Mackey on Friday, 26 September 2014 15:47

@Sebastian,

XenServer is a packaged and tuned system, so we intentionally disable yum install. The biggest reason we do that is because installation of random packages could destabilize the system due to dependencies or other requirements of new packages. Updates for XenServer are delivered via a custom hotfix process which takes into account the running VMs and toolstack.

What type of package were you wanting to install?

-tim

0
@Sebastian, XenServer is a packaged and tuned system, so we intentionally disable yum install. The biggest reason we do that is because installation of random packages could destabilize the system due to dependencies or other requirements of new packages. Updates for XenServer are delivered via a custom hotfix process which takes into account the running VMs and toolstack. What type of package were you wanting to install? -tim
Guest - mt on Sunday, 28 September 2014 00:09

@TIM
Please let us admins worry about what we break, don't cripple the system to protect us from ourselves.
There are many uses for a working yum, example from the last 48hrs: update bash due to Shelshock.
IMO, this is a mistake, and will generate noise.

0
@TIM Please let us admins worry about what we break, don't cripple the system to protect us from ourselves. There are many uses for a working yum, example from the last 48hrs: update bash due to Shelshock. IMO, this is a mistake, and will generate noise.
Tim Mackey on Sunday, 28 September 2014 14:44

@mt

What you suggest would make perfect sense if XenServer was a pure CentOS distro with "Xen" stuff added to it. While it may behave like a CentOS distro at many levels, it isn't and to allow arbitrary packages to be added has been shown to destabilize the platform for the simple reason those packages were never designed to work with XenServer. Our package management system consists of .xsupdate files which are designed to work with XenServer and the kernel and packaging decisions we've made. When we release updates, you apply them using our patch system, not yum.

I completely hear you about letting admins install packages which are valuable to them. As the dev team knows, I've been pushing hard for a supported mechanism to install certain classes of applications (such as monitoring and management tools), and to have a means to preserve the configuration of those tools during an upgrade. Part of the challenge is finding a way for the package author to know if their tool changes any of the default XenServer packages. If it does, then that package is unlikely to be stable in a XenServer environment.

Put another way, with XenServer we've made decisions around our packaging to produce a highly scalable virtualization platform which is installed from a single ISO. We put a ton of energy into performance tuning XenServer, and that factors into the decisions for which packages are present by default. That's not to say that you can't bypass our safe-guards, but to do so means you're now taking responsibility for full management of the platform and that future updates we release are unlikely to preserve your modifications.

I know that's not the answer you were hoping for, but we don't "cripple" XenServer by disabling yum; we ensure its stability by doing so.

-tim

0
@mt What you suggest would make perfect sense if XenServer was a pure CentOS distro with "Xen" stuff added to it. While it may behave like a CentOS distro at many levels, it isn't and to allow arbitrary packages to be added has been shown to destabilize the platform for the simple reason those packages were never designed to work with XenServer. Our package management system consists of .xsupdate files which are designed to work with XenServer and the kernel and packaging decisions we've made. When we release updates, you apply them using our patch system, not yum. I completely hear you about letting admins install packages which are valuable to them. As the dev team knows, I've been pushing hard for a supported mechanism to install certain classes of applications (such as monitoring and management tools), and to have a means to preserve the configuration of those tools during an upgrade. Part of the challenge is finding a way for the package author to know if their tool changes any of the default XenServer packages. If it does, then that package is unlikely to be stable in a XenServer environment. Put another way, with XenServer we've made decisions around our packaging to produce a highly scalable virtualization platform which is installed from a single ISO. We put a ton of energy into performance tuning XenServer, and that factors into the decisions for which packages are present by default. That's not to say that you can't bypass our safe-guards, but to do so means you're now taking responsibility for full management of the platform and that future updates we release are unlikely to preserve your modifications. I know that's not the answer you were hoping for, but we don't "cripple" XenServer by disabling yum; we ensure its stability by doing so. -tim
Guest - mt on Monday, 29 September 2014 12:11

212.29.211.2

@Tim,

XenServer is taking advantage of a popular distribution which is a wise decision, as it allows you not to build the OS and framework from scratch. You benefit from this popularity, as many system administrators already familiar with the OS/distribution and its tools. Which is part of their decision to deploy XenServer.
Now, with this decision to disable yum ("unless you bypass the safe-guards", wasn't this the case anyway, as standard repositories were disabled already?) you are starting a process of throwing out parts of that framework, you're shooting yourself in the foot twice, first by taking away tools from the framework making it non-standard, and second by alienating system administrators and integrators as they will find themselves in a crippled non-familiar environment.
Please specify which packages destabilize the platform, then lets agree that any sysadmin that was stupid enough to install such packages on a production system without a proper backup should consider changing his line of work. So I'm not buying into the stability and performance argument.
I'm sure I don't need to tell you that yum can be configured to bypass/lock certain packages, or that if needed it can be enhanced with plugins, I'm certain that you can find your own way to make sure that XenServer updating mechanism is playing nice with the existing underlying environment and tools.

"the responsibility of full management of the platform" is always the system administrator's, installing extra packages or not. Its the sysadmin responsibility to make sure everything ticks, if we "force" (manually and specifically enable) repos to install packages with yum, we don't expect the configuration to be preserved, and its always our responsibility to backup, especially before upgrades.

Please keep in mind that I write this because I care about XenServer, and want it to succeed.
As I see it, you are trying to bend the base distribution to your platform, instead of making sure your platform is compatible and complying to the distribution its based on.
We've seen this in the past, there was even a company that was created for packaging, deploying and updating software stacks for "appliances". It was called rPath and it was a colossal failure.
Please don't repeat the same mistakes.

0
212.29.211.2 @Tim, XenServer is taking advantage of a popular distribution which is a wise decision, as it allows you not to build the OS and framework from scratch. You benefit from this popularity, as many system administrators already familiar with the OS/distribution and its tools. Which is part of their decision to deploy XenServer. Now, with this decision to disable yum ("unless you bypass the safe-guards", wasn't this the case anyway, as standard repositories were disabled already?) you are starting a process of throwing out parts of that framework, you're shooting yourself in the foot twice, first by taking away tools from the framework making it non-standard, and second by alienating system administrators and integrators as they will find themselves in a crippled non-familiar environment. Please specify which packages destabilize the platform, then lets agree that any sysadmin that was stupid enough to install such packages on a production system without a proper backup should consider changing his line of work. So I'm not buying into the stability and performance argument. I'm sure I don't need to tell you that yum can be configured to bypass/lock certain packages, or that if needed it can be enhanced with plugins, I'm certain that you can find your own way to make sure that XenServer updating mechanism is playing nice with the existing underlying environment and tools. "the responsibility of full management of the platform" is always the system administrator's, installing extra packages or not. Its the sysadmin responsibility to make sure everything ticks, if we "force" (manually and specifically enable) repos to install packages with yum, we don't expect the configuration to be preserved, and its always our responsibility to backup, especially before upgrades. Please keep in mind that I write this because I care about XenServer, and want it to succeed. As I see it, you are trying to bend the base distribution to your platform, instead of making sure your platform is compatible and complying to the distribution its based on. We've seen this in the past, there was even a company that was created for packaging, deploying and updating software stacks for "appliances". It was called rPath and it was a colossal failure. Please don't repeat the same mistakes.
Guest - Heinrich Huber on Wednesday, 01 October 2014 10:05

@mt

I understand and respect your arguments for customizing a system within your environment as you like.

But as a XenServer-User too, I expect that this software does exactly what it was build for. And I think it can do it best, if it is reduced to the properties which are needed for the "job".
You are right, there can be security or functional issues. In my opinion they can/must be solved within XenServer and not external instruments. And since XenServer is only based on CentOS (it isn't CentOS), even CentOS tools are external intruments.

I agree with Tim that basic things like monitoring should work. And I'm the first who's pleased about additional functionalities which serves the main goal - damn good virtualization. But after all, it's a virtualization solution, and no file server, directory server, mail server, ...

0
@mt I understand and respect your arguments for customizing a system within your environment as you like. But as a XenServer-User too, I expect that this software does exactly what it was build for. And I think it can do it best, if it is reduced to the properties which are needed for the "job". You are right, there can be security or functional issues. In my opinion they can/must be solved within XenServer and not external instruments. And since XenServer is only based on CentOS (it isn't CentOS), even CentOS tools are external intruments. I agree with Tim that basic things like monitoring should work. And I'm the first who's pleased about additional functionalities which serves the main goal - damn good virtualization. But after all, it's a virtualization solution, and no file server, directory server, mail server, ...
Tim Mackey on Monday, 29 September 2014 17:42

@mt,

I accept your points, but what you might not know is that XenServer currently ships with a highly modified kernel, is 32 bit, and that many of the packages are in fact forks with patches applied. Putting together a whitelist of external packages would be a massive time sink as it would require us to validate they are indeed compatible, and that they remain compatible as we patch.

That being said, with Creedence we are moving closer to a stock Linux system and in time what you propose might be feasible. As I mentioned, I'm advocating an ability to incorporate external packages. It's a work in progress, but we needed to first get ourselves to a 64bit platform and kernel.org kernel. With post-Creedence, we're setting ourselves up to better handle these external packages.

I do recall rPath, and actually attempted to publish multiple demo "appliances" with them. Your point about avoiding their mistakes is well made.

-tim

1
@mt, I accept your points, but what you might not know is that XenServer currently ships with a highly modified kernel, is 32 bit, and that many of the packages are in fact forks with patches applied. Putting together a whitelist of external packages would be a massive time sink as it would require us to validate they are indeed compatible, and that they remain compatible as we patch. That being said, with Creedence we are moving closer to a stock Linux system and in time what you propose might be feasible. As I mentioned, I'm advocating an ability to incorporate external packages. It's a work in progress, but we needed to first get ourselves to a 64bit platform and kernel.org kernel. With post-Creedence, we're setting ourselves up to better handle these external packages. I do recall rPath, and actually attempted to publish multiple demo "appliances" with them. Your point about avoiding their mistakes is well made. -tim
Sebastian on Friday, 26 September 2014 17:59

@TIM

we run drbd on our xenserver 6.2 hosts and export the drbd device via iscsi... so i need some packages installed... e.g. scsi-target-utils

and i'm trying to get a local rbd / ceph volume mounted :-)

i tryied turning it on first, ( works on 6.2 ):

sed -i -e "s/enabled=0/enabled=1/" /etc/yum.repos.d/CentOS-Base.repo
yum install ksh libmcrypt bind-utils libtool-libs scsi-target-utils

Thanks !

1
@TIM we run drbd on our xenserver 6.2 hosts and export the drbd device via iscsi... so i need some packages installed... e.g. scsi-target-utils and i'm trying to get a local rbd / ceph volume mounted :-) i tryied turning it on first, ( works on 6.2 ): sed -i -e "s/enabled=0/enabled=1/" /etc/yum.repos.d/CentOS-Base.repo yum install ksh libmcrypt bind-utils libtool-libs scsi-target-utils Thanks !
Tim Mackey on Sunday, 28 September 2014 14:47

@Sebastian,

I'm glad you're moving forward, but please do be aware that applying any hotfixes or doing an upgrade isn't guaranteed to preserve your configuration. Most importantly, since Creedence is now 64 bit, you may have additional testing to perform prior to an upgrade.

-tim

0
@Sebastian, I'm glad you're moving forward, but please do be aware that applying any hotfixes or doing an upgrade isn't guaranteed to preserve your configuration. Most importantly, since Creedence is now 64 bit, you may have additional testing to perform prior to an upgrade. -tim
Guest - Mr. T on Sunday, 28 September 2014 17:57

Getting more support for other storage mechanisms like DRBD, Ceph and GlusterFS would be really good to see. The SR mechanism is very restrictive in many ways. So many other storage solutions have the ability to create snapshots, deal with backups, and not have to physically move disk content to be able to migrate VMs across servers or pools. In spite of the major improvements in disk I/O, it is still not close to what a native OS is capable of or what some competitors can offer. Citrix should be putting more efforts into improving storage options, now that XenServer has the power and robustness to take this on.

1
Getting more support for other storage mechanisms like DRBD, Ceph and GlusterFS would be really good to see. The SR mechanism is very restrictive in many ways. So many other storage solutions have the ability to create snapshots, deal with backups, and not have to physically move disk content to be able to migrate VMs across servers or pools. In spite of the major improvements in disk I/O, it is still not close to what a native OS is capable of or what some competitors can offer. Citrix should be putting more efforts into improving storage options, now that XenServer has the power and robustness to take this on.
Tim Mackey on Monday, 29 September 2014 17:44

@Mr. T

If you read through the many comments on the "post-Creedence" planning, improved flexibility with storage is a recurring theme. We've taken that to heart and are working storage flexibility into the requirements.

-tim

1
@Mr. T If you read through the many comments on the "post-Creedence" planning, improved flexibility with storage is a recurring theme. We've taken that to heart and are working storage flexibility into the requirements. -tim
JK Benedict on Saturday, 27 September 2014 09:10

Tim --

I forgot to hat-tip you on CIS (formerly TAAS - https://cis.citrix.com) - as I have been working with that team for some time now. The more Server Status Reports that myself the community can generate -- even on an healthy system -- the better. The primary reason for this is one thing:

You cannot beat real world scenarios and that (to me) affords us all the opportunity to converge on something as to ensure it is isolated, the community has a means to perform their own self-health checks, and... it just makes for a better experience/product in the end.

I have a few plugins to knock out for Creedence already, sir! :)

--jkbs | @xenfomation

0
Tim -- I forgot to hat-tip you on CIS (formerly TAAS - https://cis.citrix.com) - as I have been working with that team for some time now. The more Server Status Reports that myself the community can generate -- even on an healthy system -- the better. The primary reason for this is one thing: You cannot beat real world scenarios and that (to me) affords us all the opportunity to converge on something as to ensure it is isolated, the community has a means to perform their own self-health checks, and... it just makes for a better experience/product in the end. I have a few plugins to knock out for Creedence already, sir! :) --jkbs | @xenfomation
Ralf Weyer on Tuesday, 30 September 2014 07:44

I've asked a question about 2 hours ago, which seems to have been vanished now...

Ok, here we go again:

Is there a possiblity to update from beta 2 to beta 3?

When the final is released will it be possible to update from beta to final, or is this not recommendate at all?

Best regards,

Ralf

0
I've asked a question about 2 hours ago, which seems to have been vanished now... Ok, here we go again: Is there a possiblity to update from beta 2 to beta 3? When the final is released will it be possible to update from beta to final, or is this not recommendate at all? Best regards, Ralf
Tim Mackey on Tuesday, 30 September 2014 19:54

@Ralf,

While it may be possible to upgrade from one pre-release build to another, it shouldn't be expected to work. I definitely wouldn't make any claims that any upgrade attempt from pre-release to final release would work 100% of the time. It's just something we don't place a priority on testing. Upgrade from release to release; that's a different story as you'd expect.

btw, your original question didn't vanish so much as it was in the moderation queue. Moderation is one of many methods we use to keep the content on this site about XenServer and not random stuff ;)

-tim

0
@Ralf, While it may be possible to upgrade from one pre-release build to another, it shouldn't be expected to work. I definitely wouldn't make any claims that any upgrade attempt from pre-release to final release would work 100% of the time. It's just something we don't place a priority on testing. Upgrade from release to release; that's a different story as you'd expect. btw, your original question didn't vanish so much as it was in the moderation queue. Moderation is one of many methods we use to keep the content on this site about XenServer and not random stuff ;) -tim
Guest - james on Tuesday, 30 September 2014 18:33

yes, beta2 -> beta3 works fine and is tested.
beta3 to final should be possible, though may not be official yet. Please refer to wiki pages on "Creedence release"

0
yes, beta2 -> beta3 works fine and is tested. beta3 to final should be possible, though may not be official yet. Please refer to wiki pages on "Creedence release"
Guest - james on Friday, 14 November 2014 16:52

@Tim,
Just curious, when is the XS Creedence release planned ?

0
@Tim, Just curious, when is the XS Creedence release planned ?
Guest - Marcos on Monday, 12 January 2015 18:24

I have been searching high and low on how to do something is simple as shutdown when the UPS senses a power outage. Or at least inform the guest OSes to suspend, and let the server wait for imminent death.

Since last year, having chosen XenServer Credence, work *DAYS* have been lost to recovering critical paravirtual servers. Unfortunately we still live in a Polish city where power outages are not uncommon due to strong winds/weather, etc. in our SOHO setting.

If only the usb connected UPS could be used to talk to Xenserver! Directly, or indirectly via usbip or pvusb, then via xl or XenCenter from the guest (examples only). All hardware is common and modern but not too new; within 2-4yrs old. That is hopefully not asking too much to support.

Recently upgraded to XenServer.Creedence.20141212.RC90239.iso, but already suffered two outages since then, leading to abrupt crashes and extended downtimes.

Does anyone know of a way, either officially, or eg. via the CentOS repos to activate any toolstacks and kernel modules needed for this USB UPS, then configure so that events are used by the Xen API?

[Cypress Semiconductor USB to Serial]

dmesg:

[111819.506536] usb 4-1.7: USB disconnect, device number 7
[111820.729407] usb 4-1.7: new low-speed USB device number 8 using ehci-pci
[111820.834855] hid-generic 0003:0665:5161.0005: hiddev0,hidraw2: USB HID v1.00 Device [Cypress Semiconductor USB to Serial] on usb-0000:00:1d.0-1.7/input0


[root@xenjux ~]# lsmod |grep usb
usbhid 49012 0
hid 101674 2 hid_generic,usbhid

0
I have been searching high and low on how to do something is simple as shutdown when the UPS senses a power outage. Or at least inform the guest OSes to suspend, and let the server wait for imminent death. Since last year, having chosen XenServer Credence, work *DAYS* have been lost to recovering critical paravirtual servers. Unfortunately we still live in a Polish city where power outages are not uncommon due to strong winds/weather, etc. in our SOHO setting. If only the usb connected UPS could be used to talk to Xenserver! Directly, or indirectly via usbip or pvusb, then via xl or XenCenter from the guest (examples only). All hardware is common and modern but not too new; within 2-4yrs old. That is hopefully not asking too much to support. Recently upgraded to XenServer.Creedence.20141212.RC90239.iso, but already suffered two outages since then, leading to abrupt crashes and extended downtimes. Does anyone know of a way, either officially, or eg. via the CentOS repos to activate any toolstacks and kernel modules needed for this USB UPS, then configure so that events are used by the Xen API? [Cypress Semiconductor USB to Serial] dmesg: [111819.506536] usb 4-1.7: USB disconnect, device number 7 [111820.729407] usb 4-1.7: new low-speed USB device number 8 using ehci-pci [111820.834855] hid-generic 0003:0665:5161.0005: hiddev0,hidraw2: USB HID v1.00 Device [Cypress Semiconductor USB to Serial] on usb-0000:00:1d.0-1.7/input0 [root@xenjux ~]# lsmod |grep usb usbhid 49012 0 hid 101674 2 hid_generic,usbhid

About XenServer

XenServer is the leading open source virtualization platform, powered by the Xen Project hypervisor and the XAPI toolstack. It is used in the world's largest clouds and enterprises.
 
Commercial support for XenServer is available from Citrix.