Virtualization Blog

Discussions and observations on virtualization.

Validation of the Creedence Alpha

On Monday May 19th early access to XenServer Creedence builds started from xenserver.org.  The  xenserver.org community has access to XenServer pre-release installation media of an alpha quality and is invited to provide feedback on it.

This blog describes the validation and system testing performed on the first alpha build.

Test Inventory

The XenServer development process incorporates daily automated regression testing complemented by various additional layers of testing, both automated and manual, that are executed less frequently.

In outline, these are the test suites and test cycles executed during XenServer development.

Automated short-cycle regression testing (“BVT”) for fast feedback to developers – on every build on every branch.

Automated medium-cycle regression testing (“BST”) to maintain quality on team branches

Automated long-cycle system regression test (“Nightly”) – on select builds on select branches, aimed at providing wide regression coverage on a daily basis

Automated performance regression test, measures several hundred key performance indicators – run on select builds on select branches on a regular basis

Automated stress test (huge numbers of lifecycle operations on single hosts) – run once per week on average

Automated pool stress test (huge numbers of lifecycle and storage operations on XS pools) - run once per week on average

Automated long-cycle system regression test (“Full regression”) – on select builds on select branches, aimed at providing extensive wide test coverage, this comprises a huge number of tests and takes several days to run, frequency of run is therefore one every two weeks on average

Automated large scale stability test (huge numbers of VMs on large XS pools, boot storm and other key ‘scale’ use-cases – run on demand, usually several times in the run up to a product release and ahead of key internal milestone including deliveries to other Citrix product groups

Automated soak test – run on demand, usually several times in the run up to a product release and ahead of key internal milestone including deliveries to other Citrix product groups, this comprises long-running tests aimed at validating XS over an extended time period

Automated upgrade test – run ahead of key milestones and deliveries to validate upgrade procedures for new releases.

Manual test – exploratory testing using XenCenter, aimed principally at testing edge cases and scenarios that are not well covered by automation, cycles of manual testing are carried out on a regular basis, and ahead of key milestones and deliveries.

Exit Criteria

Each stage of a XenServer release project requires different test suites to have been run “successfully” (usually meaning a particular pass-rate has been achieved and/or failures are understood and deemed acceptable).

However test pass rates are only a barometer of quality – if one test out of a hundred fails then that may not matter, but on the other what if that one test case failure represents a high impact problem affecting a common use-case? For this reason we also use defect counts and impact analyses as part of the exit criteria.

XenServer engineering maintains a high quality bar throughout the release cycle – the “Nightly” automated regression suite comprising several thousand test cases must always achieve over 95%. If it does not, then new feature development stops while bugs are fixed and code reverted until a high pass rate is restored.

The Alpha.next release is a drop from the Creedence project branch that has achieved the following pass rates on the following test suites.:

  • ·         Nightly regression  – 96.5%
  • ·         Stress –  passed (no pass rate for this suite)
  • ·         Pool stress –  passed (no pass rate for this suite)
  • ·         General regression –  91.3%

Drops later in the project lifecycle (e.g. Tech Preview) will be subjected to more testing and with more stringent exit criteria.

More Info

For more information on the automation framework used for these tests, please read my blog about XenRT!

Recent comment in this post
Rob Gilson
We downloaded and installed it earlier this month. Outwardly no changes but a lot of under-the-covers fixes and upgrades that were... Read More
Sunday, 22 June 2014 22:33
Continue reading
8597 Hits
1 Comment

Introducing Open Source XenRT

As a follow up activity to the open sourcing of XenServer, Citrix is pleased to announce the open sourcing of its automated test platform, XenRT.

XenRT ("Xen Regression Test") is a test automation framework, written in Python, providing abstractions for the various components under test (pool, host, VM, storage, network etc). The library code which makes up these abstractions simplifies the process of writing tests, allowing quite complex operations to be performed in a single method call.

In a full deployment, XenRT handles all aspects of the testing process - it will schedule a test job onto a host, bootstrap it (via DHCP/PXE), install the build to be tested, carry out the testing, and collect all necessary logs for troubleshooting, without any user interaction required.

In addition to basic functional, regression, and stress testing, XenRT has suites of tests that are used for testing performance, scalability, and interoperability.

Within Citrix, XenRT is used with a distributed lab comprised of an extremely wide range of hardware, and is developed and maintained by a team of some 25 developers. Tests are also written and executed directly by the wider XenServer engineering team, in a true "Test-as-a-Service" platform - see this post on the Citrix blog for more information.

XenRT has been open sourced to leverage Citrix's experience and resources in test automation to help improve the quality of open source Xen and XenServer releases, to benefit the entire community.

To get started with XenRT, follow the links below to the code and a README document (which contains getting started instructions - further documentation will follow in the near future). For discussion a mailing list has been created - information about this can be found at https://lists.xenserver.org/sympa/info/xenrt-users.

Download links:
README document
Main XenRT tarball
Third party test resource tarball
Source for third party resources (not required for normal operation)

Continue reading
13937 Hits
1 Comment

About XenServer

XenServer is the leading open source virtualization platform, powered by the Xen Project hypervisor and the XAPI toolstack. It is used in the world's largest clouds and enterprises.
 
Commercial support for XenServer is available from Citrix.