Useful tools for working with GPUs

Monitoring GPUs

GPU Monitoring Tools
  • nvidia-smi (looping mode via the -l option has minimal impact on the GPU performance)
  • GPU-Z
  • Fraps - a product that can measure framerates in guest
  • If working with Citrix technologies involving HDX, such as XenDesktop you can use HDX monitor 3.3 (Advanced thin wire graphics) to monitor many performance metrics including framerates. See http://support.citrix.com/article/CTX135817
General XenServer Performance Metrics

Benchmarking Software

Factors to consider:
  • OpenGL or DX
  • Level of automation
  • Licensing
  • Whether suitable for the GPU/vGPU being used (does the GPU/vGPU have sufficient framebuffer?) 
  • Do you manually have to configure / turn on hardware (GPU) acceleration in the application
  • Full screen / Windows
Some Benchmarks that may be useful (in no particular order)

Application

Test mode OpenGL or DirectX

Notes

Google Earth

DX or OGL

The “Tour” mode is somewhat automated.

Viewperf 11

OGL

Free, automated, produces benchmark results.
Available from http://www.spec.org/benchmarks.html#gwpg

Autodesk Maya 2013

OGL

Requires a licensed version. It has a benchmark mode.

Dassault Catia V6

OGL

Requires a licensed version, the VM environment may causes some challenges with licensing and this should be checked with the vendor.
The app can be automated with some minimal setup.

Dassault  Solidworks 2013

DX

Requires a licensed version. Includes a performance test that runs automatically.
(Separately, Viewperf includes a Solidworks benchmark capture that runs without requiring Solidworks be installed)

Autodesk 3DSMax 2013

DX

Requires a licensed version. Includes automated benchmark modes. Also includes an i-ray benchmark that’s not currently run on vGPU as it has a dependency on CUDA but may be suitable for GPU pass-through.

FutureMark 3DMark 06

DX

DX9 benchmark, used for basic checkout and stress.
Fullscreen only so probably won’t work with XenDesktop. complements Unigine (see below).
There is a licensed version and also free version available from www.futuremark.com

Unigine Heaven

DX or OGL

DX9/10/11 benchmark, used for basic checkout and stress.
Runs in fullscreen or windowed mode, i.e. works with XenDesktop.
There is a licensed version, free version also available from www.unigine.com

Adobe Photoshop CS6

OGL

Requires a licensed version. The app doesn’t currently have any automated test modes.

Batman Arkham City Demo

DX

Demo includes a test mode that can be started to generate load.

Windows Media Player

-

The Wildlife HD sample may be useful

DX SDK samples

DX

These are free with the DirectX SDKs from Microsoft.
In most cases they require some user interaction to generate load (e.g. zoom in/out, pan, etc.)

  • DX9 LocalDeformablePRT.exe
   
  • DX9 ShadowMap.exe
   
  • DX10 NBodyGravity.exe
   
  • DX10 HDAO10.1.exe
   
  • DX11 NBodyGravityCS11.exe
   
  • DX11 FluidCS11.exe
   

HTML5 Fishbowl

DX and OGL

Run this on IE and Chrome browsers:
http://ie.microsoft.com/testdrive/performance/fishbowl/

Windows Experience Index

Particularly well suited for knowledge worker targeted virtualised GPUS e.g. NVIDIAs K100/K200

Redway3D Turbine

http://www.redway3d.com/pages/redsdkDownloads.php

WebGL Aquarium

http://webglsamples.googlecode.com/hg/aquarium/aquarium.html

If you have feedback on any of the suggested applications, or think we should add to the list please do comment on the feedback thread for this page, here.

 

Our developers, support, QA and solutions lab often post blogs on benchmarking and test automation at http://blogs.citrix.com/product/xenserver/

Some that may be of particular interest include:

 

 

 

 

Debugging Windows Guests on XenServer

WinDbg

WinDbg is one of a number of tools available from Microsoft that can be used for debugging Windows guests in XenServer environments.

You can get QEMU to passive-open a TCP port on dom0 for serial output and wait for a connection, this method will work if you're running on a machine with a dynamically assigned IP address:
on XenServer 6.2 or later

xe vm-param-add uuid=<uuid> param-name=platform hvm_serial=tcp::<port>,server,nodelay

on earlier versions

xe vm-param-add uuid=<uuid> param-name=other-config hvm_serial=tcp::<port>,server,nodelay

Then:

  1. Grab a copy of HW Virtual Serial Port from http://www.hw-group.com/products/hw_vsp/hw_vsp2_en.html and install it.
  2. Fire up the application and configure a COM port to point at your dom0 IP address and the <port> you specified to QEMU above). Make sure that you uncheck 'NVT Enable' in the Settings tab.
  3. Start the COM port.
  4. Configure debugging inside the guest by editing boot.ini in the usual way.
  5. Reboot the VM.
  6. Start windbg and point it at the COM port you created.
An Unofficial Method

To the best of my knowledge we have never released sockpipe.exe but this is a method some of our own developers use, it isn't supported but may be useful to developers in a debug scenario.

If you're running windbg on a machine with a statically assigned IP address then you can use the following setting to get QEMU to active-open a TCP port on your machine for serial output:

on XenServer 6.2 or later

xe vm-param-add uuid=<uuid> param-name=platform hvm_serial=tcp:<machine>:<port>

on earlier versions

xe vm-param-add uuid=<uuid> param-name=other-config hvm_serial=tcp:<machine>:<port>

Then:

  1. Run sockpipe.exe (which seems to be available here) on the machine where you want to run windbg. Run it without arguments to get help; it should be fairly obvious what's going on. For example, you could use 'sockpipe <name> <port>' (where <port> is the name number as specified to QEMU above).
  2. Fire up windbg and go ``File->Kernel Debugging'', make sure that ``Pipe'' is ticked, and enter \\.\pipe\<name> in the Port box, where <name> is the pipe name specified to sockpipe. Hit OK.
  3. Configure debugging inside the guest by editing boot.ini in the usual way.
  4. Reboot the VM.
  5. Debug as normal.

I'm told it may be a bit slow, but it mostly works although it has limitations; it will not work for SMP 64 bit or SMP Vista.

 

Further Information

 

 

 

 

WMI Interface - A Supported Mechanism to access XenStore data in Windows guests

The XenStore_Client.exe is a small executable program that was (in previous versions of XenServer) distributed with the XenServer Tools and enabled users to access the value of parameters contained in XenStore. This program was removed from XenServer 6.1.0 as there was no knowledge of dependencies or other consumers. However, after the program was removed, Citrix noticed that there were users relying on this unsupported tool. XenStore_Client.exe was never officially supported or tested.

XenServer 6.1.0 introduces an alternative mechanism for extracting XenStore information by using Windows Management Instrumentation (WMI), which offers a far richer interface. This article provides information about using WMI for extracting XenStore information.

Developers wishing to use XenServer 6.2.0 and above and the Windows guest WMI interface as for communicating with XenStore and other hypervisor interfaces should read http://support.citrix.com/article/CTX136422 outlining its usage and providing code examples. The WMI interface can be used in XenServer 6.1.0 as an alternative to XenStore_Client.exe which was available in earlier versions of XenServer.

Debugging XAPI Session Leaks - An example for XS6.1

This is a script written internally by one of our developers debugging a session issues in a third party application on XenServer 6.1 available as is. The script gives an overview of all sessions that were created but not destroyed by the caller (in the timespan covered by the logs). It also shows whether the XAPI Garbage Collection (GC) process destroyed the session or not (due to exceeding the limit of 400 sessions, or because the session was not used for 24 hours). It also shows the user name used when logging in, and whether the session was created within dom0 on a unix domain socket (UNIX) or over a TCP connection (INET). In the latter case, the user-agent it listed as well. Note that the user name in this case is not relevant for authentication, but can in some cases be used to isolate what the application that created the session was.

The format of logs is not guaranteed to remain unchanged between XenServer versions and this code should be regarded as demonstrator code to aid developers writing debug scripts. XenServer logs are not a guaranteed interface and this script or the techniques used are not suitable for use in customer, production or QA automation environments.

 

The script can be executed by calling it on the log files supplied in chronological order, newest first, thus:

python do.py xensource.log.15 xensource.log.14 xensource.log.13 xensource.log.12 xensource.log.11 xensource.log.10 xensource.log.9 xensource.log.8 xensource.log.7 xensource.log.6 xensource.log.5 xensource.log.4 xensource.log.3 xensource.log.2 xensource.log.1 xensource.log > my_output_file.txt

#!/usr/bin/env python

#Copyright (c) Citrix Systems, Inc.
# All rights reserved.
# 
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 
#   1) Redistributions of source code must retain the above copyright
#      notice, this list of conditions and the following disclaimer.
# 
#   2) Redistributions in binary form must reproduce the above
#      copyright notice, this list of conditions and the following
#      disclaimer in the documentation and/or other materials
#      provided with the distribution.
# 
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
# FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
# COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
# INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
# OF THE POSSIBILITY OF SUCH DAMAGE.

# This script was written for debugging a 3rd party application on XS6.1 
# The format of logs is not guaranteed and this code should be regarded
# as demonstrator code to aid developers writing debug scripts.
# XenServer logs are not a guaranteed interface and this script or the 
# techniques used are not suitable for use in customer, production or
# QA automation environments.

# The script gives an overview of all sessions that were created but not 
# destroyed by the caller (in the timespan covered by the logs). It also 
# shows whether the XAPI Garbage Collection (GC) process destroyed the 
# session or not (due to exceeding the limit of 400 sessions, or because 
# the session was not used for 24 hours). It also shows the user name 
# used when logging in, and whether the session was created within dom0 
# on a unix domain socket (UNIX) or over a TCP connection (INET). In the 
# latter case, the user-agent it listed as well. Note that the user name 
# in this case is not relevant for authentication, but can in some cases
# be used to isolate what the application that created the session is.
import sys
import re

files = sys.argv[1:]

logins = {}
logouts = {}
gced = {}
task2agent = {}

d = -1
def check_date(dnew):
	global d
	if d > -1 and dnew < d:
		print "ERROR: please list the logs files in order (oldest first)"
		sys.exit()
	else:
		d = dnew

for filename in files:
	f = open(filename,'r')
	for s in f.readlines():
		m = re.match('.*(\d\d \d\d:\d\d:\d\d).*D:(.*)\|.*Session\.create trackid=(\w*).*(uname=\w*)', s)
		if m and 'pool=false' in s:
			check_date(m.group(1))
			if "INET" in s:
				extra = "INET"
			else:
				extra = "UNIX"
			extra += ' ' + m.group(4) + ' '
			if m.group(2) in task2agent:
				extra += 'user-agent: ' + task2agent[m.group(2)]
			logins[m.group(3)] = (m.group(1), extra)
		m = re.match('.*(\d\d \d\d:\d\d:\d\d).*Session\.destroy trackid=(\w*)', s)
		if m and m.group(2) in logins.keys():
			check_date(m.group(1))
			if 'db_gc' not in s:
				logouts[m.group(2)] = m.group(1)
			else:
				gced[m.group(2)] = m.group(1)
		m = re.match('.*D:(.*)\|.*User-Agent: (.*)', s)
		if m:
			task2agent[m.group(1)] = m.group(2)
	f.close()


print "logged in but not logged out"
print "============================"
print "Logged in\tGC'ed\t\tTrackid\t\t\t\t\tAdditional info"
for trackid, (time, info) in sorted(logins.items(), cmp=lambda (a, (b, c)), (d, (e, f)): cmp(b, e)):
	if trackid not in logouts.keys():
		gc = gced.get(trackid, '-\t')
		print "%s\t%s\t%s\t%s" % (time, gc, trackid, info)
 

 

 

 

 

Install, Upgrade and Rollback of Test XenServer Environments

Officially supported mechanism for installing XenServer can be found in the XenServer Installation Guide and can be applied to production environments.

Developer Experimental Tools for use in test environments

There are also some additional tools provided unsupported, designed for use by developers working with XenServer in test environments. This page details techniques that may be of use to developers frquently installing or tearing down test environments.

prepare_host_upgrade

This is an undocumented xe CLI mechanism avoiding PXE config (prepare_host_ugrade.py), prepare_host_upgrade is an easy way to do an in-place upgrade without the need for a CD drive of PXE config.

It is what XenCenter uses for the RPU wizard, but can also be used through the CLI, it's use within the RPU wizard is fully supported for production systems but direct access is not officially supported.

Instructions
  1. Firstly you need an HTTP location of the unpacked main.iso.
  2. Next, run this command to check the URL is OK

    xe host-call-plugin plugin=prepare_host_upgrade.py host-uuid=<host-uuid> fn=testUrl args:url=http://path_to_xenserver_install_files/

    (The trailing slashes at the end of the URL is important). This should return "true" .
  3. Then run this command to download the XenServer installer and set up a boot config:

    xe host-call-plugin plugin=prepare_host_upgrade.py host-uuid=<host-uuid> fn=main args:url=http://path_to_xenserver_install_files/

  4. Finally, reboot the host. The host should boot into the installer, run the upgrade, then reboot back into XenServer.

 

Automated host restore

XenServer can be restored manually on a per host basis. Section 8.9.2. Backing up XenServer hosts of the Administrators Guide details this process of restoring from an installation CD:

 

To use the restored version of the root filesystem, reboot the XenServer host using the XenServer installation CD and select the Restore from backup option.

After the restore from backup is completed, reboot the XenServer host and it will start up from the restored

image.                                                    

Finally, restore the VM meta-data using

xe pool-restore-database file-name=/var/backup/pool-database-*

Note:

Restoring from a backup as described here does not destroy the backup partition.

 

Developers who are familiar with this mechanism may find it useful in test situations to use the experimental feature, introduced in XenServer 6.2, which allows this backup mechanism to be accessed via an answer file. Where the contents of the answer file are:

<?xml version="1.0"?>

<restore></restore>

Or for a more sophisticated version where you have multiple
disks:

<?xml version="1.0"?>

<restore>

        <backup-disk>sdb</backup-disk>

</restore>

The answerfile should be run using the installer that the host is being rolled back from – e.g. to roll back from XS6.2 to XS6.1, the XS6.2 installer should be run.

Debugging problems with install or failed first boot

FeedBack on Developer Upgrade/Install/Rollback Tools

Whilst the techniques on this page are provided as is for test and development use. Long-term we would like to move them from experimental status to fully supported and as such welcome positive feedback and also would like to hear and resolve any problems users encounter on this forum thread.  

 

 

 

About XenServer

XenServer is the leading open source virtualization platform, powered by the Xen Project hypervisor and the XAPI toolstack. It is used in the world's largest clouds and enterprises.
 
Commercial support for XenServer is available from Citrix.