All Things Xen

General ramblings regarding Citrix XenServer & its open source counter part.

Basic Network Testing with IPERF

Purpose

I am often asked how one can perform simple network testing within, outside, and into XenServer.  This is a great question as – by itself – it is simple enough to answer.  However, depending on what one desires out of “network testing” the answer can quickly become more complex.

As such, this I have decided to answer this question using a long standing, free utility called IPERF (well, IPERF2).  It is a rather simple, straight-forward, but powerful utility I have used over many, many years.  Links to IPERF will be provided - along with documentation on its use - as it will serve in this guide as a way to:


- Test bandwidth between two or more points

- Determine bottlenecks

- Assists with black box testing or “what happens if” scenarios

- Use a tool that runs on both Linux and Windows

- And more…

IPERF: A Visual Breakdown

IPERF has to be installed on/at at least two separate end points.  One point acts a server/receiver and the other point acts as a client/transmitter.  This so network testing can be done on a simple subnet to a complex, routed network: end-to-end using TCP or UDP generated traffic:

The visual shows an IPERF client transmitting data over IPv4 to an IPERF receiver.  Packets traverse the network - from wireless routers and through firewalls - from the client side to the server side to over port 5001.

IPERF and XenServer

The key to network testing is in remembering that any device which is connected to a network infrastructure – Virtual or Physical – is a node, host, target, end point, or just simply … a networked device.

With regards to virtual machines, XenServer obviously supports Windows and Linux operating systems.  IPERF can be used to test virtual-to-virtual networking as well as virtual-to-physical networking.  If we stack virtual machines in a box to our left and stack physical machines in a box to our right – despite a common subnet or routed network – we can quickly see the permutations of how "Virtual and Physical Network Testing" can be achieved with IPERF transmitting data from one point to another:

And if one wanted, they could just as easily test networking for this:

Requirements

To illustrate a basic server/client model with IPERF, the following will be required:

- A Windows 7 VM that will act as an IPERF client

- A CentOS 5.x VM that will act as a receiver.

- IPERF2 (the latest version of IPERF, or "IPERF3" can be found at https://github.com/esnet/iperf or, more specifically, http://downloads.es.net/pub/iperf/)

The reason for using IPERF2 is quite simple: portability and compatibility on two of the most popular operating systems that I know are virtualized.  In addition, the same steps to installing IPERF2 on these hosts can be carried out on physical systems running similar operating systems, as well. 

The remainder of this article - regarding IPERF2 - will require use of the MS-DOS command-line as well as the Linux shell (of choice).  I will carefully explain all commands as so if you are “strictly a GUI” person, you should fit right in.

Disclaimer

When utilizing IPERF2, keep in mind that this is a traffic generator.  While one can control the quantity and duration of traffic, it is still network traffic

So, consider testing during non-peak hours or after hours as to not interfere with production-based network activity.

Windows and IPERF

The Windows port of IPERF 2.0.5 requires Windows XP (or greater) and can be downloaded from:

http://sourceforge.net/p/iperf/patches/_discuss/thread/20d4a4b0/5c44/attachment/Iperf.zip

Within the .zip file you will find two directories.  One is labeled DEBUG and the other is labeled RELEASE.  Export the Iperf.exe program to a directory you will remember, such as C:\iperf\

Now, accessing the command line (cmd.exe), navigate to C:\iperf\ and execute:

iperf

The following output should appear:

Linux and IPERF

If you have additional repos already configured for CentOS, you can simply execute (as root):

yum install iperf

If that fails, one will need to download the Fedora/RedHat EPEL-Release RPM file for the version of CentOS being used.  To do this (as root), execute:

wget  http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
rpm -Uvh epel-release-5-4.noarch.rpm

 

*** Note that the above EPEL-Release RPM file is just an example (a working one) ***

 

Once epel-release-5-4.noarch.rpm is installed, execute:

yum install iperf

And once complete, as root execute iperf and one should see the following output:

http://cdn.ws.citrix.com/wp-content/uploads/2014/06/CMD2.png?__utma=222274247.1078613845.1409810797.1412210514.1412210784.2&__utmb=222274247.5.8.1412227628611&__utmc=222274247&__utmx=-&__utmz=222274247.1412210514.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)&__utmv=222274247.|1=my%20account%20holder=y=1^14=industry=(Non-company%20Visitor)=1^15=sub_industry=(Non-company%20Visitor)=1^16=employee_count=(Non-company%20Visitor)=1^17=company_name=(Non-company%20Visitor)=1^18=primary_sic=(Non-company%20Visitor)=1^19=registry_dma_code=(Non-company%20Visitor)=1&__utmk=208580497

Notice that it is the same output as what is being displayed from Windows.  IPERF2 is expecting a "-s" (server) or "-c" (client) command-line option with additional arguments.

IPERF Command-Line Arguments

On either Windows or Linux, a complete list of options for IPERF2 can be listed by executing:

iperf –help

A few good resources of examples to use IPERF2 options for the server or client can be referenced at:

http://www.slashroot.in/iperf-how-test-network-speedperformancebandwidth

http://samkear.com/networking/iperf-commands-network-troubleshooting

http://www.techrepublic.com/blog/data-center/handy-iperf-commands-for-quick-network-testing/

For now, we will focus on the options needed for our server and client:

-f, –format    [kmKM]   format to report: Kbits, Mbits, KBytes, MBytes
-m, –print_mss          print TCP maximum segment size (MTU – TCP/IP header)
-i, –interval  #        seconds between periodic bandwidth reports
-s, –server             run in server mode
-c, –client    <host>   run in client mode, connecting to <host>
-t, –time      #        time in seconds to transmit for (default 10 secs)

Lastly, there is a TCP/IP Window setting.  This goes beyond the scope of this document as it relates to the TCP frame/windowing of data.  I highly recommend reading either of the two following links – especially for Linux – as there has always been some debate as what is “best to be used”:

https://kb.doit.wisc.edu/wiscnet/page.php?id=11779

http://kb.pert.geant.net/PERTKB/IperfTool

Running An IPERF Test

So, we have IPERF2 installed on Windows 7 and on CentOS 5.10.  Before one performs any testing, ensure any AV does not block iperf.exe from running as well as port 5001 being opened across the network network.

Again, another port can be specified, but the default port IPERF2 uses for both client and server is 5001.

Server/Receiver Side

The Server/Receiver side will be on the CentOS VM.

Following the commands above, we want to execute the following to run IPERF2 as a server/receiver from our Windows 7 client machine:

iperf -s -f M -m -i 10

The output should show:

————————————————————
Server listening on TCP port 5001
TCP window size: 0.08 MByte (default)
————————————————————

The TCP window size has been previously commented on and the server is now ready to accept connections (press Control+C or Control+Z to exit).

Client/Transmission Side

Let us now focus on the client side to start sending data from the Windows 7 VM to the CentOS VM.

From Windows 7, the command line to start transmitting data for 30 seconds to our CentOS host (x.x.x.48) is:

iperf -c x.x.x.48 -t 30 -f M

Pressing enter, the traffic flow begins and the output from the client side looks like this:

From the server side, the output looks something like this:

And there we have it – a first successful test from a Windows 7 VM (located on one XenServer) to a CentOS 5.10 VM (located on another XenServer).

Understanding the Results

From either the client side or server side, results are shown by time and average.  The key item to look for from either side is:

0.0-30.0 sec  55828 MBytes  1861 MBytes/sec

Why?  This shows the average over the course of 0.0 to 30.0 seconds in terms of total megabytes transmitted as well as average megabytes of data sent per second.  In addition, since the "-f M" argument was passed as a command-line option, the output is calculated in megabytes accordingly.

In this particular case, we simply illustrated that from one VM to another VM, we transferred data at 1861 megabytes per second.

*** Note that this test was performed in a local lab with lower-end hardware than what you probably have! ***

--jkbs | @xenfomation

 

Average Queue Size and Storage IO Metrics
Increasing Ubuntu's Resolution

Related Posts

 

Comments 15

chaitanya on Monday, 10 November 2014 16:59

Hi,

Nice article..
I have a simple question.. you did this test for windows and linux os. Any specific requirement on that? I dont think it is mandatory to perform this test between win and linux.. any other reasons? or you just wanted to show that this tool/test can be used on cross platforms?

Chaitanya.

1
Hi, Nice article.. I have a simple question.. you did this test for windows and linux os. Any specific requirement on that? I dont think it is mandatory to perform this test between win and linux.. any other reasons? or you just wanted to show that this tool/test can be used on cross platforms? Chaitanya.
JK Benedict on Wednesday, 12 November 2014 03:08

Exactly: to show that IPERF can be used in any configuration, any school of thought, etc!

Windows Windows
Linux Linux
Linux Windows

--jkbs | @xenfomation

0
Exactly: to show that IPERF can be used in any configuration, any school of thought, etc! Windows Windows Linux Linux Linux Windows --jkbs | @xenfomation
Massimo De Nadal on Tuesday, 11 November 2014 12:24

Hi,
your throughput is 1861 MB/sec which means more than 14Gb !!!! :o
Can I ask you what kind of server/setup are you using ???
I've never seen such performance between 2 VMs !

Massimo

1
Hi, your throughput is 1861 MB/sec which means more than 14Gb !!!! :o Can I ask you what kind of server/setup are you using ??? I've never seen such performance between 2 VMs ! Massimo
JK Benedict on Wednesday, 12 November 2014 03:38

Hey, Massimo --

Thanks for posting! I was speaking with Tobias (thanks for letting me know about this) and in short, the issue is that IPERF has bits/bytes mixed up... or that my testing is flawed?

It's actually neither (well, I can't speak for IPERF until I look at the code), but this is a good question as I have seen impressive speeds - especially in Creedence - between VM to VM.

Now, per my calculation and trusting that IPERF is using the International System of Units, then 1861 MBytes is roughly 1.9-2.0 Gibabits (International System of Units, not the other metrics). Per second. Not unreasonable at all.

Second, the local hardware I used during testing were two, humble Dual Core systems: simple Intel Gigabit NICs, etc. Nothing up my sleeve - most sincerely and I challenge anyone to make a visit to our office in Georgia for a demonstration! :p

Massimo - and anyone else who might be interested - the following were the test scenarios I ran through. Where two separate XenServers were used, it is important to note that these were behind the SAME Gigabit switch (not routed).

Test one was from VM VM on the the same hypervisor
Test two was from VM real world physical machine w/ a 1GB NIC
Test three was from VM VM (asynchronous).
Test four was from VM VM (synchronous).

Lastly, I have to throw back to this article as while VM "advertised" speed may be reported as 1GB, please take a look at this:

http://blogs.citrix.com/2014/06/24/from-my-virtual-desktop-pv-nic-speed-advertisement-vs-reality/

So, Massimo - thank you very much for asking. I do have a question for you -- what is the usual performance that you do see between VMs?

Thanks!

--jkbs | @xenfomation

0
Hey, Massimo -- Thanks for posting! I was speaking with Tobias (thanks for letting me know about this) and in short, the issue is that IPERF has bits/bytes mixed up... or that my testing is flawed? It's actually neither (well, I can't speak for IPERF until I look at the code), but this is a good question as I have seen impressive speeds - especially in Creedence - between VM to VM. Now, per my calculation and trusting that IPERF is using the International System of Units, then 1861 MBytes is roughly 1.9-2.0 Gibabits (International System of Units, not the other metrics). Per second. Not unreasonable at all. Second, the local hardware I used during testing were two, humble Dual Core systems: simple Intel Gigabit NICs, etc. Nothing up my sleeve - most sincerely and I challenge anyone to make a visit to our office in Georgia for a demonstration! :p Massimo - and anyone else who might be interested - the following were the test scenarios I ran through. Where two separate XenServers were used, it is important to note that these were behind the SAME Gigabit switch (not routed). Test one was from VM VM on the the same hypervisor Test two was from VM real world physical machine w/ a 1GB NIC Test three was from VM VM (asynchronous). Test four was from VM VM (synchronous). Lastly, I have to throw back to this article as while VM "advertised" speed may be reported as 1GB, please take a look at this: http://blogs.citrix.com/2014/06/24/from-my-virtual-desktop-pv-nic-speed-advertisement-vs-reality/ So, Massimo - thank you very much for asking. I do have a question for you -- what is the usual performance that you do see between VMs? Thanks! --jkbs | @xenfomation
Tobias Kreidl on Wednesday, 12 November 2014 05:37

Hi, Jesse. Thanks first off for the very interesting article, which as usual, sheds light on areas where those wondering what is really going on behind the scene can get some useful tools to help better understand or diagnose their environments.

I would discount any internal server/VM transfers, since these will be super fast and bypass any physical NICs. Going from a server to a VM on a different physical box, it should certainly be possible to look at the perfmeter in XenCenter for the VM and get a rough idea if the speed matched that of iperf. I frankly have a hard time seeing how more than about 80-90% of the physical speed rating of a NIC can ever be exceeded, i.e. 800 - 900 Mbits/sec for a 1 Gb NIC. Otherwise, it would appear that buffering or some other cache mechanism is kicking in. Likewise, in various other iperf user reports, I have not seen rates over around 8.5 to 9 Mbits/sec reported for a 10 Gb NIC connection.

I do wonder if the flags are being properly interpreted. It would be enlightening to see similar output originating from the most basic command line string, something like "iperf -c IP-of-client" targeting a different physical box.

Finally, iperf 3 is the latest version, available from GitHub. It has been pretty much rewritten from scratch but retains much of the original functionality, while adding some new features.

1
Hi, Jesse. Thanks first off for the very interesting article, which as usual, sheds light on areas where those wondering what is really going on behind the scene can get some useful tools to help better understand or diagnose their environments. I would discount any internal server/VM transfers, since these will be super fast and bypass any physical NICs. Going from a server to a VM on a different physical box, it should certainly be possible to look at the perfmeter in XenCenter for the VM and get a rough idea if the speed matched that of iperf. I frankly have a hard time seeing how more than about 80-90% of the physical speed rating of a NIC can ever be exceeded, i.e. 800 - 900 Mbits/sec for a 1 Gb NIC. Otherwise, it would appear that buffering or some other cache mechanism is kicking in. Likewise, in various other iperf user reports, I have not seen rates over around 8.5 to 9 Mbits/sec reported for a 10 Gb NIC connection. I do wonder if the flags are being properly interpreted. It would be enlightening to see similar output originating from the most basic command line string, something like "iperf -c IP-of-client" targeting a different physical box. Finally, iperf 3 is the latest version, available from GitHub. It has been pretty much rewritten from scratch but retains much of the original functionality, while adding some new features.
JK Benedict on Sunday, 16 November 2014 02:34

I do see why VM - VM might be something one would not like to test, but I cannot tell you how often I receive questions about VM to VM network performance.

In short, I appreciate the feedback and this IPERF overview is really just that -- a means to getting an idea of testing just one of many items related to XenServer.

At present, I am incorporating this as well as disk performance in my next article.

Thanks!!

--jkbs | @xenfomation

0
I do see why VM - VM might be something one would not like to test, but I cannot tell you how often I receive questions about VM to VM network performance. In short, I appreciate the feedback and this IPERF overview is really just that -- a means to getting an idea of testing just one of many items related to XenServer. At present, I am incorporating this as well as disk performance in my next article. Thanks!! --jkbs | @xenfomation
Massimo De Nadal on Wednesday, 12 November 2014 09:23

Hi JK,
maybe I'm wrong... but 55828 MBytes in 30 sec means 1861 MBytes/sec
1861 MBytes means 14887 MBits.... that sounds like 14,54 Gbits !

So... definetely it's time to try Creedence in depth !!! ;)

These are my actual numbers, the HV is a decent Dell R320 (Xeon E5-2420), Xenserver 6.2 and No openvswitch:

Linux - Linux (both paravirtualized)
[ 3] 0.0-10.0 sec 10.2 GBytes 8.80 Gbits/sec

Linux (paravirtual) - Windows (paravirtual drivers)
[188] 0.0-10.0 sec 8.63 GBytes 7.41 Gbits/sec

Other server from my customers show similar numbers.

Cheers,
Massimo

0
Hi JK, maybe I'm wrong... but 55828 MBytes in 30 sec means 1861 MBytes/sec 1861 MBytes means 14887 MBits.... that sounds like 14,54 Gbits ! So... definetely it's time to try Creedence in depth !!! ;) These are my actual numbers, the HV is a decent Dell R320 (Xeon E5-2420), Xenserver 6.2 and No openvswitch: Linux - Linux (both paravirtualized) [ 3] 0.0-10.0 sec 10.2 GBytes 8.80 Gbits/sec Linux (paravirtual) - Windows (paravirtual drivers) [188] 0.0-10.0 sec 8.63 GBytes 7.41 Gbits/sec Other server from my customers show similar numbers. Cheers, Massimo
Tobias Kreidl on Wednesday, 12 November 2014 17:13

Hi, Massimo:
8.80 Gbits/sec = about 1 GByte/sec and is still nowhere close to 14 Gbits/sec which even a 10 Gb NIC should not be capable of. Were your tests across two physical boxes? I ask because internal transfers are going to use VIFs and will not even flow over the network cards themselves, and hence the numbers will be way greater than what a physical NIC can handle and depend more on the server architecture. Also, I did a lot of testing a while back with OVS vs. bridge mode and saw virtually no difference in speed. So again, please clarify where your server and client resided with your own iperf tests.
Regards,
-=Tobias

0
Hi, Massimo: 8.80 Gbits/sec = about 1 GByte/sec and is still nowhere close to 14 Gbits/sec which even a 10 Gb NIC should not be capable of. Were your tests across two physical boxes? I ask because internal transfers are going to use VIFs and will not even flow over the network cards themselves, and hence the numbers will be way greater than what a physical NIC can handle and depend more on the server architecture. Also, I did a lot of testing a while back with OVS vs. bridge mode and saw virtually no difference in speed. So again, please clarify where your server and client resided with your own iperf tests. Regards, -=Tobias
Guest - james on Tuesday, 18 November 2014 18:18

Hi JK, thanks for good post. Just curious what was the tool used for creating those visuals in this article? are these any special stencils (visio)?
thanks.

0
Hi JK, thanks for good post. Just curious what was the tool used for creating those visuals in this article? are these any special stencils (visio)? thanks.
JK Benedict on Saturday, 22 November 2014 15:29

Thanks!

Indeed, I use Visio. However, I use The Gimp and Mspaint for just about all visuals. Each tool is slick, but I suppose my OCD leads me to seek perfection and reusable images.

--jkbs @xenfomation

0
Thanks! Indeed, I use Visio. However, I use The Gimp and Mspaint for just about all visuals. Each tool is slick, but I suppose my OCD leads me to seek perfection and reusable images. --jkbs @xenfomation
Felipe Franciosi on Wednesday, 03 December 2014 12:37

Thanks for writing this up, JK. I would like to add that it is imperative to ensure iperf versions match on the sending and destination hosts. We have had problems in the past where versions mismatched (even though they were installed from the same source -- rpmforge in that case) and, as a result, measurements were all over the place.

0
Thanks for writing this up, JK. I would like to add that it is imperative to ensure iperf versions match on the sending and destination hosts. We have had problems in the past where versions mismatched (even though they were installed from the same source -- rpmforge in that case) and, as a result, measurements were all over the place.
JK Benedict on Sunday, 09 August 2015 16:03

FELIPE!!!

I was preparing to respond to Gabriel's recent comment and found I NEVER replied to your comment. I owe you a pint, kind sir and apologies! :(

You are absolutely correct, sir - ABSOLUTELY CORRECT - and I do need to brush up this blog entry with that information!

So, what Felipe is emphasizing is that, despite your configuration, IPERF should originate from the same RPM/Source Repository. This is, as Tobias had also mentioned earlier, is that IPERF is a very sweet tool for pushing traffic...

However, it is free open-source, but has recently had a resurgence of active development. As such, for the sake of precision, make sure that IPERF for all hosts involved in testing:

  • Are the same version
  • Come from the same repository/source
  • And, where a Windows IPERF binary will be used against a Linux IPERF binary, make CERTAIN these "ported versions" come from the same "software maintainer"

  • This is all for the sake of science, consistency and so forth. The world will not end, but depending on the type/length of testing you are doing, a version/binary mismatch can taint results.

    Cheers!!

    0
    FELIPE!!! I was preparing to respond to Gabriel's recent comment and found I NEVER replied to your comment. I owe you a pint, kind sir and apologies! :( [b]You are absolutely correct, sir - ABSOLUTELY CORRECT - and I do need to brush up this blog entry with that information![/b] So, what Felipe is emphasizing is that, despite your configuration, IPERF should originate from the same RPM/Source Repository. This is, as Tobias had also mentioned earlier, is that IPERF is a very sweet tool for pushing traffic... However, it is free open-source, but has recently had a resurgence of active development. As such, for the sake of precision, make sure that IPERF for all hosts involved in testing: [*] Are the same version [*] Come from the same repository/source [*] And, where a Windows IPERF binary will be used against a Linux IPERF binary, make CERTAIN these "ported versions" come from the same "software maintainer" This is all for the sake of science, consistency and so forth. The world will not end, but depending on the type/length of testing you are doing, a version/binary mismatch can taint results. Cheers!!
    Guest - gabriel on Friday, 07 August 2015 19:27

    why did you use all those parameter?
    why not just -s and -c and that's it?

    1
    why did you use all those parameter? why not just -s and -c and that's it?
    JK Benedict on Sunday, 09 August 2015 16:20

    Gabriel,

    Thanks for the questions, but honestly I think the better question is "Why did you even test this, write it up and share it in the first place?" :D

    Sincerely, the main reason I did this was to satisfy my OCD as well as give a basic overview of how you can use IPERF.

    The second reason is that I wanted to make absolutely certain that the tests I ran on various hosts were as consistent as possible. So, despite how wonderful any software may be, I specifically used "all those parameters" so I control what IPERF is doing on both sides.

    While -s and -c would suffice for run of the mill testing, I wanted to expose some of the options...

    https://iperf.fr/iperf-doc.html#doc

    Cheers!

    0
    Gabriel, Thanks for the questions, but honestly I think the better question is "Why did you even test this, write it up and share it in the first place?" :D Sincerely, the main reason I did this was to satisfy my OCD as well as give a basic overview of how you can use IPERF. The second reason is that I wanted to make absolutely certain that the tests I ran on various hosts were as consistent as possible. So, despite how wonderful any software may be, I specifically used "all those parameters" so I control what IPERF is doing on both sides. While -s and -c would suffice for run of the mill testing, I wanted to expose some of the options... https://iperf.fr/iperf-doc.html#doc Cheers!
    Guest - Julianich on Tuesday, 12 January 2016 16:16

    i guess you should install KrojamSoft FilesSearch and see if it can help

    0
    i guess you should install KrojamSoft FilesSearch and see if it can help

    About XenServer

    XenServer is the leading open source virtualization platform, powered by the Xen Project hypervisor and the XAPI toolstack. It is used in the world's largest clouds and enterprises.
     
    Commercial support for XenServer is available from Citrix.