You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
4150 lines
178 KiB
4150 lines
178 KiB
\input texinfo @c -*-texinfo-*- |
|
@c %**start of header |
|
@setfilename netperf.info |
|
@settitle Care and Feeding of Netperf 2.6.X |
|
@c %**end of header |
|
|
|
@copying |
|
This is Rick Jones' feeble attempt at a Texinfo-based manual for the |
|
netperf benchmark. |
|
|
|
Copyright @copyright{} 2005-2012 Hewlett-Packard Company |
|
@quotation |
|
Permission is granted to copy, distribute and/or modify this document |
|
per the terms of the netperf source license, a copy of which can be |
|
found in the file @file{COPYING} of the basic netperf distribution. |
|
@end quotation |
|
@end copying |
|
|
|
@titlepage |
|
@title Care and Feeding of Netperf |
|
@subtitle Versions 2.6.0 and Later |
|
@author Rick Jones @email{rick.jones2@@hp.com} |
|
@c this is here to start the copyright page |
|
@page |
|
@vskip 0pt plus 1filll |
|
@insertcopying |
|
@end titlepage |
|
|
|
@c begin with a table of contents |
|
@contents |
|
|
|
@ifnottex |
|
@node Top, Introduction, (dir), (dir) |
|
@top Netperf Manual |
|
|
|
@insertcopying |
|
@end ifnottex |
|
|
|
@menu |
|
* Introduction:: An introduction to netperf - what it |
|
is and what it is not. |
|
* Installing Netperf:: How to go about installing netperf. |
|
* The Design of Netperf:: |
|
* Global Command-line Options:: |
|
* Using Netperf to Measure Bulk Data Transfer:: |
|
* Using Netperf to Measure Request/Response :: |
|
* Using Netperf to Measure Aggregate Performance:: |
|
* Using Netperf to Measure Bidirectional Transfer:: |
|
* The Omni Tests:: |
|
* Other Netperf Tests:: |
|
* Address Resolution:: |
|
* Enhancing Netperf:: |
|
* Netperf4:: |
|
* Concept Index:: |
|
* Option Index:: |
|
@end menu |
|
|
|
@node Introduction, Installing Netperf, Top, Top |
|
@chapter Introduction |
|
|
|
@cindex Introduction |
|
|
|
Netperf is a benchmark that can be use to measure various aspect of |
|
networking performance. The primary foci are bulk (aka |
|
unidirectional) data transfer and request/response performance using |
|
either TCP or UDP and the Berkeley Sockets interface. As of this |
|
writing, the tests available either unconditionally or conditionally |
|
include: |
|
|
|
@itemize @bullet |
|
@item |
|
TCP and UDP unidirectional transfer and request/response over IPv4 and |
|
IPv6 using the Sockets interface. |
|
@item |
|
TCP and UDP unidirectional transfer and request/response over IPv4 |
|
using the XTI interface. |
|
@item |
|
Link-level unidirectional transfer and request/response using the DLPI |
|
interface. |
|
@item |
|
Unix domain sockets |
|
@item |
|
SCTP unidirectional transfer and request/response over IPv4 and IPv6 |
|
using the sockets interface. |
|
@end itemize |
|
|
|
While not every revision of netperf will work on every platform |
|
listed, the intention is that at least some version of netperf will |
|
work on the following platforms: |
|
|
|
@itemize @bullet |
|
@item |
|
Unix - at least all the major variants. |
|
@item |
|
Linux |
|
@item |
|
Windows |
|
@item |
|
Others |
|
@end itemize |
|
|
|
Netperf is maintained and informally supported primarily by Rick |
|
Jones, who can perhaps be best described as Netperf Contributing |
|
Editor. Non-trivial and very appreciated assistance comes from others |
|
in the network performance community, who are too numerous to mention |
|
here. While it is often used by them, netperf is NOT supported via any |
|
of the formal Hewlett-Packard support channels. You should feel free |
|
to make enhancements and modifications to netperf to suit your |
|
nefarious porpoises, so long as you stay within the guidelines of the |
|
netperf copyright. If you feel so inclined, you can send your changes |
|
to |
|
@email{netperf-feedback@@netperf.org,netperf-feedback} for possible |
|
inclusion into subsequent versions of netperf. |
|
|
|
It is the Contributing Editor's belief that the netperf license walks |
|
like open source and talks like open source. However, the license was |
|
never submitted for ``certification'' as an open source license. If |
|
you would prefer to make contributions to a networking benchmark using |
|
a certified open source license, please consider netperf4, which is |
|
distributed under the terms of the GPLv2. |
|
|
|
The @email{netperf-talk@@netperf.org,netperf-talk} mailing list is |
|
available to discuss the care and feeding of netperf with others who |
|
share your interest in network performance benchmarking. The |
|
netperf-talk mailing list is a closed list (to deal with spam) and you |
|
must first subscribe by sending email to |
|
@email{netperf-talk-request@@netperf.org,netperf-talk-request}. |
|
|
|
|
|
@menu |
|
* Conventions:: |
|
@end menu |
|
|
|
@node Conventions, , Introduction, Introduction |
|
@section Conventions |
|
|
|
A @dfn{sizespec} is a one or two item, comma-separated list used as an |
|
argument to a command-line option that can set one or two, related |
|
netperf parameters. If you wish to set both parameters to separate |
|
values, items should be separated by a comma: |
|
|
|
@example |
|
parameter1,parameter2 |
|
@end example |
|
|
|
If you wish to set the first parameter without altering the value of |
|
the second from its default, you should follow the first item with a |
|
comma: |
|
|
|
@example |
|
parameter1, |
|
@end example |
|
|
|
|
|
Likewise, precede the item with a comma if you wish to set only the |
|
second parameter: |
|
|
|
@example |
|
,parameter2 |
|
@end example |
|
|
|
An item with no commas: |
|
|
|
@example |
|
parameter1and2 |
|
@end example |
|
|
|
will set both parameters to the same value. This last mode is one of |
|
the most frequently used. |
|
|
|
There is another variant of the comma-separated, two-item list called |
|
a @dfn{optionspec} which is like a sizespec with the exception that a |
|
single item with no comma: |
|
|
|
@example |
|
parameter1 |
|
@end example |
|
|
|
will only set the value of the first parameter and will leave the |
|
second parameter at its default value. |
|
|
|
Netperf has two types of command-line options. The first are global |
|
command line options. They are essentially any option not tied to a |
|
particular test or group of tests. An example of a global |
|
command-line option is the one which sets the test type - @option{-t}. |
|
|
|
The second type of options are test-specific options. These are |
|
options which are only applicable to a particular test or set of |
|
tests. An example of a test-specific option would be the send socket |
|
buffer size for a TCP_STREAM test. |
|
|
|
Global command-line options are specified first with test-specific |
|
options following after a @code{--} as in: |
|
|
|
@example |
|
netperf <global> -- <test-specific> |
|
@end example |
|
|
|
|
|
@node Installing Netperf, The Design of Netperf, Introduction, Top |
|
@chapter Installing Netperf |
|
|
|
@cindex Installation |
|
|
|
Netperf's primary form of distribution is source code. This allows |
|
installation on systems other than those to which the authors have |
|
ready access and thus the ability to create binaries. There are two |
|
styles of netperf installation. The first runs the netperf server |
|
program - netserver - as a child of inetd. This requires the |
|
installer to have sufficient privileges to edit the files |
|
@file{/etc/services} and @file{/etc/inetd.conf} or their |
|
platform-specific equivalents. |
|
|
|
The second style is to run netserver as a standalone daemon. This |
|
second method does not require edit privileges on @file{/etc/services} |
|
and @file{/etc/inetd.conf} but does mean you must remember to run the |
|
netserver program explicitly after every system reboot. |
|
|
|
This manual assumes that those wishing to measure networking |
|
performance already know how to use anonymous FTP and/or a web |
|
browser. It is also expected that you have at least a passing |
|
familiarity with the networking protocols and interfaces involved. In |
|
all honesty, if you do not have such familiarity, likely as not you |
|
have some experience to gain before attempting network performance |
|
measurements. The excellent texts by authors such as Stevens, Fenner |
|
and Rudoff and/or Stallings would be good starting points. There are |
|
likely other excellent sources out there as well. |
|
|
|
@menu |
|
* Getting Netperf Bits:: |
|
* Installing Netperf Bits:: |
|
* Verifying Installation:: |
|
@end menu |
|
|
|
@node Getting Netperf Bits, Installing Netperf Bits, Installing Netperf, Installing Netperf |
|
@section Getting Netperf Bits |
|
|
|
Gzipped tar files of netperf sources can be retrieved via |
|
@uref{ftp://ftp.netperf.org/netperf,anonymous FTP} |
|
for ``released'' versions of the bits. Pre-release versions of the |
|
bits can be retrieved via anonymous FTP from the |
|
@uref{ftp://ftp.netperf.org/netperf/experimental,experimental} subdirectory. |
|
|
|
For convenience and ease of remembering, a link to the download site |
|
is provided via the |
|
@uref{http://www.netperf.org/, NetperfPage} |
|
|
|
The bits corresponding to each discrete release of netperf are |
|
@uref{http://www.netperf.org/svn/netperf2/tags,tagged} for retrieval |
|
via subversion. For example, there is a tag for the first version |
|
corresponding to this version of the manual - |
|
@uref{http://www.netperf.org/svn/netperf2/tags/netperf-2.6.0,netperf |
|
2.6.0}. Those wishing to be on the bleeding edge of netperf |
|
development can use subversion to grab the |
|
@uref{http://www.netperf.org/svn/netperf2/trunk,top of trunk}. When |
|
fixing bugs or making enhancements, patches against the top-of-trunk |
|
are preferred. |
|
|
|
There are likely other places around the Internet from which one can |
|
download netperf bits. These may be simple mirrors of the main |
|
netperf site, or they may be local variants on netperf. As with |
|
anything one downloads from the Internet, take care to make sure it is |
|
what you really wanted and isn't some malicious Trojan or whatnot. |
|
Caveat downloader. |
|
|
|
As a general rule, binaries of netperf and netserver are not |
|
distributed from ftp.netperf.org. From time to time a kind soul or |
|
souls has packaged netperf as a Debian package available via the |
|
apt-get mechanism or as an RPM. I would be most interested in |
|
learning how to enhance the makefiles to make that easier for people. |
|
|
|
@node Installing Netperf Bits, Verifying Installation, Getting Netperf Bits, Installing Netperf |
|
@section Installing Netperf |
|
|
|
Once you have downloaded the tar file of netperf sources onto your |
|
system(s), it is necessary to unpack the tar file, cd to the netperf |
|
directory, run configure and then make. Most of the time it should be |
|
sufficient to just: |
|
|
|
@example |
|
gzcat netperf-<version>.tar.gz | tar xf - |
|
cd netperf-<version> |
|
./configure |
|
make |
|
make install |
|
@end example |
|
|
|
Most of the ``usual'' configure script options should be present |
|
dealing with where to install binaries and whatnot. |
|
@example |
|
./configure --help |
|
@end example |
|
should list all of those and more. You may find the @code{--prefix} |
|
option helpful in deciding where the binaries and such will be put |
|
during the @code{make install}. |
|
|
|
@vindex --enable-cpuutil, Configure |
|
If the netperf configure script does not know how to automagically |
|
detect which CPU utilization mechanism to use on your platform you may |
|
want to add a @code{--enable-cpuutil=mumble} option to the configure |
|
command. If you have knowledge and/or experience to contribute to |
|
that area, feel free to contact @email{netperf-feedback@@netperf.org}. |
|
|
|
@vindex --enable-xti, Configure |
|
@vindex --enable-unixdomain, Configure |
|
@vindex --enable-dlpi, Configure |
|
@vindex --enable-sctp, Configure |
|
Similarly, if you want tests using the XTI interface, Unix Domain |
|
Sockets, DLPI or SCTP it will be necessary to add one or more |
|
@code{--enable-[xti|unixdomain|dlpi|sctp]=yes} options to the configure |
|
command. As of this writing, the configure script will not include |
|
those tests automagically. |
|
|
|
@vindex --enable-omni, Configure |
|
Starting with version 2.5.0, netperf began migrating most of the |
|
``classic'' netperf tests found in @file{src/nettest_bsd.c} to the |
|
so-called ``omni'' tests (aka ``two routines to run them all'') found |
|
in @file{src/nettest_omni.c}. This migration enables a number of new |
|
features such as greater control over what output is included, and new |
|
things to output. The ``omni'' test is enabled by default in 2.5.0 |
|
and a number of the classic tests are migrated - you can tell if a |
|
test has been migrated |
|
from the presence of @code{MIGRATED} in the test banner. If you |
|
encounter problems with either the omni or migrated tests, please |
|
first attempt to obtain resolution via |
|
@email{netperf-talk@@netperf.org} or |
|
@email{netperf-feedback@@netperf.org}. If that is unsuccessful, you |
|
can add a @code{--enable-omni=no} to the configure command and the |
|
omni tests will not be compiled-in and the classic tests will not be |
|
migrated. |
|
|
|
Starting with version 2.5.0, netperf includes the ``burst mode'' |
|
functionality in a default compilation of the bits. If you encounter |
|
problems with this, please first attempt to obtain help via |
|
@email{netperf-talk@@netperf.org} or |
|
@email{netperf-feedback@@netperf.org}. If that is unsuccessful, you |
|
can add a @code{--enable-burst=no} to the configure command and the |
|
burst mode functionality will not be compiled-in. |
|
|
|
On some platforms, it may be necessary to precede the configure |
|
command with a CFLAGS and/or LIBS variable as the netperf configure |
|
script is not yet smart enough to set them itself. Whenever possible, |
|
these requirements will be found in @file{README.@var{platform}} files. |
|
Expertise and assistance in making that more automagic in the |
|
configure script would be most welcome. |
|
|
|
@cindex Limiting Bandwidth |
|
@cindex Bandwidth Limitation |
|
@vindex --enable-intervals, Configure |
|
@vindex --enable-histogram, Configure |
|
Other optional configure-time settings include |
|
@code{--enable-intervals=yes} to give netperf the ability to ``pace'' |
|
its _STREAM tests and @code{--enable-histogram=yes} to have netperf |
|
keep a histogram of interesting times. Each of these will have some |
|
effect on the measured result. If your system supports |
|
@code{gethrtime()} the effect of the histogram measurement should be |
|
minimized but probably still measurable. For example, the histogram |
|
of a netperf TCP_RR test will be of the individual transaction times: |
|
@example |
|
netperf -t TCP_RR -H lag -v 2 |
|
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to lag.hpl.hp.com (15.4.89.214) port 0 AF_INET : histogram |
|
Local /Remote |
|
Socket Size Request Resp. Elapsed Trans. |
|
Send Recv Size Size Time Rate |
|
bytes Bytes bytes bytes secs. per sec |
|
|
|
16384 87380 1 1 10.00 3538.82 |
|
32768 32768 |
|
Alignment Offset |
|
Local Remote Local Remote |
|
Send Recv Send Recv |
|
8 0 0 0 |
|
Histogram of request/response times |
|
UNIT_USEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0 |
|
TEN_USEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0 |
|
HUNDRED_USEC : 0: 34480: 111: 13: 12: 6: 9: 3: 4: 7 |
|
UNIT_MSEC : 0: 60: 50: 51: 44: 44: 72: 119: 100: 101 |
|
TEN_MSEC : 0: 105: 0: 0: 0: 0: 0: 0: 0: 0 |
|
HUNDRED_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0 |
|
UNIT_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0 |
|
TEN_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0 |
|
>100_SECS: 0 |
|
HIST_TOTAL: 35391 |
|
@end example |
|
|
|
The histogram you see above is basically a base-10 log histogram where |
|
we can see that most of the transaction times were on the order of one |
|
hundred to one-hundred, ninety-nine microseconds, but they were |
|
occasionally as long as ten to nineteen milliseconds |
|
|
|
The @option{--enable-demo=yes} configure option will cause code to be |
|
included to report interim results during a test run. The rate at |
|
which interim results are reported can then be controlled via the |
|
global @option{-D} option. Here is an example of @option{-D} output: |
|
|
|
@example |
|
$ src/netperf -D 1.35 -H tardy.hpl.hp.com -f M |
|
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to tardy.hpl.hp.com (15.9.116.144) port 0 AF_INET : demo |
|
Interim result: 5.41 MBytes/s over 1.35 seconds ending at 1308789765.848 |
|
Interim result: 11.07 MBytes/s over 1.36 seconds ending at 1308789767.206 |
|
Interim result: 16.00 MBytes/s over 1.36 seconds ending at 1308789768.566 |
|
Interim result: 20.66 MBytes/s over 1.36 seconds ending at 1308789769.922 |
|
Interim result: 22.74 MBytes/s over 1.36 seconds ending at 1308789771.285 |
|
Interim result: 23.07 MBytes/s over 1.36 seconds ending at 1308789772.647 |
|
Interim result: 23.77 MBytes/s over 1.37 seconds ending at 1308789774.016 |
|
Recv Send Send |
|
Socket Socket Message Elapsed |
|
Size Size Size Time Throughput |
|
bytes bytes bytes secs. MBytes/sec |
|
|
|
87380 16384 16384 10.06 17.81 |
|
@end example |
|
|
|
Notice how the units of the interim result track that requested by the |
|
@option{-f} option. Also notice that sometimes the interval will be |
|
longer than the value specified in the @option{-D} option. This is |
|
normal and stems from how demo mode is implemented not by relying on |
|
interval timers or frequent calls to get the current time, but by |
|
calculating how many units of work must be performed to take at least |
|
the desired interval. |
|
|
|
Those familiar with this option in earlier versions of netperf will |
|
note the addition of the ``ending at'' text. This is the time as |
|
reported by a @code{gettimeofday()} call (or its emulation) with a |
|
@code{NULL} timezone pointer. This addition is intended to make it |
|
easier to insert interim results into an |
|
@uref{http://oss.oetiker.ch/rrdtool/doc/rrdtool.en.html,rrdtool} |
|
Round-Robin Database (RRD). A likely bug-riddled example of doing so |
|
can be found in @file{doc/examples/netperf_interim_to_rrd.sh}. The |
|
time is reported out to milliseconds rather than microseconds because |
|
that is the most rrdtool understands as of the time of this writing. |
|
|
|
As of this writing, a @code{make install} will not actually update the |
|
files @file{/etc/services} and/or @file{/etc/inetd.conf} or their |
|
platform-specific equivalents. It remains necessary to perform that |
|
bit of installation magic by hand. Patches to the makefile sources to |
|
effect an automagic editing of the necessary files to have netperf |
|
installed as a child of inetd would be most welcome. |
|
|
|
Starting the netserver as a standalone daemon should be as easy as: |
|
@example |
|
$ netserver |
|
Starting netserver at port 12865 |
|
Starting netserver at hostname 0.0.0.0 port 12865 and family 0 |
|
@end example |
|
|
|
Over time the specifics of the messages netserver prints to the screen |
|
may change but the gist will remain the same. |
|
|
|
If the compilation of netperf or netserver happens to fail, feel free |
|
to contact @email{netperf-feedback@@netperf.org} or join and ask in |
|
@email{netperf-talk@@netperf.org}. However, it is quite important |
|
that you include the actual compilation errors and perhaps even the |
|
configure log in your email. Otherwise, it will be that much more |
|
difficult for someone to assist you. |
|
|
|
@node Verifying Installation, , Installing Netperf Bits, Installing Netperf |
|
@section Verifying Installation |
|
|
|
Basically, once netperf is installed and netserver is configured as a |
|
child of inetd, or launched as a standalone daemon, simply typing: |
|
@example |
|
netperf |
|
@end example |
|
should result in output similar to the following: |
|
@example |
|
$ netperf |
|
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost.localdomain (127.0.0.1) port 0 AF_INET |
|
Recv Send Send |
|
Socket Socket Message Elapsed |
|
Size Size Size Time Throughput |
|
bytes bytes bytes secs. 10^6bits/sec |
|
|
|
87380 16384 16384 10.00 2997.84 |
|
@end example |
|
|
|
|
|
@node The Design of Netperf, Global Command-line Options, Installing Netperf, Top |
|
@chapter The Design of Netperf |
|
|
|
@cindex Design of Netperf |
|
|
|
Netperf is designed around a basic client-server model. There are |
|
two executables - netperf and netserver. Generally you will only |
|
execute the netperf program, with the netserver program being invoked |
|
by the remote system's inetd or having been previously started as its |
|
own standalone daemon. |
|
|
|
When you execute netperf it will establish a ``control connection'' to |
|
the remote system. This connection will be used to pass test |
|
configuration information and results to and from the remote system. |
|
Regardless of the type of test to be run, the control connection will |
|
be a TCP connection using BSD sockets. The control connection can use |
|
either IPv4 or IPv6. |
|
|
|
Once the control connection is up and the configuration information |
|
has been passed, a separate ``data'' connection will be opened for the |
|
measurement itself using the API's and protocols appropriate for the |
|
specified test. When the test is completed, the data connection will |
|
be torn-down and results from the netserver will be passed-back via the |
|
control connection and combined with netperf's result for display to |
|
the user. |
|
|
|
Netperf places no traffic on the control connection while a test is in |
|
progress. Certain TCP options, such as SO_KEEPALIVE, if set as your |
|
systems' default, may put packets out on the control connection while |
|
a test is in progress. Generally speaking this will have no effect on |
|
the results. |
|
|
|
@menu |
|
* CPU Utilization:: |
|
@end menu |
|
|
|
@node CPU Utilization, , The Design of Netperf, The Design of Netperf |
|
@section CPU Utilization |
|
@cindex CPU Utilization |
|
|
|
CPU utilization is an important, and alas all-too infrequently |
|
reported component of networking performance. Unfortunately, it can |
|
be one of the most difficult metrics to measure accurately and |
|
portably. Netperf will do its level best to report accurate |
|
CPU utilization figures, but some combinations of processor, OS and |
|
configuration may make that difficult. |
|
|
|
CPU utilization in netperf is reported as a value between 0 and 100% |
|
regardless of the number of CPUs involved. In addition to CPU |
|
utilization, netperf will report a metric called a @dfn{service |
|
demand}. The service demand is the normalization of CPU utilization |
|
and work performed. For a _STREAM test it is the microseconds of CPU |
|
time consumed to transfer on KB (K == 1024) of data. For a _RR test |
|
it is the microseconds of CPU time consumed processing a single |
|
transaction. For both CPU utilization and service demand, lower is |
|
better. |
|
|
|
Service demand can be particularly useful when trying to gauge the |
|
effect of a performance change. It is essentially a measure of |
|
efficiency, with smaller values being more efficient and thus |
|
``better.'' |
|
|
|
Netperf is coded to be able to use one of several, generally |
|
platform-specific CPU utilization measurement mechanisms. Single |
|
letter codes will be included in the CPU portion of the test banner to |
|
indicate which mechanism was used on each of the local (netperf) and |
|
remote (netserver) system. |
|
|
|
As of this writing those codes are: |
|
|
|
@table @code |
|
@item U |
|
The CPU utilization measurement mechanism was unknown to netperf or |
|
netperf/netserver was not compiled to include CPU utilization |
|
measurements. The code for the null CPU utilization mechanism can be |
|
found in @file{src/netcpu_none.c}. |
|
@item I |
|
An HP-UX-specific CPU utilization mechanism whereby the kernel |
|
incremented a per-CPU counter by one for each trip through the idle |
|
loop. This mechanism was only available on specially-compiled HP-UX |
|
kernels prior to HP-UX 10 and is mentioned here only for the sake of |
|
historical completeness and perhaps as a suggestion to those who might |
|
be altering other operating systems. While rather simple, perhaps even |
|
simplistic, this mechanism was quite robust and was not affected by |
|
the concerns of statistical methods, or methods attempting to track |
|
time in each of user, kernel, interrupt and idle modes which require |
|
quite careful accounting. It can be thought-of as the in-kernel |
|
version of the looper @code{L} mechanism without the context switch |
|
overhead. This mechanism required calibration. |
|
@item P |
|
An HP-UX-specific CPU utilization mechanism whereby the kernel |
|
keeps-track of time (in the form of CPU cycles) spent in the kernel |
|
idle loop (HP-UX 10.0 to 11.31 inclusive), or where the kernel keeps |
|
track of time spent in idle, user, kernel and interrupt processing |
|
(HP-UX 11.23 and later). The former requires calibration, the latter |
|
does not. Values in either case are retrieved via one of the pstat(2) |
|
family of calls, hence the use of the letter @code{P}. The code for |
|
these mechanisms is found in @file{src/netcpu_pstat.c} and |
|
@file{src/netcpu_pstatnew.c} respectively. |
|
@item K |
|
A Solaris-specific CPU utilization mechanism whereby the kernel keeps |
|
track of ticks (eg HZ) spent in the idle loop. This method is |
|
statistical and is known to be inaccurate when the interrupt rate is |
|
above epsilon as time spent processing interrupts is not subtracted |
|
from idle. The value is retrieved via a kstat() call - hence the use |
|
of the letter @code{K}. Since this mechanism uses units of ticks (HZ) |
|
the calibration value should invariably match HZ. (Eg 100) The code |
|
for this mechanism is implemented in @file{src/netcpu_kstat.c}. |
|
@item M |
|
A Solaris-specific mechanism available on Solaris 10 and latter which |
|
uses the new microstate accounting mechanisms. There are two, alas, |
|
overlapping, mechanisms. The first tracks nanoseconds spent in user, |
|
kernel, and idle modes. The second mechanism tracks nanoseconds spent |
|
in interrupt. Since the mechanisms overlap, netperf goes through some |
|
hand-waving to try to ``fix'' the problem. Since the accuracy of the |
|
handwaving cannot be completely determined, one must presume that |
|
while better than the @code{K} mechanism, this mechanism too is not |
|
without issues. The values are retrieved via kstat() calls, but the |
|
letter code is set to @code{M} to distinguish this mechanism from the |
|
even less accurate @code{K} mechanism. The code for this mechanism is |
|
implemented in @file{src/netcpu_kstat10.c}. |
|
@item L |
|
A mechanism based on ``looper''or ``soaker'' processes which sit in |
|
tight loops counting as fast as they possibly can. This mechanism |
|
starts a looper process for each known CPU on the system. The effect |
|
of processor hyperthreading on the mechanism is not yet known. This |
|
mechanism definitely requires calibration. The code for the |
|
``looper''mechanism can be found in @file{src/netcpu_looper.c} |
|
@item N |
|
A Microsoft Windows-specific mechanism, the code for which can be |
|
found in @file{src/netcpu_ntperf.c}. This mechanism too is based on |
|
what appears to be a form of micro-state accounting and requires no |
|
calibration. On laptops, or other systems which may dynamically alter |
|
the CPU frequency to minimize power consumption, it has been suggested |
|
that this mechanism may become slightly confused, in which case using |
|
BIOS/uEFI settings to disable the power saving would be indicated. |
|
|
|
@item S |
|
This mechanism uses @file{/proc/stat} on Linux to retrieve time |
|
(ticks) spent in idle mode. It is thought but not known to be |
|
reasonably accurate. The code for this mechanism can be found in |
|
@file{src/netcpu_procstat.c}. |
|
@item C |
|
A mechanism somewhat similar to @code{S} but using the sysctl() call |
|
on BSD-like Operating systems (*BSD and MacOS X). The code for this |
|
mechanism can be found in @file{src/netcpu_sysctl.c}. |
|
@item Others |
|
Other mechanisms included in netperf in the past have included using |
|
the times() and getrusage() calls. These calls are actually rather |
|
poorly suited to the task of measuring CPU overhead for networking as |
|
they tend to be process-specific and much network-related processing |
|
can happen outside the context of a process, in places where it is not |
|
a given it will be charged to the correct, or even a process. They |
|
are mentioned here as a warning to anyone seeing those mechanisms used |
|
in other networking benchmarks. These mechanisms are not available in |
|
netperf 2.4.0 and later. |
|
@end table |
|
|
|
For many platforms, the configure script will chose the best available |
|
CPU utilization mechanism. However, some platforms have no |
|
particularly good mechanisms. On those platforms, it is probably best |
|
to use the ``LOOPER'' mechanism which is basically some number of |
|
processes (as many as there are processors) sitting in tight little |
|
loops counting as fast as they can. The rate at which the loopers |
|
count when the system is believed to be idle is compared with the rate |
|
when the system is running netperf and the ratio is used to compute |
|
CPU utilization. |
|
|
|
In the past, netperf included some mechanisms that only reported CPU |
|
time charged to the calling process. Those mechanisms have been |
|
removed from netperf versions 2.4.0 and later because they are |
|
hopelessly inaccurate. Networking can and often results in CPU time |
|
being spent in places - such as interrupt contexts - that do not get |
|
charged to a or the correct process. |
|
|
|
In fact, time spent in the processing of interrupts is a common issue |
|
for many CPU utilization mechanisms. In particular, the ``PSTAT'' |
|
mechanism was eventually known to have problems accounting for certain |
|
interrupt time prior to HP-UX 11.11 (11iv1). HP-UX 11iv2 and later |
|
are known/presumed to be good. The ``KSTAT'' mechanism is known to |
|
have problems on all versions of Solaris up to and including Solaris |
|
10. Even the microstate accounting available via kstat in Solaris 10 |
|
has issues, though perhaps not as bad as those of prior versions. |
|
|
|
The /proc/stat mechanism under Linux is in what the author would |
|
consider an ``uncertain'' category as it appears to be statistical, |
|
which may also have issues with time spent processing interrupts. |
|
|
|
In summary, be sure to ``sanity-check'' the CPU utilization figures |
|
with other mechanisms. However, platform tools such as top, vmstat or |
|
mpstat are often based on the same mechanisms used by netperf. |
|
|
|
@menu |
|
* CPU Utilization in a Virtual Guest:: |
|
@end menu |
|
|
|
@node CPU Utilization in a Virtual Guest, , CPU Utilization, CPU Utilization |
|
@subsection CPU Utilization in a Virtual Guest |
|
|
|
The CPU utilization mechanisms used by netperf are ``inline'' in that |
|
they are run by the same netperf or netserver process as is running |
|
the test itself. This works just fine for ``bare iron'' tests but |
|
runs into a problem when using virtual machines. |
|
|
|
The relationship between virtual guest and hypervisor can be thought |
|
of as being similar to that between a process and kernel in a bare |
|
iron system. As such, (m)any CPU utilization mechanisms used in the |
|
virtual guest are similar to ``process-local'' mechanisms in a bare |
|
iron situation. However, just as with bare iron and process-local |
|
mechanisms, much networking processing happens outside the context of |
|
the virtual guest. It takes place in the hypervisor, and is not |
|
visible to mechanisms running in the guest(s). For this reason, one |
|
should not really trust CPU utilization figures reported by netperf or |
|
netserver when running in a virtual guest. |
|
|
|
If one is looking to measure the added overhead of a virtualization |
|
mechanism, rather than rely on CPU utilization, one can rely instead |
|
on netperf _RR tests - path-lengths and overheads can be a significant |
|
fraction of the latency, so increases in overhead should appear as |
|
decreases in transaction rate. Whatever you do, @b{DO NOT} rely on |
|
the throughput of a _STREAM test. Achieving link-rate can be done via |
|
a multitude of options that mask overhead rather than eliminate it. |
|
|
|
@node Global Command-line Options, Using Netperf to Measure Bulk Data Transfer, The Design of Netperf, Top |
|
@chapter Global Command-line Options |
|
|
|
This section describes each of the global command-line options |
|
available in the netperf and netserver binaries. Essentially, it is |
|
an expanded version of the usage information displayed by netperf or |
|
netserver when invoked with the @option{-h} global command-line |
|
option. |
|
|
|
@menu |
|
* Command-line Options Syntax:: |
|
* Global Options:: |
|
@end menu |
|
|
|
@node Command-line Options Syntax, Global Options, Global Command-line Options, Global Command-line Options |
|
@comment node-name, next, previous, up |
|
@section Command-line Options Syntax |
|
|
|
Revision 1.8 of netperf introduced enough new functionality to overrun |
|
the English alphabet for mnemonic command-line option names, and the |
|
author was not and is not quite ready to switch to the contemporary |
|
@option{--mumble} style of command-line options. (Call him a Luddite |
|
if you wish :). |
|
|
|
For this reason, the command-line options were split into two parts - |
|
the first are the global command-line options. They are options that |
|
affect nearly any and every test type of netperf. The second type are |
|
the test-specific command-line options. Both are entered on the same |
|
command line, but they must be separated from one another by a @code{--} |
|
for correct parsing. Global command-line options come first, followed |
|
by the @code{--} and then test-specific command-line options. If there |
|
are no test-specific options to be set, the @code{--} may be omitted. If |
|
there are no global command-line options to be set, test-specific |
|
options must still be preceded by a @code{--}. For example: |
|
@example |
|
netperf <global> -- <test-specific> |
|
@end example |
|
sets both global and test-specific options: |
|
@example |
|
netperf <global> |
|
@end example |
|
sets just global options and: |
|
@example |
|
netperf -- <test-specific> |
|
@end example |
|
sets just test-specific options. |
|
|
|
@node Global Options, , Command-line Options Syntax, Global Command-line Options |
|
@comment node-name, next, previous, up |
|
@section Global Options |
|
|
|
@table @code |
|
@vindex -a, Global |
|
@item -a <sizespec> |
|
This option allows you to alter the alignment of the buffers used in |
|
the sending and receiving calls on the local system.. Changing the |
|
alignment of the buffers can force the system to use different copy |
|
schemes, which can have a measurable effect on performance. If the |
|
page size for the system were 4096 bytes, and you want to pass |
|
page-aligned buffers beginning on page boundaries, you could use |
|
@samp{-a 4096}. By default the units are bytes, but suffix of ``G,'' |
|
``M,'' or ``K'' will specify the units to be 2^30 (GB), 2^20 (MB) or |
|
2^10 (KB) respectively. A suffix of ``g,'' ``m'' or ``k'' will specify |
|
units of 10^9, 10^6 or 10^3 bytes respectively. [Default: 8 bytes] |
|
|
|
@vindex -A, Global |
|
@item -A <sizespec> |
|
This option is identical to the @option{-a} option with the difference |
|
being it affects alignments for the remote system. |
|
|
|
@vindex -b, Global |
|
@item -b <size> |
|
This option is only present when netperf has been configure with |
|
--enable-intervals=yes prior to compilation. It sets the size of the |
|
burst of send calls in a _STREAM test. When used in conjunction with |
|
the @option{-w} option it can cause the rate at which data is sent to |
|
be ``paced.'' |
|
|
|
@vindex -B, Global |
|
@item -B <string> |
|
This option will cause @option{<string>} to be appended to the brief |
|
(see -P) output of netperf. |
|
|
|
@vindex -c, Global |
|
@item -c [rate] |
|
This option will ask that CPU utilization and service demand be |
|
calculated for the local system. For those CPU utilization mechanisms |
|
requiring calibration, the options rate parameter may be specified to |
|
preclude running another calibration step, saving 40 seconds of time. |
|
For those CPU utilization mechanisms requiring no calibration, the |
|
optional rate parameter will be utterly and completely ignored. |
|
[Default: no CPU measurements] |
|
|
|
@vindex -C, Global |
|
@item -C [rate] |
|
This option requests CPU utilization and service demand calculations |
|
for the remote system. It is otherwise identical to the @option{-c} |
|
option. |
|
|
|
@vindex -d, Global |
|
@item -d |
|
Each instance of this option will increase the quantity of debugging |
|
output displayed during a test. If the debugging output level is set |
|
high enough, it may have a measurable effect on performance. |
|
Debugging information for the local system is printed to stdout. |
|
Debugging information for the remote system is sent by default to the |
|
file @file{/tmp/netperf.debug}. [Default: no debugging output] |
|
|
|
@vindex -D, Global |
|
@item -D [interval,units] |
|
This option is only available when netperf is configured with |
|
--enable-demo=yes. When set, it will cause netperf to emit periodic |
|
reports of performance during the run. [@var{interval},@var{units}] |
|
follow the semantics of an optionspec. If specified, |
|
@var{interval} gives the minimum interval in real seconds, it does not |
|
have to be whole seconds. The @var{units} value can be used for the |
|
first guess as to how many units of work (bytes or transactions) must |
|
be done to take at least @var{interval} seconds. If omitted, |
|
@var{interval} defaults to one second and @var{units} to values |
|
specific to each test type. |
|
|
|
@vindex -f, Global |
|
@item -f G|M|K|g|m|k|x |
|
This option can be used to change the reporting units for _STREAM |
|
tests. Arguments of ``G,'' ``M,'' or ``K'' will set the units to |
|
2^30, 2^20 or 2^10 bytes/s respectively (EG power of two GB, MB or |
|
KB). Arguments of ``g,'' ``,m'' or ``k'' will set the units to 10^9, |
|
10^6 or 10^3 bits/s respectively. An argument of ``x'' requests the |
|
units be transactions per second and is only meaningful for a |
|
request-response test. [Default: ``m'' or 10^6 bits/s] |
|
|
|
@vindex -F, Global |
|
@item -F <fillfile> |
|
This option specified the file from which send which buffers will be |
|
pre-filled . While the buffers will contain data from the specified |
|
file, the file is not fully transferred to the remote system as the |
|
receiving end of the test will not write the contents of what it |
|
receives to a file. This can be used to pre-fill the send buffers |
|
with data having different compressibility and so is useful when |
|
measuring performance over mechanisms which perform compression. |
|
|
|
While previously required for a TCP_SENDFILE test, later versions of |
|
netperf removed that restriction, creating a temporary file as |
|
needed. While the author cannot recall exactly when that took place, |
|
it is known to be unnecessary in version 2.5.0 and later. |
|
|
|
@vindex -h, Global |
|
@item -h |
|
This option causes netperf to display its ``global'' usage string and |
|
exit to the exclusion of all else. |
|
|
|
@vindex -H, Global |
|
@item -H <optionspec> |
|
This option will set the name of the remote system and or the address |
|
family used for the control connection. For example: |
|
@example |
|
-H linger,4 |
|
@end example |
|
will set the name of the remote system to ``linger'' and tells netperf to |
|
use IPv4 addressing only. |
|
@example |
|
-H ,6 |
|
@end example |
|
will leave the name of the remote system at its default, and request |
|
that only IPv6 addresses be used for the control connection. |
|
@example |
|
-H lag |
|
@end example |
|
will set the name of the remote system to ``lag'' and leave the |
|
address family to AF_UNSPEC which means selection of IPv4 vs IPv6 is |
|
left to the system's address resolution. |
|
|
|
A value of ``inet'' can be used in place of ``4'' to request IPv4 only |
|
addressing. Similarly, a value of ``inet6'' can be used in place of |
|
``6'' to request IPv6 only addressing. A value of ``0'' can be used |
|
to request either IPv4 or IPv6 addressing as name resolution dictates. |
|
|
|
By default, the options set with the global @option{-H} option are |
|
inherited by the test for its data connection, unless a test-specific |
|
@option{-H} option is specified. |
|
|
|
If a @option{-H} option follows either the @option{-4} or @option{-6} |
|
options, the family setting specified with the -H option will override |
|
the @option{-4} or @option{-6} options for the remote address |
|
family. If no address family is specified, settings from a previous |
|
@option{-4} or @option{-6} option will remain. In a nutshell, the |
|
last explicit global command-line option wins. |
|
|
|
[Default: ``localhost'' for the remote name/IP address and ``0'' (eg |
|
AF_UNSPEC) for the remote address family.] |
|
|
|
@vindex -I, Global |
|
@item -I <optionspec> |
|
This option enables the calculation of confidence intervals and sets |
|
the confidence and width parameters with the first half of the |
|
optionspec being either 99 or 95 for 99% or 95% confidence |
|
respectively. The second value of the optionspec specifies the width |
|
of the desired confidence interval. For example |
|
@example |
|
-I 99,5 |
|
@end example |
|
asks netperf to be 99% confident that the measured mean values for |
|
throughput and CPU utilization are within +/- 2.5% of the ``real'' |
|
mean values. If the @option{-i} option is specified and the |
|
@option{-I} option is omitted, the confidence defaults to 99% and the |
|
width to 5% (giving +/- 2.5%) |
|
|
|
If classic netperf test calculates that the desired confidence |
|
intervals have not been met, it emits a noticeable warning that cannot |
|
be suppressed with the @option{-P} or @option{-v} options: |
|
|
|
@example |
|
netperf -H tardy.cup -i 3 -I 99,5 |
|
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to tardy.cup.hp.com (15.244.44.58) port 0 AF_INET : +/-2.5% @ 99% conf. |
|
!!! WARNING |
|
!!! Desired confidence was not achieved within the specified iterations. |
|
!!! This implies that there was variability in the test environment that |
|
!!! must be investigated before going further. |
|
!!! Confidence intervals: Throughput : 6.8% |
|
!!! Local CPU util : 0.0% |
|
!!! Remote CPU util : 0.0% |
|
|
|
Recv Send Send |
|
Socket Socket Message Elapsed |
|
Size Size Size Time Throughput |
|
bytes bytes bytes secs. 10^6bits/sec |
|
|
|
32768 16384 16384 10.01 40.23 |
|
@end example |
|
|
|
In the example above we see that netperf did not meet the desired |
|
confidence intervals. Instead of being 99% confident it was within |
|
+/- 2.5% of the real mean value of throughput it is only confident it |
|
was within +/-3.4%. In this example, increasing the @option{-i} |
|
option (described below) and/or increasing the iteration length with |
|
the @option{-l} option might resolve the situation. |
|
|
|
In an explicit ``omni'' test, failure to meet the confidence intervals |
|
will not result in netperf emitting a warning. To verify the hitting, |
|
or not, of the confidence intervals one will need to include them as |
|
part of an @ref{Omni Output Selection,output selection} in the |
|
test-specific @option{-o}, @option{-O} or @option{k} output selection |
|
options. The warning about not hitting the confidence intervals will |
|
remain in a ``migrated'' classic netperf test. |
|
|
|
@vindex -i, Global |
|
@item -i <sizespec> |
|
This option enables the calculation of confidence intervals and sets |
|
the minimum and maximum number of iterations to run in attempting to |
|
achieve the desired confidence interval. The first value sets the |
|
maximum number of iterations to run, the second, the minimum. The |
|
maximum number of iterations is silently capped at 30 and the minimum |
|
is silently floored at 3. Netperf repeats the measurement the minimum |
|
number of iterations and continues until it reaches either the |
|
desired confidence interval, or the maximum number of iterations, |
|
whichever comes first. A classic or migrated netperf test will not |
|
display the actual number of iterations run. An @ref{The Omni |
|
Tests,omni test} will emit the number of iterations run if the |
|
@code{CONFIDENCE_ITERATION} output selector is included in the |
|
@ref{Omni Output Selection,output selection}. |
|
|
|
If the @option{-I} option is specified and the @option{-i} option |
|
omitted the maximum number of iterations is set to 10 and the minimum |
|
to three. |
|
|
|
Output of a warning upon not hitting the desired confidence intervals |
|
follows the description provided for the @option{-I} option. |
|
|
|
The total test time will be somewhere between the minimum and maximum |
|
number of iterations multiplied by the test length supplied by the |
|
@option{-l} option. |
|
|
|
@vindex -j, Global |
|
@item -j |
|
This option instructs netperf to keep additional timing statistics |
|
when explicitly running an @ref{The Omni Tests,omni test}. These can |
|
be output when the test-specific @option{-o}, @option{-O} or |
|
@option{-k} @ref{Omni Output Selectors,output selectors} include one |
|
or more of: |
|
|
|
@itemize |
|
@item MIN_LATENCY |
|
@item MAX_LATENCY |
|
@item P50_LATENCY |
|
@item P90_LATENCY |
|
@item P99_LATENCY |
|
@item MEAN_LATENCY |
|
@item STDDEV_LATENCY |
|
@end itemize |
|
|
|
These statistics will be based on an expanded (100 buckets per row |
|
rather than 10) histogram of times rather than a terribly long list of |
|
individual times. As such, there will be some slight error thanks to |
|
the bucketing. However, the reduction in storage and processing |
|
overheads is well worth it. When running a request/response test, one |
|
might get some idea of the error by comparing the @ref{Omni Output |
|
Selectors,@code{MEAN_LATENCY}} calculated from the histogram with the |
|
@code{RT_LATENCY} calculated from the number of request/response |
|
transactions and the test run time. |
|
|
|
In the case of a request/response test the latencies will be |
|
transaction latencies. In the case of a receive-only test they will |
|
be time spent in the receive call. In the case of a send-only test |
|
they will be time spent in the send call. The units will be |
|
microseconds. Added in netperf 2.5.0. |
|
|
|
@vindex -l, Global |
|
@item -l testlen |
|
This option controls the length of any @b{one} iteration of the requested |
|
test. A positive value for @var{testlen} will run each iteration of |
|
the test for at least @var{testlen} seconds. A negative value for |
|
@var{testlen} will run each iteration for the absolute value of |
|
@var{testlen} transactions for a _RR test or bytes for a _STREAM test. |
|
Certain tests, notably those using UDP can only be timed, they cannot |
|
be limited by transaction or byte count. This limitation may be |
|
relaxed in an @ref{The Omni Tests,omni} test. |
|
|
|
In some situations, individual iterations of a test may run for longer |
|
for the number of seconds specified by the @option{-l} option. In |
|
particular, this may occur for those tests where the socket buffer |
|
size(s) are significantly longer than the bandwidthXdelay product of |
|
the link(s) over which the data connection passes, or those tests |
|
where there may be non-trivial numbers of retransmissions. |
|
|
|
If confidence intervals are enabled via either @option{-I} or |
|
@option{-i} the total length of the netperf test will be somewhere |
|
between the minimum and maximum iteration count multiplied by |
|
@var{testlen}. |
|
|
|
@vindex -L, Global |
|
@item -L <optionspec> |
|
This option is identical to the @option{-H} option with the difference |
|
being it sets the _local_ hostname/IP and/or address family |
|
information. This option is generally unnecessary, but can be useful |
|
when you wish to make sure that the netperf control and data |
|
connections go via different paths. It can also come-in handy if one |
|
is trying to run netperf through those evil, end-to-end breaking |
|
things known as firewalls. |
|
|
|
[Default: 0.0.0.0 (eg INADDR_ANY) for IPv4 and ::0 for IPv6 for the |
|
local name. AF_UNSPEC for the local address family.] |
|
|
|
@vindex -n, Global |
|
@item -n numcpus |
|
This option tells netperf how many CPUs it should ass-u-me are active |
|
on the system running netperf. In particular, this is used for the |
|
@ref{CPU Utilization,CPU utilization} and service demand calculations. |
|
On certain systems, netperf is able to determine the number of CPU's |
|
automagically. This option will override any number netperf might be |
|
able to determine on its own. |
|
|
|
Note that this option does _not_ set the number of CPUs on the system |
|
running netserver. When netperf/netserver cannot automagically |
|
determine the number of CPUs that can only be set for netserver via a |
|
netserver @option{-n} command-line option. |
|
|
|
As it is almost universally possible for netperf/netserver to |
|
determine the number of CPUs on the system automagically, 99 times out |
|
of 10 this option should not be necessary and may be removed in a |
|
future release of netperf. |
|
|
|
@vindex -N, Global |
|
@item -N |
|
This option tells netperf to forgo establishing a control |
|
connection. This makes it is possible to run some limited netperf |
|
tests without a corresponding netserver on the remote system. |
|
|
|
With this option set, the test to be run is to get all the addressing |
|
information it needs to establish its data connection from the command |
|
line or internal defaults. If not otherwise specified by |
|
test-specific command line options, the data connection for a |
|
``STREAM'' or ``SENDFILE'' test will be to the ``discard'' port, an |
|
``RR'' test will be to the ``echo'' port, and a ``MEARTS'' test will |
|
be to the chargen port. |
|
|
|
The response size of an ``RR'' test will be silently set to be the |
|
same as the request size. Otherwise the test would hang if the |
|
response size was larger than the request size, or would report an |
|
incorrect, inflated transaction rate if the response size was less |
|
than the request size. |
|
|
|
Since there is no control connection when this option is specified, it |
|
is not possible to set ``remote'' properties such as socket buffer |
|
size and the like via the netperf command line. Nor is it possible to |
|
retrieve such interesting remote information as CPU utilization. |
|
These items will be displayed as values which should make it |
|
immediately obvious that was the case. |
|
|
|
The only way to change remote characteristics such as socket buffer |
|
size or to obtain information such as CPU utilization is to employ |
|
platform-specific methods on the remote system. Frankly, if one has |
|
access to the remote system to employ those methods one aught to be |
|
able to run a netserver there. However, that ability may not be |
|
present in certain ``support'' situations, hence the addition of this |
|
option. |
|
|
|
Added in netperf 2.4.3. |
|
|
|
@vindex -o, Global |
|
@item -o <sizespec> |
|
The value(s) passed-in with this option will be used as an offset |
|
added to the alignment specified with the @option{-a} option. For |
|
example: |
|
@example |
|
-o 3 -a 4096 |
|
@end example |
|
will cause the buffers passed to the local (netperf) send and receive |
|
calls to begin three bytes past an address aligned to 4096 |
|
bytes. [Default: 0 bytes] |
|
|
|
@vindex -O, Global |
|
@item -O <sizespec> |
|
This option behaves just as the @option{-o} option but on the remote |
|
(netserver) system and in conjunction with the @option{-A} |
|
option. [Default: 0 bytes] |
|
|
|
@vindex -p, Global |
|
@item -p <optionspec> |
|
The first value of the optionspec passed-in with this option tells |
|
netperf the port number at which it should expect the remote netserver |
|
to be listening for control connections. The second value of the |
|
optionspec will request netperf to bind to that local port number |
|
before establishing the control connection. For example |
|
@example |
|
-p 12345 |
|
@end example |
|
tells netperf that the remote netserver is listening on port 12345 and |
|
leaves selection of the local port number for the control connection |
|
up to the local TCP/IP stack whereas |
|
@example |
|
-p ,32109 |
|
@end example |
|
leaves the remote netserver port at the default value of 12865 and |
|
causes netperf to bind to the local port number 32109 before |
|
connecting to the remote netserver. |
|
|
|
In general, setting the local port number is only necessary when one |
|
is looking to run netperf through those evil, end-to-end breaking |
|
things known as firewalls. |
|
|
|
@vindex -P, Global |
|
@item -P 0|1 |
|
A value of ``1'' for the @option{-P} option will enable display of |
|
the test banner. A value of ``0'' will disable display of the test |
|
banner. One might want to disable display of the test banner when |
|
running the same basic test type (eg TCP_STREAM) multiple times in |
|
succession where the test banners would then simply be redundant and |
|
unnecessarily clutter the output. [Default: 1 - display test banners] |
|
|
|
@vindex -s, Global |
|
@item -s <seconds> |
|
This option will cause netperf to sleep @samp{<seconds>} before |
|
actually transferring data over the data connection. This may be |
|
useful in situations where one wishes to start a great many netperf |
|
instances and do not want the earlier ones affecting the ability of |
|
the later ones to get established. |
|
|
|
Added somewhere between versions 2.4.3 and 2.5.0. |
|
|
|
@vindex -S, Global |
|
@item -S |
|
This option will cause an attempt to be made to set SO_KEEPALIVE on |
|
the data socket of a test using the BSD sockets interface. The |
|
attempt will be made on the netperf side of all tests, and will be |
|
made on the netserver side of an @ref{The Omni Tests,omni} or |
|
@ref{Migrated Tests,migrated} test. No indication of failure is given |
|
unless debug output is enabled with the global @option{-d} option. |
|
|
|
Added in version 2.5.0. |
|
|
|
@vindex -t, Global |
|
@item -t testname |
|
This option is used to tell netperf which test you wish to run. As of |
|
this writing, valid values for @var{testname} include: |
|
@itemize |
|
@item |
|
@ref{TCP_STREAM}, @ref{TCP_MAERTS}, @ref{TCP_SENDFILE}, @ref{TCP_RR}, @ref{TCP_CRR}, @ref{TCP_CC} |
|
@item |
|
@ref{UDP_STREAM}, @ref{UDP_RR} |
|
@item |
|
@ref{XTI_TCP_STREAM}, @ref{XTI_TCP_RR}, @ref{XTI_TCP_CRR}, @ref{XTI_TCP_CC} |
|
@item |
|
@ref{XTI_UDP_STREAM}, @ref{XTI_UDP_RR} |
|
@item |
|
@ref{SCTP_STREAM}, @ref{SCTP_RR} |
|
@item |
|
@ref{DLCO_STREAM}, @ref{DLCO_RR}, @ref{DLCL_STREAM}, @ref{DLCL_RR} |
|
@item |
|
@ref{Other Netperf Tests,LOC_CPU}, @ref{Other Netperf Tests,REM_CPU} |
|
@item |
|
@ref{The Omni Tests,OMNI} |
|
@end itemize |
|
Not all tests are always compiled into netperf. In particular, the |
|
``XTI,'' ``SCTP,'' ``UNIXDOMAIN,'' and ``DL*'' tests are only included in |
|
netperf when configured with |
|
@option{--enable-[xti|sctp|unixdomain|dlpi]=yes}. |
|
|
|
Netperf only runs one type of test no matter how many @option{-t} |
|
options may be present on the command-line. The last @option{-t} |
|
global command-line option will determine the test to be |
|
run. [Default: TCP_STREAM] |
|
|
|
@vindex -T, Global |
|
@item -T <optionspec> |
|
This option controls the CPU, and probably by extension memory, |
|
affinity of netperf and/or netserver. |
|
@example |
|
netperf -T 1 |
|
@end example |
|
will bind both netperf and netserver to ``CPU 1'' on their respective |
|
systems. |
|
@example |
|
netperf -T 1, |
|
@end example |
|
will bind just netperf to ``CPU 1'' and will leave netserver unbound. |
|
@example |
|
netperf -T ,2 |
|
@end example |
|
will leave netperf unbound and will bind netserver to ``CPU 2.'' |
|
@example |
|
netperf -T 1,2 |
|
@end example |
|
will bind netperf to ``CPU 1'' and netserver to ``CPU 2.'' |
|
|
|
This can be particularly useful when investigating performance issues |
|
involving where processes run relative to where NIC interrupts are |
|
processed or where NICs allocate their DMA buffers. |
|
|
|
@vindex -v, Global |
|
@item -v verbosity |
|
This option controls how verbose netperf will be in its output, and is |
|
often used in conjunction with the @option{-P} option. If the |
|
verbosity is set to a value of ``0'' then only the test's SFM (Single |
|
Figure of Merit) is displayed. If local @ref{CPU Utilization,CPU |
|
utilization} is requested via the @option{-c} option then the SFM is |
|
the local service demand. Othersise, if remote CPU utilization is |
|
requested via the @option{-C} option then the SFM is the remote |
|
service demand. If neither local nor remote CPU utilization are |
|
requested the SFM will be the measured throughput or transaction rate |
|
as implied by the test specified with the @option{-t} option. |
|
|
|
If the verbosity level is set to ``1'' then the ``normal'' netperf |
|
result output for each test is displayed. |
|
|
|
If the verbosity level is set to ``2'' then ``extra'' information will |
|
be displayed. This may include, but is not limited to the number of |
|
send or recv calls made and the average number of bytes per send or |
|
recv call, or a histogram of the time spent in each send() call or for |
|
each transaction if netperf was configured with |
|
@option{--enable-histogram=yes}. [Default: 1 - normal verbosity] |
|
|
|
In an @ref{The Omni Tests,omni} test the verbosity setting is largely |
|
ignored, save for when asking for the time histogram to be displayed. |
|
In version 2.5.0 and later there is no @ref{Omni Output Selectors,output |
|
selector} for the histogram and so it remains displayed only when the |
|
verbosity level is set to 2. |
|
|
|
@vindex -V, Global |
|
@item -V |
|
This option displays the netperf version and then exits. |
|
|
|
Added in netperf 2.4.4. |
|
|
|
@vindex -w, Global |
|
@item -w time |
|
If netperf was configured with @option{--enable-intervals=yes} then |
|
this value will set the inter-burst time to time milliseconds, and the |
|
@option{-b} option will set the number of sends per burst. The actual |
|
inter-burst time may vary depending on the system's timer resolution. |
|
|
|
@vindex -W, Global |
|
@item -W <sizespec> |
|
This option controls the number of buffers in the send (first or only |
|
value) and or receive (second or only value) buffer rings. Unlike |
|
some benchmarks, netperf does not continuously send or receive from a |
|
single buffer. Instead it rotates through a ring of |
|
buffers. [Default: One more than the size of the send or receive |
|
socket buffer sizes (@option{-s} and/or @option{-S} options) divided |
|
by the send @option{-m} or receive @option{-M} buffer size |
|
respectively] |
|
|
|
@vindex -4, Global |
|
@item -4 |
|
Specifying this option will set both the local and remote address |
|
families to AF_INET - that is use only IPv4 addresses on the control |
|
connection. This can be overridden by a subsequent @option{-6}, |
|
@option{-H} or @option{-L} option. Basically, the last option |
|
explicitly specifying an address family wins. Unless overridden by a |
|
test-specific option, this will be inherited for the data connection |
|
as well. |
|
|
|
@vindex -6, Global |
|
@item -6 |
|
Specifying this option will set both local and and remote address |
|
families to AF_INET6 - that is use only IPv6 addresses on the control |
|
connection. This can be overridden by a subsequent @option{-4}, |
|
@option{-H} or @option{-L} option. Basically, the last address family |
|
explicitly specified wins. Unless overridden by a test-specific |
|
option, this will be inherited for the data connection as well. |
|
|
|
@end table |
|
|
|
|
|
@node Using Netperf to Measure Bulk Data Transfer, Using Netperf to Measure Request/Response , Global Command-line Options, Top |
|
@chapter Using Netperf to Measure Bulk Data Transfer |
|
|
|
The most commonly measured aspect of networked system performance is |
|
that of bulk or unidirectional transfer performance. Everyone wants |
|
to know how many bits or bytes per second they can push across the |
|
network. The classic netperf convention for a bulk data transfer test |
|
name is to tack a ``_STREAM'' suffix to a test name. |
|
|
|
@menu |
|
* Issues in Bulk Transfer:: |
|
* Options common to TCP UDP and SCTP tests:: |
|
@end menu |
|
|
|
@node Issues in Bulk Transfer, Options common to TCP UDP and SCTP tests, Using Netperf to Measure Bulk Data Transfer, Using Netperf to Measure Bulk Data Transfer |
|
@comment node-name, next, previous, up |
|
@section Issues in Bulk Transfer |
|
|
|
There are any number of things which can affect the performance of a |
|
bulk transfer test. |
|
|
|
Certainly, absent compression, bulk-transfer tests can be limited by |
|
the speed of the slowest link in the path from the source to the |
|
destination. If testing over a gigabit link, you will not see more |
|
than a gigabit :) Such situations can be described as being |
|
@dfn{network-limited} or @dfn{NIC-limited}. |
|
|
|
CPU utilization can also affect the results of a bulk-transfer test. |
|
If the networking stack requires a certain number of instructions or |
|
CPU cycles per KB of data transferred, and the CPU is limited in the |
|
number of instructions or cycles it can provide, then the transfer can |
|
be described as being @dfn{CPU-bound}. |
|
|
|
A bulk-transfer test can be CPU bound even when netperf reports less |
|
than 100% CPU utilization. This can happen on an MP system where one |
|
or more of the CPUs saturate at 100% but other CPU's remain idle. |
|
Typically, a single flow of data, such as that from a single instance |
|
of a netperf _STREAM test cannot make use of much more than the power |
|
of one CPU. Exceptions to this generally occur when netperf and/or |
|
netserver run on CPU(s) other than the CPU(s) taking interrupts from |
|
the NIC(s). In that case, one might see as much as two CPUs' worth of |
|
processing being used to service the flow of data. |
|
|
|
Distance and the speed-of-light can affect performance for a |
|
bulk-transfer; often this can be mitigated by using larger windows. |
|
One common limit to the performance of a transport using window-based |
|
flow-control is: |
|
@example |
|
Throughput <= WindowSize/RoundTripTime |
|
@end example |
|
As the sender can only have a window's-worth of data outstanding on |
|
the network at any one time, and the soonest the sender can receive a |
|
window update from the receiver is one RoundTripTime (RTT). TCP and |
|
SCTP are examples of such protocols. |
|
|
|
Packet losses and their effects can be particularly bad for |
|
performance. This is especially true if the packet losses result in |
|
retransmission timeouts for the protocol(s) involved. By the time a |
|
retransmission timeout has happened, the flow or connection has sat |
|
idle for a considerable length of time. |
|
|
|
On many platforms, some variant on the @command{netstat} command can |
|
be used to retrieve statistics about packet loss and |
|
retransmission. For example: |
|
@example |
|
netstat -p tcp |
|
@end example |
|
will retrieve TCP statistics on the HP-UX Operating System. On other |
|
platforms, it may not be possible to retrieve statistics for a |
|
specific protocol and something like: |
|
@example |
|
netstat -s |
|
@end example |
|
would be used instead. |
|
|
|
Many times, such network statistics are keep since the time the stack |
|
started, and we are only really interested in statistics from when |
|
netperf was running. In such situations something along the lines of: |
|
@example |
|
netstat -p tcp > before |
|
netperf -t TCP_mumble... |
|
netstat -p tcp > after |
|
@end example |
|
is indicated. The |
|
@uref{ftp://ftp.cup.hp.com/dist/networking/tools/,beforeafter} utility |
|
can be used to subtract the statistics in @file{before} from the |
|
statistics in @file{after}: |
|
@example |
|
beforeafter before after > delta |
|
@end example |
|
and then one can look at the statistics in @file{delta}. Beforeafter |
|
is distributed in source form so one can compile it on the platform(s) |
|
of interest. |
|
|
|
If running a version 2.5.0 or later ``omni'' test under Linux one can |
|
include either or both of: |
|
@itemize |
|
@item LOCAL_TRANSPORT_RETRANS |
|
@item REMOTE_TRANSPORT_RETRANS |
|
@end itemize |
|
|
|
in the values provided via a test-specific @option{-o}, @option{-O}, |
|
or @option{-k} output selction option and netperf will report the |
|
retransmissions experienced on the data connection, as reported via a |
|
@code{getsockopt(TCP_INFO)} call. If confidence intervals have been |
|
requested via the global @option{-I} or @option{-i} options, the |
|
reported value(s) will be for the last iteration. If the test is over |
|
a protocol other than TCP, or on a platform other than Linux, the |
|
results are undefined. |
|
|
|
While it was written with HP-UX's netstat in mind, the |
|
@uref{ftp://ftp.cup.hp.com/dist/networking/briefs/annotated_netstat.txt,annotated |
|
netstat} writeup may be helpful with other platforms as well. |
|
|
|
@node Options common to TCP UDP and SCTP tests, , Issues in Bulk Transfer, Using Netperf to Measure Bulk Data Transfer |
|
@comment node-name, next, previous, up |
|
@section Options common to TCP UDP and SCTP tests |
|
|
|
Many ``test-specific'' options are actually common across the |
|
different tests. For those tests involving TCP, UDP and SCTP, whether |
|
using the BSD Sockets or the XTI interface those common options |
|
include: |
|
|
|
@table @code |
|
@vindex -h, Test-specific |
|
@item -h |
|
Display the test-suite-specific usage string and exit. For a TCP_ or |
|
UDP_ test this will be the usage string from the source file |
|
nettest_bsd.c. For an XTI_ test, this will be the usage string from |
|
the source file nettest_xti.c. For an SCTP test, this will be the |
|
usage string from the source file nettest_sctp.c. |
|
|
|
@item -H <optionspec> |
|
Normally, the remote hostname|IP and address family information is |
|
inherited from the settings for the control connection (eg global |
|
command-line @option{-H}, @option{-4} and/or @option{-6} options). |
|
The test-specific @option{-H} will override those settings for the |
|
data (aka test) connection only. Settings for the control connection |
|
are left unchanged. |
|
|
|
@vindex -L, Test-specific |
|
@item -L <optionspec> |
|
The test-specific @option{-L} option is identical to the test-specific |
|
@option{-H} option except it affects the local hostname|IP and address |
|
family information. As with its global command-line counterpart, this |
|
is generally only useful when measuring though those evil, end-to-end |
|
breaking things called firewalls. |
|
|
|
@vindex -m, Test-specific |
|
@item -m bytes |
|
Set the size of the buffer passed-in to the ``send'' calls of a |
|
_STREAM test. Note that this may have only an indirect effect on the |
|
size of the packets sent over the network, and certain Layer 4 |
|
protocols do _not_ preserve or enforce message boundaries, so setting |
|
@option{-m} for the send size does not necessarily mean the receiver |
|
will receive that many bytes at any one time. By default the units are |
|
bytes, but suffix of ``G,'' ``M,'' or ``K'' will specify the units to |
|
be 2^30 (GB), 2^20 (MB) or 2^10 (KB) respectively. A suffix of ``g,'' |
|
``m'' or ``k'' will specify units of 10^9, 10^6 or 10^3 bytes |
|
respectively. For example: |
|
@example |
|
@code{-m 32K} |
|
@end example |
|
will set the size to 32KB or 32768 bytes. [Default: the local send |
|
socket buffer size for the connection - either the system's default or |
|
the value set via the @option{-s} option.] |
|
|
|
@vindex -M, Test-specific |
|
@item -M bytes |
|
Set the size of the buffer passed-in to the ``recv'' calls of a |
|
_STREAM test. This will be an upper bound on the number of bytes |
|
received per receive call. By default the units are bytes, but suffix |
|
of ``G,'' ``M,'' or ``K'' will specify the units to be 2^30 (GB), 2^20 |
|
(MB) or 2^10 (KB) respectively. A suffix of ``g,'' ``m'' or ``k'' |
|
will specify units of 10^9, 10^6 or 10^3 bytes respectively. For |
|
example: |
|
@example |
|
@code{-M 32K} |
|
@end example |
|
will set the size to 32KB or 32768 bytes. [Default: the remote receive |
|
socket buffer size for the data connection - either the system's |
|
default or the value set via the @option{-S} option.] |
|
|
|
@vindex -P, Test-specific |
|
@item -P <optionspec> |
|
Set the local and/or remote port numbers for the data connection. |
|
|
|
@vindex -s, Test-specific |
|
@item -s <sizespec> |
|
This option sets the local (netperf) send and receive socket buffer |
|
sizes for the data connection to the value(s) specified. Often, this |
|
will affect the advertised and/or effective TCP or other window, but |
|
on some platforms it may not. By default the units are bytes, but |
|
suffix of ``G,'' ``M,'' or ``K'' will specify the units to be 2^30 |
|
(GB), 2^20 (MB) or 2^10 (KB) respectively. A suffix of ``g,'' ``m'' |
|
or ``k'' will specify units of 10^9, 10^6 or 10^3 bytes |
|
respectively. For example: |
|
@example |
|
@code{-s 128K} |
|
@end example |
|
Will request the local send and receive socket buffer sizes to be |
|
128KB or 131072 bytes. |
|
|
|
While the historic expectation is that setting the socket buffer size |
|
has a direct effect on say the TCP window, today that may not hold |
|
true for all stacks. Further, while the historic expectation is that |
|
the value specified in a @code{setsockopt()} call will be the value returned |
|
via a @code{getsockopt()} call, at least one stack is known to deliberately |
|
ignore history. When running under Windows a value of 0 may be used |
|
which will be an indication to the stack the user wants to enable a |
|
form of copy avoidance. [Default: -1 - use the system's default socket |
|
buffer sizes] |
|
|
|
@vindex -S Test-specific |
|
@item -S <sizespec> |
|
This option sets the remote (netserver) send and/or receive socket |
|
buffer sizes for the data connection to the value(s) specified. |
|
Often, this will affect the advertised and/or effective TCP or other |
|
window, but on some platforms it may not. By default the units are |
|
bytes, but suffix of ``G,'' ``M,'' or ``K'' will specify the units to |
|
be 2^30 (GB), 2^20 (MB) or 2^10 (KB) respectively. A suffix of ``g,'' |
|
``m'' or ``k'' will specify units of 10^9, 10^6 or 10^3 bytes |
|
respectively. For example: |
|
@example |
|
@code{-S 128K} |
|
@end example |
|
Will request the remote send and receive socket buffer sizes to be |
|
128KB or 131072 bytes. |
|
|
|
While the historic expectation is that setting the socket buffer size |
|
has a direct effect on say the TCP window, today that may not hold |
|
true for all stacks. Further, while the historic expectation is that |
|
the value specified in a @code{setsockopt()} call will be the value returned |
|
via a @code{getsockopt()} call, at least one stack is known to deliberately |
|
ignore history. When running under Windows a value of 0 may be used |
|
which will be an indication to the stack the user wants to enable a |
|
form of copy avoidance. [Default: -1 - use the system's default socket |
|
buffer sizes] |
|
|
|
@vindex -4, Test-specific |
|
@item -4 |
|
Set the local and remote address family for the data connection to |
|
AF_INET - ie use IPv4 addressing only. Just as with their global |
|
command-line counterparts the last of the @option{-4}, @option{-6}, |
|
@option{-H} or @option{-L} option wins for their respective address |
|
families. |
|
|
|
@vindex -6, Test-specific |
|
@item -6 |
|
This option is identical to its @option{-4} cousin, but requests IPv6 |
|
addresses for the local and remote ends of the data connection. |
|
|
|
@end table |
|
|
|
|
|
@menu |
|
* TCP_STREAM:: |
|
* TCP_MAERTS:: |
|
* TCP_SENDFILE:: |
|
* UDP_STREAM:: |
|
* XTI_TCP_STREAM:: |
|
* XTI_UDP_STREAM:: |
|
* SCTP_STREAM:: |
|
* DLCO_STREAM:: |
|
* DLCL_STREAM:: |
|
* STREAM_STREAM:: |
|
* DG_STREAM:: |
|
@end menu |
|
|
|
@node TCP_STREAM, TCP_MAERTS, Options common to TCP UDP and SCTP tests, Options common to TCP UDP and SCTP tests |
|
@subsection TCP_STREAM |
|
|
|
The TCP_STREAM test is the default test in netperf. It is quite |
|
simple, transferring some quantity of data from the system running |
|
netperf to the system running netserver. While time spent |
|
establishing the connection is not included in the throughput |
|
calculation, time spent flushing the last of the data to the remote at |
|
the end of the test is. This is how netperf knows that all the data |
|
it sent was received by the remote. In addition to the @ref{Options |
|
common to TCP UDP and SCTP tests,options common to STREAM tests}, the |
|
following test-specific options can be included to possibly alter the |
|
behavior of the test: |
|
|
|
@table @code |
|
@item -C |
|
This option will set TCP_CORK mode on the data connection on those |
|
systems where TCP_CORK is defined (typically Linux). A full |
|
description of TCP_CORK is beyond the scope of this manual, but in a |
|
nutshell it forces sub-MSS sends to be buffered so every segment sent |
|
is Maximum Segment Size (MSS) unless the application performs an |
|
explicit flush operation or the connection is closed. At present |
|
netperf does not perform any explicit flush operations. Setting |
|
TCP_CORK may improve the bitrate of tests where the ``send size'' |
|
(@option{-m} option) is smaller than the MSS. It should also improve |
|
(make smaller) the service demand. |
|
|
|
The Linux tcp(7) manpage states that TCP_CORK cannot be used in |
|
conjunction with TCP_NODELAY (set via the @option{-d} option), however |
|
netperf does not validate command-line options to enforce that. |
|
|
|
@item -D |
|
This option will set TCP_NODELAY on the data connection on those |
|
systems where TCP_NODELAY is defined. This disables something known |
|
as the Nagle Algorithm, which is intended to make the segments TCP |
|
sends as large as reasonably possible. Setting TCP_NODELAY for a |
|
TCP_STREAM test should either have no effect when the send size |
|
(@option{-m} option) is larger than the MSS or will decrease reported |
|
bitrate and increase service demand when the send size is smaller than |
|
the MSS. This stems from TCP_NODELAY causing each sub-MSS send to be |
|
its own TCP segment rather than being aggregated with other small |
|
sends. This means more trips up and down the protocol stack per KB of |
|
data transferred, which means greater CPU utilization. |
|
|
|
If setting TCP_NODELAY with @option{-D} affects throughput and/or |
|
service demand for tests where the send size (@option{-m}) is larger |
|
than the MSS it suggests the TCP/IP stack's implementation of the |
|
Nagle Algorithm _may_ be broken, perhaps interpreting the Nagle |
|
Algorithm on a segment by segment basis rather than the proper user |
|
send by user send basis. However, a better test of this can be |
|
achieved with the @ref{TCP_RR} test. |
|
|
|
@end table |
|
|
|
Here is an example of a basic TCP_STREAM test, in this case from a |
|
Debian Linux (2.6 kernel) system to an HP-UX 11iv2 (HP-UX 11.23) |
|
system: |
|
|
|
@example |
|
$ netperf -H lag |
|
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to lag.hpl.hp.com (15.4.89.214) port 0 AF_INET |
|
Recv Send Send |
|
Socket Socket Message Elapsed |
|
Size Size Size Time Throughput |
|
bytes bytes bytes secs. 10^6bits/sec |
|
|
|
32768 16384 16384 10.00 80.42 |
|
@end example |
|
|
|
We see that the default receive socket buffer size for the receiver |
|
(lag - HP-UX 11.23) is 32768 bytes, and the default socket send buffer |
|
size for the sender (Debian 2.6 kernel) is 16384 bytes, however Linux |
|
does ``auto tuning'' of socket buffer and TCP window sizes, which |
|
means the send socket buffer size may be different at the end of the |
|
test than it was at the beginning. This is addressed in the @ref{The |
|
Omni Tests,omni tests} added in version 2.5.0 and @ref{Omni Output |
|
Selection,output selection}. Throughput is expressed as 10^6 (aka |
|
Mega) bits per second, and the test ran for 10 seconds. IPv4 |
|
addresses (AF_INET) were used. |
|
|
|
@node TCP_MAERTS, TCP_SENDFILE, TCP_STREAM, Options common to TCP UDP and SCTP tests |
|
@comment node-name, next, previous, up |
|
@subsection TCP_MAERTS |
|
|
|
A TCP_MAERTS (MAERTS is STREAM backwards) test is ``just like'' a |
|
@ref{TCP_STREAM} test except the data flows from the netserver to the |
|
netperf. The global command-line @option{-F} option is ignored for |
|
this test type. The test-specific command-line @option{-C} option is |
|
ignored for this test type. |
|
|
|
Here is an example of a TCP_MAERTS test between the same two systems |
|
as in the example for the @ref{TCP_STREAM} test. This time we request |
|
larger socket buffers with @option{-s} and @option{-S} options: |
|
|
|
@example |
|
$ netperf -H lag -t TCP_MAERTS -- -s 128K -S 128K |
|
TCP MAERTS TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to lag.hpl.hp.com (15.4.89.214) port 0 AF_INET |
|
Recv Send Send |
|
Socket Socket Message Elapsed |
|
Size Size Size Time Throughput |
|
bytes bytes bytes secs. 10^6bits/sec |
|
|
|
221184 131072 131072 10.03 81.14 |
|
@end example |
|
|
|
Where we see that Linux, unlike HP-UX, may not return the same value |
|
in a @code{getsockopt()} as was requested in the prior @code{setsockopt()}. |
|
|
|
This test is included more for benchmarking convenience than anything |
|
else. |
|
|
|
@node TCP_SENDFILE, UDP_STREAM, TCP_MAERTS, Options common to TCP UDP and SCTP tests |
|
@comment node-name, next, previous, up |
|
@subsection TCP_SENDFILE |
|
|
|
The TCP_SENDFILE test is ``just like'' a @ref{TCP_STREAM} test except |
|
netperf the platform's @code{sendfile()} call instead of calling |
|
@code{send()}. Often this results in a @dfn{zero-copy} operation |
|
where data is sent directly from the filesystem buffer cache. This |
|
_should_ result in lower CPU utilization and possibly higher |
|
throughput. If it does not, then you may want to contact your |
|
vendor(s) because they have a problem on their hands. |
|
|
|
Zero-copy mechanisms may also alter the characteristics (size and |
|
number of buffers per) of packets passed to the NIC. In many stacks, |
|
when a copy is performed, the stack can ``reserve'' space at the |
|
beginning of the destination buffer for things like TCP, IP and Link |
|
headers. This then has the packet contained in a single buffer which |
|
can be easier to DMA to the NIC. When no copy is performed, there is |
|
no opportunity to reserve space for headers and so a packet will be |
|
contained in two or more buffers. |
|
|
|
As of some time before version 2.5.0, the @ref{Global Options,global |
|
@option{-F} option} is no longer required for this test. If it is not |
|
specified, netperf will create a temporary file, which it will delete |
|
at the end of the test. If the @option{-F} option is specified it |
|
must reference a file of at least the size of the send ring |
|
(@xref{Global Options,the global @option{-W} option}.) multiplied by |
|
the send size (@xref{Options common to TCP UDP and SCTP tests,the |
|
test-specific @option{-m} option}.). All other TCP-specific options |
|
remain available and optional. |
|
|
|
In this first example: |
|
@example |
|
$ netperf -H lag -F ../src/netperf -t TCP_SENDFILE -- -s 128K -S 128K |
|
TCP SENDFILE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to lag.hpl.hp.com (15.4.89.214) port 0 AF_INET |
|
alloc_sendfile_buf_ring: specified file too small. |
|
file must be larger than send_width * send_size |
|
@end example |
|
|
|
we see what happens when the file is too small. Here: |
|
|
|
@example |
|
$ netperf -H lag -F /boot/vmlinuz-2.6.8-1-686 -t TCP_SENDFILE -- -s 128K -S 128K |
|
TCP SENDFILE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to lag.hpl.hp.com (15.4.89.214) port 0 AF_INET |
|
Recv Send Send |
|
Socket Socket Message Elapsed |
|
Size Size Size Time Throughput |
|
bytes bytes bytes secs. 10^6bits/sec |
|
|
|
131072 221184 221184 10.02 81.83 |
|
@end example |
|
|
|
we resolve that issue by selecting a larger file. |
|
|
|
|
|
@node UDP_STREAM, XTI_TCP_STREAM, TCP_SENDFILE, Options common to TCP UDP and SCTP tests |
|
@subsection UDP_STREAM |
|
|
|
A UDP_STREAM test is similar to a @ref{TCP_STREAM} test except UDP is |
|
used as the transport rather than TCP. |
|
|
|
@cindex Limiting Bandwidth |
|
A UDP_STREAM test has no end-to-end flow control - UDP provides none |
|
and neither does netperf. However, if you wish, you can configure |
|
netperf with @code{--enable-intervals=yes} to enable the global |
|
command-line @option{-b} and @option{-w} options to pace bursts of |
|
traffic onto the network. |
|
|
|
This has a number of implications. |
|
|
|
The biggest of these implications is the data which is sent might not |
|
be received by the remote. For this reason, the output of a |
|
UDP_STREAM test shows both the sending and receiving throughput. On |
|
some platforms, it may be possible for the sending throughput to be |
|
reported as a value greater than the maximum rate of the link. This |
|
is common when the CPU(s) are faster than the network and there is no |
|
@dfn{intra-stack} flow-control. |
|
|
|
Here is an example of a UDP_STREAM test between two systems connected |
|
by a 10 Gigabit Ethernet link: |
|
@example |
|
$ netperf -t UDP_STREAM -H 192.168.2.125 -- -m 32768 |
|
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.2.125 (192.168.2.125) port 0 AF_INET |
|
Socket Message Elapsed Messages |
|
Size Size Time Okay Errors Throughput |
|
bytes bytes secs # # 10^6bits/sec |
|
|
|
124928 32768 10.00 105672 0 2770.20 |
|
135168 10.00 104844 2748.50 |
|
|
|
@end example |
|
|
|
The first line of numbers are statistics from the sending (netperf) |
|
side. The second line of numbers are from the receiving (netserver) |
|
side. In this case, 105672 - 104844 or 828 messages did not make it |
|
all the way to the remote netserver process. |
|
|
|
If the value of the @option{-m} option is larger than the local send |
|
socket buffer size (@option{-s} option) netperf will likely abort with |
|
an error message about how the send call failed: |
|
|
|
@example |
|
netperf -t UDP_STREAM -H 192.168.2.125 |
|
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.2.125 (192.168.2.125) port 0 AF_INET |
|
udp_send: data send error: Message too long |
|
@end example |
|
|
|
If the value of the @option{-m} option is larger than the remote |
|
socket receive buffer, the reported receive throughput will likely be |
|
zero as the remote UDP will discard the messages as being too large to |
|
fit into the socket buffer. |
|
|
|
@example |
|
$ netperf -t UDP_STREAM -H 192.168.2.125 -- -m 65000 -S 32768 |
|
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.2.125 (192.168.2.125) port 0 AF_INET |
|
Socket Message Elapsed Messages |
|
Size Size Time Okay Errors Throughput |
|
bytes bytes secs # # 10^6bits/sec |
|
|
|
124928 65000 10.00 53595 0 2786.99 |
|
65536 10.00 0 0.00 |
|
@end example |
|
|
|
The example above was between a pair of systems running a ``Linux'' |
|
kernel. Notice that the remote Linux system returned a value larger |
|
than that passed-in to the @option{-S} option. In fact, this value |
|
was larger than the message size set with the @option{-m} option. |
|
That the remote socket buffer size is reported as 65536 bytes would |
|
suggest to any sane person that a message of 65000 bytes would fit, |
|
but the socket isn't _really_ 65536 bytes, even though Linux is |
|
telling us so. Go figure. |
|
|
|
@node XTI_TCP_STREAM, XTI_UDP_STREAM, UDP_STREAM, Options common to TCP UDP and SCTP tests |
|
@subsection XTI_TCP_STREAM |
|
|
|
An XTI_TCP_STREAM test is simply a @ref{TCP_STREAM} test using the XTI |
|
rather than BSD Sockets interface. The test-specific @option{-X |
|
<devspec>} option can be used to specify the name of the local and/or |
|
remote XTI device files, which is required by the @code{t_open()} call |
|
made by netperf XTI tests. |
|
|
|
The XTI_TCP_STREAM test is only present if netperf was configured with |
|
@code{--enable-xti=yes}. The remote netserver must have also been |
|
configured with @code{--enable-xti=yes}. |
|
|
|
@node XTI_UDP_STREAM, SCTP_STREAM, XTI_TCP_STREAM, Options common to TCP UDP and SCTP tests |
|
@subsection XTI_UDP_STREAM |
|
|
|
An XTI_UDP_STREAM test is simply a @ref{UDP_STREAM} test using the XTI |
|
rather than BSD Sockets Interface. The test-specific @option{-X |
|
<devspec>} option can be used to specify the name of the local and/or |
|
remote XTI device files, which is required by the @code{t_open()} call |
|
made by netperf XTI tests. |
|
|
|
The XTI_UDP_STREAM test is only present if netperf was configured with |
|
@code{--enable-xti=yes}. The remote netserver must have also been |
|
configured with @code{--enable-xti=yes}. |
|
|
|
@node SCTP_STREAM, DLCO_STREAM, XTI_UDP_STREAM, Options common to TCP UDP and SCTP tests |
|
@subsection SCTP_STREAM |
|
|
|
An SCTP_STREAM test is essentially a @ref{TCP_STREAM} test using the SCTP |
|
rather than TCP. The @option{-D} option will set SCTP_NODELAY, which |
|
is much like the TCP_NODELAY option for TCP. The @option{-C} option |
|
is not applicable to an SCTP test as there is no corresponding |
|
SCTP_CORK option. The author is still figuring-out what the |
|
test-specific @option{-N} option does :) |
|
|
|
The SCTP_STREAM test is only present if netperf was configured with |
|
@code{--enable-sctp=yes}. The remote netserver must have also been |
|
configured with @code{--enable-sctp=yes}. |
|
|
|
@node DLCO_STREAM, DLCL_STREAM, SCTP_STREAM, Options common to TCP UDP and SCTP tests |
|
@subsection DLCO_STREAM |
|
|
|
A DLPI Connection Oriented Stream (DLCO_STREAM) test is very similar |
|
in concept to a @ref{TCP_STREAM} test. Both use reliable, |
|
connection-oriented protocols. The DLPI test differs from the TCP |
|
test in that its protocol operates only at the link-level and does not |
|
include TCP-style segmentation and reassembly. This last difference |
|
means that the value passed-in with the @option{-m} option must be |
|
less than the interface MTU. Otherwise, the @option{-m} and |
|
@option{-M} options are just like their TCP/UDP/SCTP counterparts. |
|
|
|
Other DLPI-specific options include: |
|
|
|
@table @code |
|
@item -D <devspec> |
|
This option is used to provide the fully-qualified names for the local |
|
and/or remote DLPI device files. The syntax is otherwise identical to |
|
that of a @dfn{sizespec}. |
|
@item -p <ppaspec> |
|
This option is used to specify the local and/or remote DLPI PPA(s). |
|
The PPA is used to identify the interface over which traffic is to be |
|
sent/received. The syntax of a @dfn{ppaspec} is otherwise the same as |
|
a @dfn{sizespec}. |
|
@item -s sap |
|
This option specifies the 802.2 SAP for the test. A SAP is somewhat |
|
like either the port field of a TCP or UDP header or the protocol |
|
field of an IP header. The specified SAP should not conflict with any |
|
other active SAPs on the specified PPA's (@option{-p} option). |
|
@item -w <sizespec> |
|
This option specifies the local send and receive window sizes in units |
|
of frames on those platforms which support setting such things. |
|
@item -W <sizespec> |
|
This option specifies the remote send and receive window sizes in |
|
units of frames on those platforms which support setting such things. |
|
@end table |
|
|
|
The DLCO_STREAM test is only present if netperf was configured with |
|
@code{--enable-dlpi=yes}. The remote netserver must have also been |
|
configured with @code{--enable-dlpi=yes}. |
|
|
|
|
|
@node DLCL_STREAM, STREAM_STREAM, DLCO_STREAM, Options common to TCP UDP and SCTP tests |
|
@subsection DLCL_STREAM |
|
|
|
A DLPI ConnectionLess Stream (DLCL_STREAM) test is analogous to a |
|
@ref{UDP_STREAM} test in that both make use of unreliable/best-effort, |
|
connection-less transports. The DLCL_STREAM test differs from the |
|
@ref{UDP_STREAM} test in that the message size (@option{-m} option) must |
|
always be less than the link MTU as there is no IP-like fragmentation |
|
and reassembly available and netperf does not presume to provide one. |
|
|
|
The test-specific command-line options for a DLCL_STREAM test are the |
|
same as those for a @ref{DLCO_STREAM} test. |
|
|
|
The DLCL_STREAM test is only present if netperf was configured with |
|
@code{--enable-dlpi=yes}. The remote netserver must have also been |
|
configured with @code{--enable-dlpi=yes}. |
|
|
|
@node STREAM_STREAM, DG_STREAM, DLCL_STREAM, Options common to TCP UDP and SCTP tests |
|
@comment node-name, next, previous, up |
|
@subsection STREAM_STREAM |
|
|
|
A Unix Domain Stream Socket Stream test (STREAM_STREAM) is similar in |
|
concept to a @ref{TCP_STREAM} test, but using Unix Domain sockets. It is, |
|
naturally, limited to intra-machine traffic. A STREAM_STREAM test |
|
shares the @option{-m}, @option{-M}, @option{-s} and @option{-S} |
|
options of the other _STREAM tests. In a STREAM_STREAM test the |
|
@option{-p} option sets the directory in which the pipes will be |
|
created rather than setting a port number. The default is to create |
|
the pipes in the system default for the @code{tempnam()} call. |
|
|
|
The STREAM_STREAM test is only present if netperf was configured with |
|
@code{--enable-unixdomain=yes}. The remote netserver must have also been |
|
configured with @code{--enable-unixdomain=yes}. |
|
|
|
@node DG_STREAM, , STREAM_STREAM, Options common to TCP UDP and SCTP tests |
|
@comment node-name, next, previous, up |
|
@subsection DG_STREAM |
|
|
|
A Unix Domain Datagram Socket Stream test (SG_STREAM) is very much |
|
like a @ref{TCP_STREAM} test except that message boundaries are preserved. |
|
In this way, it may also be considered similar to certain flavors of |
|
SCTP test which can also preserve message boundaries. |
|
|
|
All the options of a @ref{STREAM_STREAM} test are applicable to a DG_STREAM |
|
test. |
|
|
|
The DG_STREAM test is only present if netperf was configured with |
|
@code{--enable-unixdomain=yes}. The remote netserver must have also been |
|
configured with @code{--enable-unixdomain=yes}. |
|
|
|
|
|
@node Using Netperf to Measure Request/Response , Using Netperf to Measure Aggregate Performance, Using Netperf to Measure Bulk Data Transfer, Top |
|
@chapter Using Netperf to Measure Request/Response |
|
|
|
Request/response performance is often overlooked, yet it is just as |
|
important as bulk-transfer performance. While things like larger |
|
socket buffers and TCP windows, and stateless offloads like TSO and |
|
LRO can cover a multitude of latency and even path-length sins, those |
|
sins cannot easily hide from a request/response test. The convention |
|
for a request/response test is to have a _RR suffix. There are |
|
however a few ``request/response'' tests that have other suffixes. |
|
|
|
A request/response test, particularly synchronous, one transaction at |
|
a time test such as those found by default in netperf, is particularly |
|
sensitive to the path-length of the networking stack. An _RR test can |
|
also uncover those platforms where the NICs are strapped by default |
|
with overbearing interrupt avoidance settings in an attempt to |
|
increase the bulk-transfer performance (or rather, decrease the CPU |
|
utilization of a bulk-transfer test). This sensitivity is most acute |
|
for small request and response sizes, such as the single-byte default |
|
for a netperf _RR test. |
|
|
|
While a bulk-transfer test reports its results in units of bits or |
|
bytes transferred per second, by default a mumble_RR test reports |
|
transactions per second where a transaction is defined as the |
|
completed exchange of a request and a response. One can invert the |
|
transaction rate to arrive at the average round-trip latency. If one |
|
is confident about the symmetry of the connection, the average one-way |
|
latency can be taken as one-half the average round-trip latency. As of |
|
version 2.5.0 (actually slightly before) netperf still does not do the |
|
latter, but will do the former if one sets the verbosity to 2 for a |
|
classic netperf test, or includes the appropriate @ref{Omni Output |
|
Selectors,output selector} in an @ref{The Omni Tests,omni test}. It |
|
will also allow the user to switch the throughput units from |
|
transactions per second to bits or bytes per second with the global |
|
@option{-f} option. |
|
|
|
@menu |
|
* Issues in Request/Response:: |
|
* Options Common to TCP UDP and SCTP _RR tests:: |
|
@end menu |
|
|
|
@node Issues in Request/Response, Options Common to TCP UDP and SCTP _RR tests, Using Netperf to Measure Request/Response , Using Netperf to Measure Request/Response |
|
@comment node-name, next, previous, up |
|
@section Issues in Request/Response |
|
|
|
Most if not all the @ref{Issues in Bulk Transfer} apply to |
|
request/response. The issue of round-trip latency is even more |
|
important as netperf generally only has one transaction outstanding at |
|
a time. |
|
|
|
A single instance of a one transaction outstanding _RR test should |
|
_never_ completely saturate the CPU of a system. If testing between |
|
otherwise evenly matched systems, the symmetric nature of a _RR test |
|
with equal request and response sizes should result in equal CPU |
|
loading on both systems. However, this may not hold true on MP |
|
systems, particularly if one CPU binds the netperf and netserver |
|
differently via the global @option{-T} option. |
|
|
|
For smaller request and response sizes packet loss is a bigger issue |
|
as there is no opportunity for a @dfn{fast retransmit} or |
|
retransmission prior to a retransmission timer expiring. |
|
|
|
Virtualization may considerably increase the effective path length of |
|
a networking stack. While this may not preclude achieving link-rate |
|
on a comparatively slow link (eg 1 Gigabit Ethernet) on a _STREAM |
|
test, it can show-up as measurably fewer transactions per second on an |
|
_RR test. However, this may still be masked by interrupt coalescing |
|
in the NIC/driver. |
|
|
|
Certain NICs have ways to minimize the number of interrupts sent to |
|
the host. If these are strapped badly they can significantly reduce |
|
the performance of something like a single-byte request/response test. |
|
Such setups are distinguished by seriously low reported CPU utilization |
|
and what seems like a low (even if in the thousands) transaction per |
|
second rate. Also, if you run such an OS/driver combination on faster |
|
or slower hardware and do not see a corresponding change in the |
|
transaction rate, chances are good that the driver is strapping the |
|
NIC with aggressive interrupt avoidance settings. Good for bulk |
|
throughput, but bad for latency. |
|
|
|
Some drivers may try to automagically adjust the interrupt avoidance |
|
settings. If they are not terribly good at it, you will see |
|
considerable run-to-run variation in reported transaction rates. |
|
Particularly if you ``mix-up'' _STREAM and _RR tests. |
|
|
|
|
|
@node Options Common to TCP UDP and SCTP _RR tests, , Issues in Request/Response, Using Netperf to Measure Request/Response |
|
@comment node-name, next, previous, up |
|
@section Options Common to TCP UDP and SCTP _RR tests |
|
|
|
Many ``test-specific'' options are actually common across the |
|
different tests. For those tests involving TCP, UDP and SCTP, whether |
|
using the BSD Sockets or the XTI interface those common options |
|
include: |
|
|
|
@table @code |
|
@vindex -h, Test-specific |
|
@item -h |
|
Display the test-suite-specific usage string and exit. For a TCP_ or |
|
UDP_ test this will be the usage string from the source file |
|
@file{nettest_bsd.c}. For an XTI_ test, this will be the usage string |
|
from the source file @file{src/nettest_xti.c}. For an SCTP test, this |
|
will be the usage string from the source file |
|
@file{src/nettest_sctp.c}. |
|
|
|
@vindex -H, Test-specific |
|
@item -H <optionspec> |
|
Normally, the remote hostname|IP and address family information is |
|
inherited from the settings for the control connection (eg global |
|
command-line @option{-H}, @option{-4} and/or @option{-6} options. |
|
The test-specific @option{-H} will override those settings for the |
|
data (aka test) connection only. Settings for the control connection |
|
are left unchanged. This might be used to cause the control and data |
|
connections to take different paths through the network. |
|
|
|
@vindex -L, Test-specific |
|
@item -L <optionspec> |
|
The test-specific @option{-L} option is identical to the test-specific |
|
@option{-H} option except it affects the local hostname|IP and address |
|
family information. As with its global command-line counterpart, this |
|
is generally only useful when measuring though those evil, end-to-end |
|
breaking things called firewalls. |
|
|
|
@vindex -P, Test-specific |
|
@item -P <optionspec> |
|
Set the local and/or remote port numbers for the data connection. |
|
|
|
@vindex -r, Test-specific |
|
@item -r <sizespec> |
|
This option sets the request (first value) and/or response (second |
|
value) sizes for an _RR test. By default the units are bytes, but a |
|
suffix of ``G,'' ``M,'' or ``K'' will specify the units to be 2^30 |
|
(GB), 2^20 (MB) or 2^10 (KB) respectively. A suffix of ``g,'' ``m'' |
|
or ``k'' will specify units of 10^9, 10^6 or 10^3 bytes |
|
respectively. For example: |
|
@example |
|
@code{-r 128,16K} |
|
@end example |
|
Will set the request size to 128 bytes and the response size to 16 KB |
|
or 16384 bytes. [Default: 1 - a single-byte request and response ] |
|
|
|
@vindex -s, Test-specific |
|
@item -s <sizespec> |
|
This option sets the local (netperf) send and receive socket buffer |
|
sizes for the data connection to the value(s) specified. Often, this |
|
will affect the advertised and/or effective TCP or other window, but |
|
on some platforms it may not. By default the units are bytes, but a |
|
suffix of ``G,'' ``M,'' or ``K'' will specify the units to be 2^30 |
|
(GB), 2^20 (MB) or 2^10 (KB) respectively. A suffix of ``g,'' ``m'' |
|
or ``k'' will specify units of 10^9, 10^6 or 10^3 bytes |
|
respectively. For example: |
|
@example |
|
@code{-s 128K} |
|
@end example |
|
Will request the local send (netperf) and receive socket buffer sizes |
|
to be 128KB or 131072 bytes. |
|
|
|
While the historic expectation is that setting the socket buffer size |
|
has a direct effect on say the TCP window, today that may not hold |
|
true for all stacks. When running under Windows a value of 0 may be |
|
used which will be an indication to the stack the user wants to enable |
|
a form of copy avoidance. [Default: -1 - use the system's default |
|
socket buffer sizes] |
|
|
|
@vindex -S, Test-specific |
|
@item -S <sizespec> |
|
This option sets the remote (netserver) send and/or receive socket |
|
buffer sizes for the data connection to the value(s) specified. |
|
Often, this will affect the advertised and/or effective TCP or other |
|
window, but on some platforms it may not. By default the units are |
|
bytes, but a suffix of ``G,'' ``M,'' or ``K'' will specify the units |
|
to be 2^30 (GB), 2^20 (MB) or 2^10 (KB) respectively. A suffix of |
|
``g,'' ``m'' or ``k'' will specify units of 10^9, 10^6 or 10^3 bytes |
|
respectively. For example: |
|
@example |
|
@code{-S 128K} |
|
@end example |
|
Will request the remote (netserver) send and receive socket buffer |
|
sizes to be 128KB or 131072 bytes. |
|
|
|
While the historic expectation is that setting the socket buffer size |
|
has a direct effect on say the TCP window, today that may not hold |
|
true for all stacks. When running under Windows a value of 0 may be |
|
used which will be an indication to the stack the user wants to enable |
|
a form of copy avoidance. [Default: -1 - use the system's default |
|
socket buffer sizes] |
|
|
|
@vindex -4, Test-specific |
|
@item -4 |
|
Set the local and remote address family for the data connection to |
|
AF_INET - ie use IPv4 addressing only. Just as with their global |
|
command-line counterparts the last of the @option{-4}, @option{-6}, |
|
@option{-H} or @option{-L} option wins for their respective address |
|
families. |
|
|
|
@vindex -6 Test-specific |
|
@item -6 |
|
This option is identical to its @option{-4} cousin, but requests IPv6 |
|
addresses for the local and remote ends of the data connection. |
|
|
|
@end table |
|
|
|
@menu |
|
* TCP_RR:: |
|
* TCP_CC:: |
|
* TCP_CRR:: |
|
* UDP_RR:: |
|
* XTI_TCP_RR:: |
|
* XTI_TCP_CC:: |
|
* XTI_TCP_CRR:: |
|
* XTI_UDP_RR:: |
|
* DLCL_RR:: |
|
* DLCO_RR:: |
|
* SCTP_RR:: |
|
@end menu |
|
|
|
@node TCP_RR, TCP_CC, Options Common to TCP UDP and SCTP _RR tests, Options Common to TCP UDP and SCTP _RR tests |
|
@subsection TCP_RR |
|
@cindex Measuring Latency |
|
@cindex Latency, Request-Response |
|
|
|
A TCP_RR (TCP Request/Response) test is requested by passing a value |
|
of ``TCP_RR'' to the global @option{-t} command-line option. A TCP_RR |
|
test can be thought-of as a user-space to user-space @code{ping} with |
|
no think time - it is by default a synchronous, one transaction at a |
|
time, request/response test. |
|
|
|
The transaction rate is the number of complete transactions exchanged |
|
divided by the length of time it took to perform those transactions. |
|
|
|
If the two Systems Under Test are otherwise identical, a TCP_RR test |
|
with the same request and response size should be symmetric - it |
|
should not matter which way the test is run, and the CPU utilization |
|
measured should be virtually the same on each system. If not, it |
|
suggests that the CPU utilization mechanism being used may have some, |
|
well, issues measuring CPU utilization completely and accurately. |
|
|
|
Time to establish the TCP connection is not counted in the result. If |
|
you want connection setup overheads included, you should consider the |
|
@ref{TCP_CC,TPC_CC} or @ref{TCP_CRR,TCP_CRR} tests. |
|
|
|
If specifying the @option{-D} option to set TCP_NODELAY and disable |
|
the Nagle Algorithm increases the transaction rate reported by a |
|
TCP_RR test, it implies the stack(s) over which the TCP_RR test is |
|
running have a broken implementation of the Nagle Algorithm. Likely |
|
as not they are interpreting Nagle on a segment by segment basis |
|
rather than a user send by user send basis. You should contact your |
|
stack vendor(s) to report the problem to them. |
|
|
|
Here is an example of two systems running a basic TCP_RR test over a |
|
10 Gigabit Ethernet link: |
|
|
|
@example |
|
netperf -t TCP_RR -H 192.168.2.125 |
|
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.2.125 (192.168.2.125) port 0 AF_INET |
|
Local /Remote |
|
Socket Size Request Resp. Elapsed Trans. |
|
Send Recv Size Size Time Rate |
|
bytes Bytes bytes bytes secs. per sec |
|
|
|
16384 87380 1 1 10.00 29150.15 |
|
16384 87380 |
|
@end example |
|
|
|
In this example the request and response sizes were one byte, the |
|
socket buffers were left at their defaults, and the test ran for all |
|
of 10 seconds. The transaction per second rate was rather good for |
|
the time :) |
|
|
|
@node TCP_CC, TCP_CRR, TCP_RR, Options Common to TCP UDP and SCTP _RR tests |
|
@subsection TCP_CC |
|
@cindex Connection Latency |
|
@cindex Latency, Connection Establishment |
|
|
|
A TCP_CC (TCP Connect/Close) test is requested by passing a value of |
|
``TCP_CC'' to the global @option{-t} option. A TCP_CC test simply |
|
measures how fast the pair of systems can open and close connections |
|
between one another in a synchronous (one at a time) manner. While |
|
this is considered an _RR test, no request or response is exchanged |
|
over the connection. |
|
|
|
@cindex Port Reuse |
|
@cindex TIME_WAIT |
|
The issue of TIME_WAIT reuse is an important one for a TCP_CC test. |
|
Basically, TIME_WAIT reuse is when a pair of systems churn through |
|
connections fast enough that they wrap the 16-bit port number space in |
|
less time than the length of the TIME_WAIT state. While it is indeed |
|
theoretically possible to ``reuse'' a connection in TIME_WAIT, the |
|
conditions under which such reuse is possible are rather rare. An |
|
attempt to reuse a connection in TIME_WAIT can result in a non-trivial |
|
delay in connection establishment. |
|
|
|
Basically, any time the connection churn rate approaches: |
|
|
|
Sizeof(clientportspace) / Lengthof(TIME_WAIT) |
|
|
|
there is the risk of TIME_WAIT reuse. To minimize the chances of this |
|
happening, netperf will by default select its own client port numbers |
|
from the range of 5000 to 65535. On systems with a 60 second |
|
TIME_WAIT state, this should allow roughly 1000 transactions per |
|
second. The size of the client port space used by netperf can be |
|
controlled via the test-specific @option{-p} option, which takes a |
|
@dfn{sizespec} as a value setting the minimum (first value) and |
|
maximum (second value) port numbers used by netperf at the client end. |
|
|
|
Since no requests or responses are exchanged during a TCP_CC test, |
|
only the @option{-H}, @option{-L}, @option{-4} and @option{-6} of the |
|
``common'' test-specific options are likely to have an effect, if any, |
|
on the results. The @option{-s} and @option{-S} options _may_ have |
|
some effect if they alter the number and/or type of options carried in |
|
the TCP SYNchronize segments, such as Window Scaling or Timestamps. |
|
The @option{-P} and @option{-r} options are utterly ignored. |
|
|
|
Since connection establishment and tear-down for TCP is not symmetric, |
|
a TCP_CC test is not symmetric in its loading of the two systems under |
|
test. |
|
|
|
@node TCP_CRR, UDP_RR, TCP_CC, Options Common to TCP UDP and SCTP _RR tests |
|
@subsection TCP_CRR |
|
@cindex Latency, Connection Establishment |
|
@cindex Latency, Request-Response |
|
|
|
The TCP Connect/Request/Response (TCP_CRR) test is requested by |
|
passing a value of ``TCP_CRR'' to the global @option{-t} command-line |
|
option. A TCP_CRR test is like a merger of a @ref{TCP_RR} and |
|
@ref{TCP_CC} test which measures the performance of establishing a |
|
connection, exchanging a single request/response transaction, and |
|
tearing-down that connection. This is very much like what happens in |
|
an HTTP 1.0 or HTTP 1.1 connection when HTTP Keepalives are not used. |
|
In fact, the TCP_CRR test was added to netperf to simulate just that. |
|
|
|
Since a request and response are exchanged the @option{-r}, |
|
@option{-s} and @option{-S} options can have an effect on the |
|
performance. |
|
|
|
The issue of TIME_WAIT reuse exists for the TCP_CRR test just as it |
|
does for the TCP_CC test. Similarly, since connection establishment |
|
and tear-down is not symmetric, a TCP_CRR test is not symmetric even |
|
when the request and response sizes are the same. |
|
|
|
@node UDP_RR, XTI_TCP_RR, TCP_CRR, Options Common to TCP UDP and SCTP _RR tests |
|
@subsection UDP_RR |
|
@cindex Latency, Request-Response |
|
@cindex Packet Loss |
|
|
|
A UDP Request/Response (UDP_RR) test is requested by passing a value |
|
of ``UDP_RR'' to a global @option{-t} option. It is very much the |
|
same as a TCP_RR test except UDP is used rather than TCP. |
|
|
|
UDP does not provide for retransmission of lost UDP datagrams, and |
|
netperf does not add anything for that either. This means that if |
|
_any_ request or response is lost, the exchange of requests and |
|
responses will stop from that point until the test timer expires. |
|
Netperf will not really ``know'' this has happened - the only symptom |
|
will be a low transaction per second rate. If @option{--enable-burst} |
|
was included in the @code{configure} command and a test-specific |
|
@option{-b} option used, the UDP_RR test will ``survive'' the loss of |
|
requests and responses until the sum is one more than the value passed |
|
via the @option{-b} option. It will though almost certainly run more |
|
slowly. |
|
|
|
The netperf side of a UDP_RR test will call @code{connect()} on its |
|
data socket and thenceforth use the @code{send()} and @code{recv()} |
|
socket calls. The netserver side of a UDP_RR test will not call |
|
@code{connect()} and will use @code{recvfrom()} and @code{sendto()} |
|
calls. This means that even if the request and response sizes are the |
|
same, a UDP_RR test is _not_ symmetric in its loading of the two |
|
systems under test. |
|
|
|
Here is an example of a UDP_RR test between two otherwise |
|
identical two-CPU systems joined via a 1 Gigabit Ethernet network: |
|
|
|
@example |
|
$ netperf -T 1 -H 192.168.1.213 -t UDP_RR -c -C |
|
UDP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.213 (192.168.1.213) port 0 AF_INET |
|
Local /Remote |
|
Socket Size Request Resp. Elapsed Trans. CPU CPU S.dem S.dem |
|
Send Recv Size Size Time Rate local remote local remote |
|
bytes bytes bytes bytes secs. per sec % I % I us/Tr us/Tr |
|
|
|
65535 65535 1 1 10.01 15262.48 13.90 16.11 18.221 21.116 |
|
65535 65535 |
|
@end example |
|
|
|
This example includes the @option{-c} and @option{-C} options to |
|
enable CPU utilization reporting and shows the asymmetry in CPU |
|
loading. The @option{-T} option was used to make sure netperf and |
|
netserver ran on a given CPU and did not move around during the test. |
|
|
|
@node XTI_TCP_RR, XTI_TCP_CC, UDP_RR, Options Common to TCP UDP and SCTP _RR tests |
|
@subsection XTI_TCP_RR |
|
@cindex Latency, Request-Response |
|
|
|
An XTI_TCP_RR test is essentially the same as a @ref{TCP_RR} test only |
|
using the XTI rather than BSD Sockets interface. It is requested by |
|
passing a value of ``XTI_TCP_RR'' to the @option{-t} global |
|
command-line option. |
|
|
|
The test-specific options for an XTI_TCP_RR test are the same as those |
|
for a TCP_RR test with the addition of the @option{-X <devspec>} option to |
|
specify the names of the local and/or remote XTI device file(s). |
|
|
|
@node XTI_TCP_CC, XTI_TCP_CRR, XTI_TCP_RR, Options Common to TCP UDP and SCTP _RR tests |
|
@comment node-name, next, previous, up |
|
@subsection XTI_TCP_CC |
|
@cindex Latency, Connection Establishment |
|
|
|
An XTI_TCP_CC test is essentially the same as a @ref{TCP_CC,TCP_CC} |
|
test, only using the XTI rather than BSD Sockets interface. |
|
|
|
The test-specific options for an XTI_TCP_CC test are the same as those |
|
for a TCP_CC test with the addition of the @option{-X <devspec>} option to |
|
specify the names of the local and/or remote XTI device file(s). |
|
|
|
@node XTI_TCP_CRR, XTI_UDP_RR, XTI_TCP_CC, Options Common to TCP UDP and SCTP _RR tests |
|
@comment node-name, next, previous, up |
|
@subsection XTI_TCP_CRR |
|
@cindex Latency, Connection Establishment |
|
@cindex Latency, Request-Response |
|
|
|
The XTI_TCP_CRR test is essentially the same as a |
|
@ref{TCP_CRR,TCP_CRR} test, only using the XTI rather than BSD Sockets |
|
interface. |
|
|
|
The test-specific options for an XTI_TCP_CRR test are the same as those |
|
for a TCP_RR test with the addition of the @option{-X <devspec>} option to |
|
specify the names of the local and/or remote XTI device file(s). |
|
|
|
@node XTI_UDP_RR, DLCL_RR, XTI_TCP_CRR, Options Common to TCP UDP and SCTP _RR tests |
|
@subsection XTI_UDP_RR |
|
@cindex Latency, Request-Response |
|
|
|
An XTI_UDP_RR test is essentially the same as a UDP_RR test only using |
|
the XTI rather than BSD Sockets interface. It is requested by passing |
|
a value of ``XTI_UDP_RR'' to the @option{-t} global command-line |
|
option. |
|
|
|
The test-specific options for an XTI_UDP_RR test are the same as those |
|
for a UDP_RR test with the addition of the @option{-X <devspec>} |
|
option to specify the name of the local and/or remote XTI device |
|
file(s). |
|
|
|
@node DLCL_RR, DLCO_RR, XTI_UDP_RR, Options Common to TCP UDP and SCTP _RR tests |
|
@comment node-name, next, previous, up |
|
@subsection DLCL_RR |
|
@cindex Latency, Request-Response |
|
|
|
@node DLCO_RR, SCTP_RR, DLCL_RR, Options Common to TCP UDP and SCTP _RR tests |
|
@comment node-name, next, previous, up |
|
@subsection DLCO_RR |
|
@cindex Latency, Request-Response |
|
|
|
@node SCTP_RR, , DLCO_RR, Options Common to TCP UDP and SCTP _RR tests |
|
@comment node-name, next, previous, up |
|
@subsection SCTP_RR |
|
@cindex Latency, Request-Response |
|
|
|
@node Using Netperf to Measure Aggregate Performance, Using Netperf to Measure Bidirectional Transfer, Using Netperf to Measure Request/Response , Top |
|
@comment node-name, next, previous, up |
|
@chapter Using Netperf to Measure Aggregate Performance |
|
@cindex Aggregate Performance |
|
@vindex --enable-burst, Configure |
|
|
|
Ultimately, @ref{Netperf4,Netperf4} will be the preferred benchmark to |
|
use when one wants to measure aggregate performance because netperf |
|
has no support for explicit synchronization of concurrent tests. Until |
|
netperf4 is ready for prime time, one can make use of the heuristics |
|
and procedures mentioned here for the 85% solution. |
|
|
|
There are a few ways to measure aggregate performance with netperf. |
|
The first is to run multiple, concurrent netperf tests and can be |
|
applied to any of the netperf tests. The second is to configure |
|
netperf with @code{--enable-burst} and is applicable to the TCP_RR |
|
test. The third is a variation on the first. |
|
|
|
@menu |
|
* Running Concurrent Netperf Tests:: |
|
* Using --enable-burst:: |
|
* Using --enable-demo:: |
|
@end menu |
|
|
|
@node Running Concurrent Netperf Tests, Using --enable-burst, Using Netperf to Measure Aggregate Performance, Using Netperf to Measure Aggregate Performance |
|
@comment node-name, next, previous, up |
|
@section Running Concurrent Netperf Tests |
|
|
|
@ref{Netperf4,Netperf4} is the preferred benchmark to use when one |
|
wants to measure aggregate performance because netperf has no support |
|
for explicit synchronization of concurrent tests. This leaves |
|
netperf2 results vulnerable to @dfn{skew} errors. |
|
|
|
However, since there are times when netperf4 is unavailable it may be |
|
necessary to run netperf. The skew error can be minimized by making |
|
use of the confidence interval functionality. Then one simply |
|
launches multiple tests from the shell using a @code{for} loop or the |
|
like: |
|
|
|
@example |
|
for i in 1 2 3 4 |
|
do |
|
netperf -t TCP_STREAM -H tardy.cup.hp.com -i 10 -P 0 & |
|
done |
|
@end example |
|
|
|
which will run four, concurrent @ref{TCP_STREAM,TCP_STREAM} tests from |
|
the system on which it is executed to tardy.cup.hp.com. Each |
|
concurrent netperf will iterate 10 times thanks to the @option{-i} |
|
option and will omit the test banners (option @option{-P}) for |
|
brevity. The output looks something like this: |
|
|
|
@example |
|
87380 16384 16384 10.03 235.15 |
|
87380 16384 16384 10.03 235.09 |
|
87380 16384 16384 10.03 235.38 |
|
87380 16384 16384 10.03 233.96 |
|
@end example |
|
|
|
We can take the sum of the results and be reasonably confident that |
|
the aggregate performance was 940 Mbits/s. This method does not need |
|
to be limited to one system speaking to one other system. It can be |
|
extended to one system talking to N other systems. It could be as simple as: |
|
@example |
|
for host in 'foo bar baz bing' |
|
do |
|
netperf -t TCP_STREAM -H $hosts -i 10 -P 0 & |
|
done |
|
@end example |
|
A more complicated/sophisticated example can be found in |
|
@file{doc/examples/runemomniagg2.sh} where. |
|
|
|
If you see warnings about netperf not achieving the confidence |
|
intervals, the best thing to do is to increase the number of |
|
iterations with @option{-i} and/or increase the run length of each |
|
iteration with @option{-l}. |
|
|
|
You can also enable local (@option{-c}) and/or remote (@option{-C}) |
|
CPU utilization: |
|
|
|
@example |
|
for i in 1 2 3 4 |
|
do |
|
netperf -t TCP_STREAM -H tardy.cup.hp.com -i 10 -P 0 -c -C & |
|
done |
|
|
|
87380 16384 16384 10.03 235.47 3.67 5.09 10.226 14.180 |
|
87380 16384 16384 10.03 234.73 3.67 5.09 10.260 14.225 |
|
87380 16384 16384 10.03 234.64 3.67 5.10 10.263 14.231 |
|
87380 16384 16384 10.03 234.87 3.67 5.09 10.253 14.215 |
|
@end example |
|
|
|
If the CPU utilizations reported for the same system are the same or |
|
very very close you can be reasonably confident that skew error is |
|
minimized. Presumably one could then omit @option{-i} but that is |
|
not advised, particularly when/if the CPU utilization approaches 100 |
|
percent. In the example above we see that the CPU utilization on the |
|
local system remains the same for all four tests, and is only off by |
|
0.01 out of 5.09 on the remote system. As the number of CPUs in the |
|
system increases, and so too the odds of saturating a single CPU, the |
|
accuracy of similar CPU utilization implying little skew error is |
|
diminished. This is also the case for those increasingly rare single |
|
CPU systems if the utilization is reported as 100% or very close to |
|
it. |
|
|
|
@quotation |
|
@b{NOTE: It is very important to remember that netperf is calculating |
|
system-wide CPU utilization. When calculating the service demand |
|
(those last two columns in the output above) each netperf assumes it |
|
is the only thing running on the system. This means that for |
|
concurrent tests the service demands reported by netperf will be |
|
wrong. One has to compute service demands for concurrent tests by |
|
hand.} |
|
@end quotation |
|
|
|
If you wish you can add a unique, global @option{-B} option to each |
|
command line to append the given string to the output: |
|
|
|
@example |
|
for i in 1 2 3 4 |
|
do |
|
netperf -t TCP_STREAM -H tardy.cup.hp.com -B "this is test $i" -i 10 -P 0 & |
|
done |
|
|
|
87380 16384 16384 10.03 234.90 this is test 4 |
|
87380 16384 16384 10.03 234.41 this is test 2 |
|
87380 16384 16384 10.03 235.26 this is test 1 |
|
87380 16384 16384 10.03 235.09 this is test 3 |
|
@end example |
|
|
|
You will notice that the tests completed in an order other than they |
|
were started from the shell. This underscores why there is a threat |
|
of skew error and why netperf4 will eventually be the preferred tool |
|
for aggregate tests. Even if you see the Netperf Contributing Editor |
|
acting to the contrary!-) |
|
|
|
@menu |
|
* Issues in Running Concurrent Tests:: |
|
@end menu |
|
|
|
@node Issues in Running Concurrent Tests, , Running Concurrent Netperf Tests, Running Concurrent Netperf Tests |
|
@subsection Issues in Running Concurrent Tests |
|
|
|
In addition to the aforementioned issue of skew error, there can be |
|
other issues to consider when running concurrent netperf tests. |
|
|
|
For example, when running concurrent tests over multiple interfaces, |
|
one is not always assured that the traffic one thinks went over a |
|
given interface actually did so. In particular, the Linux networking |
|
stack takes a particularly strong stance on its following the so |
|
called @samp{weak end system model}. As such, it is willing to answer |
|
ARP requests for any of its local IP addresses on any of its |
|
interfaces. If multiple interfaces are connected to the same |
|
broadcast domain, then even if they are configured into separate IP |
|
subnets there is no a priori way of knowing which interface was |
|
actually used for which connection(s). This can be addressed by |
|
setting the @samp{arp_ignore} sysctl before configuring interfaces. |
|
|
|
As it is quite important, we will repeat that it is very important to |
|
remember that each concurrent netperf instance is calculating |
|
system-wide CPU utilization. When calculating the service demand each |
|
netperf assumes it is the only thing running on the system. This |
|
means that for concurrent tests the service demands reported by |
|
netperf @b{will be wrong}. One has to compute service demands for |
|
concurrent tests by hand |
|
|
|
Running concurrent tests can also become difficult when there is no |
|
one ``central'' node. Running tests between pairs of systems may be |
|
more difficult, calling for remote shell commands in the for loop |
|
rather than netperf commands. This introduces more skew error, which |
|
the confidence intervals may not be able to sufficiently mitigate. |
|
One possibility is to actually run three consecutive netperf tests on |
|
each node - the first being a warm-up, the last being a cool-down. |
|
The idea then is to ensure that the time it takes to get all the |
|
netperfs started is less than the length of the first netperf command |
|
in the sequence of three. Similarly, it assumes that all ``middle'' |
|
netperfs will complete before the first of the ``last'' netperfs |
|
complete. |
|
|
|
@node Using --enable-burst, Using --enable-demo, Running Concurrent Netperf Tests, Using Netperf to Measure Aggregate Performance |
|
@comment node-name, next, previous, up |
|
@section Using - -enable-burst |
|
|
|
Starting in version 2.5.0 @code{--enable-burst=yes} is the default, |
|
which means one no longer must: |
|
|
|
@example |
|
configure --enable-burst |
|
@end example |
|
|
|
To have burst-mode functionality present in netperf. This enables a |
|
test-specific @option{-b num} option in @ref{TCP_RR,TCP_RR}, |
|
@ref{UDP_RR,UDP_RR} and @ref{The Omni Tests,omni} tests. |
|
|
|
Normally, netperf will attempt to ramp-up the number of outstanding |
|
requests to @option{num} plus one transactions in flight at one time. |
|
The ramp-up is to avoid transactions being smashed together into a |
|
smaller number of segments when the transport's congestion window (if |
|
any) is smaller at the time than what netperf wants to have |
|
outstanding at one time. If, however, the user specifies a negative |
|
value for @option{num} this ramp-up is bypassed and the burst of sends |
|
is made without consideration of transport congestion window. |
|
|
|
This burst-mode is used as an alternative to or even in conjunction |
|
with multiple-concurrent _RR tests and as a way to implement a |
|
single-connection, bidirectional bulk-transfer test. When run with |
|
just a single instance of netperf, increasing the burst size can |
|
determine the maximum number of transactions per second which can be |
|
serviced by a single process: |
|
|
|
@example |
|
for b in 0 1 2 4 8 16 32 |
|
do |
|
netperf -v 0 -t TCP_RR -B "-b $b" -H hpcpc108 -P 0 -- -b $b |
|
done |
|
|
|
9457.59 -b 0 |
|
9975.37 -b 1 |
|
10000.61 -b 2 |
|
20084.47 -b 4 |
|
29965.31 -b 8 |
|
71929.27 -b 16 |
|
109718.17 -b 32 |
|
@end example |
|
|
|
The global @option{-v} and @option{-P} options were used to minimize |
|
the output to the single figure of merit which in this case the |
|
transaction rate. The global @code{-B} option was used to more |
|
clearly label the output, and the test-specific @option{-b} option |
|
enabled by @code{--enable-burst} increase the number of transactions |
|
in flight at one time. |
|
|
|
Now, since the test-specific @option{-D} option was not specified to |
|
set TCP_NODELAY, the stack was free to ``bundle'' requests and/or |
|
responses into TCP segments as it saw fit, and since the default |
|
request and response size is one byte, there could have been some |
|
considerable bundling even in the absence of transport congestion |
|
window issues. If one wants to try to achieve a closer to |
|
one-to-one correspondence between a request and response and a TCP |
|
segment, add the test-specific @option{-D} option: |
|
|
|
@example |
|
for b in 0 1 2 4 8 16 32 |
|
do |
|
netperf -v 0 -t TCP_RR -B "-b $b -D" -H hpcpc108 -P 0 -- -b $b -D |
|
done |
|
|
|
8695.12 -b 0 -D |
|
19966.48 -b 1 -D |
|
20691.07 -b 2 -D |
|
49893.58 -b 4 -D |
|
62057.31 -b 8 -D |
|
108416.88 -b 16 -D |
|
114411.66 -b 32 -D |
|
@end example |
|
|
|
You can see that this has a rather large effect on the reported |
|
transaction rate. In this particular instance, the author believes it |
|
relates to interactions between the test and interrupt coalescing |
|
settings in the driver for the NICs used. |
|
|
|
@quotation |
|
@b{NOTE: Even if you set the @option{-D} option that is still not a |
|
guarantee that each transaction is in its own TCP segments. You |
|
should get into the habit of verifying the relationship between the |
|
transaction rate and the packet rate via other means.} |
|
@end quotation |
|
|
|
You can also combine @code{--enable-burst} functionality with |
|
concurrent netperf tests. This would then be an ``aggregate of |
|
aggregates'' if you like: |
|
|
|
@example |
|
|
|
for i in 1 2 3 4 |
|
do |
|
netperf -H hpcpc108 -v 0 -P 0 -i 10 -B "aggregate $i -b 8 -D" -t TCP_RR -- -b 8 -D & |
|
done |
|
|
|
46668.38 aggregate 4 -b 8 -D |
|
44890.64 aggregate 2 -b 8 -D |
|
45702.04 aggregate 1 -b 8 -D |
|
46352.48 aggregate 3 -b 8 -D |
|
|
|
@end example |
|
|
|
Since each netperf did hit the confidence intervals, we can be |
|
reasonably certain that the aggregate transaction per second rate was |
|
the sum of all four concurrent tests, or something just shy of 184,000 |
|
transactions per second. To get some idea if that was also the packet |
|
per second rate, we could bracket that @code{for} loop with something |
|
to gather statistics and run the results through |
|
@uref{ftp://ftp.cup.hp.com/dist/networking/tools,beforeafter}: |
|
|
|
@example |
|
/usr/sbin/ethtool -S eth2 > before |
|
for i in 1 2 3 4 |
|
do |
|
netperf -H 192.168.2.108 -l 60 -v 0 -P 0 -B "aggregate $i -b 8 -D" -t TCP_RR -- -b 8 -D & |
|
done |
|
wait |
|
/usr/sbin/ethtool -S eth2 > after |
|
|
|
52312.62 aggregate 2 -b 8 -D |
|
50105.65 aggregate 4 -b 8 -D |
|
50890.82 aggregate 1 -b 8 -D |
|
50869.20 aggregate 3 -b 8 -D |
|
|
|
beforeafter before after > delta |
|
|
|
grep packets delta |
|
rx_packets: 12251544 |
|
tx_packets: 12251550 |
|
|
|
@end example |
|
|
|
This example uses @code{ethtool} because the system being used is |
|
running Linux. Other platforms have other tools - for example HP-UX |
|
has lanadmin: |
|
|
|
@example |
|
lanadmin -g mibstats <ppa> |
|
@end example |
|
|
|
and of course one could instead use @code{netstat}. |
|
|
|
The @code{wait} is important because we are launching concurrent |
|
netperfs in the background. Without it, the second ethtool command |
|
would be run before the tests finished and perhaps even before the |
|
last of them got started! |
|
|
|
The sum of the reported transaction rates is 204178 over 60 seconds, |
|
which is a total of 12250680 transactions. Each transaction is the |
|
exchange of a request and a response, so we multiply that by 2 to |
|
arrive at 24501360. |
|
|
|
The sum of the ethtool stats is 24503094 packets which matches what |
|
netperf was reporting very well. |
|
|
|
Had the request or response size differed, we would need to know how |
|
it compared with the @dfn{MSS} for the connection. |
|
|
|
Just for grins, here is the exercise repeated, using @code{netstat} |
|
instead of @code{ethtool} |
|
|
|
@example |
|
netstat -s -t > before |
|
for i in 1 2 3 4 |
|
do |
|
netperf -l 60 -H 192.168.2.108 -v 0 -P 0 -B "aggregate $i -b 8 -D" -t TCP_RR -- -b 8 -D & done |
|
wait |
|
netstat -s -t > after |
|
|
|
51305.88 aggregate 4 -b 8 -D |
|
51847.73 aggregate 2 -b 8 -D |
|
50648.19 aggregate 3 -b 8 -D |
|
53605.86 aggregate 1 -b 8 -D |
|
|
|
beforeafter before after > delta |
|
|
|
grep segments delta |
|
12445708 segments received |
|
12445730 segments send out |
|
1 segments retransmited |
|
0 bad segments received. |
|
@end example |
|
|
|
The sums are left as an exercise to the reader :) |
|
|
|
Things become considerably more complicated if there are non-trvial |
|
packet losses and/or retransmissions. |
|
|
|
Of course all this checking is unnecessary if the test is a UDP_RR |
|
test because UDP ``never'' aggregates multiple sends into the same UDP |
|
datagram, and there are no ACKnowledgements in UDP. The loss of a |
|
single request or response will not bring a ``burst'' UDP_RR test to a |
|
screeching halt, but it will reduce the number of transactions |
|
outstanding at any one time. A ``burst'' UDP_RR test @b{will} come to a |
|
halt if the sum of the lost requests and responses reaches the value |
|
specified in the test-specific @option{-b} option. |
|
|
|
@node Using --enable-demo, , Using --enable-burst, Using Netperf to Measure Aggregate Performance |
|
@section Using - -enable-demo |
|
|
|
One can |
|
@example |
|
configure --enable-demo |
|
@end example |
|
and compile netperf to enable netperf to emit ``interim results'' at |
|
semi-regular intervals. This enables a global @code{-D} option which |
|
takes a reporting interval as an argument. With that specified, the |
|
output of netperf will then look something like |
|
|
|
@example |
|
$ src/netperf -D 1.25 |
|
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost.localdomain () port 0 AF_INET : demo |
|
Interim result: 25425.52 10^6bits/s over 1.25 seconds ending at 1327962078.405 |
|
Interim result: 25486.82 10^6bits/s over 1.25 seconds ending at 1327962079.655 |
|
Interim result: 25474.96 10^6bits/s over 1.25 seconds ending at 1327962080.905 |
|
Interim result: 25523.49 10^6bits/s over 1.25 seconds ending at 1327962082.155 |
|
Interim result: 25053.57 10^6bits/s over 1.27 seconds ending at 1327962083.429 |
|
Interim result: 25349.64 10^6bits/s over 1.25 seconds ending at 1327962084.679 |
|
Interim result: 25292.84 10^6bits/s over 1.25 seconds ending at 1327962085.932 |
|
Recv Send Send |
|
Socket Socket Message Elapsed |
|
Size Size Size Time Throughput |
|
bytes bytes bytes secs. 10^6bits/sec |
|
|
|
87380 16384 16384 10.00 25375.66 |
|
@end example |
|
The units of the ``Interim result'' lines will follow the units |
|
selected via the global @code{-f} option. If the test-specific |
|
@code{-o} option is specified on the command line, the format will be |
|
CSV: |
|
@example |
|
... |
|
2978.81,MBytes/s,1.25,1327962298.035 |
|
... |
|
@end example |
|
If the test-specific @code{-k} option is used the format will be |
|
keyval with each keyval being given an index: |
|
@example |
|
... |
|
NETPERF_INTERIM_RESULT[2]=25.00 |
|
NETPERF_UNITS[2]=10^9bits/s |
|
NETPERF_INTERVAL[2]=1.25 |
|
NETPERF_ENDING[2]=1327962357.249 |
|
... |
|
@end example |
|
The expectation is it may be easier to utilize the keyvals if they |
|
have indices. |
|
|
|
But how does this help with aggregate tests? Well, what one can do is |
|
start the netperfs via a script, giving each a Very Long (tm) run |
|
time. Direct the output to a file per instance. Then, once all the |
|
netperfs have been started, take a timestamp and wait for some desired |
|
test interval. Once that interval expires take another timestamp and |
|
then start terminating the netperfs by sending them a SIGALRM signal |
|
via the likes of the @code{kill} or @code{pkill} command. The |
|
netperfs will terminate and emit the rest of the ``usual'' output, and |
|
you can then bring the files to a central location for post |
|
processing to find the aggregate performance over the ``test interval.'' |
|
|
|
This method has the advantage that it does not require advance |
|
knowledge of how long it takes to get netperf tests started and/or |
|
stopped. It does though require sufficiently synchronized clocks on |
|
all the test systems. |
|
|
|
While calls to get the current time can be inexpensive, that neither |
|
has been nor is universally true. For that reason netperf tries to |
|
minimize the number of such ``timestamping'' calls (eg |
|
@code{gettimeofday}) calls it makes when in demo mode. Rather than |
|
take a timestamp after each @code{send} or @code{recv} call completes |
|
netperf tries to guess how many units of work will be performed over |
|
the desired interval. Only once that many units of work have been |
|
completed will netperf check the time. If the reporting interval has |
|
passed, netperf will emit an ``interim result.'' If the interval has |
|
not passed, netperf will update its estimate for units and continue. |
|
|
|
After a bit of thought one can see that if things ``speed-up'' netperf |
|
will still honor the interval. However, if things ``slow-down'' |
|
netperf may be late with an ``interim result.'' Here is an example of |
|
both of those happening during a test - with the interval being |
|
honored while throughput increases, and then about half-way through |
|
when another netperf (not shown) is started we see things slowing down |
|
and netperf not hitting the interval as desired. |
|
@example |
|
$ src/netperf -D 2 -H tardy.hpl.hp.com -l 20 |
|
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to tardy.hpl.hp.com () port 0 AF_INET : demo |
|
Interim result: 36.46 10^6bits/s over 2.01 seconds ending at 1327963880.565 |
|
Interim result: 59.19 10^6bits/s over 2.00 seconds ending at 1327963882.569 |
|
Interim result: 73.39 10^6bits/s over 2.01 seconds ending at 1327963884.576 |
|
Interim result: 84.01 10^6bits/s over 2.03 seconds ending at 1327963886.603 |
|
Interim result: 75.63 10^6bits/s over 2.21 seconds ending at 1327963888.814 |
|
Interim result: 55.52 10^6bits/s over 2.72 seconds ending at 1327963891.538 |
|
Interim result: 70.94 10^6bits/s over 2.11 seconds ending at 1327963893.650 |
|
Interim result: 80.66 10^6bits/s over 2.13 seconds ending at 1327963895.777 |
|
Interim result: 86.42 10^6bits/s over 2.12 seconds ending at 1327963897.901 |
|
Recv Send Send |
|
Socket Socket Message Elapsed |
|
Size Size Size Time Throughput |
|
bytes bytes bytes secs. 10^6bits/sec |
|
|
|
87380 16384 16384 20.34 68.87 |
|
@end example |
|
So long as your post-processing mechanism can account for that, there |
|
should be no problem. As time passes there may be changes to try to |
|
improve the netperf's honoring the interval but one should not |
|
ass-u-me it will always do so. One should not assume the precision |
|
will remain fixed - future versions may change it - perhaps going |
|
beyond tenths of seconds in reporting the interval length etc. |
|
|
|
@node Using Netperf to Measure Bidirectional Transfer, The Omni Tests, Using Netperf to Measure Aggregate Performance, Top |
|
@comment node-name, next, previous, up |
|
@chapter Using Netperf to Measure Bidirectional Transfer |
|
|
|
There are two ways to use netperf to measure the performance of |
|
bidirectional transfer. The first is to run concurrent netperf tests |
|
from the command line. The second is to configure netperf with |
|
@code{--enable-burst} and use a single instance of the |
|
@ref{TCP_RR,TCP_RR} test. |
|
|
|
While neither method is more ``correct'' than the other, each is doing |
|
so in different ways, and that has possible implications. For |
|
instance, using the concurrent netperf test mechanism means that |
|
multiple TCP connections and multiple processes are involved, whereas |
|
using the single instance of TCP_RR there is only one TCP connection |
|
and one process on each end. They may behave differently, especially |
|
on an MP system. |
|
|
|
@menu |
|
* Bidirectional Transfer with Concurrent Tests:: |
|
* Bidirectional Transfer with TCP_RR:: |
|
* Implications of Concurrent Tests vs Burst Request/Response:: |
|
@end menu |
|
|
|
@node Bidirectional Transfer with Concurrent Tests, Bidirectional Transfer with TCP_RR, Using Netperf to Measure Bidirectional Transfer, Using Netperf to Measure Bidirectional Transfer |
|
@comment node-name, next, previous, up |
|
@section Bidirectional Transfer with Concurrent Tests |
|
|
|
If we had two hosts Fred and Ethel, we could simply run a netperf |
|
@ref{TCP_STREAM,TCP_STREAM} test on Fred pointing at Ethel, and a |
|
concurrent netperf TCP_STREAM test on Ethel pointing at Fred, but |
|
since there are no mechanisms to synchronize netperf tests and we |
|
would be starting tests from two different systems, there is a |
|
considerable risk of skew error. |
|
|
|
Far better would be to run simultaneous TCP_STREAM and |
|
@ref{TCP_MAERTS,TCP_MAERTS} tests from just @b{one} system, using the |
|
concepts and procedures outlined in @ref{Running Concurrent Netperf |
|
Tests,Running Concurrent Netperf Tests}. Here then is an example: |
|
|
|
@example |
|
for i in 1 |
|
do |
|
netperf -H 192.168.2.108 -t TCP_STREAM -B "outbound" -i 10 -P 0 -v 0 \ |
|
-- -s 256K -S 256K & |
|
netperf -H 192.168.2.108 -t TCP_MAERTS -B "inbound" -i 10 -P 0 -v 0 \ |
|
-- -s 256K -S 256K & |
|
done |
|
|
|
892.66 outbound |
|
891.34 inbound |
|
@end example |
|
|
|
We have used a @code{for} loop in the shell with just one iteration |
|
because that will be @b{much} easier to get both tests started at more or |
|
less the same time than doing it by hand. The global @option{-P} and |
|
@option{-v} options are used because we aren't interested in anything |
|
other than the throughput, and the global @option{-B} option is used |
|
to tag each output so we know which was inbound and which outbound |
|
relative to the system on which we were running netperf. Of course |
|
that sense is switched on the system running netserver :) The use of |
|
the global @option{-i} option is explained in @ref{Running Concurrent |
|
Netperf Tests,Running Concurrent Netperf Tests}. |
|
|
|
Beginning with version 2.5.0 we can accomplish a similar result with |
|
the @ref{The Omni Tests,the omni tests} and @ref{Omni Output |
|
Selectors,output selectors}: |
|
|
|
@example |
|
for i in 1 |
|
do |
|
netperf -H 192.168.1.3 -t omni -l 10 -P 0 -- \ |
|
-d stream -s 256K -S 256K -o throughput,direction & |
|
netperf -H 192.168.1.3 -t omni -l 10 -P 0 -- \ |
|
-d maerts -s 256K -S 256K -o throughput,direction & |
|
done |
|
|
|
805.26,Receive |
|
828.54,Send |
|
@end example |
|
|
|
@node Bidirectional Transfer with TCP_RR, Implications of Concurrent Tests vs Burst Request/Response, Bidirectional Transfer with Concurrent Tests, Using Netperf to Measure Bidirectional Transfer |
|
@comment node-name, next, previous, up |
|
@section Bidirectional Transfer with TCP_RR |
|
|
|
Starting with version 2.5.0 the @code{--enable-burst} configure option |
|
defaults to @code{yes}, and starting some time before version 2.5.0 |
|
but after 2.4.0 the global @option{-f} option would affect the |
|
``throughput'' reported by request/response tests. If one uses the |
|
test-specific @option{-b} option to have several ``transactions'' in |
|
flight at one time and the test-specific @option{-r} option to |
|
increase their size, the test looks more and more like a |
|
single-connection bidirectional transfer than a simple |
|
request/response test. |
|
|
|
So, putting it all together one can do something like: |
|
|
|
@example |
|
netperf -f m -t TCP_RR -H 192.168.1.3 -v 2 -- -b 6 -r 32K -S 256K -S 256K |
|
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.3 (192.168.1.3) port 0 AF_INET : interval : first burst 6 |
|
Local /Remote |
|
Socket Size Request Resp. Elapsed |
|
Send Recv Size Size Time Throughput |
|
bytes Bytes bytes bytes secs. 10^6bits/sec |
|
|
|
16384 87380 32768 32768 10.00 1821.30 |
|
524288 524288 |
|
Alignment Offset RoundTrip Trans Throughput |
|
Local Remote Local Remote Latency Rate 10^6bits/s |
|
Send Recv Send Recv usec/Tran per sec Outbound Inbound |
|
8 0 0 0 2015.402 3473.252 910.492 910.492 |
|
@end example |
|
|
|
to get a bidirectional bulk-throughput result. As one can see, the -v |
|
2 output will include a number of interesting, related values. |
|
|
|
@quotation |
|
@b{NOTE: The logic behind @code{--enable-burst} is very simple, and there |
|
are no calls to @code{poll()} or @code{select()} which means we want |
|
to make sure that the @code{send()} calls will never block, or we run |
|
the risk of deadlock with each side stuck trying to call @code{send()} |
|
and neither calling @code{recv()}.} |
|
@end quotation |
|
|
|
Fortunately, this is easily accomplished by setting a ``large enough'' |
|
socket buffer size with the test-specific @option{-s} and @option{-S} |
|
options. Presently this must be performed by the user. Future |
|
versions of netperf might attempt to do this automagically, but there |
|
are some issues to be worked-out. |
|
|
|
@node Implications of Concurrent Tests vs Burst Request/Response, , Bidirectional Transfer with TCP_RR, Using Netperf to Measure Bidirectional Transfer |
|
@section Implications of Concurrent Tests vs Burst Request/Response |
|
|
|
There are perhaps subtle but important differences between using |
|
concurrent unidirectional tests vs a burst-mode request to measure |
|
bidirectional performance. |
|
|
|
Broadly speaking, a single ``connection'' or ``flow'' of traffic |
|
cannot make use of the services of more than one or two CPUs at either |
|
end. Whether one or two CPUs will be used processing a flow will |
|
depend on the specifics of the stack(s) involved and whether or not |
|
the global @option{-T} option has been used to bind netperf/netserver |
|
to specific CPUs. |
|
|
|
When using concurrent tests there will be two concurrent connections |
|
or flows, which means that upwards of four CPUs will be employed |
|
processing the packets (global @option{-T} used, no more than two if |
|
not), however, with just a single, bidirectional request/response test |
|
no more than two CPUs will be employed (only one if the global |
|
@option{-T} is not used). |
|
|
|
If there is a CPU bottleneck on either system this may result in |
|
rather different results between the two methods. |
|
|
|
Also, with a bidirectional request/response test there is something of |
|
a natural balance or synchronization between inbound and outbound - a |
|
response will not be sent until a request is received, and (once the |
|
burst level is reached) a subsequent request will not be sent until a |
|
response is received. This may mask favoritism in the NIC between |
|
inbound and outbound processing. |
|
|
|
With two concurrent unidirectional tests there is no such |
|
synchronization or balance and any favoritism in the NIC may be exposed. |
|
|
|
@node The Omni Tests, Other Netperf Tests, Using Netperf to Measure Bidirectional Transfer, Top |
|
@chapter The Omni Tests |
|
|
|
Beginning with version 2.5.0, netperf begins a migration to the |
|
@samp{omni} tests or ``Two routines to measure them all.'' The code for |
|
the omni tests can be found in @file{src/nettest_omni.c} and the goal |
|
is to make it easier for netperf to support multiple protocols and |
|
report a great many additional things about the systems under test. |
|
Additionally, a flexible output selection mechanism is present which |
|
allows the user to chose specifically what values she wishes to have |
|
reported and in what format. |
|
|
|
The omni tests are included by default in version 2.5.0. To disable |
|
them, one must: |
|
@example |
|
./configure --enable-omni=no ... |
|
@end example |
|
|
|
and remake netperf. Remaking netserver is optional because even in |
|
2.5.0 it has ``unmigrated'' netserver side routines for the classic |
|
(eg @file{src/nettest_bsd.c}) tests. |
|
|
|
@menu |
|
* Native Omni Tests:: |
|
* Migrated Tests:: |
|
* Omni Output Selection:: |
|
@end menu |
|
|
|
@node Native Omni Tests, Migrated Tests, The Omni Tests, The Omni Tests |
|
@section Native Omni Tests |
|
|
|
One access the omni tests ``natively'' by using a value of ``OMNI'' |
|
with the global @option{-t} test-selection option. This will then |
|
cause netperf to use the code in @file{src/nettest_omni.c} and in |
|
particular the test-specific options parser for the omni tests. The |
|
test-specific options for the omni tests are a superset of those for |
|
``classic'' tests. The options added by the omni tests are: |
|
|
|
@table @code |
|
@vindex -c, Test-specific |
|
@item -c |
|
This explicitly declares that the test is to include connection |
|
establishment and tear-down as in either a TCP_CRR or TCP_CC test. |
|
|
|
@vindex -d, Test-specific |
|
@item -d <direction> |
|
This option sets the direction of the test relative to the netperf |
|
process. As of version 2.5.0 one can use the following in a |
|
case-insensitive manner: |
|
|
|
@table @code |
|
@item send, stream, transmit, xmit or 2 |
|
Any of which will cause netperf to send to the netserver. |
|
@item recv, receive, maerts or 4 |
|
Any of which will cause netserver to send to netperf. |
|
@item rr or 6 |
|
Either of which will cause a request/response test. |
|
@end table |
|
|
|
Additionally, one can specify two directions separated by a '|' |
|
character and they will be OR'ed together. In this way one can use |
|
the ''Send|Recv'' that will be emitted by the @ref{Omni Output |
|
Selectors,DIRECTION} @ref{Omni Output Selection,output selector} when |
|
used with a request/response test. |
|
|
|
@vindex -k, Test-specific |
|
@item -k [@ref{Omni Output Selection,output selector}] |
|
This option sets the style of output to ``keyval'' where each line of |
|
output has the form: |
|
@example |
|
key=value |
|
@end example |
|
For example: |
|
@example |
|
$ netperf -t omni -- -d rr -k "THROUGHPUT,THROUGHPUT_UNITS" |
|
OMNI TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost.localdomain (127.0.0.1) port 0 AF_INET : demo |
|
THROUGHPUT=59092.65 |
|
THROUGHPUT_UNITS=Trans/s |
|
@end example |
|
|
|
Using the @option{-k} option will override any previous, test-specific |
|
@option{-o} or @option{-O} option. |
|
|
|
@vindex -o, Test-specific |
|
@item -o [@ref{Omni Output Selection,output selector}] |
|
This option sets the style of output to ``CSV'' where there will be |
|
one line of comma-separated values, preceded by one line of column |
|
names unless the global @option{-P} option is used with a value of 0: |
|
@example |
|
$ netperf -t omni -- -d rr -o "THROUGHPUT,THROUGHPUT_UNITS" |
|
OMNI TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost.localdomain (127.0.0.1) port 0 AF_INET : demo |
|
Throughput,Throughput Units |
|
60999.07,Trans/s |
|
@end example |
|
|
|
Using the @option{-o} option will override any previous, test-specific |
|
@option{-k} or @option{-O} option. |
|
|
|
@vindex -O, Test-specific |
|
@item -O [@ref{Omni Output Selection,output selector}] |
|
This option sets the style of output to ``human readable'' which will |
|
look quite similar to classic netperf output: |
|
@example |
|
$ netperf -t omni -- -d rr -O "THROUGHPUT,THROUGHPUT_UNITS" |
|
OMNI TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost.localdomain (127.0.0.1) port 0 AF_INET : demo |
|
Throughput Throughput |
|
Units |
|
|
|
|
|
60492.57 Trans/s |
|
@end example |
|
|
|
Using the @option{-O} option will override any previous, test-specific |
|
@option{-k} or @option{-o} option. |
|
|
|
@vindex -t, Test-specific |
|
@item -t |
|
This option explicitly sets the socket type for the test's data |
|
connection. As of version 2.5.0 the known socket types include |
|
``stream'' and ``dgram'' for SOCK_STREAM and SOCK_DGRAM respectively. |
|
|
|
@vindex -T, Test-specific |
|
@item -T <protocol> |
|
This option is used to explicitly set the protocol used for the |
|
test. It is case-insensitive. As of version 2.5.0 the protocols known |
|
to netperf include: |
|
@table @code |
|
@item TCP |
|
Select the Transmission Control Protocol |
|
@item UDP |
|
Select the User Datagram Protocol |
|
@item SDP |
|
Select the Sockets Direct Protocol |
|
@item DCCP |
|
Select the Datagram Congestion Control Protocol |
|
@item SCTP |
|
Select the Stream Control Transport Protocol |
|
@item udplite |
|
Select UDP Lite |
|
@end table |
|
|
|
The default is implicit based on other settings. |
|
@end table |
|
|
|
The omni tests also extend the interpretation of some of the classic, |
|
test-specific options for the BSD Sockets tests: |
|
|
|
@table @code |
|
@item -m <optionspec> |
|
This can set the send size for either or both of the netperf and |
|
netserver sides of the test: |
|
@example |
|
-m 32K |
|
@end example |
|
sets only the netperf-side send size to 32768 bytes, and or's-in |
|
transmit for the direction. This is effectively the same behaviour as |
|
for the classic tests. |
|
@example |
|
-m ,32K |
|
@end example |
|
sets only the netserver side send size to 32768 bytes and or's-in |
|
receive for the direction. |
|
@example |
|
-m 16K,32K |
|
sets the netperf side send size to 16284 bytes, the netserver side |
|
send size to 32768 bytes and the direction will be "Send|Recv." |
|
@end example |
|
@item -M <optionspec> |
|
This can set the receive size for either or both of the netperf and |
|
netserver sides of the test: |
|
@example |
|
-M 32K |
|
@end example |
|
sets only the netserver side receive size to 32768 bytes and or's-in |
|
send for the test direction. |
|
@example |
|
-M ,32K |
|
@end example |
|
sets only the netperf side receive size to 32768 bytes and or's-in |
|
receive for the test direction. |
|
@example |
|
-M 16K,32K |
|
@end example |
|
sets the netserver side receive size to 16384 bytes and the netperf |
|
side receive size to 32768 bytes and the direction will be "Send|Recv." |
|
@end table |
|
|
|
@node Migrated Tests, Omni Output Selection, Native Omni Tests, The Omni Tests |
|
@section Migrated Tests |
|
|
|
As of version 2.5.0 several tests have been migrated to use the omni |
|
code in @file{src/nettest_omni.c} for the core of their testing. A |
|
migrated test retains all its previous output code and so should still |
|
``look and feel'' just like a pre-2.5.0 test with one exception - the |
|
first line of the test banners will include the word ``MIGRATED'' at |
|
the beginning as in: |
|
|
|
@example |
|
$ netperf |
|
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost.localdomain (127.0.0.1) port 0 AF_INET : demo |
|
Recv Send Send |
|
Socket Socket Message Elapsed |
|
Size Size Size Time Throughput |
|
bytes bytes bytes secs. 10^6bits/sec |
|
|
|
87380 16384 16384 10.00 27175.27 |
|
@end example |
|
|
|
The tests migrated in version 2.5.0 are: |
|
@itemize |
|
@item TCP_STREAM |
|
@item TCP_MAERTS |
|
@item TCP_RR |
|
@item TCP_CRR |
|
@item UDP_STREAM |
|
@item UDP_RR |
|
@end itemize |
|
|
|
It is expected that future releases will have additional tests |
|
migrated to use the ``omni'' functionality. |
|
|
|
If one uses ``omni-specific'' test-specific options in conjunction |
|
with a migrated test, instead of using the classic output code, the |
|
new omni output code will be used. For example if one uses the |
|
@option{-k} test-specific option with a value of |
|
``MIN_LATENCY,MAX_LATENCY'' with a migrated TCP_RR test one will see: |
|
|
|
@example |
|
$ netperf -t tcp_rr -- -k THROUGHPUT,THROUGHPUT_UNITS |
|
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost.localdomain (127.0.0.1) port 0 AF_INET : demo |
|
THROUGHPUT=60074.74 |
|
THROUGHPUT_UNITS=Trans/s |
|
@end example |
|
rather than: |
|
@example |
|
$ netperf -t tcp_rr |
|
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost.localdomain (127.0.0.1) port 0 AF_INET : demo |
|
Local /Remote |
|
Socket Size Request Resp. Elapsed Trans. |
|
Send Recv Size Size Time Rate |
|
bytes Bytes bytes bytes secs. per sec |
|
|
|
16384 87380 1 1 10.00 59421.52 |
|
16384 87380 |
|
@end example |
|
|
|
@node Omni Output Selection, , Migrated Tests, The Omni Tests |
|
@section Omni Output Selection |
|
|
|
The omni test-specific @option{-k}, @option{-o} and @option{-O} |
|
options take an optional @code{output selector} by which the user can |
|
configure what values are reported. The output selector can take |
|
several forms: |
|
|
|
@table @code |
|
@item @file{filename} |
|
The output selections will be read from the named file. Within the |
|
file there can be up to four lines of comma-separated output |
|
selectors. This controls how many multi-line blocks of output are emitted |
|
when the @option{-O} option is used. This output, while not identical to |
|
``classic'' netperf output, is inspired by it. Multiple lines have no |
|
effect for @option{-k} and @option{-o} options. Putting output |
|
selections in a file can be useful when the list of selections is long. |
|
@item comma and/or semi-colon-separated list |
|
The output selections will be parsed from a comma and/or |
|
semi-colon-separated list of output selectors. When the list is given |
|
to a @option{-O} option a semi-colon specifies a new output block |
|
should be started. Semi-colons have the same meaning as commas when |
|
used with the @option{-k} or @option{-o} options. Depending on the |
|
command interpreter being used, the semi-colon may have to be escaped |
|
somehow to keep it from being interpreted by the command interpreter. |
|
This can often be done by enclosing the entire list in quotes. |
|
@item all |
|
If the keyword @b{all} is specified it means that all known output |
|
values should be displayed at the end of the test. This can be a |
|
great deal of output. As of version 2.5.0 there are 157 different |
|
output selectors. |
|
@item ? |
|
If a ``?'' is given as the output selection, the list of all known |
|
output selectors will be displayed and no test actually run. When |
|
passed to the @option{-O} option they will be listed one per |
|
line. Otherwise they will be listed as a comma-separated list. It may |
|
be necessary to protect the ``?'' from the command interpreter by |
|
escaping it or enclosing it in quotes. |
|
@item no selector |
|
If nothing is given to the @option{-k}, @option{-o} or @option{-O} |
|
option then the code selects a default set of output selectors |
|
inspired by classic netperf output. The format will be the @samp{human |
|
readable} format emitted by the test-specific @option{-O} option. |
|
@end table |
|
|
|
The order of evaluation will first check for an output selection. If |
|
none is specified with the @option{-k}, @option{-o} or @option{-O} |
|
option netperf will select a default based on the characteristics of the |
|
test. If there is an output selection, the code will first check for |
|
@samp{?}, then check to see if it is the magic @samp{all} keyword. |
|
After that it will check for either @samp{,} or @samp{;} in the |
|
selection and take that to mean it is a comma and/or |
|
semi-colon-separated list. If none of those checks match, netperf will then |
|
assume the output specification is a filename and attempt to open and |
|
parse the file. |
|
|
|
@menu |
|
* Omni Output Selectors:: |
|
@end menu |
|
|
|
@node Omni Output Selectors, , Omni Output Selection, Omni Output Selection |
|
@subsection Omni Output Selectors |
|
|
|
As of version 2.5.0 the output selectors are: |
|
|
|
@table @code |
|
@item OUTPUT_NONE |
|
This is essentially a null output. For @option{-k} output it will |
|
simply add a line that reads ``OUTPUT_NONE='' to the output. For |
|
@option{-o} it will cause an empty ``column'' to be included. For |
|
@option{-O} output it will cause extra spaces to separate ``real'' output. |
|
@item SOCKET_TYPE |
|
This will cause the socket type (eg SOCK_STREAM, SOCK_DGRAM) for the |
|
data connection to be output. |
|
@item PROTOCOL |
|
This will cause the protocol used for the data connection to be displayed. |
|
@item DIRECTION |
|
This will display the data flow direction relative to the netperf |
|
process. Units: Send or Recv for a unidirectional bulk-transfer test, |
|
or Send|Recv for a request/response test. |
|
@item ELAPSED_TIME |
|
This will display the elapsed time in seconds for the test. |
|
@item THROUGHPUT |
|
This will display the throughput for the test. Units: As requested via |
|
the global @option{-f} option and displayed by the THROUGHPUT_UNITS |
|
output selector. |
|
@item THROUGHPUT_UNITS |
|
This will display the units for what is displayed by the |
|
@code{THROUGHPUT} output selector. |
|
@item LSS_SIZE_REQ |
|
This will display the local (netperf) send socket buffer size (aka |
|
SO_SNDBUF) requested via the command line. Units: Bytes. |
|
@item LSS_SIZE |
|
This will display the local (netperf) send socket buffer size |
|
(SO_SNDBUF) immediately after the data connection socket was created. |
|
Peculiarities of different networking stacks may lead to this |
|
differing from the size requested via the command line. Units: Bytes. |
|
@item LSS_SIZE_END |
|
This will display the local (netperf) send socket buffer size |
|
(SO_SNDBUF) immediately before the data connection socket is closed. |
|
Peculiarities of different networking stacks may lead this to differ |
|
from the size requested via the command line and/or the size |
|
immediately after the data connection socket was created. Units: Bytes. |
|
@item LSR_SIZE_REQ |
|
This will display the local (netperf) receive socket buffer size (aka |
|
SO_RCVBUF) requested via the command line. Units: Bytes. |
|
@item LSR_SIZE |
|
This will display the local (netperf) receive socket buffer size |
|
(SO_RCVBUF) immediately after the data connection socket was created. |
|
Peculiarities of different networking stacks may lead to this |
|
differing from the size requested via the command line. Units: Bytes. |
|
@item LSR_SIZE_END |
|
This will display the local (netperf) receive socket buffer size |
|
(SO_RCVBUF) immediately before the data connection socket is closed. |
|
Peculiarities of different networking stacks may lead this to differ |
|
from the size requested via the command line and/or the size |
|
immediately after the data connection socket was created. Units: Bytes. |
|
@item RSS_SIZE_REQ |
|
This will display the remote (netserver) send socket buffer size (aka |
|
SO_SNDBUF) requested via the command line. Units: Bytes. |
|
@item RSS_SIZE |
|
This will display the remote (netserver) send socket buffer size |
|
(SO_SNDBUF) immediately after the data connection socket was created. |
|
Peculiarities of different networking stacks may lead to this |
|
differing from the size requested via the command line. Units: Bytes. |
|
@item RSS_SIZE_END |
|
This will display the remote (netserver) send socket buffer size |
|
(SO_SNDBUF) immediately before the data connection socket is closed. |
|
Peculiarities of different networking stacks may lead this to differ |
|
from the size requested via the command line and/or the size |
|
immediately after the data connection socket was created. Units: Bytes. |
|
@item RSR_SIZE_REQ |
|
This will display the remote (netserver) receive socket buffer size (aka |
|
SO_RCVBUF) requested via the command line. Units: Bytes. |
|
@item RSR_SIZE |
|
This will display the remote (netserver) receive socket buffer size |
|
(SO_RCVBUF) immediately after the data connection socket was created. |
|
Peculiarities of different networking stacks may lead to this |
|
differing from the size requested via the command line. Units: Bytes. |
|
@item RSR_SIZE_END |
|
This will display the remote (netserver) receive socket buffer size |
|
(SO_RCVBUF) immediately before the data connection socket is closed. |
|
Peculiarities of different networking stacks may lead this to differ |
|
from the size requested via the command line and/or the size |
|
immediately after the data connection socket was created. Units: Bytes. |
|
@item LOCAL_SEND_SIZE |
|
This will display the size of the buffers netperf passed in any |
|
``send'' calls it made on the data connection for a |
|
non-request/response test. Units: Bytes. |
|
@item LOCAL_RECV_SIZE |
|
This will display the size of the buffers netperf passed in any |
|
``receive'' calls it made on the data connection for a |
|
non-request/response test. Units: Bytes. |
|
@item REMOTE_SEND_SIZE |
|
This will display the size of the buffers netserver passed in any |
|
``send'' calls it made on the data connection for a |
|
non-request/response test. Units: Bytes. |
|
@item REMOTE_RECV_SIZE |
|
This will display the size of the buffers netserver passed in any |
|
``receive'' calls it made on the data connection for a |
|
non-request/response test. Units: Bytes. |
|
@item REQUEST_SIZE |
|
This will display the size of the requests netperf sent in a |
|
request-response test. Units: Bytes. |
|
@item RESPONSE_SIZE |
|
This will display the size of the responses netserver sent in a |
|
request-response test. Units: Bytes. |
|
@item LOCAL_CPU_UTIL |
|
This will display the overall CPU utilization during the test as |
|
measured by netperf. Units: 0 to 100 percent. |
|
@item LOCAL_CPU_PERCENT_USER |
|
This will display the CPU fraction spent in user mode during the test |
|
as measured by netperf. Only supported by netcpu_procstat. Units: 0 to |
|
100 percent. |
|
@item LOCAL_CPU_PERCENT_SYSTEM |
|
This will display the CPU fraction spent in system mode during the test |
|
as measured by netperf. Only supported by netcpu_procstat. Units: 0 to |
|
100 percent. |
|
@item LOCAL_CPU_PERCENT_IOWAIT |
|
This will display the fraction of time waiting for I/O to complete |
|
during the test as measured by netperf. Only supported by |
|
netcpu_procstat. Units: 0 to 100 percent. |
|
@item LOCAL_CPU_PERCENT_IRQ |
|
This will display the fraction of time servicing interrupts during the |
|
test as measured by netperf. Only supported by netcpu_procstat. Units: |
|
0 to 100 percent. |
|
@item LOCAL_CPU_PERCENT_SWINTR |
|
This will display the fraction of time servicing softirqs during the |
|
test as measured by netperf. Only supported by netcpu_procstat. Units: |
|
0 to 100 percent. |
|
@item LOCAL_CPU_METHOD |
|
This will display the method used by netperf to measure CPU |
|
utilization. Units: single character denoting method. |
|
@item LOCAL_SD |
|
This will display the service demand, or units of CPU consumed per |
|
unit of work, as measured by netperf. Units: microseconds of CPU |
|
consumed per either KB (K==1024) of data transferred or request/response |
|
transaction. |
|
@item REMOTE_CPU_UTIL |
|
This will display the overall CPU utilization during the test as |
|
measured by netserver. Units 0 to 100 percent. |
|
@item REMOTE_CPU_PERCENT_USER |
|
This will display the CPU fraction spent in user mode during the test |
|
as measured by netserver. Only supported by netcpu_procstat. Units: 0 to |
|
100 percent. |
|
@item REMOTE_CPU_PERCENT_SYSTEM |
|
This will display the CPU fraction spent in system mode during the test |
|
as measured by netserver. Only supported by netcpu_procstat. Units: 0 to |
|
100 percent. |
|
@item REMOTE_CPU_PERCENT_IOWAIT |
|
This will display the fraction of time waiting for I/O to complete |
|
during the test as measured by netserver. Only supported by |
|
netcpu_procstat. Units: 0 to 100 percent. |
|
@item REMOTE_CPU_PERCENT_IRQ |
|
This will display the fraction of time servicing interrupts during the |
|
test as measured by netserver. Only supported by netcpu_procstat. Units: |
|
0 to 100 percent. |
|
@item REMOTE_CPU_PERCENT_SWINTR |
|
This will display the fraction of time servicing softirqs during the |
|
test as measured by netserver. Only supported by netcpu_procstat. Units: |
|
0 to 100 percent. |
|
@item REMOTE_CPU_METHOD |
|
This will display the method used by netserver to measure CPU |
|
utilization. Units: single character denoting method. |
|
@item REMOTE_SD |
|
This will display the service demand, or units of CPU consumed per |
|
unit of work, as measured by netserver. Units: microseconds of CPU |
|
consumed per either KB (K==1024) of data transferred or |
|
request/response transaction. |
|
@item SD_UNITS |
|
This will display the units for LOCAL_SD and REMOTE_SD |
|
@item CONFIDENCE_LEVEL |
|
This will display the confidence level requested by the user either |
|
explicitly via the global @option{-I} option, or implicitly via the |
|
global @option{-i} option. The value will be either 95 or 99 if |
|
confidence intervals have been requested or 0 if they were not. Units: |
|
Percent |
|
@item CONFIDENCE_INTERVAL |
|
This will display the width of the confidence interval requested |
|
either explicitly via the global @option{-I} option or implicitly via |
|
the global @option{-i} option. Units: Width in percent of mean value |
|
computed. A value of -1.0 means that confidence intervals were not requested. |
|
@item CONFIDENCE_ITERATION |
|
This will display the number of test iterations netperf undertook, |
|
perhaps while attempting to achieve the requested confidence interval |
|
and level. If confidence intervals were requested via the command line |
|
then the value will be between 3 and 30. If confidence intervals were |
|
not requested the value will be 1. Units: Iterations |
|
@item THROUGHPUT_CONFID |
|
This will display the width of the confidence interval actually |
|
achieved for @code{THROUGHPUT} during the test. Units: Width of |
|
interval as percentage of reported throughput value. |
|
@item LOCAL_CPU_CONFID |
|
This will display the width of the confidence interval actually |
|
achieved for overall CPU utilization on the system running netperf |
|
(@code{LOCAL_CPU_UTIL}) during the test, if CPU utilization measurement |
|
was enabled. Units: Width of interval as percentage of reported CPU |
|
utilization. |
|
@item REMOTE_CPU_CONFID |
|
This will display the width of the confidence interval actually |
|
achieved for overall CPU utilization on the system running netserver |
|
(@code{REMOTE_CPU_UTIL}) during the test, if CPU utilization |
|
measurement was enabled. Units: Width of interval as percentage of |
|
reported CPU utilization. |
|
@item TRANSACTION_RATE |
|
This will display the transaction rate in transactions per second for |
|
a request/response test even if the user has requested a throughput in |
|
units of bits or bytes per second via the global @option{-f} |
|
option. It is undefined for a non-request/response test. Units: |
|
Transactions per second. |
|
@item RT_LATENCY |
|
This will display the average round-trip latency for a |
|
request/response test, accounting for number of transactions in flight |
|
at one time. It is undefined for a non-request/response test. Units: |
|
Microseconds per transaction |
|
@item BURST_SIZE |
|
This will display the ``burst size'' or added transactions in flight |
|
in a request/response test as requested via a test-specific |
|
@option{-b} option. The number of transactions in flight at one time |
|
will be one greater than this value. It is undefined for a |
|
non-request/response test. Units: added Transactions in flight. |
|
@item LOCAL_TRANSPORT_RETRANS |
|
This will display the number of retransmissions experienced on the |
|
data connection during the test as determined by netperf. A value of |
|
-1 means the attempt to determine the number of retransmissions failed |
|
or the concept was not valid for the given protocol or the mechanism |
|
is not known for the platform. A value of -2 means it was not |
|
attempted. As of version 2.5.0 the meaning of values are in flux and |
|
subject to change. Units: number of retransmissions. |
|
@item REMOTE_TRANSPORT_RETRANS |
|
This will display the number of retransmissions experienced on the |
|
data connection during the test as determined by netserver. A value |
|
of -1 means the attempt to determine the number of retransmissions |
|
failed or the concept was not valid for the given protocol or the |
|
mechanism is not known for the platform. A value of -2 means it was |
|
not attempted. As of version 2.5.0 the meaning of values are in flux |
|
and subject to change. Units: number of retransmissions. |
|
@item TRANSPORT_MSS |
|
This will display the Maximum Segment Size (aka MSS) or its equivalent |
|
for the protocol being used during the test. A value of -1 means |
|
either the concept of an MSS did not apply to the protocol being used, |
|
or there was an error in retrieving it. Units: Bytes. |
|
@item LOCAL_SEND_THROUGHPUT |
|
The throughput as measured by netperf for the successful ``send'' |
|
calls it made on the data connection. Units: as requested via the |
|
global @option{-f} option and displayed via the @code{THROUGHPUT_UNITS} |
|
output selector. |
|
@item LOCAL_RECV_THROUGHPUT |
|
The throughput as measured by netperf for the successful ``receive'' |
|
calls it made on the data connection. Units: as requested via the |
|
global @option{-f} option and displayed via the @code{THROUGHPUT_UNITS} |
|
output selector. |
|
@item REMOTE_SEND_THROUGHPUT |
|
The throughput as measured by netserver for the successful ``send'' |
|
calls it made on the data connection. Units: as requested via the |
|
global @option{-f} option and displayed via the @code{THROUGHPUT_UNITS} |
|
output selector. |
|
@item REMOTE_RECV_THROUGHPUT |
|
The throughput as measured by netserver for the successful ``receive'' |
|
calls it made on the data connection. Units: as requested via the |
|
global @option{-f} option and displayed via the @code{THROUGHPUT_UNITS} |
|
output selector. |
|
@item LOCAL_CPU_BIND |
|
The CPU to which netperf was bound, if at all, during the test. A |
|
value of -1 means that netperf was not explicitly bound to a CPU |
|
during the test. Units: CPU ID |
|
@item LOCAL_CPU_COUNT |
|
The number of CPUs (cores, threads) detected by netperf. Units: CPU count. |
|
@item LOCAL_CPU_PEAK_UTIL |
|
The utilization of the CPU most heavily utilized during the test, as |
|
measured by netperf. This can be used to see if any one CPU of a |
|
multi-CPU system was saturated even though the overall CPU utilization |
|
as reported by @code{LOCAL_CPU_UTIL} was low. Units: 0 to 100% |
|
@item LOCAL_CPU_PEAK_ID |
|
The id of the CPU most heavily utilized during the test as determined |
|
by netperf. Units: CPU ID. |
|
@item LOCAL_CPU_MODEL |
|
Model information for the processor(s) present on the system running |
|
netperf. Assumes all processors in the system (as perceived by |
|
netperf) on which netperf is running are the same model. Units: Text |
|
@item LOCAL_CPU_FREQUENCY |
|
The frequency of the processor(s) on the system running netperf, at |
|
the time netperf made the call. Assumes that all processors present |
|
in the system running netperf are running at the same |
|
frequency. Units: MHz |
|
@item REMOTE_CPU_BIND |
|
The CPU to which netserver was bound, if at all, during the test. A |
|
value of -1 means that netperf was not explicitly bound to a CPU |
|
during the test. Units: CPU ID |
|
@item REMOTE_CPU_COUNT |
|
The number of CPUs (cores, threads) detected by netserver. Units: CPU |
|
count. |
|
@item REMOTE_CPU_PEAK_UTIL |
|
The utilization of the CPU most heavily utilized during the test, as |
|
measured by netserver. This can be used to see if any one CPU of a |
|
multi-CPU system was saturated even though the overall CPU utilization |
|
as reported by @code{REMOTE_CPU_UTIL} was low. Units: 0 to 100% |
|
@item REMOTE_CPU_PEAK_ID |
|
The id of the CPU most heavily utilized during the test as determined |
|
by netserver. Units: CPU ID. |
|
@item REMOTE_CPU_MODEL |
|
Model information for the processor(s) present on the system running |
|
netserver. Assumes all processors in the system (as perceived by |
|
netserver) on which netserver is running are the same model. Units: |
|
Text |
|
@item REMOTE_CPU_FREQUENCY |
|
The frequency of the processor(s) on the system running netserver, at |
|
the time netserver made the call. Assumes that all processors present |
|
in the system running netserver are running at the same |
|
frequency. Units: MHz |
|
@item SOURCE_PORT |
|
The port ID/service name to which the data socket created by netperf |
|
was bound. A value of 0 means the data socket was not explicitly |
|
bound to a port number. Units: ASCII text. |
|
@item SOURCE_ADDR |
|
The name/address to which the data socket created by netperf was |
|
bound. A value of 0.0.0.0 means the data socket was not explicitly |
|
bound to an address. Units: ASCII text. |
|
@item SOURCE_FAMILY |
|
The address family to which the data socket created by netperf was |
|
bound. A value of 0 means the data socket was not explicitly bound to |
|
a given address family. Units: ASCII text. |
|
@item DEST_PORT |
|
The port ID to which the data socket created by netserver was bound. A |
|
value of 0 means the data socket was not explicitly bound to a port |
|
number. Units: ASCII text. |
|
@item DEST_ADDR |
|
The name/address of the data socket created by netserver. Units: |
|
ASCII text. |
|
@item DEST_FAMILY |
|
The address family to which the data socket created by netserver was |
|
bound. A value of 0 means the data socket was not explicitly bound to |
|
a given address family. Units: ASCII text. |
|
@item LOCAL_SEND_CALLS |
|
The number of successful ``send'' calls made by netperf against its |
|
data socket. Units: Calls. |
|
@item LOCAL_RECV_CALLS |
|
The number of successful ``receive'' calls made by netperf against its |
|
data socket. Units: Calls. |
|
@item LOCAL_BYTES_PER_RECV |
|
The average number of bytes per ``receive'' call made by netperf |
|
against its data socket. Units: Bytes. |
|
@item LOCAL_BYTES_PER_SEND |
|
The average number of bytes per ``send'' call made by netperf against |
|
its data socket. Units: Bytes. |
|
@item LOCAL_BYTES_SENT |
|
The number of bytes successfully sent by netperf through its data |
|
socket. Units: Bytes. |
|
@item LOCAL_BYTES_RECVD |
|
The number of bytes successfully received by netperf through its data |
|
socket. Units: Bytes. |
|
@item LOCAL_BYTES_XFERD |
|
The sum of bytes sent and received by netperf through its data |
|
socket. Units: Bytes. |
|
@item LOCAL_SEND_OFFSET |
|
The offset from the alignment of the buffers passed by netperf in its |
|
``send'' calls. Specified via the global @option{-o} option and |
|
defaults to 0. Units: Bytes. |
|
@item LOCAL_RECV_OFFSET |
|
The offset from the alignment of the buffers passed by netperf in its |
|
``receive'' calls. Specified via the global @option{-o} option and |
|
defaults to 0. Units: Bytes. |
|
@item LOCAL_SEND_ALIGN |
|
The alignment of the buffers passed by netperf in its ``send'' calls |
|
as specified via the global @option{-a} option. Defaults to 8. Units: |
|
Bytes. |
|
@item LOCAL_RECV_ALIGN |
|
The alignment of the buffers passed by netperf in its ``receive'' |
|
calls as specified via the global @option{-a} option. Defaults to |
|
8. Units: Bytes. |
|
@item LOCAL_SEND_WIDTH |
|
The ``width'' of the ring of buffers through which netperf cycles as |
|
it makes its ``send'' calls. Defaults to one more than the local send |
|
socket buffer size divided by the send size as determined at the time |
|
the data socket is created. Can be used to make netperf more processor |
|
data cache unfriendly. Units: number of buffers. |
|
@item LOCAL_RECV_WIDTH |
|
The ``width'' of the ring of buffers through which netperf cycles as |
|
it makes its ``receive'' calls. Defaults to one more than the local |
|
receive socket buffer size divided by the receive size as determined |
|
at the time the data socket is created. Can be used to make netperf |
|
more processor data cache unfriendly. Units: number of buffers. |
|
@item LOCAL_SEND_DIRTY_COUNT |
|
The number of bytes to ``dirty'' (write to) before netperf makes a |
|
``send'' call. Specified via the global @option{-k} option, which |
|
requires that --enable-dirty=yes was specified with the configure |
|
command prior to building netperf. Units: Bytes. |
|
@item LOCAL_RECV_DIRTY_COUNT |
|
The number of bytes to ``dirty'' (write to) before netperf makes a |
|
``recv'' call. Specified via the global @option{-k} option which |
|
requires that --enable-dirty was specified with the configure command |
|
prior to building netperf. Units: Bytes. |
|
@item LOCAL_RECV_CLEAN_COUNT |
|
The number of bytes netperf should read ``cleanly'' before making a |
|
``receive'' call. Specified via the global @option{-k} option which |
|
requires that --enable-dirty was specified with configure command |
|
prior to building netperf. Clean reads start were dirty writes ended. |
|
Units: Bytes. |
|
@item LOCAL_NODELAY |
|
Indicates whether or not setting the test protocol-specific ``no |
|
delay'' (eg TCP_NODELAY) option on the data socket used by netperf was |
|
requested by the test-specific @option{-D} option and |
|
successful. Units: 0 means no, 1 means yes. |
|
@item LOCAL_CORK |
|
Indicates whether or not TCP_CORK was set on the data socket used by |
|
netperf as requested via the test-specific @option{-C} option. 1 means |
|
yes, 0 means no/not applicable. |
|
@item REMOTE_SEND_CALLS |
|
@item REMOTE_RECV_CALLS |
|
@item REMOTE_BYTES_PER_RECV |
|
@item REMOTE_BYTES_PER_SEND |
|
@item REMOTE_BYTES_SENT |
|
@item REMOTE_BYTES_RECVD |
|
@item REMOTE_BYTES_XFERD |
|
@item REMOTE_SEND_OFFSET |
|
@item REMOTE_RECV_OFFSET |
|
@item REMOTE_SEND_ALIGN |
|
@item REMOTE_RECV_ALIGN |
|
@item REMOTE_SEND_WIDTH |
|
@item REMOTE_RECV_WIDTH |
|
@item REMOTE_SEND_DIRTY_COUNT |
|
@item REMOTE_RECV_DIRTY_COUNT |
|
@item REMOTE_RECV_CLEAN_COUNT |
|
@item REMOTE_NODELAY |
|
@item REMOTE_CORK |
|
These are all like their ``LOCAL_'' counterparts only for the |
|
netserver rather than netperf. |
|
@item LOCAL_SYSNAME |
|
The name of the OS (eg ``Linux'') running on the system on which |
|
netperf was running. Units: ASCII Text |
|
@item LOCAL_SYSTEM_MODEL |
|
The model name of the system on which netperf was running. Units: |
|
ASCII Text. |
|
@item LOCAL_RELEASE |
|
The release name/number of the OS running on the system on which |
|
netperf was running. Units: ASCII Text |
|
@item LOCAL_VERSION |
|
The version number of the OS running on the system on which netperf |
|
was running. Units: ASCII Text |
|
@item LOCAL_MACHINE |
|
The machine architecture of the machine on which netperf was |
|
running. Units: ASCII Text. |
|
@item REMOTE_SYSNAME |
|
@item REMOTE_SYSTEM_MODEL |
|
@item REMOTE_RELEASE |
|
@item REMOTE_VERSION |
|
@item REMOTE_MACHINE |
|
These are all like their ``LOCAL_'' counterparts only for the |
|
netserver rather than netperf. |
|
@item LOCAL_INTERFACE_NAME |
|
The name of the probable egress interface through which the data |
|
connection went on the system running netperf. Example: eth0. Units: |
|
ASCII Text. |
|
@item LOCAL_INTERFACE_VENDOR |
|
The vendor ID of the probable egress interface through which traffic |
|
on the data connection went on the system running netperf. Units: |
|
Hexadecimal IDs as might be found in a @file{pci.ids} file or at |
|
@uref{http://pciids.sourceforge.net/,the PCI ID Repository}. |
|
@item LOCAL_INTERFACE_DEVICE |
|
The device ID of the probable egress interface through which traffic |
|
on the data connection went on the system running netperf. Units: |
|
Hexadecimal IDs as might be found in a @file{pci.ids} file or at |
|
@uref{http://pciids.sourceforge.net/,the PCI ID Repository}. |
|
@item LOCAL_INTERFACE_SUBVENDOR |
|
The sub-vendor ID of the probable egress interface through which |
|
traffic on the data connection went on the system running |
|
netperf. Units: Hexadecimal IDs as might be found in a @file{pci.ids} |
|
file or at @uref{http://pciids.sourceforge.net/,the PCI ID |
|
Repository}. |
|
@item LOCAL_INTERFACE_SUBDEVICE |
|
The sub-device ID of the probable egress interface through which |
|
traffic on the data connection went on the system running |
|
netperf. Units: Hexadecimal IDs as might be found in a @file{pci.ids} |
|
file or at @uref{http://pciids.sourceforge.net/,the PCI ID |
|
Repository}. |
|
@item LOCAL_DRIVER_NAME |
|
The name of the driver used for the probable egress interface through |
|
which traffic on the data connection went on the system running |
|
netperf. Units: ASCII Text. |
|
@item LOCAL_DRIVER_VERSION |
|
The version string for the driver used for the probable egress |
|
interface through which traffic on the data connection went on the |
|
system running netperf. Units: ASCII Text. |
|
@item LOCAL_DRIVER_FIRMWARE |
|
The firmware version for the driver used for the probable egress |
|
interface through which traffic on the data connection went on the |
|
system running netperf. Units: ASCII Text. |
|
@item LOCAL_DRIVER_BUS |
|
The bus address of the probable egress interface through which traffic |
|
on the data connection went on the system running netperf. Units: |
|
ASCII Text. |
|
@item LOCAL_INTERFACE_SLOT |
|
The slot ID of the probable egress interface through which traffic |
|
on the data connection went on the system running netperf. Units: |
|
ASCII Text. |
|
@item REMOTE_INTERFACE_NAME |
|
@item REMOTE_INTERFACE_VENDOR |
|
@item REMOTE_INTERFACE_DEVICE |
|
@item REMOTE_INTERFACE_SUBVENDOR |
|
@item REMOTE_INTERFACE_SUBDEVICE |
|
@item REMOTE_DRIVER_NAME |
|
@item REMOTE_DRIVER_VERSION |
|
@item REMOTE_DRIVER_FIRMWARE |
|
@item REMOTE_DRIVER_BUS |
|
@item REMOTE_INTERFACE_SLOT |
|
These are all like their ``LOCAL_'' counterparts only for the |
|
netserver rather than netperf. |
|
@item LOCAL_INTERVAL_USECS |
|
The interval at which bursts of operations (sends, receives, |
|
transactions) were attempted by netperf. Specified by the |
|
global @option{-w} option which requires --enable-intervals to have |
|
been specified with the configure command prior to building |
|
netperf. Units: Microseconds (though specified by default in |
|
milliseconds on the command line) |
|
@item LOCAL_INTERVAL_BURST |
|
The number of operations (sends, receives, transactions depending on |
|
the test) which were attempted by netperf each LOCAL_INTERVAL_USECS |
|
units of time. Specified by the global @option{-b} option which |
|
requires --enable-intervals to have been specified with the configure |
|
command prior to building netperf. Units: number of operations per burst. |
|
@item REMOTE_INTERVAL_USECS |
|
The interval at which bursts of operations (sends, receives, |
|
transactions) were attempted by netserver. Specified by the |
|
global @option{-w} option which requires --enable-intervals to have |
|
been specified with the configure command prior to building |
|
netperf. Units: Microseconds (though specified by default in |
|
milliseconds on the command line) |
|
@item REMOTE_INTERVAL_BURST |
|
The number of operations (sends, receives, transactions depending on |
|
the test) which were attempted by netperf each LOCAL_INTERVAL_USECS |
|
units of time. Specified by the global @option{-b} option which |
|
requires --enable-intervals to have been specified with the configure |
|
command prior to building netperf. Units: number of operations per burst. |
|
@item LOCAL_SECURITY_TYPE_ID |
|
@item LOCAL_SECURITY_TYPE |
|
@item LOCAL_SECURITY_ENABLED_NUM |
|
@item LOCAL_SECURITY_ENABLED |
|
@item LOCAL_SECURITY_SPECIFIC |
|
@item REMOTE_SECURITY_TYPE_ID |
|
@item REMOTE_SECURITY_TYPE |
|
@item REMOTE_SECURITY_ENABLED_NUM |
|
@item REMOTE_SECURITY_ENABLED |
|
@item REMOTE_SECURITY_SPECIFIC |
|
A bunch of stuff related to what sort of security mechanisms (eg |
|
SELINUX) were enabled on the systems during the test. |
|
@item RESULT_BRAND |
|
The string specified by the user with the global @option{-B} |
|
option. Units: ASCII Text. |
|
@item UUID |
|
The universally unique identifier associated with this test, either |
|
generated automagically by netperf, or passed to netperf via an omni |
|
test-specific @option{-u} option. Note: Future versions may make this |
|
a global command-line option. Units: ASCII Text. |
|
@item MIN_LATENCY |
|
The minimum ``latency'' or operation time (send, receive or |
|
request/response exchange depending on the test) as measured on the |
|
netperf side when the global @option{-j} option was specified. Units: |
|
Microseconds. |
|
@item MAX_LATENCY |
|
The maximum ``latency'' or operation time (send, receive or |
|
request/response exchange depending on the test) as measured on the |
|
netperf side when the global @option{-j} option was specified. Units: |
|
Microseconds. |
|
@item P50_LATENCY |
|
The 50th percentile value of ``latency'' or operation time (send, receive or |
|
request/response exchange depending on the test) as measured on the |
|
netperf side when the global @option{-j} option was specified. Units: |
|
Microseconds. |
|
@item P90_LATENCY |
|
The 90th percentile value of ``latency'' or operation time (send, receive or |
|
request/response exchange depending on the test) as measured on the |
|
netperf side when the global @option{-j} option was specified. Units: |
|
Microseconds. |
|
@item P99_LATENCY |
|
The 99th percentile value of ``latency'' or operation time (send, receive or |
|
request/response exchange depending on the test) as measured on the |
|
netperf side when the global @option{-j} option was specified. Units: |
|
Microseconds. |
|
@item MEAN_LATENCY |
|
The average ``latency'' or operation time (send, receive or |
|
request/response exchange depending on the test) as measured on the |
|
netperf side when the global @option{-j} option was specified. Units: |
|
Microseconds. |
|
@item STDDEV_LATENCY |
|
The standard deviation of ``latency'' or operation time (send, receive or |
|
request/response exchange depending on the test) as measured on the |
|
netperf side when the global @option{-j} option was specified. Units: |
|
Microseconds. |
|
@item COMMAND_LINE |
|
The full command line used when invoking netperf. Units: ASCII Text. |
|
@item OUTPUT_END |
|
While emitted with the list of output selectors, it is ignored when |
|
specified as an output selector. |
|
@end table |
|
|
|
@node Other Netperf Tests, Address Resolution, The Omni Tests, Top |
|
@chapter Other Netperf Tests |
|
|
|
Apart from the typical performance tests, netperf contains some tests |
|
which can be used to streamline measurements and reporting. These |
|
include CPU rate calibration (present) and host identification (future |
|
enhancement). |
|
|
|
@menu |
|
* CPU rate calibration:: |
|
* UUID Generation:: |
|
@end menu |
|
|
|
@node CPU rate calibration, UUID Generation, Other Netperf Tests, Other Netperf Tests |
|
@section CPU rate calibration |
|
|
|
Some of the CPU utilization measurement mechanisms of netperf work by |
|
comparing the rate at which some counter increments when the system is |
|
idle with the rate at which that same counter increments when the |
|
system is running a netperf test. The ratio of those rates is used to |
|
arrive at a CPU utilization percentage. |
|
|
|
This means that netperf must know the rate at which the counter |
|
increments when the system is presumed to be ``idle.'' If it does not |
|
know the rate, netperf will measure it before starting a data transfer |
|
test. This calibration step takes 40 seconds for each of the local or |
|
remote systems, and if repeated for each netperf test would make taking |
|
repeated measurements rather slow. |
|
|
|
Thus, the netperf CPU utilization options @option{-c} and and |
|
@option{-C} can take an optional calibration value. This value is |
|
used as the ``idle rate'' and the calibration step is not |
|
performed. To determine the idle rate, netperf can be used to run |
|
special tests which only report the value of the calibration - they |
|
are the LOC_CPU and REM_CPU tests. These return the calibration value |
|
for the local and remote system respectively. A common way to use |
|
these tests is to store their results into an environment variable and |
|
use that in subsequent netperf commands: |
|
|
|
@example |
|
LOC_RATE=`netperf -t LOC_CPU` |
|
REM_RATE=`netperf -H <remote> -t REM_CPU` |
|
netperf -H <remote> -c $LOC_RATE -C $REM_RATE ... -- ... |
|
... |
|
netperf -H <remote> -c $LOC_RATE -C $REM_RATE ... -- ... |
|
@end example |
|
|
|
If you are going to use netperf to measure aggregate results, it is |
|
important to use the LOC_CPU and REM_CPU tests to get the calibration |
|
values first to avoid issues with some of the aggregate netperf tests |
|
transferring data while others are ``idle'' and getting bogus |
|
calibration values. When running aggregate tests, it is very |
|
important to remember that any one instance of netperf does not know |
|
about the other instances of netperf. It will report global CPU |
|
utilization and will calculate service demand believing it was the |
|
only thing causing that CPU utilization. So, you can use the CPU |
|
utilization reported by netperf in an aggregate test, but you have to |
|
calculate service demands by hand. |
|
|
|
@node UUID Generation, , CPU rate calibration, Other Netperf Tests |
|
@section UUID Generation |
|
|
|
Beginning with version 2.5.0 netperf can generate Universally Unique |
|
IDentifiers (UUIDs). This can be done explicitly via the ``UUID'' |
|
test: |
|
@example |
|
$ netperf -t UUID |
|
2c8561ae-9ebd-11e0-a297-0f5bfa0349d0 |
|
@end example |
|
|
|
In and of itself, this is not terribly useful, but used in conjunction |
|
with the test-specific @option{-u} option of an ``omni'' test to set |
|
the UUID emitted by the @ref{Omni Output Selectors,UUID} output |
|
selector, it can be used to tie-together the separate instances of an |
|
aggregate netperf test. Say, for instance if they were inserted into |
|
a database of some sort. |
|
|
|
@node Address Resolution, Enhancing Netperf, Other Netperf Tests, Top |
|
@comment node-name, next, previous, up |
|
@chapter Address Resolution |
|
|
|
Netperf versions 2.4.0 and later have merged IPv4 and IPv6 tests so |
|
the functionality of the tests in @file{src/nettest_ipv6.c} has been |
|
subsumed into the tests in @file{src/nettest_bsd.c} This has been |
|
accomplished in part by switching from @code{gethostbyname()}to |
|
@code{getaddrinfo()} exclusively. While it was theoretically possible |
|
to get multiple results for a hostname from @code{gethostbyname()} it |
|
was generally unlikely and netperf's ignoring of the second and later |
|
results was not much of an issue. |
|
|
|
Now with @code{getaddrinfo} and particularly with AF_UNSPEC it is |
|
increasingly likely that a given hostname will have multiple |
|
associated addresses. The @code{establish_control()} routine of |
|
@file{src/netlib.c} will indeed attempt to chose from among all the |
|
matching IP addresses when establishing the control connection. |
|
Netperf does not _really_ care if the control connection is IPv4 or |
|
IPv6 or even mixed on either end. |
|
|
|
However, the individual tests still ass-u-me that the first result in |
|
the address list is the one to be used. Whether or not this will |
|
turn-out to be an issue has yet to be determined. |
|
|
|
If you do run into problems with this, the easiest workaround is to |
|
specify IP addresses for the data connection explicitly in the |
|
test-specific @option{-H} and @option{-L} options. At some point, the |
|
netperf tests _may_ try to be more sophisticated in their parsing of |
|
returns from @code{getaddrinfo()} - straw-man patches to |
|
@email{netperf-feedback@@netperf.org} would of course be most welcome |
|
:) |
|
|
|
Netperf has leveraged code from other open-source projects with |
|
amenable licensing to provide a replacement @code{getaddrinfo()} call |
|
on those platforms where the @command{configure} script believes there |
|
is no native getaddrinfo call. As of this writing, the replacement |
|
@code{getaddrinfo()} as been tested on HP-UX 11.0 and then presumed to |
|
run elsewhere. |
|
|
|
@node Enhancing Netperf, Netperf4, Address Resolution, Top |
|
@comment node-name, next, previous, up |
|
@chapter Enhancing Netperf |
|
|
|
Netperf is constantly evolving. If you find you want to make |
|
enhancements to netperf, by all means do so. If you wish to add a new |
|
``suite'' of tests to netperf the general idea is to: |
|
|
|
@enumerate |
|
@item |
|
Add files @file{src/nettest_mumble.c} and @file{src/nettest_mumble.h} |
|
where mumble is replaced with something meaningful for the test-suite. |
|
@item |
|
Add support for an appropriate @option{--enable-mumble} option in |
|
@file{configure.ac}. |
|
@item |
|
Edit @file{src/netperf.c}, @file{netsh.c}, and @file{netserver.c} as |
|
required, using #ifdef WANT_MUMBLE. |
|
@item |
|
Compile and test |
|
@end enumerate |
|
|
|
However, with the addition of the ``omni'' tests in version 2.5.0 it |
|
is preferred that one attempt to make the necessary changes to |
|
@file{src/nettest_omni.c} rather than adding new source files, unless |
|
this would make the omni tests entirely too complicated. |
|
|
|
If you wish to submit your changes for possible inclusion into the |
|
mainline sources, please try to base your changes on the latest |
|
available sources. (@xref{Getting Netperf Bits}.) and then send email |
|
describing the changes at a high level to |
|
@email{netperf-feedback@@netperf.org} or perhaps |
|
@email{netperf-talk@@netperf.org}. If the consensus is positive, then |
|
sending context @command{diff} results to |
|
@email{netperf-feedback@@netperf.org} is the next step. From that |
|
point, it is a matter of pestering the Netperf Contributing Editor |
|
until he gets the changes incorporated :) |
|
|
|
@node Netperf4, Concept Index, Enhancing Netperf, Top |
|
@comment node-name, next, previous, up |
|
@chapter Netperf4 |
|
|
|
Netperf4 is the shorthand name given to version 4.X.X of netperf. |
|
This is really a separate benchmark more than a newer version of |
|
netperf, but it is a descendant of netperf so the netperf name is |
|
kept. The facetious way to describe netperf4 is to say it is the |
|
egg-laying-woolly-milk-pig version of netperf :) The more respectful |
|
way to describe it is to say it is the version of netperf with support |
|
for synchronized, multiple-thread, multiple-test, multiple-system, |
|
network-oriented benchmarking. |
|
|
|
Netperf4 is still undergoing evolution. Those wishing to work with or |
|
on netperf4 are encouraged to join the |
|
@uref{http://www.netperf.org/cgi-bin/mailman/listinfo/netperf-dev,netperf-dev} |
|
mailing list and/or peruse the |
|
@uref{http://www.netperf.org/svn/netperf4/trunk,current sources}. |
|
|
|
@node Concept Index, Option Index, Netperf4, Top |
|
@unnumbered Concept Index |
|
|
|
@printindex cp |
|
|
|
@node Option Index, , Concept Index, Top |
|
@comment node-name, next, previous, up |
|
@unnumbered Option Index |
|
|
|
@printindex vr |
|
@bye |
|
|
|
@c LocalWords: texinfo setfilename settitle titlepage vskip pt filll ifnottex |
|
@c LocalWords: insertcopying cindex dfn uref printindex cp
|
|
|