With corporate Web sites getting millions of hits per day, the choice of
Web-server software can be critical. Web servers that don't respond
crisply under heavy loads can bog down network connections, deny service
for visitors, and cause network failures. · For administrators of
large sites or others seeking to differentiate among the wide variety of
Web-server software packages, the Standard Performance Evaluation
Corp.'s SPECweb96 benchmark can in many cases help determine which
Web-server software performs best on a particular set of hardware systems
and network connections. · Though SPEC officially released SPECweb96
1.0 in September, production problems with the Windows NT version delayed
release of the CD-ROM software until the end of December. This software can
evaluate the performance of Web-server software running on virtually any
Unix system or Windows NT platform.· Like SPEC's other
benchmarks, SPECweb96 is a standardized performance benchmark created in a
spirit of cooperation among competitors. As
Copyright© 1997 by CMP Media Inc., 600 Community Drive, Manhasset, NY
11030. Reprinted from INFORMATIONWEEK with permission.
a nonprofit organization, SPEC oversees the development and creation of
benchmarks by using the technical resources of member companies. The result
is a benchmark that has been examined by all interested parties and is
considered a fair test of performance by them.
For SPECweb96, companies as diverse as Digital Equipment, Hewlett-Packard,
IBM, Intel, Netscape Communications, Siemens Nixdorf, Silicon Graphics, Sun
Microsystems, and others helped shape the benchmark.
How It Works
A SPECweb96 test bed consists of a server machine that runs the Web-server
software to be tested and a set number of client machines. The client
machines use the SPECweb96 software to generate a workload that stresses
the server software. The workload is gradually increased until the server
software is saturated with hits and the response time degrades
The point at which the server is saturated is the maximum number of HTTP
operations per second that the Web-server software can sustain. That
maximum number of HTTP operations per second is the SPECweb96 performance
metric that is reported. For example, the SPECweb96 performance metric of
the sample machine in the table and graph on page 58 is 626 HTTP operations
Although SPECweb96 can do an admirable job of testing Web-server
performance, you may not want to rush out and pay $800 for the CD-ROM
without first considering what SPECweb96 is testing and what kind of test
bed you'll need. It can take some serious network resources to properly
test Web-server software, and your site may not be ready to take on such
software configuration responsibility. Also, as with most benchmarks,
you'll need close-to-laboratory conditions to make accurate comparisons
That means you can't expect to
test the server over a WAN such as the Internet, since it is difficult to
account for all possible network bottlenecks that might prevent saturation
of the Web-server software. To get accurate results, you need a relatively
quiet LAN between your test clients and server system.
How many client systems will you need to saturate your Web-server software
with hits? That answer depends on your configuration. For example, in
results reported to SPEC, Digital used two AlphaServer 4000 clients-each
with four 400-MHz CPUs and 256 Mbytes of RAM-to saturate the Zeus
Web-server software from Zeus Technology in England, running on an
AlphaServer 4000 with a single 400-MHz CPU.
To sustain the throughput necessary to saturate the Web server, the company
used a 100-Mbps FDDI LAN. This configuration produced an impressive
SPECweb96 performance metric of 1,157 HTTP operations per second.
On the other hand, to saturate Zeus Web-server software running on an HP
9000 with a single 132-MHz PA-RISC CPU, HP needed only three HP 9000
clients, each with a single 160-MHz CPU and 128 Mbytes of RAM. This
configuration produced a SPECweb96 performance metric of 426 HTTP
operations per second. In other words, you need enough client horsepower to
saturate the server system.
At the same time, you don't want the LAN to be a performance
bottleneck. If the network becomes saturated before the server system does,
you won't be able to adequately test the server.
You can determine the amount of network throughput you'll need by
taking a guess at your saturation mark and then inserting it into this
formula (saturation point * 6/1,000) to determine megabits per second. For
the HP machine mentioned above, the company needed a LAN that could sustain
2.6 Mbps (426 * 6/1,000). The SPECweb96 documentation has details on the
derivation of this formula.
A benchmark is just a benchmark. It doesn't tell you everything you
want to know about the tested systems, and sometimes it doesn't tell
you much at all. Fortunately, SPEC is very clear about the limitations of
the SPECweb96 benchmark.
Perhaps the biggest limitation is the nature of the workload. In the real
world, Web servers are hit with a multitude of different workloads. Some
Web-server software may spend most of its time handling CGI (Common Gateway
Interface) requests. Other Web-server software may be handling dynamic
The workload for SPECweb96 was determined by looking at log files for major
sites such as NCSA, HP, CommerceNet, and Netscape. The SPEC committee then
came up with a set of file sizes, access frequencies, and access patterns
that it felt accurately reflected the traffic of a hypothetical Web
provider that offers space for people to put up home pages.
The SPECweb96 clients simulate a number of Web browsers referencing pages
throughout the Web site. The SPECweb96 benchmark uses only HTTP GET
commands (the command used to retrieve a Web page), but does not do POST
commands or CGI calls, or test any security mechanisms.
Does this sound like the Web traffic your site gets? If not, SPECweb96 may
not be the best performance test for your site.
SPECweb96 uses only the HTTP 1.0 protocol. It does not support any of the
HTTP 1.1 mechanisms, such as Keep-Alive, which can dramatically improve
Web-server response time by keeping the network connection open instead of
creating a new connection for each Web-page component downloaded.
One of the best things-and one of the worst-about the SPEC benchmarks is
that they are distributed in source code form. This is great because you
can inspect the benchmark to see what it's doing or find out exactly
why it's not working. But it also means you need an ANSI C programming
environment to build the test suite before installing it on your Web server
and client systems.
The Windows NT version of SPECweb96 was built using Microsoft Visual C++
4.0, but could probably be adapted to other compilers. The Unix version of
SPECweb96 includes Make files for about 16 Unix variants, including Linux,
UnixWare, Solaris, HP/UX, and several System V and BSD variants.
SPECweb96 uses Perl language scripts both to administer the test and report
results. Fortunately, a Perl 5.0 interpreter is included on the
Obviously, you'll also need some Web-server software. Because all Web
servers use the HTTP 1.0 protocol, any Web server can be used.
Depending on the kind of traffic your site gets, SPECweb96 may be a useful
performance comparison test for your Web-server software. For many sites,
it's a useful performance metric, since HTTP GET commands comprise the
vast majority of Web-site traffic.
If your traffic doesn't fit the SPECweb-96 mold, however, you'll be
pleased to hear that the SPEC folks aren't sitting still. The next
generation of Web-server benchmark, SPECweb97, is already under way and
will support HTTP 1.1 as well as POST commands, CGI calls, and more
Accurate performance testing of network server software has always been a
tricky business, and no single benchmark can compare every facet of
performance. With SPEC's backing, experience, and rigorous reporting
requirements, SPECweb-96 is your best bet for Web-server software
More information about SPECweb96, including approved testing results, can
be found at SPEC's Web site, http://www.spec.org.