SPEC logo

SPECweb2005 Frequently Asked Questions

  1. What is SPECweb2005?
  2. What does SPECweb2005 measure?
  3. Does the SPECweb2005 score indicate how many simultaneous user sessions that a server can support?
  4. What features have been added to SPECweb2005?
  5. What kind of workloads are used in the SPECweb2005 benchmark?
  6. Does this benchmark replace the SPECweb99 and SPECweb99_SSL suites?
  7. Can SPECweb2005 results be meaningfully compared to SPECweb99 or SPECweb99_SSL results?
  8. When and where will SPECweb2005 results be available?
  9. What is the difference between SPECweb2005 and other web server benchmarks?
  10. What are the limitations of SPECweb2005?
  11. Can I use SPECweb2005 to determine the size of the server I need?
  12. How can I obtain the benchmark?
  13. What is included with SPECweb2005?
  14. What additional software is required to run SPECweb2005?
  15. What hardware is required to run the benchmark?
  16. What if I have a problem building/running the SPECweb2005 benchmarks?
  17. How can I submit SPECweb2005 results?
  18. Where are the SPECweb2005 run rules?
  19. Where can I go for more information?



1. What is SPECweb2005?

SPECweb2005 is a software benchmark product developed by the Standard Performance Evaluation Corporation (SPEC), a non-profit group of computer vendors, system integrators, universities, research organizations, publishers, and consultants. It is designed to measure a system's ability to act as a web server servicing static and dynamic page requests.

SPECweb2005 is the successor of SPECweb99 and SPECweb99_SSL, offering the capabilities of measuring both SSL and non-SSL request/response performance, and continues the tradition of giving Web users the most objective and most representative benchmark for measuring web server performance.

Rather than offering a single benchmark workload that attempts to approximate the breadth of web server workload characteristics found today, SPECweb2005 has chosen a 3-workload benchmark design: banking, ecommerce, and support. Additionally, the change from a concurrent connection-based workload metric to a simultaneous session-based workload metric is intended to offer a more direct correlation between the benchmark workload scores and the number of users a web server can support for a given workload.

SPECweb2005 continues the tradition of giving Web users the most objective, most representative benchmark for measuring secure web server performance. SPECweb2005 disclosures are governed by an extensive set of run rules to ensure fairness of results.

The benchmark clients run the application program that sends HTTP requests to the server and receives HTTP responses from the server. For portability, this application program and the prime client program have been written in Java. Note that as a logical component, one or more load-generating clients may exist on a single physical system.

The benchmark does not provide any of the web server software. That is left up to the tester. Any web server software that supports HTTP 1.1 and SSL (HTTPS) could be used. However, it should be noted that variations in implementations may lead to differences in observed performance.

To make a run of the benchmark, the tester must first set up one or more networks connecting a number of the driving "clients" to the server under test. The benchmark code is distributed to each of the drivers and the necessary fileset is created for the server. Then a test control file is configured for the specific test conditions and the benchmark is invoked with that control file.

SPECweb2005 is a generalized test, but it does make a good effort at stressing the most basic functions of a web server in a manner that has been standardized so that cross-comparisons are meaningful across similar test configurations.
 

2. What does SPECweb2005 measure?

Each SPECweb2005 benchmark workload (SPECweb2005_Banking, SPECweb2005_Ecommerce, and SPECweb2005_Support) measures the maximum number of simultaneous user sessions that a web server is able to support while still meeting specific throughput and error rate requirements. The TCP connections for each user session are made and sustained at a specified maximum bit rate with a maximum segment size intended to more realistically model conditions that will be seen on the Internet during the lifetime of this benchmark.

3. Does the SPECweb2005 score indicate how many simultaneous user sessions that a server can support?

No. SPECweb2005 consists of three separate, distinct workloads, each with its own submetric: SPECweb2005_Banking, SPECweb2005_Ecommerce, and SPECweb2005_Support. While the individual submetric scores do indicate the total number of simultaneous user sessions the server can support, the overall SPECweb2005 metric for a compliant result is the geometric mean of the three submetrics, normalized to a reference platform score. Accordingly, a SPECweb2005 score of 100, for example, would indicate the same overall performance as the reference platform. A score of 200 would indicate an overall performance that is double that of the reference platform, etc. For more information, see section 3.1 of the Run and Reporting Rules.
 
 

4. What features have been added to SPECweb2005?

SPECweb2005 includes the following improvements over its predecessors:

5. What kinds of workloads are used in the SPECweb2005 benchmark?

SPECweb2005 introduces three entirely new workloads: Banking, Ecommerce, and Support.  These were based upon statistics analyzed from web server logs and observed client-side behavior.  For more information on each of these workloads, please see the SPECweb2005 design documents.
 
 

6. Does this benchmark replace the SPECweb99 and SPECweb99_SSL suites?

Yes. SPEC will accept, and review SPECweb99 and SPECweb99_SSL results until September 27, 2005.  After that time only SPECweb2005 results will be accepted.
 
 

7. Can SPECweb2005 results be meaningfully compared to SPECweb99 or SPECweb99_SSL results?

No. SPECweb2005 is a completely redesigned benchmark with multiple workloads and different metrics.   Public comparisons of SPECweb99 or SPECweb99_SSL results to SPECweb2005 results would be considered a violation of the run and reporting rules.
 

8. When and where will SPECweb2005 results be available?

Initial SPECweb2005 results are available on SPEC's web site. Subsequent results will be posted on an ongoing basis following each two-week review cycle: results submitted by the two-week deadline are reviewed by web committee members for conformance to the run rules, and if accepted at the end of that period are then publicly released.

9. What is the difference between SPECweb2005 and other web server benchmarks?

SPECweb2005 is a standardized test. The SPEC membership - leading vendors, systems integrators, universities, research organizations, publishers and consultants - has agreed on a benchmark suite with one standardized implementation and three new workloads.

Before any SPECweb2005 results are published, SPEC requires that the system under test and the methodology used adhere to agreed-upon standards (the run rules). All results available through SPEC include full disclosure information that reveals exactly what configurations have been used to obtain a particular result.

SPECweb2005 results published on SPEC's web site provide standardized, comparable results for those who cannot run their own web server performance tests.

10. What are the limitations of SPECweb2005?

SPECweb2005 is a standardized benchmark, which means that it is an abstraction of the real world. For example, SPECweb2005 does not attempt to model latency associated with obtaining data across a wide-area network (WAN) such as the Internet. This kind of behavior is difficult to simulate at this stage, since it requires elaborate hardware and software.

11. Can I use SPECweb2005 to determine the size of the server I need?

SPECweb2005 is not the right tool for sizing a server. SPECweb2005 was not designed as a capacity planning tool. However, it does provide information on how web servers handle this specific workload. The workload uncovers several key components of a good secure web server, including LAN performance, processing power, and memory bandwidth, to name a few.

12. How can I obtain the benchmark?

SPECweb2005 is scheduled for public release in June 2005.  It can be purchased on CD-ROM from SPEC at $1,200 for new licensees, $600 for upgrades, and $300 for eligible non-profit organizations.  To order, contact SPEC's administrative office.

13. What is included with SPECweb2005?

The benchmark comes with the code necessary to run the driver system(s), the server-side file set generation tools, and dynamic content implementations. It is up to the tester to install and configure the web server and testbed.

The benchmark does not include web server software, but any web server that is HTTP/1.0 or 1.1 compliant and supports SSLv3 can be used. In addition to licensed server software, open source software can be used to run SPECweb2005.  See the run rules for more details.

The SPECweb2005 CD-ROM contains:

14. What additional software is required to run SPECweb2005?

First, of course, you'll need properly running web server software on your server. On at least one client system, you'll need a 1.4 (or above) JVM for running the prime client and client.

15. What hardware is required to run the benchmark?

16. What if I have a problem building/running the SPECweb2005 benchmark?

There may have been some issues that have been raised about the benchmark since it was released. We are keeping a SPECweb2005 issues repository.

If your issue is not amongst the known issues, then bring it to the attention of SPEC.

17. How can I submit SPECweb2005 results?

Only SPECweb2005 licensees can submit results. SPEC member companies submit results free of charge. Non-members may submit results for an additional fee. All results are subject to a two-week review by web committee members. Non-member submissions are also subject to a preliminary review. If they pass preliminary review, they may be submitted for the standard member review, and barring any issues will be published by SPEC upon payment of a fee. First-time submitters should contact SPEC's administrative office.

SPECweb2005 submissions must include both the raw output file; during the review process, other information may be requested by the subcommittee.

18. What rules govern SPECweb2005?

The current version of the run rules can be found here.

19. Where can I go for more information?

The SPECweb2005 Design Document contains design information on the benchmark and workloads. The Run and Reporting Rules and the User Guide with instructions for installing and running the benchmark are also available. See: http://www.spec.org/osg/web2005 for the available information on SPECweb2005.


Updated: July 17, 2006