Standard Performance Evaluation Corporation
SPEC Releases New Benchmark for Evaluating Java Virtual Machine (JVM) Server Performance
WARRENTON, Va., June 5, 2000 – The Standard Performance Evaluation Corp. (SPEC) has released SPECjbb2000, a new benchmark for evaluating the performance of servers running typical Java business applications. The benchmark can be used across several versions of UNIX, Windows/NT, Linux and other operating systems.
SPECjbb2000 represents an order processing application for a wholesale supplier. Written in Java, it was developed with participation from computer vendors, system integrators, universities, research organizations, publishers and consultants. Systems integrators and end users can use the benchmark to evaluate performance of hardware and software aspects of Java Virtual Machine (JVM) servers. Hardware vendors can use SPECjbb2000 to analyze their platform’s scalability when running Java applications. Software vendors can use it to evaluate the efficiency of their JVMs, just-in-time compilers (JITs), garbage collectors, thread implementations, and operating systems.
SPECjbb2000 – the "jbb" stands for Java business benchmark – is implemented as a Java program emulating a three-tier client/server system with emphasis on the middle tier. All three tiers are implemented within a single JVM. The tiers mimic a typical business application, where users in tier one generate inputs that result in the execution of business logic in the middle tier (tier two), which in turn calls to a database on the third tier. SPECjbb2000 does not depend on any package beyond the Java run-time environment (JRE).
"SPEC is continuing what it started with its Java client benchmark, SPECjvm98," says Kaivalya Dixit, SPEC president. "There are other Java server benchmarks available, but only SPECjbb2000 provides SPEC’s methodology for ensuring consistent, repeatable results."
Modeling Business Applications
SPECjbb2000 models a wholesale company, with warehouses that serve a number of districts. Customers initiate a set of operations such as placing new orders or requesting the status of an existing order. Additional operations are generated within the company, such as processing orders for delivery, entering customer payments, and checking stock levels.
SPECjbb2000 assigns one active customer per warehouse. A warehouse is a unit of about 25MB of data stored in binary trees (Btrees). Warehouses map directly to Java threads. As the number of warehouses increases during the full benchmark run, so does the number of threads.
While SPECjbb2000 is not a complete OLTP benchmark, it is a good stand-in for a large business application. The benchmark exercises the implementations of JVM, JIT, garbage collection, threads and some aspects of the operating system. It also measures the performance of CPUs, caches, memory hierarchy and the scalability of shared memory processors (SMPs).
A typical benchmark run takes about three minutes per warehouse. SPECjbb2000 measures throughput in a fixed amount of time, so faster machines do more work in the allotted time. Testing by SPEC on a variety of platforms shows that results are consistent and repeatable.
SPECjbb2000 was developed by SPEC's Open Systems Group (OSG). SPEC is a non-profit corporation formed to establish, maintain and endorse a standardized set of relevant benchmarks that can be applied to the newest generation of high-performance computers. The organization’s membership includes more than 30 computer vendors, systems integrators, publishers, consultants, and leading universities and research centers from around the world. The SPEC Web site http://www.spec.org offers, without cost, the broadest set of performance ratings for evaluating computer systems. In addition to OSG, SPEC benchmark development groups include the High-Performance Group (HPG), and the Graphics Performance Characterization (GPC) Group. For more information on SPECjbb2000 and other benchmarks, contact SPEC, 6585 Merchant Place, Suite 100, Warrenton, Va. 20187; phone: 540-349-7878; fax: 540-349-5992; e-mail: email@example.com; http://www.spec.org.