Frequently Asked Questions (FAQ)
About the SPECjbb2000 Benchmark
- What is SPECjbb2000?
- Who will use SPECjbb2000 and how will they use it?
- Why is the benchmark important for vendors?
- Why did SPEC decide to develop SPECjbb2000?
- Does this benchmark measure end-to-end Java performance?
- Does SPECjbb2000 represent real applications?
- What specific aspects of performance does SPECjbb2000 measure?
- What factors affecting Java performance are not measured by SPECjbb2000?
- Why doesn't SPECjbb2000 cover AWT/graphics performance?
- Is this benchmark suitable for measuring client performance?
- Does SPECjbb2000 use Enterprise Java Beans (EJBs), Servlets or Java Server Pages (JSPs)?
- Why doesn't the benchmark measure network and database performance?
- Does this benchmark replace any other benchmark?
- Can I compare my SPECjbb2000 results with my TPC-C results?
- Can SPECjbb2000 results be compared to results from other SPEC benchmarks?
- Is this related to any of IBM's BOB benchmarks?
- Can I use this application for non-benchmarking purposes?
- Do you provide source code for the benchmark?
- Does floating point precision matter for this benchmark?
- Are SPECjbb2000 results repeatable?
- Where can I find out more about the benchmark?
- What metrics does SPECjbb2000 use to report performance?
Setting up SPECjbb2000
- What are the requirements for running SPECjbb2000?
- What other software do I need? Web server, database, proxy server?
- Can SPECjbb2000 run under Java 2?
- Can I use this benchmark for embedded processor platforms?
- Can I run SPECjbb2000 on a uni-processor?
- Can I run SPECjbb2000 on a shared memory processor
(SMP) configuration and will it scale?
- How long does it take to run the SPECjbb2000 benchmarks?
- How much JVM heap memory is required?
- How much system memory is needed to run the benchmarks?
- Why can't I recompile this benchmark?
- What do you mean by warehouses, points and threads?
- Is there a warm-up period before each measured point?
- Where can official SPECjbb2000 results be obtained?
- Can I run SPECjbb2000 on a cluster configuration and
compare results with SMP configurations? If not, why?
- Can SPECjbb2000 help users compare Java and C++ performance?
- Does garbage collection affect SPECjbb2000 results?
- I see a lot of paging and disk I/O but I had a
compliant run. Can I publish my results?
- Why is it necessary to look at the SPECjbb2000 graph
to get a good idea of performance?
- My graph looks strange. Are there two graph formats?
- What do you get when you order SPECjbb2000?
- What is the price of a SPECjbb2000 license and when will it be available?
- How do I contact SPEC for more information?
What is SPECjbb2000?
A1: SPECjbb2000 is a Java program emulating a
3-tier system with emphasis on the middle tier. Random input selection
represents the 1st tier user interface. SPECjbb2000 fully implements the
middle tier business logic. The 3rd tier database is replaced by binary trees.
Who will use SPECjbb2000 and how will they use it?
SPECjbb2000 is inspired by the TPC-C benchmark and loosely follows the
TPC-C specification for its schema, input generation, and transaction profile.
SPECjbb2000 replaces database tables with Java classes and replaces data
records with Java objects. The objects are held by either binary trees (also
Java objects) or other data objects.
SPECjbb2000 runs in a single JVM in which threads represent terminals in a
warehouse. Each thread independently generates random input (tier 1 emulation)
before calling transaction-specific business logic. The business logic
operates on the data held in the binary trees (tier 3 emulation). The
benchmark does no disk I/O or network I/O.
For additional details, refer to the
SPECjbb2000 white paper.
A2: Buyers of middle-tier servers can compare
different platforms both in terms of differences in hardware and software
(e.g., JVM, JIT, and OS).
Why is the benchmark important for vendors?
A3: SPECjbb2000 represents a middleware
application written in Java. Hardware vendors can use the benchmark's results
to analyze their platforms' scalability when running Java applications.
Software vendors can evaluate the efficiency of their JVMs, JITs, garbage
collectors and thread implementations.
Why did SPEC decide to develop SPECjbb2000?
A4: SPEC has an existing Java client
benchmark, SPECjvm98, and wanted to offer a Java benchmark from the server
end. There are a few Java server benchmarks available, but SPECjbb2000 is the
only one that provides SPEC's methodology for ensuring consistent, repeatable
Does this benchmark measure end-to-end Java performance?
A5: No. Please see the answers that follow for
areas not covered by SPECjbb2000.
Does SPECjbb2000 represent real applications?
A6: Yes. Many companies have developed Java
application frameworks for deploying large commercial systems. SPECjbb2000
models the business logic and object caches that make up the middle tier of
these frameworks. The emulation of the other tiers isolates this tier and
simplifies the benchmark by not requiring user emulation or a database. The
implication is that combining a JVM with a high SPECjbb2000 throughput and a
database tuned for online transaction processing will provide a business with
a fast and robust multi-tier environment.
What specific aspects of performance does SPECjbb2000 measure?
In developing this benchmark, SPEC worked toward a relatively flat method
profile to avoid overly good optimization candidates. SPEC also compared the
method profiles to other Java server applications within the member companies
to make sure that SPECjbb2000 behaves like a server application.
A7: SPECjbb2000 measures the implementation of
Java Virtual Machine (JVM), Just-in-time compiler (JIT), garbage collection,
threads and some aspects of the operating system. It also measures the
performance of CPUs, caches, memory hierarchy and the scalability of Shared
Memory Processors (SMPs) platforms on the specified commercial workload.
What factors affecting Java performance are not measured by SPECjbb2000?
A8: AWT, network, I/O and graphics performance
are not measured by the benchmark.
Why doesn't SPECjbb2000 cover AWT/graphics performance?
A9: SPECjbb2000 doesn't cover AWT/graphics
because this is not considered server work.
Is this benchmark suitable for measuring client performance?
Does SPECjbb2000 use Enterprise Java Beans (EJBs), Servlets or Java Server Pages (JSPs)?
Why doesn't the benchmark measure network and database performance?
A12: A two- or three-tier configuration would
be necessary to measure network and database performance. This would be a
complex and expensive configuration to test. Many other benchmarks exist to
measure database and network performance.
Does this benchmark replace any other benchmark?
Can I compare my SPECjbb2000 results with my TPC-C results?
A14: No, absolutely not. SPECjbb2000 is
memory-resident, uses totally different data-set sizes and workload mixes,
performs no I/O to disks, and has no think times. It has a different set of
run and reporting rules, a different measure of throughput, and a different
metric. Such comparison would be a SERIOUS violation of SPEC's run and
reporting rules and of TPC's "fair use policy." Violations are subject to
Can SPECjbb2000 results be compared to results from other SPEC benchmarks?
A15: No. There is no logical way to translate
results from one benchmark to another.
Is this related to any of IBM's BOB benchmarks?
Can I use this application for non-benchmarking purposes?
Do you provide source code for the benchmark?
A18: Yes, but you are required to run with the
jar files provided with the benchmark. Recompilation is forbidden in the run
rules and will invalidate your results.
Does floating point precision matter for this benchmark?
A19: No. The SPECjbb2000 workload contains
primarily integer computation.
Are SPECjbb2000 results repeatable?
A20: Yes. Measurement on a variety of
platforms shows the SPECjbb2000 metric has a 95% confidence interval of +/- 2%
of the score.
Where can I find out more about the benchmark?
What metrics does SPECjbb2000 use to report performance?
A22: SPECjbb2000 ops/second is a composite
throughput measurement representing the averaged throughput over a range of
points. It is described in detail in the document "SPECjbb2000 Run and
Setting up SPECjbb2000
What are the requirements for running SPECjbb2000?
A23: The user needs a machine with a minimum
of 256 MB physical memory, and a JVM environment supporting the Java or Java 2
APIs. JVMs generally need 32 MB or more of disk space for the installed
software. The benchmark itself requires 10 MB of disk space. For competitive
numbers, more than 196 to 300 MB of physical memory is almost certainly
needed, depending on your JVM.
What other software do I need? Web server, database, proxy server?
A24: None of these. The benchmark does not use
a web server, database or any other additional software besides the JVM.
Can SPECjbb2000 run under Java 2?
Can I use this benchmark for embedded processor platforms?
Can I run SPECjbb2000 on a uni-processor?
A27: Yes, but you will need to run all points
from one to eight warehouses to get a reportable result.
Can I run SPECjbb2000 on a shared memory processor (SMP) configuration and
will it scale?
A28: Yes. In theory, the peak throughput point
in the SPECjbb2000 benchmark will occur when the number of warehouses (or
threads) equals the number of CPUs in an SMP system. This may not always be
the case depending your JVM, operating system and hardware. During the
development of SPECjbb2000, scaling up to 12 CPUs was measured on more than
one hardware platform and JVM.
How long does it take to run the SPECjbb2000 benchmarks?
A29: A one-to-eight warehouse SPECjbb2000 run
takes close to half an hour. Specifically, it takes just about three minutes
per point measured, where a point is a number of warehouses. It measures
throughput in this fixed amount of time, so faster machines do more work in
the allotted time.
How much JVM heap memory is required?
A30: Of the JVMs tested, the heap sizes ranged
between 196 MB and 300 MB to run up to 8 warehouses. Typically, however,
performance increases as you add more heap.
How much system memory is needed to run the benchmarks?
A31: Your system memory should always exceed
the maximum heap size for the JVM. Most systems will need about 64 MB memory
in addition to the java heap in order to run SPECjbb2000 without
Why can't I recompile this benchmark?
A32: The intent of this benchmark is to
measure the performance of the Java platform, given a known set of application
bytecodes. Therefore, you may not recompile the source code if you want to
produce a valid result.
If, however, you wish to experiment with the code, be aware that to
successfully compile the spec.reporter package you will need a Java 2
developer kit that contains the com.sun.image.codec.jpeg package.
What do you mean by warehouses, points and threads?
A33: A warehouse is a unit of stored data. It
contains roughly 22 MB of data stored in many objects in many Btrees.
Is there a warm-up period before each measured point?
A thread represents a terminal user within a warehouse. There is a
one-to-one mapping between warehouses and threads plus a few threads for
SPECjbb2000 main and various JVM functions. As the number of warehouses
increases during the full benchmark run, so does the number of threads.
By "point", we mean a two-minute measurement at a given number of
warehouses. A full benchmark run consists of a sequence of measurement points
with an increasing number of warehouses (and thus an increasing number of
A34: For each point, the benchmark creates
data for that many warehouses, then starts the threads making requests as fast
as possible. The threads run for 30 seconds (the warm-up) before starting the
two-minute measurement. This warm-up may get records into memory, but is
probably not as influential as warm-up time on an actual database benchmark.
Where can official SPECjbb2000 results be obtained?
Can I run SPECjbb2000 on a cluster configuration and compare results with
SMP configurations? If not, why?
A36: No, SPECjbb2000 was not designed to be an
appropriate benchmark for clustered environments.
Can SPECjbb2000 help users compare Java and C++ performance?
A37: No. SPECjbb2000 is a JVM server
benchmark, not a language benchmark.
Does garbage collection affect SPECjbb2000 results?
Yes. Garbage collection definitely affects throughput,
especially as the throughput increases. Increasing heap may
lessen the effects of garbage collection in your tests.
I see a lot of paging and disk I/O but I had a compliant run.
Can I publish my results?
Based on the throughput, the heap size, and whether
input.forcegc is set to true or false, it is possible to
decide how active the garbage collection is when measuring
the SPECjbb2000 performance.
For SPECjbb2000, the size of the garbage generated during
each warehouse run is determined by the throughput in terms
of OPS/sec. Based on our measurements, there are about 270KB
of garbage for every SPECjbb2000 OPS/sec during the 120
seconds when the performance is actually measured. For
example, for a test running at 10,000 OPS/sec, there is
about 2.7GB of garbage that needs collection during the 120
seconds when the performance is being measured.
The garbage is made up of short lived objects created during
each warehouse run, and can be effectively collected during
and after the run. You may find that forcing garbage
collection between run points influences performance. You
may set the property input.forcegc to "true" or "false" to
control whether the test will force a garbage collection
between each run point. If there is a forced garbage
collection in between warehouse runs, the garbage is
practically cleaned up before each warehouse run.
Your JVM may offer you flags to specify garbage collector
algorithm or parameters. The -verbosegc switch will tell you
more about the characteristics of your garbage collector
during this benchmark.
A39: Yes, but it would be unwise. This is a
memory-resident server benchmark and you should have adequate memory for your
OS, JVM, the benchmark and heap. If you observe a lot of disk activity, your
results will be non-competitive and should not be published.
Why is it necessary to look at the SPECjbb2000 graph to get a
good idea of performance?
A40: SPECjbb2000 measures the efficiency of
the JVM, JIT compiler and garbage collector. As the amount of parallelism
increases with more warehouses (and therefore more threads), the balance
between these components changes, particularly when you get to more threads
than CPUs. The shape of the curve shows you whether the system handles
My graph looks strange. Are there two graph formats?
A41: The reporter tools can make two
types of html: html with an html graph, and html with a jpg graph. The jpg
graph creation requires Java 2 and, under UNIX, the DISPLAY environment
variable set to a working X display (this may require setting xhost +
on the display). The html graph is provided for those without Java 2 or
without the display device.
If, initially, the result appears in the html with an html graph format,
you can make the other format later (after adding Java 2 or an X host display)
by rerunning the reporter. See the
User's Guide for instructions.
Compliant results sent for publication to SPEC will appear on the SPECjbb2000 website in the
html with a jpg graph format.
What do you get when you order SPECjbb2000?
A42: You get a CD-ROM of the SPEC-licensed
What is the price of a SPECjbb2000 license and when will it be available?
How do I contact SPEC for more information?
SPEC (Standard Performance Evaluation Corporation)
6585 Merchant Place, Suite 100
Warrenton, VA 20187, USA
Phone: 1 540 349 7878
FAX: 1 540 349 5992
This FAQ document was prepared by:
- Kaivalya Dixit
- Anne Troop
- Rajiv Arora
- Robert Reynolds
- Steve Munroe
- Walter Bays
- Ruth Morgenstein
Last Modification 10/04/2001 - Chris Beer