Open Systems Group
The Open Systems Group is the oldest group within SPEC. The OSG is
the current embodiment of the original founders of SPEC. The OSG
is governed by the policies and principles described in the SPEC
Open Systems Group Policies and Procedures Document. The Open
Systems Steering Committee (OSSC) provides oversight and management
for a number of technical subcommittees that investigate and develop
the benchmarks, metrics, run and reporting rules, and so on.
OSG focus is on component- and systems-level benchmarks for desktop
systems, workstations and servers running open operating system environments.
This is the group responsible for the processor metrics SPECint2006
and SPECfp2006 (and their predecessors in 2000, 1995, 1992 and the
original SPECmarks from 1989).
In addition to the SPEC CPU2006 suite, the
group has also developed Java benchmarks: SPECjAppServer2004,
SPECjAppServer2002, SPECjAppServer2001,
SPECjbb2005, SPECjbb2000,
SPECjms2007, SPECjvm2008, and
SPEC JVM98; web server benchmarks:
SPECweb2005, SPECweb99_SSL,
SPECweb99 and SPECweb96;
mail server benchmarks: SPECmail2008 and SPECmail2001;
file server benchmarks: SPECsfs2008 and SPEC
SFS97_R1; and a power and performance benchmark: SPECpower_ssj2008.
More in-depth information may be found in the OSG
FAQ and under the benchmark specific links shown above.
The Current Benchmarks
- SPEC CPU2006
- The current release of SPEC's popular processor performance
tests; the successor to SPEC CPU2000.
- SPECjAppServer2004
- Based on ECperf™ (developed under the Java Community Process),
a set of industry-standardized benchmarks designed to measure
the performance of J2EE 1.2 and 1.3 application servers.
- SPECjbb2005
- The successor to SPEC's original server-side Java benchmark
emulating middle tier business logic, JBB2000.
- SPECjms2007
- SPEC's benchmark for enterprise message-oriented middleware
servers based on JMS (Java Message Service).
- SPECjvm2008
- The current Java virtual machine (JVM) benchmark, with multithreaded
workloads that represent a broad range of application areas
- SPECmail2008
- SPEC's industry standard benchmark designed to measure the performance
of corporate e-mail servers processing requests based on the Internet
standard protocols SMTP and IMAP4.
- SPECmail2001
- SPEC's first industry-standardized mail server benchmark
designed to measure a system's ability to act as a mail server
processing email requests, based on the Internet standard protocols
SMTP and POP3.
- SPECpower_ssj2008
- SPECpower_ssj2008 is the first industry-standard SPEC benchmark
that evaluates the power and performance characteristics of volume
server class computers. The first subset of server workloads
addresses the performance of server-side Java; additional workloads
are planned.
- SPECsfs2008
- "System: File Server" benchmark, a test of server
performance utilizing NFS version 3 and CIFS. Follow-on to the
SFS97_R1 benchmark.
- SPECweb2005
- A standardized performance test for WWW servers, successor of
SPECweb99 and SPECweb99_SSL, the benchmark consists of different
workloads (both SSL and non-SSL), such as banking and e-commerce,
and writes dynamic content in scripting languages to more closely
model real-world deployments. The web server also communicates
with a lightweight backend to simulate an application/database
server.
- Results Search
- A search-engine configured across all the available OSG benchmark
results.
- Purchase a Benchmark
- Order SPEC/OSG products. SPECjvm2008 is also available
as a free download.
- Submitting Results for
Publication
- Information on how benchmark licensees can submit their results
for publication on the SPEC web site.
Future Benchmarks
These are the projects that SPEC has currently under active development.
There are also a variety of other ideas in the works at any given
time.
- SPECsip
- SPEC has formed a new subcommittee to develop standard methods
of comparing performance for servers using the Session Initiation
Protocol (SIP).
- SPECvirtualization
- SPEC has formed a new subcommittee to develop standard methods
of comparing virtualization performance for data center servers.
The group is investigating the use of heterogeneous workloads
that are spread across multiple virtual machines on a single
server, and the methods and metrics used by the benchmark will
be defined as part of the working group's efforts.
Retired Benchmarks
- SPEC CPU2000
- The previous release of SPEC's popular processor performance
tests; replaced by CPU2006.
- SPEC CPU89
- Predecessor to the CPU92 suites, now long obsolete. No material
available here at this time...
- SPEC CPU92
- The now obsolete processor performance benchmarks. See the CPU95 benchmark suite for current processor tests.
- SPEC CPU95
- The popular processor performance tests. Successor to CPU92
and replaced by CPU2000.
- SPECjAppServer2001
SPECjAppServer2002
- Based on ECperf™ (developed under the Java Community Process),
a set of industry-standardized benchmarks designed to measure
the performance of J2EE 1.2 and 1.3 application servers.
- SPEC JBB2000
- A Java Business Benchmark - SPEC's first server-side Java
benchmark emulating middle tier business logic.
- SPEC JVM98
- Benchmark suite for comparing Java virtual machine (JVM) client
platforms.
- SPEC SDM 1
- Our multi-user, UN*X commands level performance suite (sdet
and kenbus1).
- SPEC SFS93
- "System: File Server", aka LADDIS, test of NFS performance.
Replaced by the SFS97 V2.0 benchmark.
- SPEC SFS97 (2.0)
- "System: File Server", a test of NFS server performance.
Replaced by SFS97_R1 V3.0.
- SPEC SFS97_R1 (3.0)
- "System: File Server", a test of NFS server performance.
Replaced by SPECsfs2008.
- SPEC SMT97
- System-MultiTasking. The next generation for the SDM benchmarks. A flexible workload harness with
a workload based upon public domain system utilities (e.g. gcc,
groff, gzip, etc.) performing basic code development activities.
Should provide a good measure of a system's ability to manage
large numbers of active users.
- SPEC WEB96
- A standardized performance test for WWW servers. Succeeded by SPEC WEB99.
- SPECweb99
- A standardized performance test for WWW servers. Succeeded by SPECweb2005.
- SPECweb99_SSL
- A standardized performance test for WWW servers utilizing SSL.
Succeeded by SPECweb2005.
Joining SPEC/OSG
We welcome your interest in joining the Open Systems Group. For more
detailed information on benefits, membership fees, and the application
process, please go to our membership page.
The SPEC Newsletter
The SPEC Newsletter was the official voice of
the OSG. This included news and analysis along with each quarter's
performance results. Results are now published electronically on
the SPEC web site.
Questions
General questions about the OSG benchmarks can be answered by SPEC's
main office staff, most easily reached as info@spec.org.
Technical questions about one of OSG's benchmark products should
be directed to SPEC's tech-support staff at support@spec.org.
Complete contact information can be found in the SPEC Contact List.
