Skip navigation

Standard Performance Evaluation Corporation

 
 

Open Systems Group

The Open Systems Group is the oldest group within SPEC. The OSG is the current embodiment of the original founders of SPEC. The OSG is governed by the policies and principles described in the SPEC Open Systems Group Policies and Procedures Document. The Open Systems Steering Committee (OSSC) provides oversight and management for a number of technical subcommittees that investigate and develop the benchmarks, metrics, run and reporting rules, and so on.

OSG focus is on component- and systems-level benchmarks for desktop systems, workstations and servers running open operating system environments. This is the group responsible for the processor metrics SPECint2006 and SPECfp2006 (and their predecessors in 2000, 1995, 1992 and the original SPECmarks from 1989).

In addition to the SPEC CPU2006 suite, the group has also developed Java benchmarks: SPECjAppServer2004, SPECjAppServer2002, SPECjAppServer2001, SPECjbb2013, SPECjbb2005, SPECjbb2000, SPECjEnterprise2010, SPECjms2007, SPECjvm2008, and SPEC JVM98; a SIP benchmark: SPECsip_Infrastructure2011; web server benchmarks: SPECweb2009, SPECweb2005, SPECweb99_SSL, SPECweb99 and SPECweb96; mail server benchmarks: SPECmail2009, SPECmail2008 and SPECmail2001; file server benchmarks: SPECsfs2008 and SPEC SFS97_R1; a power and performance benchmark: SPECpower_ssj2008; and the virtualization benchmarks: SPECvirt_sc2013 and SPECvirt_sc2010. More in-depth information may be found in the OSG FAQ and under the benchmark specific links shown above.

The Current Benchmarks and Tools

SPEC CPU2006
The current release of SPEC's popular processor performance tests; the successor to SPEC CPU2000.
SPECjbb2013
The latest server-side Java benchmark, developed from the ground up to measure performance based on the latest Java application features.
SPECjEnterprise2010
The full system benchmark which allows performance measurement and characterization of Java EE 5.0 servers and supporting infrastructure such as JVM, Database, CPU, disk, and servers.
SPECjms2007
SPEC's benchmark for enterprise message-oriented middleware servers based on JMS (Java Message Service).
SPECjvm2008
The current Java virtual machine (JVM) benchmark, with multithreaded workloads that represent a broad range of application areas
SPECpower_ssj2008
SPECpower_ssj2008 is the first industry-standard SPEC benchmark that evaluates the power and performance characteristics of volume server class computers. The first subset of server workloads addresses the performance of server-side Java; additional workloads are planned.
SPECsfs2008
"System: File Server" benchmark, a test of server performance utilizing NFS version 3 and CIFS. Follow-on to the SFS97_R1 benchmark.
SPECsip_Infrastructure2011
A standardized performance test designed to evaluate a system's ability to act as a SIP server supporting a particular SIP application. The application modeled is a VoIP deployment for an enterprise, telco, or service provider, where the SIP server performs proxying and registration.
SPECvirt_sc2013
SPEC's updated benchmark addressing performance evaluation of datacenter servers used in virtualized server consolidation, measuring end-to-end performance of all system components including the hardware, virtualization platform, and the virtualized guest operating system and application software. In addition to major workload upgrades, the SPECvirt web server workload has been modified to require SSL (HTTPS) between the client system and the web server.
Server Efficiency Rating Tool (SERT)
Intended to measure server energy efficiency, initially as part of the second generation of the US Environmental Protection Agency (EPA) ENERGY STAR for Computer Servers program. Designed to be simple to configure and use via a comprehensive graphical user interface, the SERT uses a set of synthetic worklets to test discrete system components such as memory and storage, providing detailed power consumption data at different load levels.
Chauffeur Worklet Development Kit (WDK)
Chauffeur was designed to simplify the development of workloads for measuring both performance and energy efficiency. Because Chauffeur contains functions that are common to most workloads, developers of new workloads can focus on the actual business logic of the application, and take advantage of Chauffeur's capabilities for configuration, run-time, data collection, validation, and reporting.

Chauffeur was initially designed to meet the requirements of the SERT. However, SPEC recognized that the framework would also be useful for research and development purposes. The Chauffeur framework is now being made available as the Chauffeur Worklet Development Kit (WDK). This kit can be used to develop new workloads (or "worklets" in Chauffeur terminology). Researchers can also use the WDK to configure worklets to run in different ways, in order to mimic the behavior of different types of applications. These features can be used in the development and assessment of new technologies such as power management capabilities.
SPEC PTDaemon
The SPEC PTDaemon software is used to control power analyzers in benchmarks which contain a power measurement component.
Results Search
A search-engine configured across all the available OSG benchmark results.

Purchase a Benchmark
Order SPEC/OSG products. SPECjvm2008 is also available as a free download.
Submitting Results for Publication
Information on how benchmark licensees can submit their results for publication on the SPEC web site.

Future Benchmarks

These are the projects that SPEC has currently under active development. There are also a variety of other ideas in the works at any given time.

Cloud
OSG has formed a subcommittee to develop a cloud services benchmark. The intended audience for this benchmark includes hardware and software vendors providing products that enable cloud services; cloud providers that offer cloud services such as IaaS, PaaS or SaaS; and business customers who would use benchmark results to help them select cloud providers.
SPEC CPUv6
The next revision of the CPU-intensive benchmark suite, currently designated as CPUv6.
Handheld
OSG has formed a subcommittee to develop a compute-intensive benchmark suite for general consumer handheld devices which support an open community of application developers with access to tools and libraries for native applications.
SOA
SPEC has formed a new subcommittee to develop standard methods of measuring performance for typical middleware, database and hardware deployments of applications based on the Service Oriented Architecture (SOA).

Retired Benchmarks

SPEC CPU2000
The previous release of SPEC's popular processor performance tests; replaced by CPU2006.
SPEC CPU89
Predecessor to the CPU92 suites, now long obsolete. No material available here at this time...
SPEC CPU92
The now obsolete processor performance benchmarks. See the CPU95 benchmark suite for current processor tests.
SPEC CPU95
The popular processor performance tests. Successor to CPU92 and replaced by CPU2000.
SPECjAppServer2001
SPECjAppServer2002
SPECjAppServer2004
Based on ECperf™ (developed under the Java Community Process), a set of industry-standardized benchmarks designed to measure the performance of J2EE 1.2 and 1.3 application servers.
SPECjbb2005
The successor to SPEC's original server-side Java benchmark emulating middle tier business logic, JBB2000.
SPEC JBB2000
A Java Business Benchmark - SPEC's first server-side Java benchmark emulating middle tier business logic.
SPEC JVM98
Benchmark suite for comparing Java virtual machine (JVM) client platforms.
SPECmail2009
SPEC's industry standard benchmark designed to measure the performance of corporate e-mail servers processing requests based on the Internet standard protocols SMTP and IMAP4, with support for SSL v3.0 and TLS 1.0.
SPECmail2008
SPEC's industry standard benchmark designed to measure the performance of corporate e-mail servers processing requests based on the Internet standard protocols SMTP and IMAP4.
SPECmail2001
SPEC's first industry-standardized mail server benchmark designed to measure a system's ability to act as a mail server processing email requests, based on the Internet standard protocols SMTP and POP3.
SPEC SDM 1
Our multi-user, UN*X commands level performance suite (sdet and kenbus1).
SPEC SFS93
"System: File Server", aka LADDIS, test of NFS performance. Replaced by the SFS97 V2.0 benchmark.
SPEC SFS97 (2.0)
"System: File Server", a test of NFS server performance. Replaced by SFS97_R1 V3.0.
SPEC SFS97_R1 (3.0)
"System: File Server", a test of NFS server performance. Replaced by SPECsfs2008.
SPEC SMT97
System-MultiTasking. The next generation for the SDM benchmarks. A flexible workload harness with a workload based upon public domain system utilities (e.g. gcc, groff, gzip, etc.) performing basic code development activities. Should provide a good measure of a system's ability to manage large numbers of active users.
SPECvirt_sc2010
SPEC's benchmark addressing performance evaluation of datacenter servers used in virtualized server consolidation, measuring end-to-end performance of all system components including the hardware, virtualization platform, and the virtualized guest operating system and application software.
SPEC WEB96
A standardized performance test for WWW servers. Succeeded by SPEC WEB99.
SPECweb99
A standardized performance test for WWW servers. Succeeded by SPECweb2005.
SPECweb99_SSL
A standardized performance test for WWW servers utilizing SSL. Succeeded by SPECweb2005.
SPECweb2009
A standardized performance test for web servers, the benchmark consists of different workloads (both SSL and non-SSL), such as banking and e-commerce. The web server also communicates with a lightweight backend to simulate an application/database server. New in this benchmark is the addition of a power workload with a measurement methodology taken from the SPECpower group.
SPECweb2005
A standardized performance test for WWW servers, successor of SPECweb99 and SPECweb99_SSL, the benchmark consists of different workloads (both SSL and non-SSL), such as banking and e-commerce, and writes dynamic content in scripting languages to more closely model real-world deployments. The web server also communicates with a lightweight backend to simulate an application/database server.

Joining SPEC/OSG

We welcome your interest in joining the Open Systems Group. For more detailed information on benefits, membership fees, and the application process, please go to our membership page.

The SPEC Newsletter

The SPEC Newsletter was the official voice of the OSG. This included news and analysis along with each quarter's performance results. Results are now published electronically on the SPEC web site.

Questions

General questions about the OSG benchmarks can be answered by SPEC's main office staff, most easily reached as info@spec.org.

Technical questions about one of OSG's benchmark products should be directed to SPEC's tech-support staff at support@spec.org.

Complete contact information can be found in the SPEC Contact List.