Skip navigation

Standard Performance Evaluation Corporation

Facebook logo LinkedIn logo Twitter logo
 
 

Open Systems Group

The Open Systems Group is the oldest group within SPEC. The OSG is the current embodiment of the original founders of SPEC. The OSG is governed by the policies and principles described in the SPEC Open Systems Group Policies and Procedures Document. The Open Systems Steering Committee (OSSC) provides oversight and management for a number of technical subcommittees that investigate and develop the benchmarks, metrics, run and reporting rules, and so on.

OSG focus is on component- and systems-level benchmarks for desktop systems, workstations and servers running open operating system environments. This is the group responsible for the processor metrics SPECint2017 and SPECfp2017 (and their predecessors in 2006, 2000, 1995, 1992 and the original SPECmarks from 1989).

In addition to the SPEC CPU 2017 suite, the group has also developed Java benchmarks, a SIP benchmark, web server benchmarks, mail server benchmark, file server benchmarks, a power and performance benchmark, and the virtualization benchmarks. More in-depth information may be found in the benchmark specific links below.

The Current Benchmarks and Tools

SPEC Cloud IaaS 2018
SPEC's infrastructure-as-a-service (IaaS) performance benchmark for public and private cloud platforms.
SPEC CPU 2017
The current release of SPEC's popular processor performance tests; the successor to SPEC CPU 2006.
SPECjbb 2015
Server-side Java benchmark, developed from the ground up to measure performance based on the latest Java application features.
SPECjEnterprise 2018 Web Profile
SPECjEnterprise 2018 Web Profile measures full-system performance for Java Enterprise Edition (Java EE) Web Profile Version 7 or later application servers, databases and supporting infrastructure.
SPECjEnterprise 2010
The full system benchmark which allows performance measurement and characterization of Java EE 5.0 servers and supporting infrastructure such as JVM, Database, CPU, disk, and servers.
SPECjvm 2008
The current Java virtual machine (JVM) benchmark, with multithreaded workloads that represent a broad range of application areas.
SPECpower_ssj 2008
SPECpower_ssj 2008 is the first industry-standard SPEC benchmark that evaluates the power and performance characteristics of volume server class computers. The first subset of server workloads addresses the performance of server-side Java; additional workloads are planned.
SPECstorage Solution 2020
A storage benchmark suite measuring file server throughput and response time, providing a standardized method for comparing performance across different vendor platforms.
SPECvirt Datacenter 2021
The next generation of virtualization benchmarking for measuring performance of a scaled-out datacenter. It is a multi-host benchmark using simulated and real-life workloads to measure the overall efficiency of virtualization solutions and their management environments.
SPEC VIRT_SC 2013
The benchmark addresses performance evaluation of datacenter servers used in virtualized server consolidation, measuring end-to-end performance of all system components including the hardware, virtualization platform, and the virtualized guest operating system and application software. In addition to major workload upgrades, the SPECvirt web server workload has been modified to require SSL (HTTPS) between the client system and the web server.
Chauffeur Worklet Development Kit (WDK)
The Chauffeur WDK 2.0 simplifies workload development for researchers analyzing computer performance and power consumption. Targeting both industry and academia, the WDK is based on the SERT 2.0 infrastructure, providing run-time & platform configuration support to simplify worklet development.

This major release includes significant enhancements to the hardware detection, customization options of generating HTML reports, and developer documentation. It has now reduced memory requirements for the Director when signing results files and reduced the size of the result output for large systems or clusters.

Chauffeur WDK 2.0 added worklet-specific normalization of results and an updated list of supported operating systems including Ubuntu (14.04 LTS and 16.04 LTS) as well as current versions of Windows Server, RHEL, SLES, AIX, and Solaris. The WDK also includes the latest PTDaemon integration for power analyzers and temperature sensors, along with data collection, validation and reporting.
SPEC PTDaemon
The SPEC PTDaemon software is used to control power analyzers in benchmarks which contain a power measurement component.
Results Search
A search-engine configured across all the available OSG benchmark results.

Purchase a Benchmark
Order SPEC/OSG products. SPECjvm2008 is also available as a free download.
Submitting Results for Publication
Information on how benchmark licensees can submit their results for publication on the SPEC web site.

Future Benchmarks

These are the projects that SPEC has currently under active development for future release.

SPEC ML
The OSG ML Committee was formed in 2021 to develop practical methodologies to benchmark machine learning (ML) performance in the context of real-world platforms and environments. The Committee also works with other SPEC committees to update their benchmarks for ML environments. The OSG ML Committee's first benchmark, SPEC ML, will measure end-to-end performance of a system under test (SUT) handling ML training and inference tasks.

Retired Benchmarks

SPEC Cloud IaaS 2016
SPEC's original benchmark addressing performance of infrastructure-as-a-service (IaaS) cloud platforms.
SPEC CPU 2000
The previous release of SPEC's popular processor performance tests; replaced by CPU2006.
SPEC CPU 89
Predecessor to the CPU 92 suites, now long obsolete. No material available here at this time...
SPEC CPU 92
The now obsolete processor performance benchmarks. See the SPEC CPU 95 benchmark suite for current processor tests.
SPEC CPU 95
The popular processor performance tests. Successor to SPEC CPU 92 and replaced by SPEC CPU 2000.
SPECjAppServer 2001
SPECjAppServer 2002
SPECjAppServer 2004
Based on ECperf™ (developed under the Java Community Process), a set of industry-standardized benchmarks designed to measure the performance of J2EE 1.2 and 1.3 application servers.
SPECjbb 2013
Server-side Java benchmark, developed from the ground up to measure performance based on the latest Java application features.
SPECjbb 2005
The successor to SPEC's original server-side Java benchmark emulating middle tier business logic, JBB2000.
SPECjbb 2000
A Java Business Benchmark - SPEC's first server-side Java benchmark emulating middle tier business logic.
SPEC JMS 2007
SPEC's benchmark for enterprise message-oriented middleware servers based on JMS (Java Message Service).
SPECjvm 98
Benchmark suite for comparing Java virtual machine (JVM) client platforms.
SPECmail 2009
SPEC's industry standard benchmark designed to measure the performance of corporate e-mail servers processing requests based on the Internet standard protocols SMTP and IMAP4, with support for SSL v3.0 and TLS 1.0.
SPECmail 2008
SPEC's industry standard benchmark designed to measure the performance of corporate e-mail servers processing requests based on the Internet standard protocols SMTP and IMAP4.
SPECmail2 001
SPEC's first industry-standardized mail server benchmark designed to measure a system's ability to act as a mail server processing email requests, based on the Internet standard protocols SMTP and POP3.
SPEC SDM 1
Our multi-user, UN*X commands level performance suite (sdet and kenbus1).
SPEC SFS 93
"System: File Server", aka LADDIS, test of NFS performance. Replaced by the SPEC SFS 97 V2.0 benchmark.
SPEC SFS 97 (2.0)
"System: File Server", a test of NFS server performance. Replaced by SPEC SFS 97_R1 V3.0.
SPEC SFS 97_R1 (3.0)
"System: File Server", a test of NFS server performance. Replaced by SPEC SFS 2008.
SPEC SFS 2008
"Solution File Server" benchmark, a test of server performance utilizing NFS version 3 and CIFS. Follow-on to the SPEC SFS 97_R1 benchmark.
SPEC SFS 2014
SPEC's "Solution File Server" benchmark, a test of file server performance.
SPEC SIP_Infrastructure 2011
A standardized performance test designed to evaluate a system's ability to act as a SIP server supporting a particular SIP application. The application modeled is a VoIP deployment for an enterprise, telco, or service provider, where the SIP server performs proxying and registration.
SPEC SMT 97
System-MultiTasking. The next generation for the SDM benchmarks. A flexible workload harness with a workload based upon public domain system utilities (e.g. gcc, groff, gzip, etc.) performing basic code development activities. Should provide a good measure of a system's ability to manage large numbers of active users.
SPEC VIRT_SC 2010
SPEC's benchmark addressing performance evaluation of datacenter servers used in virtualized server consolidation, measuring end-to-end performance of all system components including the hardware, virtualization platform, and the virtualized guest operating system and application software.
SPECweb 96
A standardized performance test for WWW servers. Succeeded by SPECweb 99.
SPECweb 99
A standardized performance test for WWW servers. Succeeded by SPECweb 2005.
SPECweb 99_SSL
A standardized performance test for WWW servers utilizing SSL. Succeeded by SPECweb 2005.
SPECweb 2009
A standardized performance test for web servers, the benchmark consists of different workloads (both SSL and non-SSL), such as banking and e-commerce. The web server also communicates with a lightweight backend to simulate an application/database server. New in this benchmark is the addition of a power workload with a measurement methodology taken from the SPECpower group.
SPECweb 2005
A standardized performance test for WWW servers, successor of SPECweb99 and SPECweb99_SSL, the benchmark consists of different workloads (both SSL and non-SSL), such as banking and e-commerce, and writes dynamic content in scripting languages to more closely model real-world deployments. The web server also communicates with a lightweight backend to simulate an application/database server.

Joining SPEC/OSG

We welcome your interest in joining the Open Systems Group. For more detailed information on benefits, membership fees, and the application process, please go to our membership page.

The SPEC Newsletter

The SPEC Newsletter was the official voice of the OSG. This included news and analysis along with each quarter's performance results. Results are now published electronically on the SPEC web site.

Questions

General questions about the OSG benchmarks can be answered by SPEC's main office staff, most easily reached as info@spec.org.

Technical questions about one of OSG's benchmark products should be directed to SPEC's tech-support staff at support@spec.org.

Complete contact information can be found in the SPEC Contact List.