Skip navigation

Standard Performance Evaluation Corporation

Facebook logo LinkedIn logo Twitter logo Google+ logo
 
 

SPEC's Benchmarks

Cloud

  • SPEC Cloud_IaaS 2016
    [benchmark info] [published results] [order benchmark]
    SPEC's first benchmark suite to measure cloud performance SPEC Cloud_IaaS 2016's use is targeted at cloud providers, cloud consumers, hardware vendors, virtualization software vendors, application software vendors, and academic researchers. The benchmark addresses the performance of infrastructure-as-a-service (IaaS) public or private cloud platforms. The benchmark is designed to stress provisioning as well as runtime aspects of a cloud using I/O and CPU intensive cloud computing workloads. SPEC selected the social media NoSQL database transaction and K-Means clustering using map/reduce as two significant and representative workload types within cloud computing.

CPU

  • SPEC CPU2017
    [benchmark info] [published results] [support] [order benchmark]
    Designed to provide performance measurements that can be used to compare compute-intensive workloads on different computer systems, SPEC CPU2017 contains 43 benchmarks organized into four suites: SPECspeed 2017 Integer, SPECspeed 2017 Floating Point, SPECrate 2017 Integer, and SPECrate 2017 Floating Point.

  • SPEC CPU2006
    [benchmark info] [published results] [support] [order benchmark]
    Designed to provide performance measurements that can be used to compare compute-intensive workloads on different computer systems, SPEC CPU2006 contains two benchmark suites: CINT2006 for measuring and comparing compute-intensive integer performance, and CFP2006 for measuring and comparing compute-intensive floating point performance.

  • SPEC CPU2000
    [Retired]

  • SPEC CPU95
    [Retired]

  • SPEC CPU92
    [Retired]

Graphics and Workstation Performance

Handheld

  • Handheld
    [info]
    In 2013 SPEC formed a committee chartered for the development of, and support for, a compute intensive benchmark suite for handheld devices. In 2017, recognizing the reduced interest in this work, this committee was dissolved.

High Performance Computing, OpenMP, MPI, OpenACC, OpenCL

  • SPEC ACCEL
    [benchmark info] [published results] [support] [order benchmark]
    SPEC ACCEL tests performance with a suite of computationally intensive parallel applications running under the OpenCL and OpenACC APIs. The suite exercises the performance of the accelerator, host CPU, memory transfer between host and accelerator, support libraries and drivers, and compilers.

  • SPEC MPI2007
    [benchmark info] [published results] [support] [order benchmark]
    MPI2007 is SPEC's benchmark suite for evaluating MPI-parallel, floating point, compute intensive performance across a wide range of cluster and SMP hardware. The suite consists of the intial MPIM2007 suite and MPIL2007, which contains larger working sets and longer run times than MPIM2007.

  • SPEC OMP2012
    [benchmark info] [published results] [support] [order benchmark]
    The successor to the OMP2001, designed for measuring performance using applications based on the OpenMP 3.1 standard for shared-memory parallel processing. OMP2012 also includes an optional metric for measuring energy consumption.

  • SPEC OMP2001
    [Retired]

  • SPEC HPC2002
    [Retired]

  • SPEC HPC96
    [Retired]

Java Client/Server

  • SPECjAppServer2004
    [Retired]

  • SPECjAppServer2002
    [Retired]

  • SPECjAppServer2001
    [Retired]

  • SPECjbb2015
    [benchmark info] [published results] [support] [order benchmark]
    The SPECjbb2015 benchmark has been developed from the ground up to measure performance based on the latest Java application features. It is relevant to all audiences who are interested in Java server performance, including JVM vendors, hardware developers, Java application developers, researchers and members of the academic community.

  • SPECjbb2013
    [Retired]

  • SPECjbb2005
    [Retired]

  • SPECjbb2000
    [Retired]

  • SPECjEnterprise2010
    [benchmark info] [published results] [support] [order benchmark]
    SPECjEnterprise2010 measures full-system performance for Java Enterprise Edition (Java EE) 5 or later application servers, databases and supporting infrastructure and expands the scope of the SPECjAppServer2004 benchmark.

  • SPECjms2007
    [benchmark info] [published results] [support] [order benchmark]
    SPECjms2007 is the first industry-standard benchmark for evaluating the performance of enterprise message-oriented middleware servers based on JMS (Java Message Service). It provides a standard workload and performance metrics for competitive product comparisons, as well as a framework for indepth performance analysis of enterprise messaging platforms.

  • SPECjvm2008
    [benchmark info] [published results] [support] [download benchmark]
    SPECjvm2008 is a benchmark suite for measuring the performance of a Java Runtime Environment (JRE), containing several real life applications and benchmarks focusing on core java functionality. The SPECjvm2008 workload mimics a variety of common general purpose application computations.

  • SPEC JVM98
    [Retired]

Mail Servers

Storage

Power

  • SPECpower_ssj2008
    [benchmark info] [published results] [support] [order benchmark]
    SPECpower_ssj2008 is the first industry-standard SPEC benchmark that evaluates the power and performance characteristics of volume server class computers. The initial benchmark addresses the performance of server-side Java, and additional workloads are planned.

Virtualization

  • SPEC VIRT_SC 2013
    [benchmark info] [published results] [support] [order benchmark]
    SPEC's updated benchmark addressing performance evaluation of datacenter servers used in virtualized server consolidation. SPEC VIRT_SC 2013 measures the end-to-end performance of all system components including the hardware, virtualization platform, and the virtualized guest operating system and application software. The benchmark supports hardware virtualization, operating system virtualization, and hardware partitioning schemes.

  • SPEC VIRT_SC 2010
    [Retired]

Web Servers

SPEC Tools

  • Server Efficiency Rating Tool (SERT) 2.0
    [benchmark info] [support] [order software]
    The Server Efficiency Rating Tool (SERT) was created by Standard Performance Evaluation Corporation (SPEC) at the request of the US Environmental Protection Agency. The SERT 2.0 adds a single-value metric, reduces runtime, improves automation and testing, and broadens device and platform support. Designed to be simple to configure and use via a comprehensive graphical user interface, the SERT suite uses a set of synthetic worklets to test discrete system components such as processors, memory and storage, providing detailed power consumption data at different load levels.

    The SERT metric, created with the support of the RG Power Working Group, rates the server efficiency of single- and multi-node servers across a broad span of configurations.

  • Server Efficiency Rating Tool (SERT) 1.1.1
    [benchmark info] [support] [order software]
    The Server Efficiency Rating Tool (SERT) was created by Standard Performance Evaluation Corporation (SPEC) at the request of the US Environmental Protection Agency. The SERT 1.1.1 is the most current SERT version supported by the U.S. EPA Energy Star v2.0 program. Designed to be simple to configure and use via a comprehensive graphical user interface, the SERT suite uses a set of synthetic worklets to test discrete system components such as processors, memory and storage, providing detailed power consumption data at different load levels.

  • Chauffeur Worklet Development Kit (WDK)
    [kit info] [forum] [order software]
    Chauffeur was designed to simplify the development of workloads for measuring both performance and energy efficiency. Because Chauffeur contains functions that are common to most workloads, developers of new workloads can focus on the actual business logic of the application, and take advantage of Chauffeur's capabilities for configuration, run-time, data collection, validation, and reporting.

    Chauffeur was initially designed to meet the requirements of the SERT. However, SPEC recognized that the framework would also be useful for research and development purposes. The Chauffeur framework is now being made available as the Chauffeur Worklet Development Kit (WDK). This kit can be used to develop new workloads (or "worklets" in Chauffeur terminology). Researchers can also use the WDK to configure worklets to run in different ways, in order to mimic the behavior of different types of applications. These features can be used in the development and assessment of new technologies such as power management capabilities.

    Version 2.0 is based on the SERT 2.0 infrastructure and includes significant enhancements to the hardware detection, customization options of generating HTML reports, and developer documentation. It has now reduced memory requirements for the Director when signing results files and reduced the size of the result output for large systems or clusters.

    Chauffeur WDK 2.0 added worklet-specific normalization of results and an updated list of supported operating systems including Ubuntu (14.04 LTS and 16.04 LTS) as well as current versions of Windows Server, RHEL, SLES, AIX, and Solaris. The WDK also includes the latest PTDaemon integration for power analyzers and temperature sensors, along with data collection, validation and reporting.

  • PTDaemon
    [info]
    The power temperature daemon (also known as PTDaemon) is used to offload the work of controlling a power analyzer or temperature sensor during measurement intervals to a system other than the SUT. It hides the details of different power analyzer interface protocols and behaviors from the benchmark software, presenting a common TCP/IP-based interface that can be readily integrated into different benchmark harnesses. Benchmarks already using PTDaemon include SPECpower_ssj2008, SPEC CPU2017, and SPECvirt_sc2013, the SERT suite, and the Chauffeur WDK.