SPEC CPU®2017 Changes

$Id: changes.html 6738 2023-01-12 23:11:44Z JohnHenning $ Latest: www.spec.org/cpu2017/Docs/

This document describes changes in SPEC CPU®2017, a product of the SPEC® non-profit corporation (about SPEC).
This document is intended as a readable summary of the key changes in each release.
The Revisions document has a more complete, terse list of changes.

All users are strongly encouraged to update to the latest version,
to improve compatibility, reliability, accuracy, and clarity (as described below).
Updates are easy to do.

Version 1.1.9 - Nov-2022

RISC-V toolset

Sysinfo improved

PTD (the Power/Temperature Daemon) v1.10.0

Version 1.1.8 - Apr-2021

output_format check

Presubmission tools

Sysinfo

Version 1.1.7 - Feb-2021

macOS arm64

Benchmark updates

502.gcc_r / 602.gcc_s

Presubmission tools

PTD (the Power/Temperature Daemon)

Intel Compiler version checking

Version 1.1.5 - Nov-2020

Updates for GCC

Updates to Run and Reporting Rules

Benchmark updates

500.perlbench_r / 600.perlbench_s

510.parest_r

Presubmission tools

Other minor updates

Version 1.1.0 - Aug-2019

Power measurement

Updates to Run and Reporting Rules

Benchmark updates

500.perlbench_r / 600.perlbench_s

502.gcc_r / 602.gcc_s

510.parest_r

520.omnetpp_r / 620.omnetpp_s

521.wrf_r / 621.wrf_s

525.x264_r/625.x264_s

526.blender_r

627.cam4_s

538.imagick_r / 638.imagick_s

Other minor updates

Version 1.1.9 - Nov-2022

RISC-V toolset

Native toolset available: SPEC CPU tools (such as runcpu, specinvoke, and specperl) are now available for RISC-V 64-bit Linux systems. The initial toolset included with v1.1.9 was built and tested on Ubuntu 22.04.1. Other versions of Linux may also work, but have not been tested. If you run into difficulties with other Linux versions, please check the list of Frequently Asked Questions and the list of Known Problems before contacting Technical Support.

Portability work remains: Please note that the SPEC CPU 2017 suite contains 43 component benchmarks. It is possible that RISC-V systems may require portability compiler options and/or portability source code changes for some of these benchmarks. Therefore, it may not be possible to do a reportable run on RISC-V until after portability problems are resolved, either by selection and approval of PORTABILITY options or by development and approval of source code patches. The procedure to suggest a source code patch is defined in the Technical Support document.

Sysinfo improved

The sysinfo utility, which automatically prints information about the system under test, has been extended to report additional information, including CPU microcode version, process limits, and selected information from systemd.

PTD (the Power/Temperature Daemon) v1.10.0

PTD (the Power/Temperature Daemon) is updated to Version 1.10.0 and supports additional devices.

Version 1.1.8 - Apr-2021

output_format check

Starting in v1.1.8, --output_format=check is automatically included when selecting reportable runs.

Usage note: Although it is allowed to run a SPEC CPU test and fill in the system details afterwards, you may find it useful to create the description before the test, because you can use it as a checklist of whether you have configured the system in the desired fashion. It was arguably an error that reportable did not formerly include output_format=check, since the definition of reportable has   always   been that it should enforce the run rules whenever practical. A valid system description is required by the rules.

Presubmission tools

The presubmission tools, which were first included in 1.1.5, have been updated. As noted below, please see the README for the tools.

There are many updates, including:

Sysinfo

The sysinfo utility now checks whether dmidecode is available in several directories, starting with whatever is on your PATH, and then looking in /usr/local/sbin, /usr/sbin/, and /sbin/. Note that if you install dmidecode from nongnu.org, by default it will be installed in /usr/local/sbin.

The sysinfo utility now checks whether your version of lscpu includes the useful --cache option.

Version 1.1.7 - Feb-2021

macOS arm64

When SPEC CPU 2017 v1.1.7 is installed on a macOS system, SPEC CPU tools will be selected that match the chip - either x86_64 or arm64 ("M1"). The config directory now includes an example config file for Apple Clang on macOS.

Benchmark updates

502.gcc_r / 602.gcc_s

The portability flag SPEC_GCC_VARIADIC_FUNCTIONS_MISMATCH_WORKAROUND is available for use on macOS, where it is neeeded to address an ABI violation.

Presubmission tools

The presubmission tools, which were first included in 1.1.5, have been updated. As noted below, please see the README for the tools.

PTD (the Power/Temperature Daemon)

Update PTD (the Power/Temperature Daemon) to v1.9.2, which supports more devices.

Intel Compiler version checking

The field sw_compiler has certain syntax requirements for how compiler versions are specified. Due to a change in how Intel Compiler versions are released, this field has been updated. If you are using a newer Intel compiler, you will need this update in order to avoid warnings when result files are generated.

Version 1.1.5 - Nov-2020

Changes to the sysinfo utility

The sysinfo utility has several fixes and enhancements. The changes include:

In addition, sysinfo output is now captured in the primary log file, which is useful if a run fails without producing any other reports.

All users of CPU 2017 are encouraged to update their copy of SPEC CPU 2017 to obtain these sysinfo changes, which may improve both accuracy and clarity of reports.

Updates for GCC

GCC version 10 requires workarounds for several benchmarks. These workarounds do not qualify as PORTABILITY (runrules.html#portability) and therefore must be applied consistently in base (runrules.html#BaseFlags). The example GCC config files have been updated to demonstrate how to apply the workarounds while complying with the rules.

Benchmarks with updated GCC notes include:

SPECrateSPECspeed

All who use GCC with SPEC CPU 2017 are encouraged to update their copy of SPEC CPU to obtain the new example config files, which are more compatible with contemporary GCC.

Updates to Run and Reporting Rules

Rule 2.2.4 regarding size changes was clarified, adding references to related rules.

Benchmark updates

500.perlbench_r / 600.perlbench_s

Removed op/time_loop from the (non-timed) test workload, which was discovered to be non-portable when a compiler user reported that undefined behavior on a conversion caused the test to always fail, independent of optimization level. The test was for a case that does not arise in the (timed) reference workload.

510.parest_r

Added portability option -DSPEC_PAREST_STD_FLUSH_WORKAROUND which works around an issue in module message_log.cc, which takes the address of std::flush and uses it in a comparison. If a compiler does not generate consistent results for the address, then the comparison fails and 510.parest_r does not generate any output.

It is also conceivable that a C++2003 compiler might forbid taking the address of std::flush, because it was not explicitly deemed acceptable to do so prior to C++20.

In either of those cases, this flag may be used. It substitutes an alternate flush function in the MessageLog namespace thereby allowing the comparison to succeed.

Presubmission tools

Results that are published at SPEC are reviewed by a series of perl scripts prior to publication. A subset of those tools is included with SPEC CPU 2017 v1.1.5 in directory
$SPEC/bin/presubmit (Unix) or
%SPEC%\bin\presubmit (Windows).

If you use the presubmit tools, your review of results may be quicker or smoother. Please see the README.txt file in that directory.

Other minor updates

Version 1.1.0 - Aug-2019

Power measurement

Power was previously introduced as an experimental feature in CPU 2017 v1.0.
As of v1.1, the feature is now fully supported.

Power measurement is optional. You can include it, or not, as you may prefer.

If you wish to measure power, you will need:

In your config file, you specify the network location for the controller system; set the expected voltage and current ranges; and describe your measurement setup for readers of your results.

For more information, see the config file documentation on Power Measurement.

Updates to Run and Reporting Rules

Benchmark updates

500.perlbench_r / 600.perlbench_s

502.gcc_r / 602.gcc_s

Removed signal handling, on the grounds that it is system-specific (non-portable) behavior, it makes debugging difficult, and the change is performance neutral (because no signals are invoked during ordinary execution). The change does not affect ordinary reportable runs.

510.parest_r

A syntax error comparing a pointer to an integer was corrected. The change is performance neutral: it affects one comparison which is done 16 times during a run; and empirical testing did not show any differences. (Median within less than a tenth of a percent.)

--- ...old.../510.parest_r/src/source/base/parameter_handler.cc
+++ ...new.../510.parest_r/src/source/base/parameter_handler.cc
@@ -749,7 +749,7 @@
   char *endptr;
   double d = std::strtod (s.c_str(), &endptr);
                                   // assert there was no error
-  AssertThrow ((s.c_str()!='\0') || (*endptr == '\0'),
+  AssertThrow ((s.c_str()[0]!='\0') || (*endptr == '\0'),
               ExcConversionError(s));

520.omnetpp_r / 620.omnetpp_s

Removed signal handling, on the grounds that it is system-specific (non-portable) behavior, it makes debugging difficult, and the change is performance neutral (because no signals are invoked during ordinary execution). The change does not affect ordinary reportable runs.

521.wrf_r / 621.wrf_s

In file src/read_grib.c, the file time.h is included from whatever the usual location is, instead of having a hard-coded (and possibly incorrect) path. The change was made after a user reported that the hard-coded path was not present on his macOS 10.14 system. If for some unlikely reason it were actually necessary to use the hard-coded path, a portability flag could be used to request the old behavior. As with any other portability flag, such usage would need to be justified.

--- ...old...521.wrf_r/src/read_grib.c
+++ ...new...521.wrf_r/src/read_grib.c
 #include <math.h>
 #include <limits.h>
-#if defined(MACOS) || defined(SPEC_MACOSX)
+#if (defined(MACOS) || defined(SPEC_MACOSX)) && defined(SPEC_TIME_H_ABSOLUTE_PATH)
 #include "/usr/include/time.h"
 #else
 #include <time.h>
 #endif

525.x264_r/625.x264_s

Removed signal handling, on the grounds that it is system-specific (non-portable) behavior, it makes debugging difficult, and the change is performance neutral (because no signals are invoked during ordinary execution). The change does not affect ordinary reportable runs.

526.blender_r

627.cam4_s

A potential data race condition was reported between multiple threads writing and reading an implicitly SHARED variable in a parallel region. To avoid the problem, line 109 of src/uv3s_update.F90 was changed to mark the variable tmp as private:

!$omp parallel do private (i, j, k, tmp)

The source code in question is executed during the reference (timed) benchmark run. Although SPEC tries to avoid changes to timed code after a benchmark has been released, in this case an exception was made because:

538.imagick_r / 638.imagick_s

Removed signal handling, on the grounds that it is system-specific (non-portable) behavior, it makes debugging difficult, and the change is performance neutral (because no signals are invoked during ordinary execution). The change does not affect ordinary reportable runs.

Other minor updates

Changes to SPEC CPU®2017 Copyright © 2019-2022 Standard Performance Evaluation Corporation (SPEC®)