Last updated: $Date: 2008/09/17 14:48:09
(To check for possible updates to this document, please see http://www.spec.org/mpi2007/Docs/ )
SPEC MPI2007 V1.1 is an incremental update to SPEC MPI2007 V1.0. Results generated with V1.1 are comparable to results from V1.0 and vice versa. V1.1 is intended to improve compatibility, stability, documentation and ease of use. Changes are intended to be useful to several kinds of users:
For users of new platforms:
For anyone who reads (or produces) a result:
For researchers and developers:
For new users of the suite:
For those who test many platforms:
(This table of contents proceeds in rough order of time for a user of the suite: you acquire a platform, ensure that you are familiar with the rules, build the benchmarks, run them, generate reports, and occasionally use utilities and other features.)
Benchmark source code changes
4.2.8 user-built systems
1. Build directories separated
2. Bundle up binaries and config file
3. Parallel builds on Windows too
4. Unexpected rebuilds reduced
Running the Suite
1. Per-benchmark basepeak and copies - behavior change
2. PreENV allows setting of environment variables
3. Runtime monitoring
4. Data Sets - name changes
1. CSV updated
2. Flag reporting - multiple files supported, flag order preserved, report readability
3. Graphs cleaned up
4. Links and attachments
5. Report names have changed
6. Seconds are reported with more digits
7. Submission check automatically included with rawformat
8. Test date automatically set
1. Convert to Development
2. Dump alternative source
4. Make alternative source
5. ogo top takes you to $GO
Other New and Changed Tools Features
1. Benchmark lists and sets can be referenced
2. Debug logs
3. Keeping temporaries
4. Submit lines continued
5. Submit notes
6. Trailing spaces in config files
Updated Feature Index
Note: links to SPEC MPI2007 documents on this web page assume that you are reading the page from a directory that also contains the other SPEC MPI2007 documents. If by some chance you are reading this web page from a location where the links do not work, try accessing the referenced documents at one of the following locations:
The following benchmark changes were made in V1.1:
Build directories separated: Benchmarks are now built in directories named benchspec/MPI2007/nnn.benchmark/build/build... (or, on Windows, benchspec\MPI2007\nnn.benchmark\build\build...), rather than under the benchmark's run subdirectory. The change is intended to make it easier to copy, backup, or delete build and run directories separately from each other. (It may also make problem diagnosis easier in some situations, since your habit of removing all the run directories will no longer destroy essential evidence 10 minutes before the compiler developer says "Wait - what exactly happened at build time?").
If you prefer the V1.0 behavior, you can revert to it by setting build_in_build_dir to 0.
You can now bundle up a set of binaries and their associated config file, for easy transportation and use on other systems.
Parallel builds on Windows too: Users of Microsoft Windows systems can now use multiple processors to do parallel builds, by setting makeflags, for example:
makeflags = -j N
This feature has worked with SPEC CPU testing on Unix for many years; what's new in MPI2007 V1.1 is the ability to use it on Windows. Note that requesting a parallel build with makeflags = -j N causes multiple processors to be used at build time. It has no effect on whether multiple processors are used at run time, and so does not affect how you report on parallelism.
Data Sets: The naming convention for the particular data sets that you want to run have been changed. The old names of test, train and ref are now considered classes of runs. They are no longer consider specific data sets to run. The new corresponding names are mtest, mtrain and mref. For reportable runs, it will automatically select mref as the data set. This was done to support the future addition of the large data sets (which are scheduled for V2.0).
Per-benchmark basepeak and copies - behavior change: If you select basepeak=1 for an individual benchmark, the number of copies in peak will be forced to be the same as in base. Note that in SPEC MPI2007 V1.0, you could set basepeak for a benchmark, and still change the number of copies in peak; this was deemed to be an error. If you want to run the same tuning in both base and peak, while changing the number of copies, you will need to build two binaries with the same compiler switches.
The PreENV config file option allows setting of environment variables prior to the exectuion of runspec.
Run-time monitoring: The monitor hooks have been a little-known feature of the SPEC CPU toolset for many years. They were first described in the ACM SIGARCH article SPEC CPU2006 Benchmark Tools and are now further described in monitors.html. The monitor hooks allow advanced users to instrument the suite in a variety of ways. SPEC can provide only limited support for their use; if your monitors break files or processes that the suite expects to find, you should be prepared to do significant diagnostic work on your own.
CSV format updated - If you populate spreadsheets from your runs, you probably shouldn't be doing cut/paste of text files; you'll get more accurate data by using --output_format csv. The V1.1 CSV output now has a format that includes much more of the information in the other reports. All runs times are now included, and the selected run times are listed separately. The flags used are also included. Although details of the new features are not shown in the documentation, you should explore them, by taking the new CSV out for a test drive. It is hoped that you will find the V1.1 format more complete and more useful.
Flag reporting - multiple files supported, flag order preserved, report readability There are several changes to reporting on compiler flags:
You can now format a single result using multiple flags files. This feature is intended to make it easier for multiple results to share what should be shared, while separating what should be separated. Common elements (such as a certain version of a compiler) can be placed into one flags file, while the elements that differ from one system to another (such as platform notes) can be maintained separately. Suggestions on use of this feature can be found in flag-description.html.
The flag reporter now does a better job of reporting flags in the same order in which they appeared on the command line.
Flag reporting has been re-organized in an attempt to improve readability:
Base Compiler Invocation Base Portability Flags Base Optimization Peak Compiler Invocation Peak Portability Flags Peak Optimization
Graphs cleaned up:
||Graphs have been changed to reduce the amount of shading, and to reduce painting of other pixels that were not essential to the data being presented (with a tip of the hat to Professor Tufte's notion of reducing "chartjunk", or apologies, depending on the reader's opinion of the change).|
Report names have changed:
In MPI2007 V1.0, final reports had names of the form
for example, MPIM2007.003.ps, MPIM2007.003.txt, MPIM2007.022.pdf, and so forth. The form of the file names has changed to now be
for example, MPIM2007.003.ref.ps, MPIM2007.003.ref.txt, MPIM2007.022.ref.pdf.
There are two reasons for this change:
Seconds are reported with more digits:
Background: For certain values, the SPEC tools print 3 significant digits. This is intentional. For example, if one system has a SPECmpiM_peak2007 performance of 1234.456 and another has a SPECmpiM_peak2007 performance of 1229.987, it is arguable that the performance of these systems is not materially different. Given the reality of run-to-run variation (which is, sometimes, on the order of 4%), it makes sense to report both systems' SPECmpiM_peak2007 as 1230.
Although there is agreement that it is acceptable to round SPEC's computed metrics to 3 significant digits, it has been noted that the argument is weaker for rounding of original observations. In particular, if we wish to acknowledge the reality of run to run variation, then it seems reasonable to report a time of 1234.456 seconds using an integral number of seconds (1234), rather than rounding to the three significant digits, which in this case would be rounding to the nearest 10 seconds (1230).
Change made: Ever since the release of V1.0 of SPEC MPI2007, results posted on SPEC's web site (such as the HTML, PDF, and text formats) have used 3 significant digits for computed metrics, and seconds larger than 1000 have been reported as an integral number of seconds. As of V1.1, reports produced on your own test systems now behave the same way.
The Submission Check report is now automatically included in the output_format list when using rawformat. This change was made because the typical use of rawformat is to create final (submission quality) reports. Even if you don't plan to submit your result to SPEC, the checks that are done by Submission Check can help you to create reports that are more complete and more readable.
The test_date is now automatically set from the system clock, and you should not set it yourself.
Convert to Development: In order to assist with compliance with the run rules (so that results are meaningful and comparable), the SPEC CPU tools perform various checks to ensure that benchmark code, workloads, and tools match the original distribution. Sometimes, though, researchers or developers may want to work in an environment without these checks, for example, when modifying code to add performance instrumentation.
Prior to V1.1, doing so typically required that you abandon the tools. With V1.1, you now have another choice: you can continue using the SPEC supplied toolset in a development sandbox, via the convert_to_development utility.
Keeping temporaries: If you are having trouble debugging your test setup (for example, if your new submit command or parallel_test option is failing), you may want to try the new keeptmp feature. When this option is set, the above-mentioned debug log is kept, along with the various temporary files that it mentions.
If you leave keeptmp at its default setting, temporary files will be automatically deleted after a successful run. If you are managing disk space on a tight budget, and keeping temporaries, you'll almost certainly need to adjust your cleaning methods.
submit lines continued: It is now possible to append a numeral to submit lines, to continue your submit commands over several lines. This feature is intended to improve the readability of your config file when using the submit feature.
Submit notes: The tools will now automatically insert a section with notes on your submit command for runs that use submit. You can customize the section.
Trailing spaces are now stripped in config files, unless preceded by a backslash, as described in the section on whitespace.
The errata items from V1.0 have been corrected. These include:
Documentation has been added for the new features mentioned in this document. Most of the changes are linked from the descriptions above. A few items might not be immediately obvious from the above links, and are called out here:
These user-visible features are new, updated, or newly documented for SPEC MPI2007 V1.1:
Copyright © 2007-2010 Standard Performance Evaluation Corporation
All Rights Reserved