Standard Performance Evaluation Corporation
About Downloaded Config Files and Their Corresponding Results
SPEC is frequently asked:
Q1. I downloaded a config file from www.spec.org/cpu2000, and it seems to have some differences from the corresponding performance result (as reported in the Text, HTML, PDF, or PS file). Does a SPEC CPU2000 config file have to match the result?
A1. There is one part of the config file that must strictly match; and there are parts that may be out of date. That is, the Text, HTML, PDF, or PS report may be more accurate for some information.
First, let's consider the part that must strictly match: the SPEC tools use gnu make to build the executable binaries exactly as described by the config file. If the tester changes the optimization level for 303.mumble from:
OPTIMIZE = -O5 to OPTIMIZE = -O4
then the tools will force a recompile prior to running 303.mumble, in order to ensure that the config file and the binaries strictly match.
But recall (from www.spec.org/cpu2000/docs/config.html) that a config file addresses several different audiences -- one of which is the human reader. Continuing our example above, suppose that the config file contains:
notes104 = Optimization level -O5 is used for 303.mumble
Here, we find an instance where the config file does not really match the result. We changed the OPTIMIZE line, but did not change the notes line. The config file works, the benchmark 303.mumble works, but the human reader sees erroneous information.
Therefore, SPEC allows the notes to be fixed, even after the test is run, because the notes lines do not affect the building of the binary, and do not affect how quickly it runs. If the tester changes the above line to:
notes104 = Optimization level -O5 is not used for 303.mumble
the only effect will be to improve the accuracy of what is presented to the human reader. The tester could even add a line such as:
notes105 = ...because 303.mumble is actually faster with -O4
after the test is complete, and both notes will be printed in the generated reports (Text, HTML, PDF, and PS).
Q2. OK, so the config file options that are addressed to the human reader can be editted after the test is complete. Does that include making changes to the testbed description?
A2. Yes. There are three common reasons why one might want to change the testbed description:
In all three of these examples, the tester can edit the description either before the test is run, or after it is complete. Such changes have no effect on the times that were recorded.
Of course, the tester must ensure that such edits are accurate. (The tools do not attempt to enforce accuracy in testbed description items such as model name; that's up to the tester.)
Q3. How exactly are changes made after a test is run?
A3. Please bring up http://www.spec.org/cpu2000/docs/utility.html and look in the section on "rawformat".
Notice there that the changes are actually made to the rawfile; the config file itself is not changed.
Therefore, if a config file is downloaded
from www.spec.org/cpu2000, the
reader will see the config file as
it existed at the beginning of the test.
Q4. So, what's the point of posting the config file on the SPEC web site, if all these changes are allowed?
A4. First, remember that the portion of the config file that actually affects the generation of the binary is locked down. If you see in a downloaded config file that benchmark 303.mumble uses:
OPTIMIZE = -O4
then you can be sure that 303.mumble does not use -O5. The config file provides an unambiguous record of how the binary was actually built.
Second, if you have a SPEC CPU2000 license, and you have the software environment described by the tester (in the final, formatted report - Text, HTML, PDF, or PS), you can download the config file and create binary versions of the SPEC CPU2000 benchmarks which are performance-equivalent to the binaries used by the tester. If you have the same hardware environment as described by the tester (in the final formatted report), you should be able to obtain performance similar to that which is claimed by the tester, within a small margin for run-to-run variation.
To summarize, if you download a config file from the SPEC web site, you should consider it to be the canonical record of how the binaries were actually built. But it is not the record of the system under test; if you want to know whether the binaries were run on the Model A4 or on the Model B5, look in the final, formatted report (Text, HTML, PDF, or PS).
Last updated 4-Oct-2001 jh