The 'runspec' Command

Last updated: $Date: 2011-09-07 11:08:21 -0400 (Wed, 07 Sep 2011) $ by $Author: CloyceS $

(To check for possible updates to this document, please see http://www.spec.org/cpu2006/Docs/ )

Contents

1 Introduction

1.1 Who Needs runspec?

1.2 About Config Files

1.2.1 Finding a config file

1.2.2 Naming your config file

1.2.3 If you change only one thing...

1.3 About Defaults

1.4 About Disk Usage and Support for Multiple Users

1.4.1 Directory tree

1.4.2 Hey! Where did all my disk space go?

1.5 Multi-user support

1.5.1 Recommended sharing method: output_root

1.5.2 Alternative sharing methods

2 Before Using runspec

2.1 Install kit

2.2 Have a config file

2.3 Undefine SPEC

2.4 Set your path: Unix

2.5 Set your path: Windows

2.6 Check your disk space

3 Using runspec

3.1 Simplest usage

3.1.1 Reportable run

3.1.2 Running selected benchmarks

3.1.3 Output files

3.2 Syntax

3.2.1 Benchmark names in run lists

3.2.2 Run order for reportable runs

3.2.3 Run order when more than one tuning is present

3.2.4 Run order when more than one suite is present

3.3 Actions

3.4 Commonly used options

--action --check_version --config --copies --flagsurl --help --ignore_errors --iterations --loose --output_format --rate --rawformat --rebuild --reportable --tune

3.5 Less commonly used options

--basepeak --nobuild --comment --define --delay --deletework --extension --fake --fakereport --fakereportable --[no]feedback --[no]graph_auto --graph_max --graph_min --http_proxy --http_timeout --info_wrap_column --keeptmp --log_timestamp --machine --make_no_clobber --make_bundle --maxcompares --notes_wrap_column --parallel_setup --parallel_setup_prefork --parallel_setup_type --parallel_test --preenv --reportonly --review --[no]setprocgroup --size --speed --test --[no]table --undef --update --update_flags --unpack_bundle --use_bundle --username --verbose --version

4 Quick reference

Note: links to SPEC CPU2006 documents on this web page assume that you are reading the page from a directory that also contains the other SPEC CPU2006 documents. If by some chance you are reading this web page from a location where the links do not work, try accessing the referenced documents at one of the following locations:

1 Introduction

1.1 Who Needs runspec?

Everyone who uses SPEC CPU2006 needs runspec. It is the primary tool in the suite. It is used to build the benchmarks, run them, and report on their results. All users of CPU2006 should read this document.

If you are a beginner, please start out by reading from the beginning through section 3.1 Simplest Usage. That will probably be enough to get you started.

1.2 About Config Files

In order to use runspec, you need a "config file", which contains detailed instructions about how to build and run the benchmarks. You may not have to learn much about config files in order to get started. Typically, you start off using a config file that someone else has previously written.

1.2.1 Finding a config file

Where can you find such a config file? There are various sources:

  1. Look in the directory $SPEC/config/ (Unix) or %SPEC%\config\ (Windows). You may find that there is a already a config file there with a name that indicates that it is appropriate for your system. You may even find that default.cfg already contains settings that would be a good starting place for your system.

  2. Look at the SPEC web site (http://www.spec.org/cpu2006/) for a CPU2006 result submission that used your system, or a system similar to yours. You can download the config file from that submission.

  3. You can review the examples in
    $SPEC/config/Example*.cfg (Unix) or
    %SPEC%\config\Example*.cfg (Windows).
  4. Alternatively, you can write your own, using the instructions in config.html

1.2.2 Naming your config file

Once you have found a config file that you would like to use as a starting point, you will probably find it convenient to copy it and modify it according to your needs. There are various options:

1.2.3 If you change only one thing...

At first, you may hesitate to change settings in config files, until you have a chance to read config.html. But there is one thing that you might want to change right away. Look for a line that says:

ext=

That line determines what extension will be added to your binaries. If there are comments next to that line giving instructions ("# set ext to A for this, or to B for that"), then set it accordingly. But if there are no such instructions, then usually you are free to set the extension to whatever you like, which can be very useful to ensure that your binaries are not accidentally over-written. You might add your name in the extension, if you are sharing a testbed with others. Or, you may find it convenient to keep binaries for a series of experiments, to facilitate later analysis; if you're naming your config files with names such as jan07a.cfg, you might choose to use "ext=jan07a" in the config file.

1.3 About Defaults

The SPEC tools have followed two principles regarding defaults:

  1. There should always be a default for everything
  2. It should be easy to change the defaults

This means (the good news) that something sensible will usually happen, even when you are not explicit about what you want. But it also means (the bad news) that if something unexpected happens, you may have to look in several places in order to figure out why it behaves differently than you expect.

The order of precedence for settings is:

Highest precedence: runspec command
Middle: config file
Lowest: the tools as shipped by SPEC

Therefore, when this document tells you that something is the default, bear in mind that your config file may have changed that setting. With luck, the author of the config file will tell you so (perhaps in the comments to the config file).

1.4 About Disk Usage

1.4.1 Directory Tree

The structure of the CPU2006 directory tree is:

$SPEC or %SPEC% - the root directory
   benchspec    - some suite-wide files
      CPU2006   - the benchmarks
   bin          - tools to run and report on the suite
   config       - config files
   result       - log files and reports
   tools        - sources for the CPU2006 tools

Within each of the individual benchmarks, the structure is:

nnn.benchmark - root for this benchmark
   Spec       - SPEC metadata about the benchmark
   data        
      all     - data used by all runs (if needed by the benchmark)
      ref     - the real data set, required for all result reporting
      test    - data for a simple test that an executable is functional
      train   - data for feedback-directed optimization
   exe        - compiled versions of the benchmark
   run        - all builds and runs take place here
   src        - the sources for the benchmark

1.4.2 Hey! Where did all my disk space go?

When you find yourself wondering "Where did all my disk space go?", the answer is "The run directories." Most (*) activity takes place in automatically created subdirectories of $SPEC/benchspec/CPU2006/*/run/ (Unix) or %SPEC%\benchspec\CPU2006\*\run\ (Windows).

For example, suppose Bob has a config file that he is using to test some new memory optimizations, and has set

ext=BobMemoryOpt

in his config file. In that case, the tools would create directories such as these:

$ pwd
/Users/bob/cpu2006/benchspec/CPU2006/401.bzip2/run
$ ls
list
run_base_test_BobMemoryOpt.0001
run_base_train_BobMemoryOpt.0001
run_base_ref_BobMemoryOpt.0001
run_peak_test_BobMemoryOpt.0001
run_peak_train_BobMemoryOpt.0001
run_peak_ref_BobMemoryOpt.0001
$ 

To get your disk space back, see the documentation of the various cleaning options, below; or issue a command such as the following (on Unix systems; Windows users can select the files with Explorer):

rm -Rf $SPEC/benchspec/CPU2006/*/run/run*BobMemory*

The effect of the above command would be to delete all the run directories associated with the benchmarks which used extension *BobMemory*. Note that the command did not delete the directories where the benchmarks were built (...CPU2006/*/build/*); sometimes it can come in handy to keep the build directories, perhaps to assist with debugging.

(*) Other space: In addition to the run directories, other consumers of disk space include: (1) temporary files; for a listing of these, see the documentation of keeptmp; and (2) the build directories. For the example above, underneath:

/Users/bob/cpu2006/benchspec/CPU2006/401.bzip2/build/

will be found:

$ ls
build_base_BobMemoryOpt.0001
build_peak_BobMemoryOpt.0001

(History: As of the release of SPEC CPU2006 V1.0, directory names include the string "build" or "run" followed by the extension; in CPU2000, they simply used numbers. As of SPEC CPU2006 1.1, there is a changed location for the build directories; if you prefer the old location, see build_in_build_dir.)

1.5 Multi-user support

(If you are not sharing a SPEC CPU2006 installation with other users, you can skip ahead to section 2.)

The SPEC CPU2006 toolset provides support for multiple users of a single installation, but the tools also rely upon users to make appropriate choices regarding setup of operating-system file protections. This section describes the multi-user features and describes ways of organizing protections. First, to the features that are always enabled:

If you have more than one user of SPEC CPU2006, you can use additional features and choose from several different ways to organize the on-disk layout to share usage of the product. The recommended way is described first.

1.5.1 Recommended sharing method: output_root

The recommended method for sharing a SPEC CPU2006 installation among multiple users has 4 steps:

StepExample (Unix)
Protect most of the SPEC tree read-only chmod -R ugo-w $SPEC
Allow shared access to the config directorychmod 1777 $SPEC/config
Keep your own config filescp config/assignment1.cfg config/alan1.cfg
Add an output_root to your config fileoutput_root=/home/${username}/spec

More detail about the steps is below.

  1. Most of the CPU2006 tree is shared, and can be protected read-only. For example, on a Unix system, you might set protections with:

    chmod -R ugo-w $SPEC
    
  2. The one exception is the config directory, $SPEC/config/ (Unix) or %SPEC%\config\ (Windows), which needs to be a read/write directory shared by all the users. It is written to by users when they create config files, and by the tools themselves: config files are updated after successful builds to associate them with their binaries.

    On Unix, the above protection command needs to be supplemented with:

    chmod 1777 $SPEC/config
    

    which will have the effect (on most Unix systems) of allowing users to create config files which they can choose to protect to allow access only by themselves.

  3. Config files usually would not be shared between users. For example, students might create their own copies of a config file.

    Alan enters:

    cd /cs403
    . ./shrc
    cp config/assignment1.cfg config/alan1.cfg
    chmod u+w config/alan1.cfg
    runspec --config alan1 --action build 456.hmmer
    

    Wendy enters:

    cd /cs403
    . ./shrc
    cp config/assignment1.cfg config/wendy1.cfg
    chmod u+w config/wendy1.cfg
    runspec --config wendy1 --action build 456.hmmer
    
  4. Set output_root in the config files to change the destinations of the outputs.

    To see the effect of output_root, consider an example with and without the feature. If $SPEC is set to /cs403 and if ext=feb27a, then normally the build directory for 456.hmmer with base tuning would be:

    /cs403/benchspec/CPU2006/456.hmmer/run/build_base_feb27a.001

    But if the config files include (near the top, before any occurrence of a section marker):

    output_root=/home/${username}/spec
    ext=feb27a
    

    then Alan's build directory for 456.hmmer will be

    /home/alan/spec/benchspec/CPU2006/456.hmmer/run/build_base_feb27a.001

    and Wendy's will be

    /home/wendy/spec/benchspec/CPU2006/456.hmmer/run/build_base_feb27a.001

    With the above setting of output_root, log files and reports that would normally go to /cs403/result instead will go to /home/alan/spec/result and /home/wendy/spec/result. Alan will find hmmer executables underneath /home/alan/spec/benchspec/CPU2006/456.hmmer/exe. And so forth.

Summary: output_root is the recommended way to separate users. Set the protection on the original tree to read-only, except for the config directory, which should be set to allow users to write, and protect, their own config files.

(History: the output_root feature was added in SPEC CPU2006 V1.0.)

1.5.2 Alternative sharing methods

An alternative is to keep all the files in a single directory tree. In this case:

Name convention: Users sharing a tree can adopt conventions to make their files more readily identifiable. As mentioned above, you can set your config file name to match your own name, and do the same for the extension.

Expid convention: Another alternative is to tag directories with labels that help to identify them based on an "experiment ID", with the config file feature expid, as described in config.html. (History: The expid was added in SPEC CPU2006 V1.0.)

Spend the disk space: A final alternative, of course, is to not share. You can simply give each user their own copy of the entire SPEC CPU2006 directory tree. This may be the easiest way to ensure that there are no surprises (at the expense of extra disk space.)


2 Before Using runspec

Before using runspec, you need to:

2.1 Install CPU2006

The runspec tool uses perl version 5.8.7, which is installed as specperl when you install CPU2006. If you haven't already installed the suite, then please see system-requirements.html followed by:

2.2 Find a config file

You won't get far unless you have a config file, but fortunately you can get started by using a pre-existing config file. See About Config Files, above.

2.3 Undefine SPEC

If the environment variable SPEC is already defined (e.g. from a run of some other SPEC benchmark suite), it may be wise to undefine it first, e.g. by logging out and logging in, or by using whatever commands your system uses for removing definitions (such as unset).

To check whether the variable is already defined, type

echo $SPEC (Unix) or
echo %SPEC% (Windows)

On Unix systems, the desired output is nothing at all; on Windows systems, the desired output is %SPEC%.

Similarly, if your PATH includes tools from some other SPEC suite, it may be wise to remove them from your path.

Next, you need to set your path appropriately for your system type.
See section 2.4 for Unix or section 2.5 for Windows.

2.4 Setting the path: Unix (and Mac OSX)

If you are using a Unix system, change your current directory to the top-level SPEC directory and source either shrc or cshrc:

Note that it is, in general, a good idea to ensure that you understand what is in your path, and that you have only what you truly need. If you have non-standard versions of commonly used utilities in your path, you may avoid unpleasant surprises by taking them out.

q. Do you have to be root? Occasionally, users of Unix systems have asked whether it is necessary to elevate privileges, or to become 'root', prior to entering the above command. SPEC recommends (*) that you do not become root, because: (1) To the best of SPEC's knowledge, no component of SPEC CPU needs to modify system directories, nor does any component need to call privileged system interfaces. (2) Therefore, if you find that it appears that there is some reason why you need to be root, the cause is likely to be outside the SPEC toolset - for example, disk protections, or quota limits. (3) For safe benchmarking, it is better to avoid being root, for the same reason that it is a good idea to wear seat belts in a car: accidents happen, humans make mistakes. For example, if you accidentally type:

kill 1

when you meant to say:

kill %1

then you will very grateful if you are not privileged at that moment.

(*) This is only a recommendation, not a requirement nor a rule.

2.5 Setting the path: Windows

If you are using a Microsoft Windows system, start a Command Prompt Window (previously known as an "MSDOS window"). Change to the directory where you have installed CPU2006, then edit shrc.bat, following the instructions contained therein. For example:

C:\> f:
F:\> cd diego\cpu2006
F:\diego\cpu2006\> copy shrc.bat shrc.bat.orig
F:\diego\cpu2006\> notepad shrc.bat

and follow the instructions in shrc.bat to make the appropriate edits for your compiler paths.

Caution: you may find that the lines are not correctly formatted (the text appears to be all run together) when you edit this file. If so, see the section "Using Text Files on Windows" in the Windows installation guide.

You will have to uncomment one of two lines:

   rem set SHRC_COMPILER_PATH_SET=yes 

or

   rem set SHRC_PRECOMPILED=yes  

by removing "rem" from the beginning of the desired line.

If you uncomment the first line, you will have to follow the instructions a few lines further on, to set up the environment for your compiler.

If you uncomment the second line, you must have pre-compiled binaries for the benchmarks

Note that it is, in general, a good idea to ensure that you understand what is in your path, and that you have only what you truly need. If you have non-standard versions of commonly used utilities in your path, you may avoid unpleasant surprises by taking them out. In order to help you understand your path, shrc.bat will print it after it is done.

When you are done, set the path using your edited shrc.bat, for example:

F:\diego\cpu2006> shrc

2.6 Make sure that you have enough disk space.

Presumably, you checked to be sure you had enough space when you read system-requirements.html, but now might be a good time to double check that you still have enough. Typically, you will want to have at least 8 GB free disk space at the start of a run. Windows users can say "dir", and will find the free space at the bottom of the directory listing. Unix users can say "df -k ." to get a measure of freespace in KB.

If you have done some runs, and you are wondering where your space has gone, see section 1.4.2.


3 Using runspec

3.1 Simplest usage

3.1.1 Reportable run

It is easiest to use runspec when:

In this lucky circumstance, all that needs to be done is to name the config file, select which benchmark suite is to be run: int for the SPECint2006 benchmarks or fp for the SPECfp2006 benchmarks, and say --reportable to attempt a full run.

For example, suppose that Wilfried wants to give Ryan a config file and compiled binaries with some new integer optimizations for a Unix system. Wilfried might type something like this:

[/usr/wilfried]$ cd $SPEC
[/bigdisk/cpu2006]$ spectar -cvf - be*/C*/*/exe/*newint* config/newint.cfg | specxz > newint.tar.xz

and then Ryan might type something like this:

ryan% cd /usr/ryan/cpu2006
cpu2006% bash
bash-2.05$ . ./shrc
bash-2.05$ specxz -dc newint.tar.xz | spectar -xf -
bash-2.05$ runspec --config newint.cfg --nobuild --reportable int

In the example above, the --nobuild emphasizes that the tools should not attempt to build the binaries; instead, the prebuilt binaries should be used. If there is some reason why the tools don't like that idea (for example: the config file does not match the binaries), they will complain and refuse to run, but with --nobuild they won't go off and try to do a build.

As a another example, suppose that Reinhold has given Kaivalya a Windows config file with changes from 12 August, and Kaivalya wants to run the floating point suite. He might say something like this:

F:\kaivalya\cpu2006\> shrc
F:\kaivalya\cpu2006\> specxz -dc reinhold_aug12a.tar.xz | spectar -xf -
F:\kaivalya\cpu2006\> runspec --config reinhold_aug12a --reportable fp

3.1.2 Running selected benchmarks

If you want to run a subset of the benchmarks, rather than running the whole suite, you can name them. Since a reportable run uses an entire suite, you will need to turn off reportable:

[/usr/mat/cpu2006]$ runspec --config mat_dec25j.cfg --noreportable 482.sphinx3

3.1.3 Output files

Look for the output of your runspec command in the directory $SPEC/result (Unix) or %SPEC%\result (Windows). There, you will find log files and result files. More information about log files can be found in config.html.

The format of the result files depends on what was selected in your config file, but will typically include at least .txt for ASCII text, and will always include .rsf, for raw (unformatted) run data. More information about result formats can be found below, under --output_format. Note that you can always re-generate the output, using the --rawformat option, also documented below.

This concludes the section on simplest usage.
If simple commands such as the above are not enough to meet your needs, you can find out about commonly used options by continuing to read the next 3 sections (3.2, 3.3, and 3.4).

3.2 Syntax

The syntax for the runspec command is:

runspec [options] [list of benchmarks to run]

Options are described in the following sections. There, you will notice that many options have both long and short names. The long names are invoked using two dashes, and the short names use only a single dash. For long names that take a parameter, you can optionally use an equals sign. Long names can also be abbreviated, provided that you still enter enough letters for uniqueness. For example, the following commands all do the same thing:

runspec --config=dianne_july25a --debug=99 fp 
runspec --config dianne_july25a --debug 99 fp 
runspec --conf dianne_july25a   --deb 99   fp 
runspec -c dianne_july25a       -v 99      fp 

The list of benchmarks to run can be:

For a reportable run, you must specify int, fp, or all.

3.2.1 Benchmark names in run lists

Individual benchmarks can be named, numbered, or both; and they can be abbreviated, as long as you enter enough characters for uniqueness. For example, each of the following commands does the same thing:

runspec -c jason_july09d --noreportable 459.GemsFDTD 465.tonto
runspec -c jason_july09d --noreportable 459 465
runspec -c jason_july09d --noreportable GemsFDTD tonto
runspec -c jason_july09d --noreportable Gem ton

It is also possible to exclude a benchmark, using a hat (^, also known as carat, typically found as shift-6). For example, suppose your system lacks a C++ compiler, and you therefore cannot run the integer benchmarks 471.omnetpp, 473.astar, and 483.xalancbmk. You could run all of the integer benchmarks except these by entering a command such as this one:

runspec --noreportable -c kathy_sep14c int ^omnet ^astar ^xalanc

Note that if hat has significance to your shell, you may need to protect it from interpretation by the shell, for example by putting it in single quotes. On Windows, you will need to use both a hat and double quotes for each benchmark you want to exclude, like this:

E:\cpu2006> runspec --noreportable -c cathy_apr21b int  "^omnet"  "^astar"  "^xalanc"

3.2.2 Run order for reportable runs

A reportable run runs all the benchmarks in a suite with the test and train data sets as an additional verification that the benchmark binaries get correct results. The test and train workloads are not timed. Then, the reference workloads are run three times, so that median run time can be determined for each benchmark. For example, here are the runs for a reportable integer run of CPU2006:

$ grep runspec: *036.log
runspec: runspec -c mar26a -T base --reportable int
$ grep Running *036.log
Running Benchmarks
  Running 400.perlbench test base mar21a.native32 default
  Running 401.bzip2 test base mar21a.native32 default
  Running 403.gcc test base mar21a.native32 default
  Running 429.mcf test base mar21a.native32 default
  Running 445.gobmk test base mar21a.native32 default
  Running 456.hmmer test base mar21a.native32 default
  Running 458.sjeng test base mar21a.native32 default
  Running 462.libquantum test base mar21a.native32 default
  Running 464.h264ref test base mar21a.native32 default
  Running 471.omnetpp test base mar21a.native32 default
  Running 473.astar test base mar21a.native32 default
  Running 483.xalancbmk test base mar21a.native32 default
  Running 999.specrand test base mar21a.native32 default
Running Benchmarks
  Running 400.perlbench train base mar21a.native32 default
  Running 401.bzip2 train base mar21a.native32 default
  Running 403.gcc train base mar21a.native32 default
  Running 429.mcf train base mar21a.native32 default
  Running 445.gobmk train base mar21a.native32 default
  Running 456.hmmer train base mar21a.native32 default
  Running 458.sjeng train base mar21a.native32 default
  Running 462.libquantum train base mar21a.native32 default
  Running 464.h264ref train base mar21a.native32 default
  Running 471.omnetpp train base mar21a.native32 default
  Running 473.astar train base mar21a.native32 default
  Running 483.xalancbmk train base mar21a.native32 default
  Running 999.specrand train base mar21a.native32 default
Running Benchmarks
  Running (#1) 400.perlbench ref base mar21a.native32 default
  Running (#1) 401.bzip2 ref base mar21a.native32 default
  Running (#1) 403.gcc ref base mar21a.native32 default
  Running (#1) 429.mcf ref base mar21a.native32 default
  Running (#1) 445.gobmk ref base mar21a.native32 default
  Running (#1) 456.hmmer ref base mar21a.native32 default
  Running (#1) 458.sjeng ref base mar21a.native32 default
  Running (#1) 462.libquantum ref base mar21a.native32 default
  Running (#1) 464.h264ref ref base mar21a.native32 default
  Running (#1) 471.omnetpp ref base mar21a.native32 default
  Running (#1) 473.astar ref base mar21a.native32 default
  Running (#1) 483.xalancbmk ref base mar21a.native32 default
  Running (#1) 999.specrand ref base mar21a.native32 default
  Running (#2) 400.perlbench ref base mar21a.native32 default
  Running (#2) 401.bzip2 ref base mar21a.native32 default
  Running (#2) 403.gcc ref base mar21a.native32 default
  Running (#2) 429.mcf ref base mar21a.native32 default
  Running (#2) 445.gobmk ref base mar21a.native32 default
  Running (#2) 456.hmmer ref base mar21a.native32 default
  Running (#2) 458.sjeng ref base mar21a.native32 default
  Running (#2) 462.libquantum ref base mar21a.native32 default
  Running (#2) 464.h264ref ref base mar21a.native32 default
  Running (#2) 471.omnetpp ref base mar21a.native32 default
  Running (#2) 473.astar ref base mar21a.native32 default
  Running (#2) 483.xalancbmk ref base mar21a.native32 default
  Running (#2) 999.specrand ref base mar21a.native32 default
  Running (#3) 400.perlbench ref base mar21a.native32 default
  Running (#3) 401.bzip2 ref base mar21a.native32 default
  Running (#3) 403.gcc ref base mar21a.native32 default
  Running (#3) 429.mcf ref base mar21a.native32 default
  Running (#3) 445.gobmk ref base mar21a.native32 default
  Running (#3) 456.hmmer ref base mar21a.native32 default
  Running (#3) 458.sjeng ref base mar21a.native32 default
  Running (#3) 462.libquantum ref base mar21a.native32 default
  Running (#3) 464.h264ref ref base mar21a.native32 default
  Running (#3) 471.omnetpp ref base mar21a.native32 default
  Running (#3) 473.astar ref base mar21a.native32 default
  Running (#3) 483.xalancbmk ref base mar21a.native32 default
  Running (#3) 999.specrand ref base mar21a.native32 default

The above order can be summarized as:

test
train 
ref1, ref2, ref3

Sometimes, it can be useful to understand when directory setup occurs. So, let's expand the list to include setup:

setup for test
test 
setup for train
train 
setup for ref
ref1, ref2, ref3

3.2.3 Run order when more than one tuning is present

If you run both base and peak tuning, base is always run first. If you do a reportable run with both base and peak, the order is:

setup for test
base test, peak test
setup for train
base train, peak train
setup for ref
base ref1, base ref2, base ref3
peak ref1, peak ref2, peak ref3

3.2.4 Run order when more than one suite is present

If you use all in your list of benchmarks to run, integer is run first, followed by floating point. If you use all in a reportable run, the order is:

setup for test
int base test, fp base test, int peak test, fp peak test
setup for train
int base train, fp base train, int peak train, fp peak train
setup for ref
int base ref1, fp base ref1
int base ref2, fp base ref2
int base ref3, fp base ref3
int peak ref1, fp peak ref1
int peak ref2, fp peak ref2
int peak ref3, fp peak ref3

3.3 Actions

When runspec is used, it normally (*) takes some kind of action for the set of benchmarks specified at the end of the command line (or defaulted from the config file). The default action is validate, which means that the benchmarks will be built if necessary, the run directories will be set up, the benchmarks will be run, and reports will be generated.

(*) Exception: if you use the --rawformat switch, then --action is ignored.

If you want to cause a different action, then you can enter one of the following runspec options:

--action buildCompile the benchmarks. More information about compiling may be found in config.html, including information about additional files that are output during a build.
--action buildsetupSet up build directories for the benchmarks, but do not attempt to compile them. (History: the buildsetup action was added in SPEC CPU2006 V1.0.)
--action configpp Preprocess the config file and dump it to stdout (History: config file preprocessing was added in SPEC CPU2006 V1.0)
--action onlyrunRun the benchmarks but do not bother to verify that they got the correct answers. Reports are always marked "invalid", since the correctness checks are skipped. Therefore, this option is rarely useful, but it can be selected if, for example, you are generating a performance trace and wish to avoid tracing some of the tools overhead. (History: for SPEC CPU2000, this option was spelled "run", but for SPEC CPU2006 the name was changed to clarify what it does.)
--action report Synonym for --fakereport; see also --fakereportable.
--action runSynonym for --action validate. (History: as of SPEC CPU2006 V1.0, the meaning of "--action run" changed in an attempt to better match what users expect "run" to do.)
--action runsetupSynonym for --action setup
--action setupSet up the run directories. Copy executables and data to work directories.
--action validateBuild (if needed), run, check for correct answers, and generate reports.

In addition, the following cleanup actions are available (in order by level of vigor):

--action cleanEmpty all run and build directories for the specified benchmark set for the current user. For example, if the current OS username is set to jeff and this command is entered:
D:\cpu2006\> runspec --action clean --config may12a fp
then the tools will remove run directories with username jeff for fp benchmarks generated by config file may12a.cfg (in nnn.benchmark\run and nnn.benchmark\build).
--action clobber Clean + remove all executables of the current type for the specified benchmark set.
--action trash Same as clean, but do it for all users of this SPEC directory tree, and all types, regardless of what's in the config file.
--action realclean A synonym for --action trash
--action scrub Remove everybody's run and build directories and all executables for the specified benchmark set.

Alternative cleaning method:

If you prefer, you can clean disk space by entering commands such as the following (on Unix systems):

      rm -Rf $SPEC/benchspec/C*/*/run
      rm -Rf $SPEC/benchspec/C*/*/exe

Notes

  1. The above commands not only empty the contents of the run and exe directories; they also delete the directories themselves. That's fine; the tools will re-create the run and exe directories if they are needed again later on.

  2. The above commands do NOT clean the build directories (unless you've set build_in_build_dir=0). Often, it's useful to preserve the build directories for debugging purposes, but if you'd like to get rid of them too, just add $SPEC/benchspec/C*/*/build to your list of directories.

Windows users:

Windows users can achieve a similar effect using Windows Explorer.

I have so much disk space, I'll never use all of it:

Run directories are automatically re-used for subsequent runs. If you prefer, you can ask the tools to never touch a used run directory. Do this by setting the environment variable:

     SPEC_CPU2006_NO_RUNDIR_DEL

In this case, you should be prepared to do frequent cleaning, perhaps after reviewing the results of each run.

3.4 Commonly used options

Most users of runspec will want to become familiar with the following options.

--action action

--check_version

--config name

--copies number

--flagsurl URL[,URL...]

--help

--ignore_errors

--iterations number

--loose

--output_format format

all implies all of the following except screen, check, and mail
cfg|config|conffile
configfile|cfgfile
config file used for this run (e.g. CINT2006.030.ref.cfg) (History: as of the release of SPEC CPU2006 V1.0, the config file is adjusted to include any changes you may have made to fields for readers, as described in )utility.html.)
check|chk|sub|
subcheck|subtest|test
Submission syntax check (automatically enabled for reportable runs). Causes many fields to be checked for acceptable formatting - e.g. hardware available "Nov-2007", not "11/07"; memory size "4 GB", not "4Gb"; and so forth. SPEC enforces consistent syntax for results submitted to its website as an aid to readers of results, and to increase the likelihood that queries find the intended results. If you select --output_format subcheck on your local system, you can find out about most formatting problems before you submit your results to SPEC. Even if you don't plan to submit your results to SPEC, the Submission Check format can help you create reports that are more complete and readable. (History: the ability to check syntax locally, prior to submitting a result to SPEC, was added in SPEC CPU2006 V1.0. As of SPEC CPU2006 V1.1, Submission Check is automatically enabled when doing rawformat.)
csv|spreadsheet

Comma-separated variable (e.g. CINT2006.030.ref.csv). If you populate spreadsheets from your runs, you probably shouldn't be doing cut/paste of text files; you'll get more accurate data by using --output_format csv.

History: the CSV format was added in SPEC CPU2006 V1.0. The format was updated with SPEC CPU2006 V1.1 to include more information: CSV output includes much of the information in the other reports. All runs times are included, and the selected run times are listed separately. The flags used are also included.

default implies HTML and text
flag|flags Flag report (e.g. CINT2006.030.flags.ref.html). Will also be produced when formats that use it are requested (PDF, HTML). (History: flag reporting was added in SPEC CPU2006 V1.0.)
html|xhtml|www|web web page (e.g. CINT2006.030.ref.html)
mail|mailto|email All generated reports will be sent to an address specified in the config file. (History: the ability to email reports was added in SPEC CPU2006 V1.0.)
pdf|adobe Portable Document Format (e.g. CINT2006.030.pdf). This format is the design center for SPEC CPU2006 reporting. Other formats contain less information: text lacks graphs, postscript lacks hyperlinks, and HTML is less structured. (It does not appear as part of "default" only because some systems may lack the ability to read PDF.)
postscript|ps|
printer|print
PostScript (e.g. CINT2006.030.ref.ps)
raw|rsf raw results, e.g. CINT2006.030.ref.rsf. Note: you will automatically get an rsf file for commands that run a test or that update a result (such as rawformat --flagsurl). (History: for SPEC CPU2000, raw results were written to ".raw" files; for SPEC CPU2006, they are written to ".rsf" files.)
screen|scr|disp|
display|terminal|term
ASCII text output to stdout. (History: report generation to stdout was added in SPEC CPU2006 V1.0.)
text|txt|ASCII|asc ASCII text, e.g. CINT2006.030.ref.txt. (History: for SPEC CPU2000, ascii output went to files of type .asc; for SPEC CPU2006, the type is .txt.)
Many of the synonyms above are newly accepted in CPU2006. Now, you don't have to scratch your head and try to remember whether to spell your desired output format as "ps" or as "postscript". SPEC CPU2006 causes less dandruff than SPEC CPU2000. Your scalp may vary.

--rate [copies]

--rawformat rawfiles

--rebuild

--reportable

--tune tuning

3.5 Less commonly used options

--basepeak [bench,bench,...]

--nobuild

--comment "comment text"

--define SYMBOL[=VALUE]
--define SYMBOL:VALUE

--delay secs

--deletework

--extension name[,name...]

--fake

--fakereport

--fakereportable

--[no]feedback

--[no]graph_auto

--graph_max N

--graph_min N

--http_proxy proxy[:port]

--http_timeout N

--info_wrap_columns N

--[no]keeptmp

--[no]log_timestamp

--machine name[,name...]

--make_no_clobber

--make_bundle name

--maxcompares N

--notes_wrap_columns N

--parallel_setup processes

--parallel_setup_prefork cmd

--parallel_setup_type <fork|submit|none>

--parallel_test processes

--preenv, --nopreenv

--review, --noreview

--setprocgroup, --nosetprocgroup

--size size[,size...]

--speed

--test

--table, --notable

--train_with WORKLOAD

--undef SYMBOL

--update

--unpack_bundle name

--use_bundle name

--username name

--verbose n

--version


4 Quick reference

(This table is organized alphabetically, without regard to upper/lower case, and without regard to the presence of a leading "no").

-a Same as --action
--action action Do: build|buildsetup|clean|clobber|configpp| onlyrun|realclean|report|run|runsetup|scrub| setup|trash|validate
--basepeak Copy base results to peak (use with --rawformat)
--nobuild Do not attempt to build binaries
-c Same as --config
-C Same as --copies
--check_version Check whether an updated version of CPU2006 is available
--comment "text"Add a comment to the log and the stored configfile.
--config file Set config file for runspec to use
--copies Set the number of copies for a SPECrate run
-D Same as --rebuild
-d Same as --deletework
--debug Same as --verbose
--define SYMBOL[=VALUE] Define a config preprocessor macro
--delay secs Add delay before and after benchmark invocation
--deletework Force work directories to be rebuilt
--dryrun Same as --fake
--dry-run Same as --fake
-e Same as --extension
--ext Same as --extension
--extension ext[,ext...] Set the extensions
-F Same as --flagsurl
--fake Show what commands would be executed.
--fakereport Generate a report without compiling codes or doing a run.
--fakereportable Generate a fake report as if "--reportable" were set.
--[no]feedback Control whether builds use feedback directed optimization
--flagupdate Same as --update
--flagsupdate Same as --update
--flagsurl url Location (url or filespec) where to find your flags file
--getflags Same as --update
--graph_auto Let the tools pick minimum and maximum for the graph
--graph_min N Set the minimum for the graph
--graph_max N Set the maximum for the graph
-h Same as --help
--help Print usage message
--http_proxy Specify the proxy for internet access
--http_timeout Timeout when attempting http access
-I Same as --ignore_errors
-i Same as --size
--ignore_errors Continue with benchmark runs even if some fail
--ignoreerror Same as --ignore_errors
--info_wrap_column NSet wrap width for non-notes informational items
--infowrap Same as --info_wrap_column
--input Same as --size
--iterations N Run each benchmark N times
--keeptmp Keep temporary files
-l Same as --loose
--loose Do not produce a reportable result
--noloose Same as --reportable
-m Same as --machine
-M Same as --make_no_clobber
--mach Same as --machine
--machine name[,name...] Set the machine types
--make_bundle Create a package of binaries and config file
--make_no_clobber Do not delete existing object files before building.
--max_active_compares Same as --maxcompares
--maxcompares N Set the number of concurrent compares to N
--mockup Same as --fakereportable
-n Same as --iterations
-N Same as --nobuild
--newflags Same as --update
--notes_wrap_column NSet wrap width for notes lines
-noteswrap Same as --notes_wrap_column
-o Same as --output_format
--output_format format[,format...] Generate: all|cfg|check|csv|flags|html|mail|pdf|ps|raw|screen|text
--parallel_setup Number of run directories to set up in parallel
--parallel_setup_preforkCommand to run before setup
--parallel_setup_typeMethod to use for parallel setup
--parallel_test Number of test/train workloads to run in parallel
--preenv Allow environment settings in config file to be applied
-R Same as --rawformat
-r Same as --rate
--rate [N] Do throughput (SPECrate) run, or rawformat with SPECrate metrics
--rawformat Format raw file
--rebuild Force a rebuild of binaries
--reportable Produce a reportable result
--noreportable Same as --loose
--reportonly Same as --fakereport
--[no]review Format results for review
-s Same as --reportable
-S SYMBOL[=VALUE] Same as --define
-S SYMBOL:VALUE Same as --define
--[no]setprocgroup [Don't] try to create all processes in one group.
--size size[,size...] Select data set(s): test|train|ref
--speed Convert SPECrate run to SPECspeed, or do a SPECspeed run
--strict Same as --reportable
--nostrict Same as --loose
-T Same as --tune
--[no]table Do [not] include a detailed table of results
--test Run various perl validation tests on specperl
--train_with Change the training workload
--tune Set the tuning levels to one of: base|peak|all
--tuning Same as --tune
--undef SYMBOL Remove any definition of this config preprocessor macro
-U Same as --username
--unpack_bundle Unpack a package of binaries and config file
--use_bundle Use a package of binaries and config file
--update Check www.spec.org for updates to benchmark and example flag files, and config files
--update_flags Same as --update
--username Name of user to tag as owner for run directories
-v Same as --verbose
--verbose Set verbosity level for messages to N
-V Same as --version
--version Output lots of version information
-? Same as --help

Copyright 1999-2011 Standard Performance Evaluation Corporation

All Rights Reserved