SPEChpc™ 2021 Config Files

$Id$ Latest: www.spec.org/hpc2021/Docs/

I. Config file basics

A. What is a config file?

B. Benchmark selection

C. Five consumers

D. Syntax

1. Sections and scope

2. Comments

3. Whitespace

4. Quoting

5. Line continuation

6. Included files

7. Section markers

E. Section merging

Precedence Examples
1. Benchmark specifier
2. Tuning specifier
3. Label specifier
4. Combining sections
5. Conflicting sections
6. Section order
7. Repeated Sections

F. Variables

1. Defining variables

2. $[square] substitution

3. Useful $[variables]

4. ${curly} interpolation

5. Useful ${variables}

6. Unsetting "%undef%"

7. Debug tips

All variable types

$(MAKEVAR)
\$SHELLVAR
$[startup]
${during_run}/$during_run
%{macro}
%{ENV_var} New with SPEChpc 2021

II. Config file options for runhpc

A. Precedence: config file vs. runhpc command line

B. Options

action allow_label_override backup_config basepeak bind check_version command_add_redirect delay deletework difflines enable_monitor env_vars expand_notes expid fail fail_build fail_run flagsurl force_monitor http_proxy http_timeout ignore_errors ignore_sigint info_wrap_columns iterations keeptmp line_width label locking log_line_width log_timestamp mail_reports mailcompress mailmethod mailport mailserver mailto make make_no_clobber makeflags mean_anyway minimize_builddirs minimize_rundirs no_input_handler no_monitor nobuild notes_wrap_columns notes_wrap_indent output_format output_root pmodel New preenv ranks rebuild reportable runlist save_build_files section_specifier_fatal setprocgroup size src.alt strict_rundir_verify table teeout threads tune use_submit_for_speed verbose verify_binaries version_url

III. Config file options for specmake

A. Commonly used Make variables

CC, CXX, FC
PORTABILITY (+ several similar options)
OPTIMIZE (+etc.)
EXTRA_CFLAGS (+etc.)
LIBS
PASS1_OPTIMIZE (+etc.)

B. Parallel Model Selection

C. Creating your own Make variables

D. += is available but use with caution

E. Using buildsetup to create a sandbox

F. About Automatic Rebuilds

{C|CXX|F}C_VERSION_OPTION New and required with SPEChpc 2021

IV. Config file options for the shell

A. \$SHELLVAR variable substitution

B. Options

bench_post_setup build_pre_bench build_post_bench mointor_X post_setup submit

C. Using submit

1. Basic usage

2. Using scripts with submit for thread binding

3. Reporting

Submit Examples:

1. mpirun

2. bind list

3. Generating scripts and command_add_redirect

4. Generating a script for numactl

5. Line continuation

6. Magical notes_submit

V. Config file options for human readers

A. Descriptive fields

Test description

hw_avail license_num notes_test prepared_by sw_avail system_class system_name system_vendor tester test_sponsor

Benchmark description

sw_base_ptrsize sw_c_compiler sw_cxx_compiler sw_f_compiler sw_mpi_library sw_mpi_other sw_notes sw_other

Node description(s)

node_tag_label node_tag_order node_tag_count node_tag_hw_vendor node_tag_hw__model node_tag_purpose node_tag_hw_cpu_name node_tag_hw_ncores node_tag_hw_ncoresorder node_tag_hw_ncoresperchip node_tag_hw_nchips node_tag_hw_nthreadspercore node_tag_hw_cpu_char node_tag_hw_cpu_mhz node_tag_hw_fpu node_tag_hw_pcache node_tag_hw_scache node_tag_hw_tcache node_tag_hw_ocache node_tag_hw_memory node_tag_sw_os node_tag_sw_state node_tag_hw_disk node_tag_sw_local_file node_tag_sw_shared_file node_tag_sw_other node_tag_hw_other node_tag_notes

Accelerator description(s)

node_tag_accel_count node_tag_accel_model node_tag_accel_vendor node_tag_accel_type node_tag_accel_connect node_tag_accel_ecc node_tag_accel_desc node_tag_accel_driver

Adaptor description(s)

node_tag1_hw_adaptor_tag2_model node_tag1_hw_adaptor_tag2_slot_type node_tag1_hw_adaptor_tag2_data_rate node_tag1_hw_adaptor_tag2_count node_tag1_hw_adaptor_tag2_ports_used node_tag1_hw_adaptor_tag2_driver node_tag1_hw_adaptor_tag2_firmware

Interconnect description(s)

interconnect_tag_hw_purpose interconnect_tag_hw_model interconnect_tag_hw_notes interconnect_tag_hw_switch interconnect_tag_hw_switch_count interconnect_tag_hw_switch_ports interconnect_tag_hw_switch_data_rate interconnect_tag_hw_switch_firmware interconnect_tag_hw_topo interconnect_tag_hw_vendor interconnect_tag_label interconnect_tag_order

B. Fields can vary by scope

C. Additional notes for the reader

1. Notes sections

notes_comp_NNN notes_port_NNN notes_base_NNN notes_peak_NNN notes_submit_NNN notes_os_NNN notes_plat_NNN notes_part_NNN notes_NNN

2. Note numbering

3. Additional tags

4. Links in notes sections

VII. The config file preprocessor

A. Introduction

1. Example: Picking Model

2. Syntax Basics

B. Defining macros

1. In a config file

2. On the command line

3. Predefined macros: endian, hostname...
%{ENV_variable_name} New with SPEChpc 2021

4. Example: Adjusting the Environment

5. Redefinition

C. Un-doing macro definition

D. Using macros

E. Conditionals

1. %ifdef .. %endif

2. %ifndef .. %endif

3. %if .. %endif

4. %else

5. %elif

F. Informational directives

1. %warning

2. %error

3. %info New with SPEChpc 2021

4. %dumpmacros New with SPEChpc 2021

VII. Output files - and how they relate to your config file

A. Help, I've got too many config files!

B. The log file

1. Useful Search Strings

2. About Temporary Debug Logs

3. Verbosity levels

D. Help, I've got too many log files!

E. Finding the build directory

F. Files in the build directory

G. For more information

Appendices

A. Other benchmark sets

B. Obsolete features

 

I. Config file basics

I.A. What is a config file?

A SPEChpc config file is a file that defines how to build, run, and report on the SPEChpc benchmarks in a particular environment. It defines how SPEChpc 2021 interacts with your test environment, and lets you describe that environment in human-readable form.

A config file provides detailed control of testing, including:

  • Tuning and portability, which you may customize for your compilers.
  • The runtime environment, which you may customize for your operating system.
  • A description of the SUT (System Under Test), which you should customize to create an accurate report.

Using your customized options, the SPEChpc tools automatically create Makefiles, build the benchmarks, run them, generate reports, and write log files.

Because they collect your options into one place, config files are key to result reproducibility.

For example, if a vendor publishes SPEChpc 2021 results for the SuperHero Model 42 at the SPEC web site www.spec.org/hpc2021, it is expected that a customer can demonstrate similar performance using only 3 ingredients:

  1. The SPEChpc 2021 media, ordered from SPEC.
  2. The software and hardware as specified in the full disclosure from the published result.
  3. The config file, which is posted along with the results.

q. This document looks big and intimidating. Where do I start?

a. Don't start here. Start with the Overview and Using SPEC SPEChpc 2021 - the 'runhpc' Command.

After that, please read section I.C carefully, which explains that config files contain options for five (5) different consumers. You need to recognize which options are for which consumers.

Please notice that config files have 3 kinds of sections. You need to know how named sections work.

From that point on, you can probably skip around among topics as they may interest you.

Tip: Most topics can be found by adding '#topic' to the URL for this document. Examples: #consumers  #shell  #readers  #sw_c_compiler 

Still lost? The config file directory contains several example config files (Prefixed as "Example_"). Copy the example that best matches the compiler version you're using to a new config file, and edit from there. The "Example_SUT.inc" file includes an example for the system description fields which you can either copy into your config file, or copy it to a new include file for your particular system.

I.B. Benchmark selection

In a config file, you can reference: One or more individual benchmarks, such as 605.lbm_s, or entire suites, using the Short Tags in the table below.

Short
Tag
Suite Contents Metrics How many ranks?
What do Higher Scores Mean?
Tiny SPEChpc 2021 Tiny Workload 9 benchmarks SPEChpc 2021_tny_base
SPEChpc 2021_tny_peak
The Tiny workloads use up to 60GB of memory and are intended for use on a single node using between 1 and 256 ranks. More nodes and ranks may be used however higher rank counts may see lower scaling as MPI communication becomes more dominant.
Higher scores indicate that less time is needed.
Small SPEChpc 2021 Small Workload 9 benchmarks SPEChpc 2021_sml_base
SPEChpc 2021_sml_peak
The Small workloads use up to 480GB of memory and are intended for use on one or more nodes using between 64 and 1024 ranks. More ranks may be used however higher rank counts may see lower scaling as MPI communication becomes more dominant.
Higher scores indicate that less time is needed.
Medium SPEChpc 2021 Medium Workload 6 benchmarks SPEChpc 2021_med_base
SPEChpc 2021_med_peak
The Medium workloads use up to 4TB of memory and are intended for use on a mid-size cluster using between 256 and 4096 ranks. More ranks may be used however higher rank counts may see lower scaling as MPI communication becomes more dominant.
Higher scores indicate that less time is needed.
Large SPEChpc 2021 Large Workload 6 benchmarks SPEChpc 2021_lrg_base
SPEChpc 2021_lrg_peak
The Large workloads use up to 14.5TB of memory and are intended for use on a larger clusters using between 2048 and 32,768 ranks. More ranks may be used however higher rank counts may see lower scaling as MPI communication becomes more dominant.
Higher scores indicate that less time is needed.
The "Short Tag" is the canonical abbreviation for use with runhpc, where context is defined by the tools. In a published document, context may not be clear.
To avoid ambiguity in published documents, the Suite Name or the Metrics should be spelled as shown above.

I.C. Five consumers

A config file has content for five (5) distinct consumers, as shown in the table.
To understand a config file, you must understand which consumer is addressed at any given point.
Column 3 below provides a few examples for each; click the roman numerals in column 2 for many more.

Consumer List of
options
Examples Brief description
runhpc II ranks
output_format
reportable
threads
pmodel

Options that change how runhpc itself works. Many should be familiar from Using SPEChpc 2021.
Putting your desired options in the config file can simplify the command line. For example, if jimmy.cfg includes the lines:

output_format = text,csv
tune          = base
reportable    = yes
ranks         = 2048
pmodel        = MPI
runlist       = medium 

then the defaults for the runhpc command would change as shown. Both of these would do the same thing:

runhpc --config=jimmy --output=text,csv --tune=base --reportable --pmodel=MPI -ranks 2048 medium 
specmake III OPTIMIZE
PORTABILITY

Make variables, to control benchmark builds via specmake.

Commonly used specmake variables: section III
Many more: Make Variables
Using specmake on the command line: Utilities

shell IV
post_setup
submit

Commands to be executed by the Unix shell (/bin/sh)
Example:

submit = mpirun -np $ranks --bind-to none $[top]/bind.sh $command  

Above, the mpirun command will use a user supplied "bind.sh" shell script to perform binding per rank.

Warning: SPEChpc config files can execute arbitrary shell commands.
Read a config file before using it.
Don't be root. Don't run as Administrator. Turn privileges off.

readers V hw_model
notes100
sw_compiler

System Under Test (SUT) description, with enough detail so that the reader can understand what was tested and can reproduce your results. If a SPEChpc 2021 result is published (whether at SPEC or independently) it must be fully described.

preprocessor VII %define
%ifdef

Preprocessing directives and definitions to adjust your config file prior to use. All preprocessor directives begin in column 1. Example:

%if %{model} eq 'openacc'
%   define model_define -DSPEC_OPENACC 
%endif
OPTIMIZE  = -O3 %{model_define} 

I.D. Config file syntax

I.D.1. Sections and scope

A config file contains: a header section, named sections, and a HASH section.

Scope: Every line is considered to be within the scope of one of these three. Lines prior to the first section marker are in the scope of the header section. All other lines are either in the scope of the most recently preceding section marker or in the HASH section.

Sections Description Example
header section

The header section is the first section, prior to the any named section. Most attempts to address runhpc itself are done in the header section. In the example, lines 1 through 6 are in the header section.

$ cat -n hiroshi.cfg
 1  flagsurl             = $[top]/config/flags/gcc.xml
 2  iterations           = 1
 3  output_format        = text
 4  runlist              = 605.lbm_s
 5  size                 = ref
 6  tune                 = peak
 7  small=peak:
 8     CC_VERSION_OPTION = -v
 9     CC                = mpicc
10     OPTIMIZE          = -O2
11  605.lbm_s=peak:
12     OPTIMIZE          = -O3
13  
14  __HASH__
15  605.lbm_s=peak=none:
16  # Last updated 2021-02-06 14:29:40
17  opthash=ff6059d6d9ec9577f7f49d05178c58688f31004089
18  baggage=
19  compiler_version=\
20  @eNo1jbEOgjAYhPc+RUcdaFEJIWwGHEhQjBLjRn7LLzapLWkL8
21  kpP9OCqkdX070hmtk0bTjMUspiuhQA9RFgfDki3brEkLdkCf0y
22  0NO36VHldDROLqTSzoNS2JfS5pT/DqUAH54cv4tAsjDEC6M9au
23  FuH/CZ+c5Q+pyRd+tUlX
24  compile_options=\
25  @eNp1T11PgzAUfe+vaPrOMjNfJGMJlLqhQBtLX3xptHYGBWoKM
26  vNp93Vjs3obadX2I+sHXZtD+0D3VXr9bX+8/InJBEMAeKBFZLq
27  KgWj06epUADLlCVqe+rFquJaKiHumJSaC1YWAuOAr/DPWvfu4I
28  mALF6zzeSpj96+RITFhJd3rG/dMaQTzEIJYV2T0DJl8RlGfl7Z
29  hjyd8vw+D9MirnY6z5LRW9NOC1yNkc/OfAJtxnuD
30  exehash=5290fe504238c6de1a13e275ab8ca11e035fbb4e7e
31  
$ 
         
  or
default:

Options for the header section may also be entered in section

default=default=default:

or a shorter spelling thereof, such as

default:

This can be helpful if you need to effectively return to the header section, perhaps when using include.

Tip: Nevertheless, it is usually easier to maintain a config files that keeps all runhpc options near the top, preferably in alphabetical order.

Named sections

A named section is a portion of the config file that follows a line that contains a section marker.

Briefly, a section marker is a 1-to-3 part string with equals signs in the middle and a colon at the end; see detail below. The example has 2 named sections, delimited by 2 section markers on lines 7 and 11.

Notice that the example sets OPTIMIZE in both of the named sections. To understand which one gets used, see the precedence rules, which describe how sections interact.

HASH section

The HASH Section is the final, automatically-generated section. It starts with __HASH__ and contains checksums. The example starts the HASH section at line 14. (For readability, the lines are truncated on the right.)

The automatically-updated checksums ensure that binaries match their config file. You can optionally disable checking, but doing so is strongly discouraged. See verify_binaries.

Config files printed by --output_format=config do not include the HASH section.

I.D.2. Comments

Comments begin with '#'. There are two types:

Syntax Type Saved? Detail
# Regular Yes Regular comments can be full-line or trailing. A copy of your config file is saved with the test results. If you submit your results to SPEC for publication, the regular comments can be read by anyone.
#> Protected No

A full-line comment that begins with #> is a protected comment and will not be saved with your results. You can use protected comments for proprietary information, for example:

#> I didn't use the C++ beta because of Bob's big back-end bug.  

Both types of comments are ignored when processing the config file.

Full-line comments: If # is the first non-blank item on a line, then the whole line is a comment. Comment lines can be placed anywhere in a config file.

Trailing comments: If a line has non-blank items, you can (usually) add regular comments. You cannot write a protected trailing comment. If you try to use a protected comment after some other element, it is treated as a regular comment.

All comments below will be saved except the one that says 'NOT saved'.

                       # New optimizers.
default=base:          # Most optimizers go up to ten.
   OPTIMIZE   = -O11   #    These go to eleven.
                       #> This comment is NOT saved
   COPTIMIZE  = -std   #> This comment is saved

Not a comment: These instances of # do not start a comment:

I.D.3. Whitespace

Blank lines can be placed anywhere in a config file. They are ignored.

Spaces at the beginning of lines are ignored, with the exception that preprocessor directives always begin with a percent sign (%) in the first column. You can put spaces after the percent sign, if you wish (example).

Spaces within a line are usually ignored. Don't try to break up a section marker, and you can't break up a single word (say 'OPTIMIZE' not 'OPT I MIZE'). If multiple spaces separate line elements, it is as if there were only one space. Each of these have the same meaning:

   OPTIMIZE=-O2 -noalias
   OPTIMIZE   =  -O2      -noalias

One place where spaces are considered significant is in notes, where you can use spaces to line up your comments. Notes are printed in a fixed-width font.

Trailing spaces and tabs are ignored, unless they are preceded by a backslash.

I.D.4. Quoting

If you use double (") or single (') quotes within a config file, runhpc leaves them alone. The assumption is that you put them there because one of the consumers (such as a shell) needs them. The quotes are not significant to runhpc but may be highly significant to the consumer. See the section on quote traps.

If you use a backslash (\) it is usually not significant. The exceptions are:

I.D.5. Line continuation

Many fields, including most reader fields, can be continued by adding a number:

node_compute_hw_tcache000= 256 MB I+D on chip per chip
node_compute_hw_tcache001 = 16 MB shared / 4 cores
node_compute_sw_os1 = SUSE Linux Enterprise Linux Server 12
node_compute_sw_os2 = 4.12.14-94.41-default

The fields which cannot be continued are the ones that are expecting a simple integer, such as node_hw_nchips and license_num; and the ones that expect a date, such as hw_avail. You can pick your own style of numbering, as in the examples above. (Note: the stored results from your test always use three-digit numbers, and have slightly different syntax, as discussed in utility.html.)

Shell-style "here documents" with double angle brackets and a delimiter word (e.g. <<EOT) can be used to set multi-line values. Backslash-continued lines are also supported. For example:

$ cat continued_lines.cfg
expand_notes   = 1
output_format  = text
output_root    = /tmp/fake_lines
runlist        = 605.lbm_s

here_continued = <<EOT
      + This is +
       + a test  +
EOT

backslash_continued = + So is +\
 + this  +

notes1 = ${here_continued}
notes2 = ${backslash_continued}
$ cat continued_lines.sh 
runhpc --config=continued_lines --fakereport | grep txt
grep '+' /tmp/fake_lines/result/hpc2021_sml*txt
$ ./continued_lines.sh 
    format: Text -> /tmp/fake_lines/result/hpc2021_sml.001.small.ref.txt
           + This is +
           + a test  +
     + So is +
     + this  +
$   

I.D.6. Included files

You can include other files in your config file using include:
Multiple files may be included.
Included files may use macros (and you can use configpp to check the effect).
Included files may write to arbitrary sections, including (effectively) the header section.

Example: a config file is developed on one system, and applied on a different System Under Test (SUT). The compilers are, of course, installed on the development system. They might not be present on the SUT.

The compiler is described in the main config file on lines 10-12.
The hardware is described in SUT.inc which is included via line 6.

$ cat -n include.cfg
 1  iterations    = 1
 2  output_format = text
 3  output_root   = /tmp/example
 4  runlist       = 605.lbm_s
 5  size          = test
 6  test_sponsor    = Turboblaster
 7  tester          = Turboblaster
 8  include: SUT.inc
 9  default:
10    CC                = mpicc
11    CC_VERSION_OPTION = -V
12    sw_compiler000   = C/C++/Fortran: Version 9.2.0 of
13    sw_compiler001   = GNU Compiler Collection
14    sw_avail         = Aug-2019
15    sw_mpi_library   = OpenMPI Version 3.1.5
$

Note the software date (sw_avail) on line 14 above.
Today's SUT is the Turboblaster SuperHero IV.
Copy its information to the expected place.

$ cp Turboblaster.inc SUT.inc
$ cat Turboblaster.inc 
default:
node_compute_label = TurboBlaster 5000
node_compute_order = 1
node_compute_count = 2
node_compute_purpose = compute
node_compute_hw_vendor = Mega Technology
node_compute_hw_model = Turblaster 5000
node_compute_hw_cpu_name = Turbo CPU
node_compute_hw_ncpuorder = 1 chips
hw_avail  = Feb-2018
$

Note the hardware date (hw_avail) in the include file.
Both dates are referenced in reports:

$ cat include.sh 
runhpc --config=include | grep txt
grep avail /tmp/example/result/*txt
$ ./include.sh 
format: Text -> /tmp/example/result/hpc2021_sml.001.small.test.txt
Test sponsor: Turboblaster      Hardware availability: Feb-2018
Tested by:    Turboblaster      Software availability: Aug-2019
$ 

I.D.7. Section markers and Named sections

A named section begins with a section marker and continues until the next section marker or the HASH section is reached.

Named sections can be entered in any order.
Section markers can be repeated. Material from repeated sections is automatically consolidated.

A section marker is a one- to three-part string of the form:

   benchmark[,...]=tuning[,...]=label[,...]:

The three parts of a section marker are called the section specifiers, with allowed values:

Section
specifier
Allowed values
benchmark default
A metric: tiny, small, medium, or large Any individual benchmark, such as 605.lbm_s
A list, such as: 505.lbm_t,605.lbm_s,705.lbm_m,805.lbm_l
tuning default
base
peak
A list of tuning levels: base,peak
label default
Any specific label: an arbitrary tag to identify binaries and directories
A list of labels, separated by commas

Trailing default section specifiers may be omitted from a section marker. In the pairs below, in each case, the second line is equivalent to the first:

small=default=default:     605.lbm_s=base=default:      default=default=default:
small:                     605.lbm_s=base:              default:

I.E. Section merging and precedence rules

By constructing section markers, you determine how you would like your options applied. Benchmarks are built according to instructions in the sections that they match, subject to rules for combining sections and resolving conflicts among them. Sections are combined using these rules.
Click down arrow to go to an example.

  • down arrow For the benchmark specifier, the precedence is:

    highest   named benchmark(s)
              suite name 
    lowest    default 

    (See also Appendix A)

  • down arrow For the tuning specifier, base or peak has higher precedence than default.
  • down arrow For the label specifier, any named label is at a higher precedence level than default.
  • Combining sections:
    1. down arrow Combine sections that apply to a benchmark, if there is no conflict among them.
    2. down arrow If sections conflict with each other, the order of precedence is:

      highest     benchmark
                  tuning
      lowest      label 
  • Section order:
    • down arrow For sections at differing precedence levels, order does not matter.

    • down arrow If a section occurs more than once, the settings are combined. If there are conflicts, the last instance wins.

Precedence Example 1: benchmark specifiers

For the benchmark specifier, the precedence is:

  highest   named benchmark(s)
            suite name
  lowest    default

The flagsurl line picks up definitions that are provided with SPEChpc 2021.

705.lbm_m is an medium benchmark, but it does not use the setting on lines 14-15. Instead, it uses the higher precedence lines 10-11 (named benchmark).

605.lbm_s is an small benchmark. The highest precedence section for it is on lines 12-13.

805.lbm_l is an large benchmark. It gets the low-precedence OPTIMIZE setting from lines 16-17.

$ cat -n precedence_example1.cfg
     1  flagsurl             = $[top]/config/flags/gcc.xml
     2  iterations           = 1
     3  output_format        = text
     4  output_root          = /tmp/ptest
     5  runlist              = 605.lbm_s,705.lbm_l,805.lbm_l
     6  size                 = test
     7  default:
     8     CC_VERSION_OPTION = -v
     9     CC                = mpicc
    10  701.lbm_s:
    11     OPTIMIZE          = -O3
    12  small:
    13     OPTIMIZE          = -O2
    14  medium:
    15     OPTIMIZE          = -O1
    16  default:
    17     OPTIMIZE          = -O0
Precedence Example 2: tuning specifiers

For the tuning specifier, base or peak has higher precedence than default.

The first few lines of the config file use similar features as the previous example.

The tuning from line 12 is used for base, and line 14 for peak.

$ cat -n precedence_example2.cfg 
     1  flagsurl             = $[top]/config/flags/gcc.xml
     2  iterations           = 1
     3  output_format        = text
     4  output_root          = /tmp/ptest2
     5  runlist              = 605.lbm_s
     6  size                 = test
     7  default=default:
     8     CC_VERSION_OPTION = -v
     9     CC                = mpicc
    10     OPTIMIZE          = -O0
    11  default=base:
    12     OPTIMIZE          = -O1
    13  default=peak:
    14     OPTIMIZE          = -O3
$ cat precedence_example2.sh 
runhpc --config=precedence_example2 --tune=base | grep txt
runhpc --config=precedence_example2 --tune=peak | grep txt
cd /tmp/ptest2/result
grep 'O[0-9]' *txt
$ ./precedence_example2.sh 
    format: Text -> /tmp/ptest2/result/hpc2021.001.small.test.txt
    format: Text -> /tmp/ptest2/result/hpc2021.002.small.test.txt
hpc2021.001.small.test.txt: 605.lbm_s: -O1
hpc2021.002.small.test.txt: 605.lbm_s: -O3
$
Precedence Example 3: label specifiers

For the label specifier, any named label has higher precedence than the default.

This config file is simpler than the previous examples, because we don't even bother to run it; instead, --fake is used.

The runhpc command uses --label=OhTwo. Therefore, the default setting for OPTIMIZE on line 6 is over-ridden by the settings on lines 7-8.

$ cat -n precedence_example3.cfg 
 1  runlist              = 605.lbm_s
 2  size                 = test
 3  small=base=default:
 4     CC_VERSION_OPTION = -v
 5     CC                = mpicc
 6     OPTIMIZE          = -O0
 7  small=base=OhTwo:
 8     OPTIMIZE          = -O2
$ cat precedence_example3.sh 
runhpc --config=precedence_example3 --fake --label=OhTwo | grep lbm.c
$ ./precedence_example3.sh 
mpicc -c -o lbm.o -DSPEC  -DNDEBUG -O2 lbm.c
$  
Precedence Example 4: Combining sections

Combine sections that apply to a benchmark, if there is no conflict among them.

Note that line 1 sets the label, and line 3 sets the tuning.

All sections -- including lines 6, 8, 10, and 12 -- contribute to the compile command, which has been wrapped for readability.

$ cat -n precedence_example4.cfg 
  1  label                = wall
  2  runlist              = 605.lbm_s
  3  tune                 = peak
  4  default:
  5     CC_VERSION_OPTION = -v
  6     CC                = mpicc
  7  small:
  8     OPTIMIZE          = -O1 
  9  default=peak:
 10     COPTIMIZE         = -ftree-vectorize
 11  default=default=wall:
 12     EXTRA_COPTIMIZE   = -Wall
$ cat precedence_example4.sh  
runhpc --config=precedence_example4 --fake | grep lbm.c
$ ./precedence_example4.sh 
mpicc -c -o lbm.o -DSPEC -DNDEBUG -O1 -ftree-vectorize           -Wall     lbm.c
$  
Precedence Example 5: Conflicting sections

If sections conflict with each other, the order of precedence is:

highest     benchmark
            tuning
lowest      label 

The first runhpc command includes --label=wall. It uses the OPTIMIZE setting from lines 13-14, which have higher precedence than the default (lines 15-16).

The second runhpc command includes both --label=wall and --tune peak. The OPTIMIZE setting from lines 11-12 is used.

The third also uses --label=wall --tune peak; and runs 705.lbm_l, which is an medium benchmark. The OPTIMIZE setting from lines 9-10 is used.

$ cat -n precedence_example5.cfg 
  1  flagsurl             = $[top]/config/flags/gcc.xml
  2  iterations           = 1
  3  output_format        = text
  4  output_root          = /tmp/ptest
  5  size                 = test
  6  default:
  7     CC_VERSION_OPTION = -v
  8     CC                = mpicc
  9  small:
 10     OPTIMIZE          = -O3 
 11  default=peak:
 12     OPTIMIZE          = -O2
 13  default=default=wall:
 14     OPTIMIZE          = -O1
 15  default:
 16     OPTIMIZE          = -O0
$ cat precedence_example5.sh 
runhpc --fake --config=precedence_example5 --label=wall 705.lbm_l         | grep lbm.c
runhpc --fake --config=precedence_example5 --label=wall -T peak  705.lbm_l | grep lbm.c
runhpc --fake --config=precedence_example5 --label=wall -T peak  605.lbm_m | grep lbm.c
$ ./precedence_example5.sh 
mpicc -c -o lbm.o -DSPEC -DNDEBUG -O1    lbm.c
mpicc -c -o lbm.o -DSPEC -DNDEBUG -O2    lbm.c
mpicc -c -o lbm.o -DSPEC -DNDEBUG -O3    lbm.c
$  
Precedence Example 6: Section order

For sections at differing precedence levels, order does not matter.

These two config files use a different order but perform the same functions when runhpc applies them..

$ diff --side-by-side precedence_example6a.cfg precedence_example6b.cfg
iterations           = 1                            iterations           = 1
output_format        = text                         output_format        = text
output_root          = /tmp/ptest                   output_root          = /tmp/ptest
runlist              = 605.lbm_s                    runlist              = 605.lbm_s
size                 = test                         size                 = test
default:                                            default:
   CC_VERSION_OPTION = -v                              CC_VERSION_OPTION = -v
   CC                = mpicc                             CC                = mpicc
605.lbm_s:                                        <
   OPTIMIZE          = -O3                        <
small:                                            <
   OPTIMIZE          = -O1                        <
default:                                            default:
   OPTIMIZE          = -O0                             OPTIMIZE          = -O0
                                                  > small:
                                                  >    OPTIMIZE          = -O1
                                                  > 605.lbm_s:
                                                  >    OPTIMIZE          = -O3
$ cat precedence_example6.sh
runhpc --fake --config=precedence_example6a | grep lbm.c
runhpc --fake --config=precedence_example6b | grep lbm.c
$ chmod +x precedence_example6.sh
$ ./precedence_example6.sh 
mpicc -c -o lbm.o -DSPEC -DNDEBUG -O3  lbm.c
mpicc -c -o lbm.o -DSPEC -DNDEBUG -O3  lbm.c
$  
Precedence Example 7: Repeated Sections

If a section occurs more than once, the settings are combined. If there are conflicts, the last instance wins.

Recall that trailing default specifiers can be dropped. Therefore, these three section markers:

 7  small:
12  small=default:
17  small=default=default:

actually name the same section, which is entered three times.

The EXTRA_CFLAGS, COPTIMIZE, and EXTRA_COPTIMIZE settings (lines 8, 14, 19) are combined.

The OPTIMIZE settings on lines 8, 13, and 18 conflict with each other. Only the last one is retained.

$ cat -n precedence_example7.cfg
  1  label                = wall
  2  runlist              = 605.lbm_s
  3  tune                 = peak
  4  default:
  5     CC_VERSION_OPTION = -v
  6     CC                = mpicc
  7  small:
  8     OPTIMIZE          = -O1 
  9     EXTRA_CFLAGS      = -finline-functions
 10  medium:
 11     OPTIMIZE          = -O0 
 12  small=default:
 13     OPTIMIZE          = -O2 
 14     COPTIMIZE         = -ftree-vectorize
 15  medium=peak:
 16     OPTIMIZE          = -O0 
 17  small=default=default:
 18     OPTIMIZE          = -O3 
 19     EXTRA_COPTIMIZE   = -Wall
$ cat precedence_example7.sh 
runhpc --config=precedence_example7 --fake | grep lbm.c 
$ ./precedence_example7.sh 
mpicc -c -o lbm.o -DSPEC -DNDEBUG  
    -O3 -ftree-vectorize  -finline-functions  -Wall   lbm.c
$  

I.F. Variables and Variable Substitution

You can do variable substitution within a config file.
As described above, the contents of a config file are directed to various consumers.
Therefore, to use variable substitution, you need to know which consumer does the substitution.
Differing syntax is used for each.

q. Wait a minute... all these choices for substitution? Which do I want?

a. Probably either the first in the list: specmake; or the last: the preprocessor.

Format Example Description + consumer Traps for the unwary
$(round) $(COMPILER_DIR) Make variables, interpreted by specmake  
\$SHELLVAR \$SPEC Shell or command interpreter variables. Quoting
$[square] $[top]/config Unchanging items, substituted by runhpc at startup. Timing
${curly}
$unbracketed
${hw_avail}ish
$hw_avail
Changeable items subject to perl interpolation.
$unbracketed is allowed if it is not ambiguous.
Try not to confuse with %{curly}
%{ENV_var} %{ENV_LIBRARY_PATH} Predefined macros for environment variables, handled by the preprocessor.  
%{curly} %{bits} macros, handled by the preprocessor Try not to confuse with ${curly}

I.F.1. Defining variables

You can create your own runhpc variables using a line of the form

name = value

The name may contain only letters, digits, and underscores (a hyphen is NOT allowed).
Start with a letter.
You may indent your definitions if you wish (see: whitespace)

Exception: preprocessor macros are different on all of the above. Hyphens are allowed, use

%define name value

and the % must appear in column 1. You can indent after the % if you wish.

Conventions: Although not required, certain conventions are usually followed:

  1. lower case is typically used for macros. When combining words, CamelCase may be useful.
  2. lower case is also typically used for runhpc variables, combining words with underscores
  3. CAPITALS are typically used when creating make variables
  4. CAPITALS are also typically used for shell or environment variables.

Examples of the above (in the same order)

%ifndef processorNumaControl
%   define processorNumaControl firstTouch # macro
%endif
default:
    my_submit_cmd       = numactl -C $BIND # runhpc variable
    MYTUNE              = -O2 --math=SIMD  # make variable
    ENV_LD_LIBRARY_PATH = /opt/lib         # environment variable

The remainder of this section I.F is about runhpc variables -- the $[square] and ${curly} rows from the table at the top.

I.F.2. $[square] substitution at startup

Immediately after preprocessing, variables that are delimited by $[square brackets] are substituted.
Any value set in the config file can be substituted, provided that is visible in the current scope.
You can access the value of additional variables that you may have created.
See the list of useful variables below.
Perhaps the most useful is $[top] for the top of the SPEChpc 2021 tree, often found in contexts such as:

    flagsurl01          = $[top]/config/compiler.xml
    flagsurl02          = $[top]/config/platform.xml
    EXTRA_LIBS          = $[top]/mylibs
    preENV_LIBRARY_PATH = $[top]/lib64:$[top]/lib

Traps for the unwary: In some cases it may be obvious to the human which value to use, but the tools aren't as smart as you.

  1. Timing: You cannot substitute variables that do not exist, or whose values aren't known when the config file is first read. For example, the label is not sorted out until later. Values that pertain to individual benchmarks (such as the current iteration number or the name of the benchmark being run) are not set until much later, when benchmarks are actually being run.
  2. Command line: If a value is mentioned on both the runhpc command line and in the config file, the substitution might not do what you expect.
  3. output_root:
    You cannot set an output_root that depends on a runhpc variable.
    You can set one that uses a macro:

    output_root=$[top]/my/path      # wrong
    output_root=${top}/my/path      # wrong
    output_root=%{ENV_SPEC}/my/path # right

Square substitution is done early. That comes in handy if you need a variable right away, for example, in order to use it with preENV.

$ cat EarlySub.cfg 
SW_DIR                 = /opt/path/to/compilers
preENV                 = 1
preENV_LD_LIBRARY_PATH = $[SW_DIR]/lib

$ cat EarlySub.sh
runhpc --config=EarlySub --fake 605.lbm_s | grep '^LD' | uniq
$ ./EarlySub.sh 
LD_LIBRARY_PATH=/opt/path/to/compilers/lib
$  

The example below uses variables defined in several named sections. The sections are delimited by section markers default: (line 8), default=base: (line 11), and default=peak: (line 15).

$ cat -n square.cfg
     1  expand_notes         = 1
     2  iterations           = 1
     3  output_format        = text
     4  output_root          = /tmp/square
     5  runlist              = 605.lbm_s
     6  size                 = test
     7  tune                 = base,peak
     8  default:
     9     CC                = mpicc
    10     CC_VERSION_OPTION = -v
    11  default=base:
    12     the_system        = STAR
    13     OPTIMIZE          = -O1
    14     notes_base_100    = base tuning uses '$[CC]' '$[OPTIMIZE]' on system '$[the_system]'
    15  default=peak:       
    16     OPTIMIZE          = -O2
    17     notes_peak_100    = peak tuning uses '$[CC]' '$[OPTIMIZE]' on system '$[the_system]'
    18  
$ cat square.sh 
runhpc --config=square | grep txt
grep tuning /tmp/square/result/hpc2021.001.small.test.txt
$ ./square.sh 
    format: Text -> /tmp/square/result/hpc2021.001.small.test.txt
     base tuning uses 'mpicc' '-O1' on system 'STAR'
     peak tuning uses 'mpicc' '-O2' on system ''
$  

Note that line 14 finds all three variables that it is looking for, but line 17 does not. If it is not clear why this happens, please see the descriptions of named sections and precedence above.

I.F.3. Useful $[square] variables

Useful $[variables] include:

$[configfile] Your config file name
$[configpath] The full path to your config file
$[dirprot] protection that is applied to directories created by runhpc
$[endian] 4321 or 87654321 for big endian; 1234 or 12345678 for little
$[flag_url_base] directory where flags files are looked up
$[OS] unix or windows
$[os_exe_ext] exe for windows, nil elsewhere
$[realuser] the user name according to the OS
$[top] the top directory of your installed SPEChpc 2021 tree
$[username] the username for purposes of tagging run directories
$[uid] the numeric user id

You can access the initial value of most options that you can enter into a config file, including:

action, allow_label_override, backup_config, basepeak, check_version, command_add_redirect, copies, delay, deletework, difflines, env_vars, expand_notes, expid, fake, flagsurl, http_proxy, http_timeout, ignore_errors, ignore_sigint, info_wrap_columns, iterations, label, line_width, locking, log_line_width, mail_reports, mailcompress, mailmethod, mailport, mailserver, mailto, make, make_no_clobber, makeflags, mean_anyway, minimize_builddirs, minimize_rundirs, no_input_handler, no_monitor, notes_wrap_columns, notes_wrap_indent, output_format, output_root, plain_train, rawformat, rebuild, reportable, runlist, section_specifier_fatal, setprocgroup, size, strict_rundir_verify, sysinfo_program, table, teeout, train_single_thread, tune, use_submit_for_speed, username, verbose, verify_binaries, version_url.

I.F.4. ${curly} substitution (interpolation) during a run
+ $unbracketed substitution

During a run, variables that are delimited by ${curly brackets} are substituted: ${command}
Usually, variables can be spelled with or without the curlies: $command or ${command}
Exception 1: curlies are required if the variable is adjacent to other text ${command}s

runhpc uses perl interpolation.
Only scalars (not: perl arrays and hashes) can be interpolated.

Example: on the notes100 line, you could optionally add say either ${lognum} or $lognum, but don't try taking the curly brackets away from ${size}.

$ cat just.cfg 
expand_notes      = 1
notes100          = Just ${size}ing, in run $lognum 
output_root       = /tmp/just
runlist           = 605.lbm_s
size              = test
CC                = mpicc
CC_VERSION_OPTION = -v
$ 
$ cat just.sh
runhpc -c just | grep txt
grep Just /tmp/just/result/hpc2021.001.small.test.txt
$   
$ ./just.sh 
    format: Text -> /tmp/just/result/hpc2021.001.small.test.txt
     Just testing, in run 001
$ 

Traps for the unwary

  1. Try not to confuse ${curly} interpolation vs. %{curly} (macros)
  2. Timing: Some variables are only defined at certain times, and a line that uses it might be interpolated at a different time. Therefore interpolation won't always do what you might wish. In particular, notes are not expanded in the context of a particular benchmark run, and therefore variables such as ${tune} are not useful within them.

  3. output_root:
    You cannot set an output_root that depends on a runhpc variable.
    You can set one that uses a macro:

    output_root=$[top]/my/path      # wrong
    output_root=${top}/my/path      # wrong
    output_root=%{ENV_SPEC}/my/path # right

I.F.5. Useful ${curly} variables

These variables may be of interest:

${baseexe} The first part of the executable name, which is <baseexe>_<tune>.<label>. For example, in "lbm_base.foo", baseexe is "lbm".
${benchmark} The number and name of the benchmark currently being run, for example 605.lbm_s
${benchname} The name of the benchmark currently being run, for example lbm
${benchnum} The number of the benchmark currently being run, for example 605
${benchtop} The top directory for the benchmark currently being run, for example /spec/hpc2021/benchspec/HPC/605.lbm_s
${command} The shell command line to run the current benchmark, for example ../run_base_test_sticky.0000/weather_base.sticky output6.ref.txt 24000 10000 3000 1250 600 100 6
${commandexe} The executable for the current command, for example ../run_base_test_none.0000/weather_base.sticky
${label} The label for the benchmark being run
${iter} The current iteration number
${logname} The complete log file name, for example /spec/hpc2021/result/hpc2021.168.log
${lognum} The log file number, for example 168
${tune} The tuning for the benchmark being run (base or peak)
${workload} The current workload number (within the iteration)

For a complete list of the available variables relative to the current config file, set

expand_notes = 1
verbose      = 35 # or higher

Then, do a run that causes a command substitution to happen.
In the log, you will find many lines of the form:

 - Variables available for interpolation that have changed since the last list:
    (From config) $runmode = "speed"
    (From config) $size = "test"
 - Variables available for interpolation that have changed since the last list:
    (From config) $size = "ref"

I.F.6. Unsetting a variable with "%undef%"

It is sometimes useful to undo the setting of a variable that would otherwise be included from another section. This can be accomplished using the special value %undef%. In the example, line 14 undefines COPTIMIZE when compiling peak:

 $ cat -n dave.cfg 
 1  flagsurl             = $[top]/config/flags/gcc.xml
 2  iterations           = 1
 3  output_format        = text
 4  output_root          = /tmp/undef
 5  runlist              = 605.lbm_s
 6  size                 = test
 7  tune                 = base,peak
 8  default:
 9     CC_VERSION_OPTION = -v
10     CC                = mpicc
11     OPTIMIZE          = -O2
12     COPTIMIZE         = -fno-tree-pre
13  605.lbm_s=peak:
14     COPTIMIZE         = %undef%
15  
$ runhpc --config=dave | grep txt
    format: Text -> /tmp/undef/result/hpc2021.001.small.test.txt
$ cd /tmp/undef/benchspec/HPC/605.lbm_r/build
$ grep OPTIMIZE build_base_none.0000/Makefile.spec 
COPTIMIZE        = -fno-tree-pre
OPTIMIZE         = -O2
$ grep OPTIMIZE build_peak_none.0000/Makefile.spec 
COPTIMIZE        = 
OPTIMIZE         = -O2
$ 

Notice that in the build directory, COPTIMIZE is present for base and absent for peak.

I.F.7. Debug tips for runhpc variables

When debugging a config file that uses runhpc variables, try:

iterations       = 1
minimize_rundirs = 0
reportable       = 0
runlist          = (one or two benchmarks)
size             = test 
verbose          = 40

Using --fake will probably be informative. Look inside the log for the (case-sensitive) word 'From'.

II. Config file options for runhpc

This section documents options that control the operation of runhpc itself.

II.A. Precedence: config file vs. runhpc command line

In the list that follows, some items are linked to the document Using SPEChpc 2021 - the 'runhpc' Command because they can be specified either in a config file, or on the runhpc command line.

New with SPEChpc 2021, If an option is specified in both places, the command line wins.

II.B. Options

In the table that follows, if an option is documented as accepting the values "no" and "yes", these may also be specified as "false" and "true", or as "0" and "1".

The "Use In" column indicates where the option can be used:

H use only in header section
N use in a named section.
H,N can be used in both the header section and in named sections. The item can therefore be applied on a global basis, and/or can be applied to individual benchmarks.
Option Use In Default Meaning
action H validate What to do. The available actions are defined in the runhpc document. See also the buildsetup example in the section on specmake.
allow_label_override H no

The runhpc command can use --label to select sections that apply to a specific label. If the label mentioned on the runhpc command does not occur in any section marker:

  • If allow_label_override is no (or 0 or false), the tools will print an error message and exit.
  • If allow_label_override is yes (or 1 or true), then the tools will attempt to build (or run) the same binary they would have built by default, but will name the binary with specified label.
This option was known as allow_extension_override prior to SPEChpc 2021.
backup_config H yes When updating the hashes in the config file, make a backup copy first. Highly recommended to defend against full-file-system errors, system crashes, or other unfortunate events.
basepeak H,N no Use base binary and/or base result for peak. If applied to the whole suite then only base is run, and its results are reported for both the base and peak metrics. If applied to a single benchmark, the same binary will be used for both base and peak runs, and the base results will be reported for both. (The reason for running the binary during both base and peak is to remove the possibility that skipping a benchmark altogether might somehow change the performance of some other benchmark.)
check_version H no (yes for
reportable
runs)

When set, before doing a reportable run, runhpc will download a small file (~15 bytes) from www.spec.org containing the current version of the suite and the date it was released, and check your copy vs. that file. In this way, you can be notified if the version of the suite that you're using is out-of-date.

Setting this variable to no will disable this check.

If you'd like to check a local file instead, you can modify version_url to point to your internal copy.

If you would like to check your version for a NON-reportable run, you will need to add --check_version to your command line. Setting check_version=yes in the config file only causes the check for reportable runs.

command_add_redirect H no

If set, the generated ${command} will include redirection operators (stdout, stderr), which are passed along to the shell that executes the command. If this variable is not set, specinvoke does the redirection. This option is commonly used with submit.

When used with submit, the command_add_redirect feature lets you choose whether redirection operators (such as <this_benchmark.in or >that_benchmark.out) are applied to your entire modified submit command (the default) or just to the portion that has ${command}.
In pseudo-code:

A benchmark runs exe; it sends output to out.
If you use:
submit = send_job job="${command}" rank_id=$rank
   Do you prefer this?
         send_job job="exe >out" rank_id=n
Or this?
         send_job job="exe" rank_id=n >out

The above is only pseudo-code; see the section on Using Submit for real examples.

Option Use In Default Meaning
delay H,N 0 Insert a delay of the specified number of seconds before and after benchmark execution. This delay does not contribute to the measured runtime of the benchmark. This delay is also not available in a reportable run.
deletework H,N no If set to yes, always delete existing benchmark working directories. An extra-careful person might want to set this to ensure no unwanted leftovers from previous benchmark runs, but the tools are already trying to enforce that property.
difflines H,N 10 Number of lines of differences to print when comparing results.
enable_monitor H,N yes

If this variable is set to no, then all of the monitoring hooks are disabled. This can be overridden by setting force_monitor. force_monitor is new with SPEChpc 2021

env_vars
or
envvars
H,N no

If set to yes, environment settings can be changed using ENV_* options in the config file. Note that you cannot change OMP_NUM_THREADS or ACC_NUM_CORES using this feature. Use threads.

Example: Consider the config file below, which sets the LD_LIBRARY_PATH to the location of the compiler runtime libraries for an alternate mpicc thats not used for the base.

 size        = test
 tune        = peak
 label       = srini
+env_vars    = 1
 #
 605.lbm_s=peak:
+   ENV_LD_LIBRARY_PATH = /tmp/compiler/lib64:%{ENV_LD_LIBRARY_PATH}
    CC                  = alt_mpicc
    CC_VERSION_OPTION   = -v
    OPTIMIZE            = -O1
 
  1. The syntax %{ENV_x} uses the preprocessor to import both the definition for SPEC and the definition for LD_LIBRARY_PATH as they were at runhpc startup. If the definitions for SPEC and LD_LIBRARY_PATH at the start of the run are, respectively, /A and /B, then runhpc will add 'LD_LIBRARY_PATH=/A/libraries:/B'  to the environment when it invokes the executable lbm_s_peak.srini.
  2. For SPEChpc 2021, the environment must be consistent during base (rule 3.4). You may vary it for peak.
  3. When developing a config file that uses env_vars, you may find it useful to set verbose to 35 (or higher), which will cause the tools to log environment settings.
  4. Which environment? If you are attempting to set the environment for your entire run, this is not the feature that you are looking for. Try preenv instead. The env_vars option and ENV* are about communication from the config file to the environment of the invoked benchmark.
expand_notes H no If set, will expand variables in notes. This capability is limited because notes are NOT processed by specmake, so you cannot do repeated substitutions.
expid H If set to a non-empty value, will cause executables, run directories, results, and log files to be put in a subdirectory (with the same name as the value set) in their normal directories. For example, setting expid = CDS will cause benchmark binaries to end up in exe/CDS, run directories to end up in run/CDS, and results and logs in $SPEC/result/CDS.
fail H,N no If set, will cause a build or run to fail.
fail_build H,N 0

If set, will cause a build to fail. For example, you could say something like this:

605.lbm_s=default:
#> I am posting this config file for use by others in the
#> company, but am forcing it to fail here because
#> I want to force users to review this section.
#>
#> Once you find your way here, you should test whether
#> bug report 234567 has been fixed, by using the first
#> line below.  If it has not been fixed, then use the
#> second.  In either case, you'll need to remove the
#> fail_build.
#>
#>   - Pney Guvaxre
#>     Boomtime, the 66th day of Confusion in the YOLD 3172

OPTIMIZE = -Osuperduper
# OPTIMIZE = -Omiddling
fail_build = yes

In the example above, the build is forced to fail until the user examines and modifies that section of the config file. Notice that Pney has used protected comments to cause the comments about the internal bug report to disappear if the config file were to be published as part of a reportable run.

fail_run H,N no If set, will cause a run to fail.
Option Use In Default Meaning
flagsurl H none

If set, retrieve the named URL or filename and use that as the "user" flags file. If the special value "noflags" is used, runhpc will not use any file and (if formatting previously run results) will remove any stored file. Automated processing of flags is explained in flag-description.html.

If you want to list more than one flagsfile, the recommended method is by using numbered continuation lines, for example:

     flagsurl1 = mycompiler.xml
     flagsurl2 = myplatform.xml 

Using other methods (such as backslash continuation) to specify multiple flags files may appear to work, but may result in unexpected differences between the original config file and the config file as written by output format config.

Multiple flags files will typically be needed, because flags files are separated into two types, "compiler", and "platform".

force_monitor H,N no

If this variable is set to yes, then all of the monitoring hooks are enabled, regardless of settings that would otherwise turn them off. This means that every invocation of specinvoke will be subject to monitor_specrun_wrapper, and all command invocations will be subject to monitor_wrapper. This includes things that would normally not be subject to monitoring, such as FDO training runs, input generation commands, and commands used for validating benchmark output such as specdiff. force_monitor is new with SPEChpc 2021.

http_proxy H

In some cases, such as when doing version checks and loading flag description files, runhpc will use HTTP to fetch a file. If you need to specify the URL of a proxy server, this is the variable to use. By default, no proxy is used. Note that this setting will override the value of the http_proxy environment variable. For example, one might set:

http_proxy = http://webcache.tom.spokewrenchdad.com:8080 
http_timeout H 30 This is the amount of time (in seconds) to wait while attempting to fetch a file via HTTP. If the connection cannot be established in this amount of time, the attempt will be aborted.
ignore_errors H no Ignore certain errors which would otherwise cause the run to stop. Very useful when debugging a new compiler and new set of options: with this option set, you'll find out about all the benchmarks that have problems, instead of only finding out about the first one.
ignore_sigint H no Ignore SIGINT. If this is set, runhpc will attempt to continue running when you interrupt one of its child processes by pressing ^C (assuming that you have ^C mapped in the common way). Note that this does NOT cause runhpc itself to ignore SIGINT.
info_wrap_columns H 50 When set to a value greater than 0, attempts to split non-notes informational lines such that they are no longer than info_wrap_columns columns wide. Lines are split on whitespace, and newly created lines are guaranteed to have at least the same indentation as the original line. If a line contains an item that is longer than info_wrap_columns, a warning is logged and the original line is left unchanged.
iterations H 3 Number of iterations to run.
keeptmp H no

Whether or not to keep various temporary files. If you leave keeptmp at its default setting, temporary files will be automatically deleted after a successful run. If not, temporary files may accumulate at a prodigious rate, and you should be prepared to clean them by hand. Temporary files include:

  • hpc2021.nnn.log.debug in your result directory ($SPEC/result, %SPEC%\result, or the corresponding location under your output_root).
  • A directory named tmp/ under one of these places:

    $SPEC/               or        %SPEC%\
    Whatever you set for an output_root directory
  • If tmp/ is not usable, you might find temporaries under tmp.hpc2021 or, possibly, tmp.hpc2021.n, where n may range from 1 to 15.
label H none

An arbitrary tag for executables, build directories, and run directories.
Legal characters are: alphanumerics, underscore, hyphen, and period. For SPEChpc 2021, this field replaces the field ext
Examples:

label=jun12.old.CC 
label=jun12.new.CC 
label=jun14-flagday 
label=jun15-robert.wants.yet.another.test

If a label is used as a section specifier, it can be referenced from the runhpc command line.
If a runhpc command uses a label that is not defined in a config file, by default the run will stop with an error:

runhpc --label=sandra [...]
ERROR: The label 'sandra' defines no settings in the config file!

The error can be disabled if you set allow_label_override=yes.

Tip: Make a habit of setting the label to anything that is meaningful to you. Doing so makes it easier to compare binaries, easier to track history, and easier to clean up by removing directories that match the label. Example:

This config file runs 605.lbm_s with 4 ranks of the (very short) test workload. It uses differing tuning depending on the label:

ranks               = 4
output_format        = text
runlist              = 605.lbm_s
default:
   CC_VERSION_OPTION = -v
   CC                = mpicc
default=base=OhZero:
   OPTIMIZE          = -O0
default=base=OhThree:
   OPTIMIZE          = -O3 

The label can be used when hunting down directories - either to analyze them or to delete them, as in the Unix find command. On other systems, try searching in a GUI file manager.

$ find . -type d -name '*OhZ*'
./benchspec/HPC/605.lbm_s/build/build_base_OhZero.0000
./benchspec/HPC/605.lbm_s/run/run_base_test_OhZero.0000
./benchspec/HPC/605.lbm_s/run/run_base_test_OhZero.0001
./benchspec/HPC/605.lbm_s/run/run_base_test_OhZero.0002
$ cd benchspec/HPC/605.lbm_s/build
$ grep OPTIMIZE */Makefile.spec
build_base_OhThree.0000/Makefile.spec:OPTIMIZE   = -O3
build_base_OhZero.0000/Makefile.spec:OPTIMIZE   = -O0
$ 
Option Use In Default Meaning
line_width H 0 Line wrap width for screen. If left at the default, 0, then lines will not be wrapped and may be arbitrarily long.
locking H yes Try to use file locking to avoid race conditions, e.g. if more than one copy of runhpc is in use. Although performance tests are typically done with only one copy of runhpc active, it can be handy to run multiple copies if you are just testing for correctness, or if you are compiling the benchmarks.
log_line_width H 0 Line wrap width for logfiles. If your editor complains about lines being too long when you look at logfiles, try setting this to some reasonable value, such as 80 or yes. If left at the default, yes, then lines will not be wrapped and may be arbitrarily long.
log_timestamp H no Whether or not to prepend time stamps to log file lines.
mailcompress H no When using the 'mail' output format, turning this on will cause the various report attachments to be compressed with gzip.
mailmethod H smtp When using the 'mail' output format, this specifies the method that should be used to send the mail. On UNIX and UNIX-like systems, there are three choices: 'smtp' (communicate directly with an SMTP server over the network), 'sendmail' (try invoking sendmail directly from locations where it is commonly installed), and 'qmail' (try invoking qmail-inject from locations where it is commonly installed).
Using a sendmail or qmail-inject program from a non-standard location is possible only by setting the PERL_MAILER environment variable. See the Mail::Mailer documentation for details.
mailport H 25 When using the 'mail' output format, and when the mailmethod is 'smtp', this specifies the port to use on the mail server. The default is the standard SMTP port and should not be changed.
mailserver H 127.0.0.1 When using the 'mail' output format, and when the mailmethod is 'smtp', this specifies the IP address or hostname of the mailserver through which to send the results.
mailto H ''

The address or addresses to which results should be sent when using the 'mail' output format. If multiple addresses are specified, they should be separated by commas or whitespace. Each address should consist only of the name@domain part (i.e. no "full name" type info). The addresses are not checked for correct formatting; if a mistake is made, the results may be sent to an unknown location. Think: comp.arch. OK, probably not there, but seriously be careful about security on this one.

Config files as posted at www.spec.org/hpc2021 will not include whatever you put on this line (thus, spambots will not see the contents of this field).

Note that to get your reports mailed to you, you need to specify both mail as an output_format and an address to which they should be mailed. For example:

   mailto=fast.guy@welovebenchmarks.org
   output_format=text,mail

If no addresses are specified, no mail will be sent.

mail_reports H all The list of report types to mail. The format and possible values are the same as for output_format, with the addition of log, which will cause the current log file to be sent. The default is for all files associated with the run to be mailed (so, this will include what you listed as your desired output_format plus log (the log file) and rsf (the rawfile). You can cut your email down to the bare essentials with something like this:
   mailto=fast.guy@welovebenchmarks.org
   output_format=text,mail
   mail_reports=text
If none of the requested report types were generated, no mail will be sent.
make H,N specmake Name of make executable. Note that the tools will enforce use of specmake for reportable results.
make_no_clobber H,N no Don't delete directories when building executables. The default is no, meaning "clobber". The "yes" option, meaning "avoid clobbering", should only be used for troubleshooting a problematic compile. The tools will not allow you to use this option when building binaries for a reportable result. Note that you could issue multiple successive runhpc commands with this option set (either in the config file, or with the --make_no_clobber switch), and the build directories will be preserved. But once you remove make_no_clobber (allowing it to default back to no), then the tools will attempt a normal build with a fresh build directory.
makeflags H,N ''

Extra flags for make (such as -j). Set this to -j n where n is the number of concurrent processes to run during a build. Omitting n or setting it to zero unlimits the number of jobs that will be run in parallel. Use with care! Make flags should be used here only if you are familiar with GNU make. (The program specmake is GNU Make under another name to ensure no accidental conflicts with other Make utilities you might have. The GNU Make Manual can be consulted, and you can also say specmake --help.)

Note that requesting a parallel build with makeflags = -j N causes multiple processors to be used at build time. It has no effect on whether multiple processors are used at run time, and so does not affect how you report on parallelism.

Option Use In Default Meaning
mean_anyway H no Calculate mean even if invalid. DANGER: this will write a mean to all reports even if no valid mean can be computed (e.g. half the benchmarks failed). A mean from an invalid run is not "reportable" (that is, it cannot be represented in public as the SPEC metric).
minimize_rundirs H no During a run, try to keep working disk size down. Cannot be used in a reportable run.
minimize_builddirs H no Try to keep working disk size down during builds.
nobuild H no Do not attempt to build benchmarks. Useful to prevent attempts to rebuild benchmarks that cannot be built. Also comes in handy when testing whether proposed config file options would potentially force an automatic rebuild.
no_monitor H,N '' Exclude the listed workloads from monitoring via the various monitor_* hooks.
no_input_handler H,N close

Method to use to simulate an empty input. Choices are:

  • close - Just close stdin before executing the benchmark (this is the default).
  • zero - Create a zero-length file and attach it to stdin.
  • file - Synonym for zero.
  • null - Open the null device (/dev/null or NUL:) and attach it to stdin.

Normally, this option should be left at the default; it was actually added to the tools for the benefit of a different SPEC suite that needed the feature. If a reportable run for SPEChpc 2021 uses this feature, an explanation should be provided as to why it was used.

notes_wrap_columns H 0 When set to a value greater than 0, attempts to split notes lines such that they are no longer than notes_wrap_columns columns wide. Lines are split on whitespace, and newly created lines are guaranteed to have at least the same indentation as the original line. If a line contains an item that is longer than notes_wrap_columns, a warning is logged and the original line is left unchanged.
notes_wrap_indent H ' ' When line wrapping is enabled (see notes_wrap_columns), this is the string that will be prepended to newly created lines after the indentation from the original line is applied. The default is four spaces, but it can be set to any arbitrary string.
output_format H all Format for reports. Valid options are listed at runhpc.html under --output_format; major options include txt (ASCII text), html, pdf, and ps. You might prefer to set this to txt if you're going to be doing lots of runs, and only create the pretty reports at the end of the series. See also the information in runhpc.html about --rawformat.
output_root H

If set to a non-empty value, all output files will be rooted under the named directory, instead of under $SPEC (or %SPEC%).
If the output_root is not an absolute path (one that begins with "/" on Unix, or a device name on Windows), the path will be created under $SPEC or %SPEC%. You probably will find it more useful if you pick some place other than the default, for convenience when cleaning directories, allocating quotas, or optimizing I/O.

You can navigate a rooted directory with ogo.
This option can be useful for sharing an installation.
It can also can be useful to optimize your I/O for a big SPECrate run: you can put the output_root on your faster device.

Example: Junjie has a 500 TB optimized file system under /local/home. If he types

runhpc --output_root=/local/home/junjie --config=mine 605.lbm_s 
These directories will be created For these purposes
log files and reports /local/home/junjie/result/
605.lbm_s build directories /local/home/junjie/benchspec/HPC/605.lbm_s/build/
605.lbm_s benchmark binaries (executables) /local/home/junjie/benchspec/HPC/605.lbm_s/exe/
605.lbm_s run directories /local/home/junjie/benchspec/HPC/605.lbm_s/run/
Option Use In Default Meaning
pmodel N MPI Optionally select the node level parallel model to use for each benchmark. If not set, the default is to build with MPI only.

Important! This option only sets the appropriate macro define flag for the model's directives to be visible in the post-processed source. You must also set the appropriate compiler option to enable the use of the model (ex. "-fopenmp") in OPTIMIZE. Also, not all models are supported by all compilers. Please consult your compiler's documentation for the appropriate options and which models they support.

pmodel setting
MPIUse MPI only without a node level parallel model.
ACCUse MPI+OpenACC.
OMPUse MPI+OpenMP using task/thread based directives.
TGTUse MPI+OpenMP using 'target' based directives.

preenv
or
pre_env
H yes

Use preENV_ lines in the config file. When this option is set (the default), lines of the form preENV_<variable> = <value> will cause runhpc to set the specified environment variable to value and re-exec runhpc to perform the run. The restart is done in order to ensure that the entire run takes place with the new settings. You can set preENV_SOME_VARIABLE = value only in the header section or a section using one these benchmark specifiers:

default:
tiny:
small:
medium:
large:

Any attempt to use preENV_ in other sections is silently ignored.
You cannot change OMP_NUM_THREADS or ACC_NUM_CORES using this feature. Use threads.
Multiple preENV_ settings may appear in the config file.
If you are looking for a way to affect the environment of an individual benchmark, try env_vars, not preenv.

Example: Above, the section on env_vars showed how a library path can be adjusted for a single benchmark.
Instead of adjusting one benchmark, the environment could be set for all benchmarks by putting a line for preENV_LD_LIBRARY_PATH at the top of the config file:

preENV_LD_LIBRARY_PATH = %{ENV_SPEC}/libraries:%{ENV_LD_LIBRARY_PATH}
ranks H yes The number of ranks to be used to run each benchmark. The value is required to be set, either in the config file or on the runhpc command line, so the reports will be formatted correctly. Use $ranks to pass the count to the MPI driver, as shown in the submit section.
reportable H yes Strictly follow reporting rules, to the extent that it is practical to enforce them by automated means. The tester remains responsible for ensuring that the runs are rule compliant. You must set reportable to generate a valid run suitable for publication and/or submission to SPEC.
rebuild H no Rebuild binaries even if they exist.
runlist H none Benchmarks or sets to run. Names can be abbreviated, just as on the command line. See the long discussion of run order in runhpc.html.
save_build_files H,N none

After a build is finished, files matching any of the space-delimited wildcard patterns in this variable will be gathered up and saved. When a non-reportable run is being set up, those files will be unpacked into the run directory when the executable is copied in. The auxiliary file package is ignored when setting up reportable runs. The save_build_files feature is new with SPEChpc 2021.

section_specifier_fatal H yes

While parsing the config file, if a section specifier is found that refers to an unknown benchmark or benchset, an error is output and the run stops. Set section_specifier_fatal=no in the header section of your config file to convert this error into a warning and allow the run to continue. The ability to convert section specifier errors into warnings is probably of use only for benchmark developers.

setprocgroup H yes Set the process group. On Unix-like systems, improves the chances that ^C gets the whole run, not just one of the children.
size H ref

Size of input set: test or ref
If you are in the early stages of testing a new compiler or new set of options, you might set this to test.
The reference workload (ref) is the only size whose time appears in reports.

You might choose to use runhpc --size=test while debugging a new set of compilation options.

Reportable runs only invoke the ref workload

Option Use In Default Meaning
src.alt N none

Name of a SPEC-approved alternate source.
Set the option in the named section for the benchmark(s) where you wish to use it.
Multiple src.alts may be specified a single benchmark. Separate them wth commas.
The effect of using a src.alt is to modify the sources in the build directory.
You may also spell this option as srcalt.

 

About alternate sources: Sometimes a portability issue may require use of different source code for a benchmark, and SPEC may issue a src.alt, which is a compressed tar file containing modifications, created by makesrcalt.

To use a src.alt, see the instructions posted with it at www.spec.org/hpc2021/src.alt.
Typically, you begin by downloading it to your $SPEC or %SPEC% directory and unpacking it:

$ cd $SPEC   
$ specxz -dc nnn.benchmark.FixMumble.tar.xz | spectar -xvf -
$ cat README.nnn.benchmark.src.alt.FixMumble.txt

The README will explain what to add to your config file and any other instructions that are needed.

After you unpack it, a directory is created under under <benchmark>/src/src.alt/ and a set of patches are stored there. You can look at the patches using dumpsrcalt, but it may be easier to just apply the src.alt and look at a build directory.

Example: This config file builds with or without a src.alt, depending on the runhpc setting for the --label.

$ cat testme.cfg
action               = buildsetup
runlist              = nnn.benchmark
tune                 = base
default:
   CC                = mpicc
   CC_VERSION_OPTION = -v
nnn.benchmark=base=without:
   OPTIMIZE          = -O2
nnn.benchmark=base=with:
   OPTIMIZE          = -O2
   srcalt            = FixMumble

To populate build directories:

runhpc --label=without --config=testme
runhpc --label=with    --config=testme

Then, if you visit benchspec/CPU/nnn.benchmark/build you can compare the directories.

Option Use In Default Meaning
strict_rundir_verify H yes

When set, the tools will verify that the file contents in existing run directories match the expected checksums. Normally, this should always be on, and reportable runs will force it to be on. Turning it off might make the setup phase go a little faster while you are tuning the benchmarks.

Developer notes: setting strict_rundir_verify=no might be useful when prototyping a change to a workload or testing the effect of differing workloads. Note, though, that once you start changing your installed tree for such purposes it is easy to get lost; you might as well keep a pristine tree without modifications, and use a second tree that you convert_to_development.

table H yes In ASCII reports, include information about each execution of the benchmark.
teeout H no Run output through tee so you can see it on the screen. Primarily affects builds, but also provides some information about progress of runtime, by showing you the specinvoke commands.
threads H,N 1 Value to be set for OMP_NUM_THREADS or ACC_NUM_CORES when benchmarks are run.
If you have set these in your environment prior to using runhpc, that setting will be ignored. To use multiple benchmark threads, you must use either this config file field or the corresponding option runhpc --threads=N.
tune H base default tuning level. In a reportable run, must be either all or base.
verbose H 5 Verbosity level. Select level 1 through 99 to control how much debugging info runhpc prints out. For more information, see the section on log files, below.
verify_binaries H yes

runhpc uses checksums to verify that executables match the config file that invokes them, and if they do not, runhpc forces a recompile. You can turn that feature off by setting verify_binaries=no.

Warning: It is strongly recommended that you keep this option at its default, yes (that is, enabled). If you disable this feature, you effectively say that you are willing to run a benchmark even if you don't know what you did or how you did it -- that is, you lack information as to how it was built!

The feature can be turned off because it may be useful to do so sometimes when debugging (for an example, see env_vars), but it should not be routinely disabled.

Since SPEC requires that you disclose how you build benchmarks, reportable runs (using the command-line switch --reportable or config file setting reportable=yes) will cause verify_binaries to be automatically enabled. For SPEChpc 2021, this field replaces the field check_md5

version_url H http://
www.
spec.org/
hpc2021/
current_
version
If version checking is enabled, this specifies the location from which the version information should be fetched.

III. Config file options for specmake

For SPEChpc you do not write Makefiles. Instead, you set Make variables in the config file, which are sent to a SPEC-supplied copy of GNU Make, known as specmake. Variables with a dollar sign and parentheses, aka "round brackets", are substituted by specmake. For example:

COMPILER_DIR = /usr/local/bin/
CC           = $(COMPILER_DIR)cc
CXX          = $(COMPILER_DIR)c++
FC           = $(COMPILER_DIR)f90

See below for more information on syntax of variables that you create and reference.

III.A. Commonly used Make variables

  • Basic specmake variables are described in this section.
  • Many more are described in the Make Variables document.
  • The specmake utility can be run directly from the command line, as described in Utilities.

The following Make variables are frequently useful. When selecting where to put a flag, please bear in mind that the run rules require that portability flags must use PORTABILITY variables.

CC How to invoke your C compiler.
CXX How to invoke your C++ compiler.
FC How to invoke your Fortran compiler.
CC_VERSION_OPTION
CXX_VERSION_OPTION
FC_VERSION_OPTION

New with SPEChpc 2021: You must specify how to ask each compiler "Please tell me your version" because the method varies from compiler to compiler. The version information is recorded contemporaneously with the build.

You must specify the option for all compilers that you use.
If you change compilers for some benchmarks, be sure to also change the corresponding version option.

PORTABILITY
EXTRA_PORTABILITY
Portability flags to be applied no matter what the compiler.
{C|CXX|F}PORTABILITY
EXTRA_{C|CXX|F}PORTABILITY
Portability flags to be applied to modules of the designated language
(For example, CXXPORTABILITY is for the C++ modules).
OPTIMIZE EXTRA_OPTIMIZE Optimization flags to be applied for all compilers.
{C|CXX|F}OPTIMIZE
EXTRA_{C|CXX|F}OPTIMIZE
Optimization flags to be applied to modules of the designated language.
EXTRA_{C|CXX|F}FLAGS Flags that are neither optimization nor portability
LIBS Libraries to add to the link line
Many more See chart in document Make Variables

III.B. Parallel Model Selection

By default, the SPEChpc benchmarks use only MPI parallelism. However, you may also enable either OpenACC or OpenMP directives to enable node-level parallelism. The benchmark will then be hybrid MPI+OpenACC or MPI+OpenMP.

For OpenMP, you may select either "traditional" OpenMP using CPU task/threads, or "Target" OpenMP which includes OpenMP 5.0 'target' constructs that may use to target a host CPU or offload to accelerator device.

Note if you choose to use a node-level parallel model, the same model must be used for all benchmarks in base. For peak, you may select the model on a per benchmark basis.

To enable a node-level parallel mode, use the appropriate 'pmodel' setting. The tools then set the appropriate preprocessor define flag which makes the directives visible in the source. Note that not all compilers support all models and the compiler flag that enables the model must be set in your OPTIMIZE variable in the config file. Please consult your compiler's documentation for the supported models and options.

pmodel settingModelDefine flag implicitly added
MPIMPI-Only (default)None.
ACCMPI+OpenACC-DSPEC_OPENACC
OMPMPI+OpenMP using CPU task/threads-DSPEC_OPENMP
TGTMPI+OpenMP using Target constructs-DSPEC_OPENMP_TARGET

Note that using the define flags directly in you OPTIMIZE or PORTABILITY flags will cause the tools to error. This is to help reduce the possibiltiy of setting the pmodel to one model but then accidently enabling a different model.

The config file fragment below demonstrates how to conditionally include a parallel model selected via runhpc --define option:

% cat -n config/pmodel.cfg
     1  # If model is not set use Only MPI
     2  %ifndef %{model}
     3  pmodel=MPI
     4  %endif
     5
     6  # Compiler flags applied to all models
     7  default:
     8    CC             = mpicc
     9    OPTIMIZE       = -O3
    10    CXXPORTABILITY = --c++14
    11    submit = mpirun --bind-to none -mp $ranks $command
    12
    13  # OpenACC flags
    14  %if %{model} eq 'acc'
    15    pmodel = ACC
    16    OPTIMIZE += -acc
    17  %endif
    18
    19  # OpenMP (CPU Threaded) flags
    20  %if %{model} eq 'omp'
    21     pmodel = OMP
    22     OPTIMIZE += -mp
    23  %endif
    24
    25  # OpenMP (Target) flags
    26  %if %{model} eq 'omptgt'
    27     pmodel=TGT
    28     OPTIMIZE += -mp=gpu
    29  %endif
    30
    31  605.lbm_s=peak:
    32    pmodel = ACC
    33    OPTIMIZE = -acc -fast
    34
    35  # This will error since the model define flags
    36  # should only be set by the tools
    37  705.lbm_m=peak:
    38    OPTIMIZE = -mp -fast -DSPEC_OPENMP
    39
% runhpc -c pmodel --fake -ranks=4 605.lbm_s | grep lbm.c
mpicc -c -o lbm.o -DSPEC -DNDEBUG   -O3                  lbm.c
% runhpc -c pmodel --fake -ranks=4 --define model=acc 605.lbm_s | grep lbm.c
mpicc -c -o lbm.o -DSPEC -DNDEBUG   -O3 -acc           -DSPEC_OPENACC        lbm.c
% runhpc -c pmodel --fake -ranks=4 --define model=omp 605.lbm_s | grep lbm.c
mpicc -c -o lbm.o -DSPEC -DNDEBUG   -O3 -mp           -DSPEC_OPENMP        lbm.c
% runhpc -c pmodel --fake -ranks=4 --define model=omptgt 605.lbm_s | grep lbm.c
mpicc -c -o lbm.o -DSPEC -DNDEBUG   -O3 -mp=gpu           -DSPEC_OPENMP_TARGET        lbm.c
% runhpc -c pmodel --fake -ranks=4 --tune peak 605.lbm_s | grep lbm.c
mpicc -c -o lbm.o -DSPEC -DNDEBUG   -acc -fast           -DSPEC_OPENACC        lbm.c

Note for peak only OpenACC is used from 605.lbm_s since it's not guarded by the define.

Using a pmodel's define flag in the config will cause the following error:

% runhpc -c pmodel --fake -ranks=4 --tune peak 705.lbm_m
SPEC HPC(r) 2021 Benchmark Suites
Copyright 1995-2021 Standard Performance Evaluation Corporation (SPEC)

runhpc v.unknown
Using 'linux-x86_64' tools
Reading file manifests... read 16870 entries from 2 files in 0.08s (200294 files/s)
... cut ...
Compiling Binaries

ERROR: pmodel define flag "-DSPEC_OPENMP" may only be set by the tools.
Please remove it from the following line of your config file.
OPTIMIZE         = -acc -fast -DSPEC_OPENMP
... cut ... 

III.C. Creating your own Make variables

Variables with a dollar sign and parentheses, aka "round brackets", are substituted by specmake.

Deprecated feature alert: Although it is also possible to pass information to specmake using curly brackets: ${COMPILER_DIR}, this is not recommended. Instead, you should consistently use curly brackets to address runhpc and round brackets to address specmake. It is possible that a future version of runhpc may insist on interpolating curly brackets itself, rather than allowing specmake to do so.

Example:

$ cat makevar.cfg 
action            = build
runlist           = 605.lbm_s
default:
   DEBUG_SYMBOLS  = --debug:symbols=expanded_info_level:42
   EXTRA_FFLAGS   = $(GEE)
small=base:
   GEE            = -g $(DEBUG_SYMBOLS)
small=peak:
   GEE            = -g 

$ cat makevar.sh
runhpc --config=makevar --fake --tune=base | grep COMP:
runhpc --config=makevar --fake --tune=peak | grep COMP:
$ ./makevar.sh 
COMP: "cc -c -o lbm.o -g --debug:symbols=expanded_info_level:42 <source>"
COMP: "cc -c -o lbm.o -g <source>"
$  

The config file above creates two variables options (DEBUG_SYMBOLS and GEE). Both are passed to specmake, which interprets them. The results are shown above using the runhpc --fake option.

For an extensive example of variable substitution handled by specmake, see the SPEC CPU®2000 example at www.spec.org/cpu2000/docs/example-advanced.cfg. Search that file for LIBS, and note the long comment which provides a walk-through of a complex substitution handled by specmake.

III.D. The operator "+=" is available (but should be used with caution)

The operator "+=" adds to specmake variables. It may be convenient; it also may cause hard-to-diagnose bugs. Example:

$ cat tmp.cfg  
action  = build
runlist = 605.lbm_s
tune    = peak
default:
   OPTIMIZE   = -O1
fprate=default:
   OPTIMIZE  += --unroll
default=peak:
   OPTIMIZE  += --inner_unroll
605.lbm_s=peak:
   OPTIMIZE  += --outer_unroll
default=default=breakfast:
   OPTIMIZE   = --jelly_roll

$ runhpc --fake --config=tmp | grep lbm.c
cc -c -o lbm.o -DSPEC -DSPEC_CPU -DNDEBUG -O1 --inner_unroll --unroll --outer_unroll lbm.c
$   

Note that the options accumulate.

Caution: although the += operator adds flexibility, it may introduce hard-to-predict behavior, depending on precedence of section specifiers, the order of your config file, and other features, such as include files. Instead of using '+=' try picking different make variables for different purposes. For an example of hard-to-predict behavior, what will happen if you add --label=breakfast to the above runhpc command? (Try it.)

Recommendations:
Avoid += to prevent surprises.
Keep it simple.
Pick different makevars for different purposes.
Create conventions for your config files and write them down, in config file comments.
If you must use += review its effects carefully (--fake is your friend).

III.E Using action=buildsetup to create a sandbox

When debugging a set of build options, it is often useful to create a "sandbox" - that is, a directory where you can play with the benchmark and its options. This example creates a build sandbox with action buildsetup.

$ cat sandbox.cfg 
action        = buildsetup
label         = fast
output_root   = /tmp/demo_buildsetup
runlist       = 605.lbm_s
tune          = peak
default=peak:
   OPTIMIZE   = -fast
$ cat sandbox.sh 
runhpc --config=sandbox | grep log
grep Makefile.spec /tmp/demo_buildsetup/result/hpc2021.001.log
$ ./sandbox.sh 
The log for this run is in /tmp/demo_buildsetup/result/hpc2021.001.log
Wrote to makefile '/tmp/demo_buildsetup/benchspec/HPC/605.lbm_s/build/build_peak_fast.0000/Makefile.spec':

The action causes a directory to be created
There, the Makefile can be examined, used in a dry run, or modified as part of a testing effort.

$ cd /tmp/demo_buildsetup/benchspec/HPC/605.lbm_s/build/build_peak_fast.0000/
$ grep OPTIMIZE Makefile.spec 
OPTIMIZE         = -fast
$ specmake --dry-run
cc -c -o lbm.o -DSPEC -DNDEBUG   -fast                  lbm.c
cc -c -o main.o -DSPEC -DNDEBUG   -fast                  main.c
cc -c -o specrand/specrand.o -DSPEC -DNDEBUG   -fast                  specrand/specrand.c
cc      -fast          lbm.o main.o specrand/specrand.o             -lm         -o lbm

See also the chapter on specmake in SPEChpc 2021 Utilities and the sandbox examples in Avoiding runhpc.

III.F. About Automatic Rebuilds

The SPEChpc tools try to keep config files and binaries synchronized with each other.(*)
Edits to a config file may cause binaries to be rebuilt, sometimes to the surprise(**) of testers.

Testing option sensitivity: The first thing that happens in a rebuild is to delete the old binary.
If that is a potential problem (perhaps it takes a long time to build), you can test whether a config file change will cause a rebuild:

  1. Copy the config file to a test version (say, mycopy.cfg)
  2. Edit the copy.
  3. Remove any lines that change defaults for ignore_errors or verify_binaries
  4. Use: runhpc --config=mycopy --nobuild --size=test --iterations=1 ...

The command:

No binaries are harmed.

Notes:

(*) Unless you change verify_binaries. Recommendation: Don't change it.

(**) Recent implementations surprise less often. For detail, see the CPU2006 version of this section.

IV. Config file options for the shell

Some options in your config file cause commands to be executed by your shell (/bin/sh).

IV.A. \$SHELLVAR variable substitution

Substitution by the shell uses backslash dollar sign.
The backslash protects the variables from interpretation by runhpc.

Example: This config file runs 605.lbm_s twice, with base and peak options. Only peak uses backslashes:

$ cat tmp.cfg
expand_notes         = 1
iterations           = 1
runlist              = 605.lbm_s
size                 = test
tune                 = base,peak
default:
   CC                = mpicc
   CC_VERSION_OPTION = -v
default=base:
   submit    = echo home=$HOME, spec=$SPEC   > /tmp/chan; ${command}
default=peak:
   submit    = echo home=\$HOME, spec=\$SPEC > /tmp/nui;  ${command}
$ runhpc --config=tmp | grep txt
    format: Text -> /Users/bo/spec/hpc2021/result/hpc2021.697.small.test.txt
$ cd /tmp
$ cat chan nui
home=, spec=
home=/Users/chris, spec=/Users/chris/spec/hpc2021
$
   

In base, $HOME and $SPEC are gobbled up by runhpc, which obediently retuns their values: nothing at all.
In peak, backslashes prevent runhpc from interpreting the variables, and the shell provides their values.

IV.B. Shell Options

Warning: SPEChpc config files can execute arbitrary shell commands.
Read a config file before using it.
Don't be root. Don't run as Administrator. Turn privileges off.

These options cause commands to be executed:

bench_post_setup

Command to be executed after each benchmark's run directory setup phase. The rules say that this feature may be used to cause data to be written to stable storage (e.g. sync). The command must be the same for all benchmarks. It will be run after each benchmark is setup, and for all workloads (test/train/ref). It is not affected by the setting of parallel_test.

Use the header section for bench_post_setup.

build_pre_bench
build_post_bench
Commands for benchmark monitoring, described in the document on the Monitoring Facility.
monitor_X Commands that allow benchmark monitoring, described in the document on the Monitoring Facility.
post_setup

Command to be executed after all benchmark run directories have been set up. The rules say that this feature may be used to cause data to be written to stable storage (e.g. sync).

Notes:

  1. This option may only be specified in the header section.
  2. The post_setup option is not executed if minimize_rundirs is set.
  3. If parallel_test is greater than 1, post_setup is not executed for test and train.
submit

Modified command to actually run a benchmark. The default is in ${command}, which the rules allow you to supplement, for example by including the mpiexec command or batch scheduler command.

IV.C. Using submit

The config file feature submit allows you specify the command needed to launch your MPI command as well as include and process to core bindings. This section provides examples to demonstrate how submit works with several other config file features. You might also want to search published results at www.spec.org/hpc2021 for systems that are similar to your system.

IV.C.1. Basic usage of submit

You can use submit to specify the MPI or batch scheduler command used to launch your MPI jobs.

Submit Example 1: ${command} and mpirun

The example below runs 4 ranks using OpenMPI's "-bind-to cores" option to bind ranks to cores.

$ cat submitmpi.cfg 
ranks   = 4
runlist  = 605.lbm_s
submit   = mpirun --bind-to cores -np ${ranks} ${command}
$ cat submitmpi.sh
runhpc --fake --config=submitmpi | grep '^mpirun' | cut -b 1-75
$ ./submitmpi.sh 
mpirun --bind-to cores -np 4 ../run_base_ref_none.0000/lbm_base.none 
$  

IV.C.2. Using scripts with submit for thread binding per rank

You can use scripts with submit to specify the thread binding for individual MPI ranks. While setting environment variables such as OMP_PLACES in your config's environment setting or using utilities such as numactl work fine when there's a single rank per node, using these with multiple ranks would cause the threads from each rank to be bound to the same core. Several MPI and batch schedulers will perform this binding correctly for you. However in cases where it doesn't, you can provide a script to perform the binding.

Submit Example 2: Using script ${command} and mpirun

The example below runs 8 ranks with each running 16 OpenMP threads. The example uses a perl script to determine the local rank number via OpenMPI's OMPI_COMM_WORLD_LOCAL_RANK environment variable, and then bind the rank's threads via the numactl. Note: please check your MPI distribution's documentation for the equivalent environment variable.

The sample for a two socket system with 128 cores with 8 NUMA nodes. You would need to modify the script for different systems.

$ cat bind8.pl
use strict;
my %bind;
$bind{0} = "0-15";
$bind{1} = "16-31";
$bind{2} = "32-47";
$bind{3} = "48-63";
$bind{4} = "64-79";
$bind{5} = "80-95";
$bind{6} = "96-111";
$bind{7} = "112-127";

my %mem;
$mem{0} = "0";
$mem{1} = "1";
$mem{2} = "2";
$mem{3} = "3";
$mem{4} = "4";
$mem{5} = "5";
$mem{6} = "6";
$mem{7} = "7";


my $rank = $ENV{OMPI_COMM_WORLD_LOCAL_RANK};
my $cmd = "numactl -C $bind{$rank} -m $mem{$rank} ";
#my $cmd = "taskset -c $bind{$rank} ";
while (my $arg = shift) {
        $cmd .= "$arg ";
}
my $rc = system($cmd);
exit($rc);

Assuming the bind8.pl is copied to the top directory in your SPEChpc installation, you would then use the following submit command to execute it:

$ cat config/submitmpi2.cdf
ranks    = 8
runlist  = 605.lbm_s
submit   = mpirun --bind-to cores -np ${ranks} perl $[top]/bind8.pl ${command}

IV.C.3. Reporting of submit usage

If you use the submit feature, a notes section will automatically be created to indicate that you have done so.

                                 Submit Notes
                                 ------------
    The config file option 'submit' was used.

You can add notes to that section, or customize it as you wish, by creating lines with notes_submit_NNN. The phrase The config file option 'submit' was used must appear somewhere in your customized notes. You can vary the capitalization of the phrase, you can even break it across multiple lines; it just needs to be present. If it is not, it will automatically be added. If you used scripts in your submit command, the full text of the script should be listed in the submit notes.

V. Config file options for readers

Whether or not you send your result to SPEC, you should fully disclose how you achieved the result. If it requires the installation of the GoFastLinker, you should say so. By setting the appropriate fields in the config file, you can cause information about the GoFastLinker to appear in the reports that are intended for humans.

V.A. Descriptive fields

Here are the fields that you can set to describe your testbed to readers:

license_num The SPEC license number for either the tester or the test_sponsor.
prepared_by Is never output. If you wish, you could set this to your own name, so that the rawfile will be tagged with your name but not the formal reports.
hw_avail The availability date of the system hardware.
system_name The name assigned to the kind of system. Usually this name collectively describes the key components of the system, e.g. SGI Altix 4700 Cluster (4 nodes at 1600 MHz) with NumaLink4, but in some cases the name may not derive from its components: Our Super Duper Cluster Computer.
system_vendor The Vendor of the particular system. Usually this is the same as the manufacturer of the node used, but a system may also be constructed from 3rd-party commodity boards and the Vendor is the manufacturer of the integrated system rather than the component boards.
system_class The type of compute system:
HomoHomogeneous cluster (all nodes identical).
HeteroHeterogeneous cluster (different kinds of nodes).
SMPSymmetric Multi-Processor (one node containing multiple CPU's).
sw_avail The availability date of the system software.
tester The entity actually carrying out the tests. An optional field; if not specified, defaults to test_sponsor. An example is given in the test_sponsor section.
test_date When the tests were run. Note that the test date will be automatically set by runhpc.
test_sponsor

The entity sponsoring this test. An optional field; if not specified, defaults to system_vendor.

For example, suppose that the Genius Compiler Company wants to show off their new compiler on the TurboBlaster 9000 computer, but does not happen to own a maxed-out system with eight thousand processors. Meanwhile, the Pawtuckaway State College Engineering department has just taken delivery of such a system. In this case, the compiler company could contract with the college to test their compiler on the big machine. The fields could be set as:

      test_sponsor = Genius Compilers
      tester       = Pawtuckaway State College
      system_vendor    = TurboBlaster

The second set of parameters can be continued to multiple lines by repeating the variable name multiple times with different suffixes:

sw_c_compiler Name and version of the C compiler.
sw_cxx_compiler Name and version of the C++ compiler.
sw_f_compiler Name and version of the Fortran compiler.
sw_mpi_library The type and version of the MPI library used.
sw_mpi_other Other information on the MPI implementation or settings.
sw_other Any other performance-relevant non-compiler software used, including third-party libraries, accelerators, pre-processors, etc.

V.A.I Component attributes

A system running MPI may be a standalone SMP or a cluster of nodes with dedicated file servers. The SPEChpc 2021 reporting conventions allow you to describe highly heterogeneous systems by specifying the attributes of multiple kinds of nodes, interconnects and file servers.

The output report will contain a separate section describing each of these components. Each one of these sections consists of a formatted table derived from the fixed-field entries in the config-file, followed by free-form notes derived from the notes entries of the config-file.

Each component is described with a prefix that defines its type, followed by the field to be assigned:

        node_IBM655_label = IBM 655 Server                      # This system has only one kind of node
        node_IBM655_count = 16                                  # and uses 16 of them.
        node_IBM655_count = compute,fileserver                                  # One of the nodes is also the file server.
        node_IBM655_notes = Node 0 is the file server for the cluster.

        node_HP2000_label = HP 2000 Server
        node_HP3000_label = HP 3000 Server                      # This system has two kinds of nodes
        node_HP2000_count = 12                                  # with different numbers of each.
        node_HP3000_count = 20
        node_HP2000_purpose = compute
        node_HP3000_purpose = compute

        interconnect_GBEthernet_data_rate = 1Gb/sec                      # Details of the GB Ethernet interconnect.
        interconnect_InfiniBand_data_rate = 1Gb/sec                      # Details of the InfiniBand interconnect.

        node_SUN1000_label = SUN 1000 Disk Array Server   # Details of the SUN 1000 file server.
        node_SUN1000_count = 1                            # Only one of them.
        node_SUN1000_purpose = fileserver

If you don't declare any components of a given type, no subsection will appear in the report. For example, on an SMT system there will be only one kind of node, and no file server, since the node uses its own local file system. The interconnect is neither replaceable nor configurable so there are no details to report, either.

V.A.II Node description(s)

An SMP system consists of one node of one type. A heterogeneous cluster may consist of many nodes of several types. Most nodes are used for computation but others may act as file servers or provide other utilities. The fields in this section describe the hardware and software configuration of a given node, and are assigned as follows:

        node_HP2000_vendor = Hewlett-Packard Corporation

The available fixed-format fields are as follows:

node_tag_label The name of the node type.
node_tag_order The output report contains a subsection for each component in the system. The node_order parameter specifies the position of this subsection relative to the others in the report.
node_tag_count The number of nodes in the system, of this type.
node_tag_hw_vendor The hardware vendor. An example of usage of this and related fields is given in the test_sponsor section.
node_tag_hw_model The model of the node type.
node_tag_purpose What the node is used for:
compute Used for computation.
fileserverA file server.
head Initiates processes but is not otherwise used for computation.
other Fills some other purpose, such as a TTY-server.
node_tag_hw_cpu_name Manufacturer-determined formal processor name.
node_tag_hw_ncores Number of CPU cores configured/enabled in the node. See the discussion of CPU counting in the run rules.
node_tag_hw_ncoresorder Valid number of processor cores orderable for this model of node, including a unit. For example: 2, 4, 6, or 8 chips.
node_tag_hw_ncoresperchip The number of processor cores per CPU chip.
node_tag_hw_nchips Number of CPU chips configured. See the discussion of CPU counting in the run rules.
node_tag_hw_nthreadspercore Number of hardware threads per core. See the discussion of CPU counting in the run rules.
node_tag_hw_cpu_char Technical characteristics to help identify the processor. (You'll find more information about the intended use of hw_cpu_name vs. hw_cpu_char in the run rules.)
node_tag_hw_cpu_mhz Speed of the CPUs, in MHz.
node_tag_hw_fpu Floating point unit.
node_tag_hw_pcache 1st level (primary) cache.
node_tag_hw_scache 2nd level cache.
node_tag_hw_tcache 3rd level cache.
node_tag_hw_ocache 4th level or other form of cache.
node_tag_hw_memory Size of main memory (and other performance-relevant information about memory, as discussed in the run rules.)
node_tag_sw_os Operating system name and version.
node_tag_sw_state Multi-user, single-user, default, etc.

Continuable fields:

node_tag_hw_disk

Disk subsystem for the SPEC run directories. Three important Notes:

  1. Only a single file system may be used for the run directories, as described in the run rules at Rule 3.1.1.
  2. If the output_root feature is used, its disk subsystem should be the one described here.
  3. Disks used for other purposes - such as the system disk - are usually not a performance-relevant variable, so need not be described. But if you wish to do so, you can extend this field using the same continuation rules as other fields (so, you could use hw_disk01, hw_disk02, etc).
node_tag_sw_local_file Local file space, used perhaps to store intermediate data files. Local file systems are not usually described since (a) the SPEC run-directory must be shared, and (b) file systems other than the SPEC run directory are usually not a performance-relevant variable.
node_tag_sw_shared_file

File system (ntfs, ufs, nfs, etc) for the SPEC run directories. Three important Notes:

  1. Only a single file system may be used for the run directories, as described in the run rules at Rule 3.1.1.
  2. If the output_root feature is used, its file system should be the one described here.
  3. File systems used for other purposes - such as for the operating system - are usually not a performance-relevant variable, so need not be described.
node_tag_sw_other Any other performance-relevant software to describe for the node. (Note that compiler and library details are covered in the Benchmark (REFERENCE) section).
node_tag_hw_other Any other performance-relevant hardware to describe for the node.
node_tag_notes Free-form notes describing other details of the node type. Note that you can use the continuation convention to separate this into node_notes_hw, node_notes_os, etc.

V.A.III Accelerator description(s)

Included as part of the node description, the following fields describe the accelerator used. The fields may be left blank or set to "Not Applicable" if no accelerator was used.

node_tag_accel_count The number of accelerators included per node.
node_tag_accel_connect Describes how the accelerator is connected to the system under test. Possible values include but not limited to integrated, PCIe, none.
node_tag_accel_desc Provide further details about he accelerator that are important to performance. Things that might be of interest include memory sizes, number of cores, number of vectors, etc.
node_tag_accel_driver The name and version of the software driver used to control the accelerator.
node_tag_accel_ecc This is a Yes or No field stating if the accelerator uses ECC for its memory.
node_tag_accel_model The model name of the accelerator. If multiple different accelerators are used, use a comma delimited list with the count of each type.
node_tag_accel_name The name of the accelerator. If multiple different accelerators are used, use a comma delimited list.
node_tag_accel_type The type of accelerator. Possible values include, but not limited to: GPU, APU, CPU, etc.
node_tag_accel_vendor The company/vendor of the accelerator.

V.A.IV. Adaptor description(s)

The nodes in a compute-cluster use adaptors to attach to the available interconnects. The disclosure convention provides the following fields to declare multiple adaptors of different kinds for each given node type, using a second level of tags

            node_HPCX_hw_adaptor_InfiniBand_model = PCI-Express DDR InfiniBand HCA
            node_HPCX_hw_adaptor_InfiniBand_count = 1

that allow for there being different kinds of adaptors. The defined fields are

node_tag1_hw_adaptor_tag2_model Adaptor model and vendor.
node_tag1_hw_adaptor_tag2_slot_type PCIe x8, HTX, etc.
node_tag1_hw_adaptor_tag2_data_rate Per port nominal data-rate and units, e.g. 1Gb/sec.
node_tag1_hw_adaptor_tag2_count Number of tag2 adaptors attached to nodes of type tag1.
node_tag1_hw_adaptor_tag2_ports_used Number fo cables connecting this kind of adaptor to the interconnect switches.
node_tag1_hw_adaptor_tag2_driver Adaptor driver.
node_tag1_hw_adaptor_tag2_firmware Adaptor firmware.

V.A.V Interconnect description(s)

The nodes of a compute-cluster are joined via one or more interconnects that will transmit various kinds of data communication. The report will contain one section for each interconnect. An SMT system with no external file server will not use an interconnect for communication and file transfers, so none of the below fields will be filles in and the report will not contain any interconnect sections. The attributes of each interconnect are prefixed as follows:

        interconnect_GBEthernet_label = Gigabit Ethernet             # If more than one interconnect.
        interconnect_mpiComm_label    = MPI traffic interconnect     # It may be named by what it's used for.

First, the fixed-format fields:

interconnect_tag_label The name of this interconnect type.
interconnect_tag_order The output report contains a subsection for each component in the system. The interconnect_order parameter specifies the position of this subsection relative to the others in the report.
interconnect_tag_hw_vendor The hardware vendor. An example of usage of this and related fields is given in the test_sponsor section.
interconnect_tag_hw_model The model of the interconnect type.
interconnect_tag1_hw_switch_tag2_count Number of switches of this type in the interconnect.
interconnect_tag1_hw_switch_tag2_ports Ports per switch of this type.
interconnect_tag1_hw_switch_tag2_data_rate Descriptive phrase for switch port speed.

Continuable fields..

interconnect_tag_hw_switch_tag2 The vendor and model of the internal aggregation or leaf switch(es) composing this interconnect.
interconnect_tag_hw_topo Description of the interconnect arrangement.
interconnect_tag1_sw_switch_tag2_firmware Switch firmware version.
interconnect_tag_purpose Major use for this interconnect.
interconnect_tag_notes Free-form notes describing interconnect details.

V.B. Fields can vary by scope.

Fields can appear and disappear based upon scope. For example, if your Small runs used two Fortran compilers (which is allowed for peak), you could construct a config file that adjusts the fields accordingly:

$ cat tmp.cfg
expand_notes      = 1
test_sponsor      = Turboblaster, Inc.
output_format     = text
output_root       = /tmp/fake
default:
   hw_avail       = Feb-2021
   sw_avail       = Jan-2021
   sw_compiler1   = C: V42.0 of TurboBlaster C/C++
   sw_compiler2   = Fortran: V42.2 of TurboBlaster Fortran 
small:
   sw_avail       = Apr-2021
   sw_compiler3   = Fortran: V9.2.0 of gfortran
   notes_comp_100 = In base, all benchmarks use Turboblaster Fortran
   notes_comp_110 = In peak, some benchmarks use Turboblaster Fortran and some
   notes_comp_120 =          benchmarks use gfortran, as noted in the report.
$ runhpc --fakereportable --config=tmp small medium | grep .txt
    format: Text -> /tmp/fake/result/hpc2021_sml.001.small.txt
    format: Text -> /tmp/fake/result/hpc2021_med.001.medium.txt
$ cd /tmp/fake/result
$ grep avail hpc2021_sml.001.small.txt 
 Test sponsor: Turboblaster, Inc.             Hardware availability: Feb-2021
 Tested by:    Turboblaster, Inc.             Software availability: Apr-2021
$ grep avail hpc2021_med.001.medium.txt 
 Test sponsor: Turboblaster, Inc.             Hardware availability: Feb-2021
 Tested by:    Turboblaster, Inc.             Software availability: Jan-2021
$ grep ortran hpc2021_sml.001.small.txt 
                      Fortran: V42.2 of TurboBlaster Fortran
                      Fortran: V9.2.0 of gfortran
     In base, all benchmarks use Turboblaster Fortran
     In peak, some benchmarks use Turboblaster Fortran and some
              benchmarks use gfortran, as noted in the report.
$ grep ortran hpc2021_med.001.medium.txt 
                      Fortran: V42.2 of TurboBlaster Fortran
$   

In the above example, notice that both the compiler information and the availability date changed in the report, depending on the metric.

V.C. Additional notes for the reader

In addition to the pre-defined fields, you can write as many notes as you wish. These notes are printed in the report, using a fixed-width font. For example, you can use notes to describe software or hardware information with more detail beyond the predefined fields:

   notes_os_001  = The operating system used service pack 2 plus patches 
   notes_os_002  = 31415, 92653, and 58979.  At installation time, the 
   notes_os_003  = optional "Numa Performance Package" was selected.

V.C.1. Notes sections

There are various notes sections. If there are no notes in a particular section, it is not output, so you don't need to worry about making sure you have something in each section.

The sections, in order of appearance, are as follows:

  1. notes_comp_NNN -- Notes about compiler invocation.
  2. notes_port_NNN -- Notes about portability options.
  3. notes_base_NNN -- Notes about base optimization options.
  4. notes_peak_NNN -- Notes about peak optimization options.
  5. notes_submit_NNN -- Notes about use of the submit option.
  6. notes_os_NNN -- Notes about operating system tuning and changes.
  7. notes_plat_NNN -- Notes about platform tuning and changes.
  8. notes_part_NNN -- Notes about component parts (for kit-built systems).
  9. notes_NNN -- General notes.

Notes about the submit command are described above, with the description of the submit option.

V.C.2. Note numbering

Start your notes with the name of the notes section where you want the note to appear, and then add numbers to define the order of the lines. Within a section, notes are sorted by line number. The NNN above is not intended to indicate that you are restricted to 3 digits; you can use a smaller or larger number of digits as you wish, and you can skip around as you like: for example, ex-BASIC programmers might naturally use line numbers 100, 110, 120... But note that if you say notes_plat782348320742972403 you just might encounter the dreaded (and highly unusual) "out of memory" error, so don't do that.

You can optionally include an underscore just before the number, but beware: if you say both notes_plat_105 and notes_plat105, both are considered to be the same line. The last one mentioned will replace the first, and it will be the only one output.

V.C.3. Additional tags

For all sections you can add an optional additional tag of your choosing before the numbers. Notes will be organized within the tags.

The intent of the feature is that it may allow you to organize your system information in a manner that better suits your own categories for describing it.

For example:

$ cat notes_tags.cfg 
iterations      = 1
output_format   = text
output_root     = /tmp/notes_tags
runlist         = 605.lbm_s
size            = test
small=base:
   CC                       = mpicc
   CC_VERSION_OPTION        = -V
   notes_part_greeting_011  = ++ how
   notes_part_greeting_20   = ++ you?
   notes_part_greeting_012  = ++ are
   notes_part_aname_1       = ++ Veronica,
   notes_part_080           = ++ hi

$ cat notes_tags.sh  
runhpc --config=notes_tags --fakereport 605.lbm_s | grep txt
grep '++' /tmp/notes_tags/result/hpc2021_sml.001.small.test.txt
$ ./notes_tags.sh
    format: Text -> /tmp/notes_tags/result/hpc2021_sml.001.small.test.txt
     ++ hi
     ++ Veronica,
     ++ how
     ++ are
     ++ you?
$ 

V.C.4. Links in notes sections

You can mention URLs in your notes section, and html reports will correctly render them as hyperlinks. For example:

notes_plat_001 = Additional detail may be found at
notes_plat_002 = http://www.turboblaster.com/servers/big/green/

If you like, you can use descriptive text for the link by preceding it by the word LINK and adding the descriptive text in square brackets:

LINK url AS [descriptive text]

The brackets may be omitted if your descriptive text is a single word, without blanks.

For example:

notes_plat_001 = Additional detail may be found at 
notes_plat_002 = LINK http://www.turboblaster.com/servers/big/green/ AS [TurboBlaster Servers]

When the above appears in an html report, it is rendered as:

 Additional detail may be found at 
 TurboBlaster Servers
      

And in a text report, it appears as:

                                Platform Notes
                                --------------
     Additional detail may found at 
     TurboBlaster Servers (http://www.turboblaster.com/servers/big/green/)

Since the text report is not a context in which the reader can click on a link, it is spelled out instead. Note that because the text report spells the link out, the text line is wider than in HTML, PS, and PDF reports. When deciding where to break your notes lines, you'll have to pick whether to plan line widths for text (which may result in thin-looking lines elsewhere) or plan your line widths for HTML/PS/PDF (which may result in lines that fall of the right edge with text). The feature notes_wrap_columns won't help you here, since it is applied before the link is spelled out.

VI. The config file preprocessor

VI.A. Introduction

The SPEChpc tools include a configuration file macro preprocessor. This section introduces preprocessor examples, and then discusses a few basics of syntax.

VI.A.1. Preprocessor Example: Picking a Model

In this example, the user is testing several configurations, and wants to pick an appropriate pmodel.

$ cat -n preprocessor_example1.cfg
     1  505.lbm_t,605.lbm_s,705.lbm_m,805.lbm_l:  
     2  %if %{model} eq "openacc"  
     3      pmodel=ACC
     4  %elif %{model} eq "openmp"
     5  %   if %{usegpu} == 1
     6         pmodel=TGT  
     7  %   else
     8         pmodel=OMP 
     9  %   endif
    10  %endif

Lines that start with percent (%) are preprocessor directives. The percent (%) character must be in column 1.

The config file preprocessor is called a macro processor because it allows you to define macros, which are brief abbreviations for longer constructs. In the example above, two macros are used: (%{model}, and %{usegpu}). Macro values are tested, and hunks of config file are included or excluded, using the conditionals %if (lines 2 and 5), and %elif (line 4). One of the conditionals includes an %else clause (line 7).

The preprocessor is automatically run whenever you use runhpc, and macros may be set on the runhpc command line. Or, you can run it separately, as configpp:

% specperl bin/harness/configpp -c preprocessor_example1.cfg --define model=openacc | grep -e lbm -e pmodel
505.lbm_t,605.lbm_s,705.lbm_m,805.lbm_l:
    pmodel=ACC
% specperl bin/harness/configpp -c preprocessor_example1.cfg --define model=openmp | grep -e lbm -e pmodel
505.lbm_t,605.lbm_s,705.lbm_m,805.lbm_l:
       pmodel=OMP
% specperl bin/harness/configpp -c preprocessor_example1.cfg -S model=openmp -S usegpu=1 | grep -e lbm -e pmodel
505.lbm_t,605.lbm_s,705.lbm_m,805.lbm_l:
       pmodel=TGT

Notice above that preprocessor variables can be set on the command line, using either --define or '-S'.

VI.A.2. Preprocessor Syntax Basics

Emphasis: Column 1. Always punch column 1.

All preprocessor directives begin with the percent sign (%) in column 1. You can -- and probably should -- use indenting, but keep the percent sign on the far left. Any amount of spaces or tabs may separate the percent from the directive. The following are okay:

%define foo
%    define bar
%		undef hello!

The following are not okay:

 %define foo           Space in the first column
	%define foo    Tab in the first column
#define foo            <--wrong!

Did you catch why the last one is wrong? This isn't CPP - use percent, not hash!

One line per directive

The macro preprocessor does NOT follow the same quoting rules described elsewhere for config files. In particular, you may not use line continuation, line appending, or block quotes. You may have a value of arbitrary length, but in the interests of config file readability and maintainability, please keep them relatively short.

VI.B. Defining macros

Macro names may only be composed of alphanumeric characters, underscores, and hyphens, and they ARE case-sensitive.

Macros can be defined in a config file or on the command line. Both ways are equivalent.

Macros set on the command-line are defined first; therefore, it is not possible to use the command line to override a macro definition that occurs in the config file itself. If you want the command line to "win", see the notes below about redefinition.

A macro that has not been defined will not be substituted.

A macro that has been defined, but not given a value,

  • behaves as if it were 1 in a numeric context
  • behaves as if it were true in a logical context (because for Perl, 1 is true)
  • behaves as an empty string in a string context

See the example in the next section.

Having no value is not the same as having an empty string as the value. The following are NOT equivalent:

--define FOO="" FOO is defined, is false, and behaves as an empty string in a string context.
--define FOO FOO is defined, is true,  and behaves as an empty string in a string context.

See the table about Truth and definition, below.

VI.B.1. In a config file

To define a macro in a config file, use the %define preprocessor directive. Note that no quoting is necessary when specifying the names of macros or their values.

# Define a simple macro
%define foo bar
# Now the macro called 'foo' has the value 'bar'

# It's not necessary for a macro to have a value to be useful
%define baz
# Now the macro called 'baz' is defined, but it has no value.

Above, the macro baz has been defined, but has not been assigned a value.
Nevertheless, if you use it in an expression, it will behave as if it has the value 1.

$ cat macroDefinedButNoValue.cfg 
%define baz
%ifdef %{baz}
%   info baz is defined. In text contexts, it looks like this: "%{baz}".
%endif
%if %{baz} 
%   info In a logical context, it behaves as if it were 1 (i.e. true)
%endif
%if %{baz} + 3  == 4
%   info In a numeric context, it behaves as if it were 1.
%endif
$ cat macroDefinedButNoValue.sh  
specperl bin/harness/configpp --config=macroDefinedButNoValue | grep INFO
$ ./macroDefinedButNoValue.sh 
  INFO: baz is defined. In text contexts, it looks like this: "".
  INFO: In a logical context, it behaves as if it were 1 (i.e. true)
  INFO: In a numeric context, it behaves as if it were 1.
$ 

VI.B.2. On the command line

Macros can also be defined on the command line, using the --define switch. You may find it convenient to abbreviate that switch, if you prefer. The following are entirely equivalent:

$ runhpc --define mymacro=something
$ runhpc -S mymacro=something

VI.B.3. Predefined macros endian, hostname... %{ENV_variable_name}

Some useful macros are predefined, as shown in the table below. New with SPEChpc 2021: The config file preprocessor can do environment variable substitution. Environment variables to be substituted use a percent sign and curly brackets, and the name is prefixed with the string "ENV_", as %{ENV_variable_name}.

%{configfile} The name of the config file specified on the command line (possibly not including directory path)
%{endian} Indicator of endian characteristics, for example '12345678' indicating a little-endian system
%{hostname} System Under Test name
%{runhpc} Your original runhpc command
%{top_config_file} The name of the top-level config file (likely the same as %{configfile}, one may have the full path and the other not)
%{current_config_file} The name of the config file currently being read (same as %{configfile} unless reading an included config file)
%{parent_config_file} The name of the config file that included the config file currently being read (empty for the top-level config file)
%{config_nesting_level} The number of nested includes in effect (0 for the top-level config file)
%{ENV_variable}

Any of your environment variables.

For a complete list, see dumpmacros.

Automating output_root: You can automatically bring an output_root into your config file using %{ENV_GO} and navigate around it, as shown in the example for the ogo utility.

VI.B.4. Example: Adjusting the Environment

Notice that %{ENV_variable_name} brings an environment variable into your config file; what if you want to push it back out to affect other things? In that case, you will also need the feature preenv, which causes runhpc to re-invoke itself with your requested variable in the environment.

For example, you could set up paths for your compiler using something like this:

$ cat -n test.cfg
     1  action       = build
     2  output_root  = /tmp/me
     3  rebuild      = 1
     4  runlist      = 605.lbm_s
     5  verbose      = 99
     6
     7  %define tentop /SW/compilers/GCC/Linux/x86_64/gcc-10.1.0
     8  %define ninetop  /SW/compilers/GCC/Linux/x86_64/gcc-9.2.0
     9
    10  %ifdef %{wantGcc9}
    11     preENV_PATH            = %{ninetop}/bin:%{ENV_PATH}
    12     preENV_LD_LIBRARY_PATH = %{ninetop}/lib64:%{ENV_LD_LIBRARY_PATH}
    13  % else
    14     preENV_PATH            = %{tentop}/bin:%{ENV_PATH}
    15     preENV_LD_LIBRARY_PATH = %{tentop}/lib64:%{ENV_LD_LIBRARY_PATH}
    16  %endif
    17
    18  default:
    19     CC                = mpicc
    20     CC_VERSION_OPTION = -v
    21
$ 

The config file above builds a single benchmark, 605.lbm_s, using the local paths for GCC 9.2 or 10.1. Notice that in both cases, the desired path is added to the existing path: for example, on line 12, the right side accesses the existing setting via %{ENV_LD_LIBRARY_PATH} and the left side applies it to the build, via preENV_LD_LIBRARY_PATH. On line 5, the config file sets verbose=99, so that runhpc will print all possible detail and we will be able to confirm the adjusted variables.

VI.B.5. Redefinition

You will receive a warning if a previously defined macro is re-defined. In the example below, the first config file just sets a macro; the second one tests before acting.

$ cat preproc.symbol.just_define.cfg 
%define build_ncpus 20
makeflags   = -j%{build_ncpus}
$ 
$ cat preproc.symbol.test_first.cfg 
%ifndef %{build_ncpus}
%   define build_ncpus 20
%endif
makeflags   = -j%{build_ncpus}
$ 

A run script uses both config files with and without a definition on the runhpc command line:

$ cat preproc.symbol.sh 
JUST_DEFINE="configpp --config=preproc.symbol.just_define.cfg"
 TEST_FIRST="configpp --config=preproc.symbol.test_first.cfg"
    GREP_IT='grep -e WARNING -e makeflags'
echo original:
$JUST_DEFINE --define build_ncpus=8 | $GREP_IT
$JUST_DEFINE                        | $GREP_IT
echo
echo test1st: 
$TEST_FIRST  --define build_ncpus=8 | $GREP_IT
$TEST_FIRST                         | $GREP_IT
$

Result: testing first with ifndef avoids a warning and does a better job of respecting the user's wishes.

$ ./preproc.symbol.sh 
original:
WARNING: Redefinition of preprocessor macro 'build_ncpus' on line 1
makeflags   = -j20
makeflags   = -j20

test1st:
makeflags   = -j8
makeflags   = -j20
$  

VI.C. Un-doing macro definition

Sometimes you want to make the preprocessor forget about a macro that you taught it. This is easily accomplished.

Macros can be undefined in two ways:

  1. On the command line, using the --undef switch, or
  2. In the configuration file, using the '%undef' preprocessor directive.
    %define foo bar
    # Now the macro called 'foo' has the value 'bar'
    
    %undef foo
    # Now it doesn't
    

    Note that no quoting is necessary when specifying the names of macros.

Like macro definition, the undefinition requests can't affect macros set in the config file because those definitions effectively happen after the un-definition. For this reason, command-line undefinition is basically useless; it can only undo macros also set on the command line.
So why was such a useless ability added to the tools? The writer likes orthogonality.

VI.D. Using macros

By now you're probably over that initial euphoric rush that comes from wantonly defining and undefining macros, and you're looking for something more. This is it!

When you want to use the value of a macro, you refer to it by name. Unfortunately, the syntax for this is not as simple as you might hope. It's not too complicated, though; to have the preprocessor expand the macro 'foo', just write

  %{foo}

in the place where you'd like it to appear. Given the following config file snippet:

%define foo Hello_
%define bar baz
%define foobar Huh?
%define foobaz What?
%define Hello_baz Please don't do this

Here's a handy table to see the various ways you can reference these values:

Macro reference Value
%{foo} Hello_
%{bar} baz
%{foobar} Huh?
%{foobaz} What?
%{Hello_baz} Please don't do this

Easy, right? The following is also possible:

%{foo%{bar}} What?
%{%{foo}%{bar}} Please don't do this

Because macro values can only be one line long, it's not possible to use the preprocessor to macro-ize large chunks of your config file at once, as may be common practice for advanced users of CPP.

A macro that has not been defined will not be substituted. Note the mention of %{FOO} on the last line of this config file.

$ cat macroDef.cfg 
action          = build
output_format   = text
output_root     = /tmp/mDef
rebuild         = 1
runlist         = 605.lbm_s
size            = test
teeout          = 1
intspeed=base:
   CXX                 = mpicc
   CXX_VERSION_OPTION  = -v 
   OPTIMIZE            = %{FOO}
$

The failure for the first test below is expected: the GNU C compiler does not know what to do with ${FOO}.

$ cat macroDef1.sh 
runhpc --config=macroDef | grep -e lbm.o -e ^Error -e uccess
$ ./macroDef1.sh 
mpicc -c -o lbm.o -DSPEC -DNDEBUG -I.  %{FOO} lbm.c 
Error with make 'specmake --output-sync build': check file 
   "/tmp/mDef/benchspec/HPC/605.lbm_s/build/build_base_none.0000/make.out"
$ cat /tmp/mDef/benchspec/HPC/605.lbm_s/build/build*/make.out
mpicc -c -o lbm.o -DSPEC -DNDEBUG -I.  %{FOO}  lbm.c
gcc: error: %{FOO}: No such file or directory
$

If you want to substitute an empty string, then assign it one.

$ cat macroDef2.sh 
runhpc --config=macroDef --define FOO="" | grep -e lbm.o -e ^Error -e uccess
$ ./macroDef2.sh 
mpicc -c -o lbm.o -DSPEC -DNDEBUG -I.      lbm.c 
Build successes for small: 605.lbm_s(base)
$ 

VI.E. Conditionals

Defining, undefining, and expanding macros is quite an enjoyable activity in and of itself, and can even be useful on occasion. However, conditionals add an entirely new dimension to config file processing: the ability to include and exclude entire sections of text based on macros and their values.

VI.E.1. %ifdef .. %endif

The %ifdef conditional provides a way to determine whether or not a particular macro has been defined. If the named macro has been defined, the conditional is true, and the text to the matching %endif is included in the text of the config file as evaluated by runhpc. Note that the matching %endif may not necessarily be the next %endif; conditionals may be nested.

For example, given the following section of a config file:

%define foo
%ifdef %{foo}
This text will be included
%endif

%ifdef %{bar}
This text will not be included
%endif

The preprocessor would produce the following output:

This text will be included

Note especially the quoting used for the macro names in the conditional; the only time macro name quoting may be omitted is when defining or undefining it.

VI.E.2. %ifndef .. %endif

The %ifndef conditional is the converse of %ifdef; If the named macro has not been defined, the conditional is true, and the text to the matching %endif is included in the text of the config file as evaluated by runhpc. Note that the matching %endif may not necessarily be the next %endif; conditionals may be nested.

Given a slightly modified version of the example from earlier:

%define foo
%ifndef %{foo}
Now THIS text will not be included
%endif

%ifndef %{bar}
This text WILL be included
%endif

The preprocessor would produce the following output:

This text WILL be included

VI.E.3. %if .. %endif

Checking whether or not a macro is defined is quite useful, but it's just a subset of the more general conditional facility available. This general form is

%if expression
...
%endif

The expression is evaluated using a subset of the Perl interpreter, so the possibilities for testing values are fairly broad. For example,

%ifdef %{foo}
...
%endif

is exactly equivalent to

%if defined(%{foo})
...
%endif

Likewise,

%ifndef %{foo}
...
%endif

is exactly equivalent to

%if !defined(%{foo})
...
%endif

Using the general form, it's possible to string conditionals together:

%if defined(%{foo}) && !defined(%{bar}) || %{baz} == 0
...
%endif

To compare versus a string value, you must supply quotes:

%if %{foo} eq 'Hello, Dave.'
...
%endif

You may perform basic math on macro values:

%if %{foo} * 2004 > 3737
...
%endif

You can do basic regular expression matching, as shown in the example for %info.

More precisely, the Perl operations allowed are the :base_core and :base_math bundles, with the ability to dereference and modify variables disallowed. For more details, see the source code for config.pl (the eval_pp_conditional subroutine) and Perl's own Opcode documentation.

Truth and definition
%ifdef %{v} and
%if defined(%{v})
  vs.
%if %{v}

Conditional expressions are evaluated using perl's rules for Truth and Falsehood.
Therefore 0 (zero) is defined, and is false:

$ cat test_define.cfg
%ifdef %{a}
   OPTIMIZE = -a
%endif
%if defined(%{b})   
   LDOPTIMIZE = -b
%endif
%if %{c}
   COPTIMIZE = -c
%endif
$ specperl bin/harness/configpp --config=test_define -S a=1 -S b=1 -S c=1 | grep OPT  
   OPTIMIZE = -a
   LDOPTIMIZE = -b
   COPTIMIZE = -c
$ specperl bin/harness/configpp --config=test_define -S a=0 -S b=0 -S c=0 | grep OPT
   OPTIMIZE = -a
   LDOPTIMIZE = -b

VI.E.4. %else

It's possible to get by without the "else" part of the classic "if .. then .. else" trio, but it's not any fun. It works as you'd expect:

%define foo
%ifndef %{foo}
This text will not be included
%else
This text WILL be included (from the else clause)
%endif

The preprocessor would produce the following output:

This text WILL be included (from the else clause)

Only one %else per conditional is allowed.

VI.E.5. %elif

%elif is another convenience that's been added. For those not familiar with CPP, it's an "else if" construct. You may have as many of these as you'd like. Given:

%define foo Hello!

%if !defined(%{foo})
This text will not be included
%elif defined(%{bar})
This text won't be included either
%elif '%{foo}' eq 'Hello!'
This text WILL be included (from the second elif clause)
%else
Alas, the else here is left out as well.
%endif

The preprocessor would produce the following output:

This text WILL be included (from the second elif clause)

VI.F. Informational Directives

It's often helpful to be able to warn or exit on certain conditions. Perhaps there's a macro that must be set to a particular value, or maybe it's just very highly recommended.

VI.F.1. %warning

%warning does just what you'd expect; when the preprocessor encounters this directive, it prints the text following to stdout and the current log file, along with its location within the file being read, and continues on.

$ cat warning.cfg 
%if !defined(%{somewhat_important_macro})
%   warning You have not defined somewhat_important_macro!
%endif
$ 
$ cat warning.sh 
specperl bin/harness/configpp --config=warning | grep -C2 WARNING
$ 
$ ./warning.sh 

***
  WARNING: You have not defined somewhat_important_macro!
           (From line 2 of
           /hpc2021/config/warning.cfg.)
$  

VI.F.2. %error

Like %warning, %error logs an error to stderr and the log file. Unlike %warning, though, it then stops the run.

Consider a slightly modified version of the previous example:

$ cat error.cfg 
%if !defined(%{REALLY_important_macro})
%   error You have not defined REALLY_important_macro!
%endif
$ 
$ cat error.sh 
specperl bin/harness/configpp --config=error > /tmp/out
echo runhpc exit code: $?
grep -C2 ERROR /tmp/out
$ ./error.sh 
runhpc exit code: 1

************************************************************************
  ERROR: You have not defined REALLY_important_macro!
         (From line 2 of
         /hpc2021/config/error.cfg.)
$ 

Unlike a warning, the error will be close to the last thing output. As you can see from the output of echo $? above, runhpc exited with an error code 1.

VI.F.3. %info

The %info directive prints a message preceded by the word INFO:. For example,


$ cat -n info.cfg 
   1  %if !defined(%{chip}) || %{chip} !~ m/(sparc|x86)/
   2  %   error Please use --define chip=sparc or --define chip=x86
   3  %endif
   4  
   5  %if  %{chip} eq "sparc"
   6  %   define default_build_ncpus 64
   7  %elif %{chip} eq "x86"
   8  %   define default_build_ncpus 20
   9  %endif
  10  %ifndef   %{build_ncpus}
  11  %   define  build_ncpus   %{default_build_ncpus}
  12  %endif
  13  
  14  %info Preprocessor selections: 
  15  %info    .    build_ncpus      %{build_ncpus}
  16  %info    .    chip             %{chip}
  17  
  18  make = specmake --jobs=%{build_ncpus} --load-average=%{build_ncpus} 
$
$ specperl bin/harness/configpp -c info --define chip=sparc | grep -e make -e INFO
INFO: Preprocessor selections:
INFO: .    build_ncpus      64
INFO: .    chip             sparc
make = specmake --jobs=64 --load-average=64
$ 
 

Note in the example above:

  • Line 1 demonstrates basic perl regular expression matching
  • Lines 14-16 use %info
  • After all preprocessing has been completed, line 18 tells the specmake utility that it can start up to 64 compile jobs, provided that the system load average is no higher than 64.

VI.F.4. %dumpmacros

The %dumpmacros prints the values of all macros currently defined at the point where it appears. Each macro value is preceded by the word 'DBG:'. For example,


$ cat -n dumpmacros.cfg 
     1	%if !defined(%{chip}) || %{chip} !~ m/(sparc|x86)/
     2	%   error Please use --define chip=sparc or --define chip=x86
     3	%endif
     4	
     5	%if  %{chip} eq "sparc"
     6	%   define default_build_ncpus 64
     7	%elif %{chip} eq "x86"
     8	%   define default_build_ncpus 20
     9	%endif
    10	%ifndef   %{build_ncpus}
    11	%   define  build_ncpus   %{default_build_ncpus}
    12	%endif
    13	
    14	%dumpmacros
$
$ specperl bin/harness/configpp -c dumpmacros --define chip=sparc | grep -e DBG
  DBG: build_ncpus: '%{default_build_ncpus}'
  DBG: chip: 'sparc'
  DBG: default_build_ncpus: '64'
  DBG: runhpc: 'configpp -c dumpmacros --define chip=sparc'
$
 

The %dumpmacros directive is new with SPEChpc 2021.

VII. Output files - and how they relate to your config file

This section describes how the location and contents of several kinds of output files are influenced by your config file.

VII.A. Help, I've got too many config files!

It was mentioned above that the HASH section of the config file is written automatically by the tools. Each time your config file is updated, a backup copy is made. Thus your config directory may soon come to look like this:

$ cd $SPEC/config
$ ls tune.cfg*
tune.cfg                        tune.cfg.2021-02-05T124831      tune.cfg.2021-02-05T125733
tune.cfg.2021-02-05T120242      tune.cfg.2021-02-05T125557      tune.cfg.2021-02-05T125738
tune.cfg.2021-02-05T122021      tune.cfg.2021-02-05T125603      tune.cfg.2021-02-05T125744
tune.cfg.2021-02-05T122026      tune.cfg.2021-02-05T125608      tune.cfg.2021-02-05T125749
tune.cfg.2021-02-05T124215      tune.cfg.2021-02-05T125614      tune.cfg.2021-02-05T125756
tune.cfg.2021-02-05T124222      tune.cfg.2021-02-05T125620      tune.cfg.2021-02-05T125802
tune.cfg.2021-02-05T124728      tune.cfg.2021-02-05T125626      tune.cfg.2021-02-05T125807
tune.cfg.2021-02-05T124739      tune.cfg.2021-02-05T125632      tune.cfg.2021-02-05T125812
tune.cfg.2021-02-05T124821      tune.cfg.2021-02-05T125727
$  

If this feels like too much clutter, you can disable the backup mechanism, as described under backup_config. Note that doing so may leave you with a risk of losing the config file in case of a filesystem overflow or system crash. A better idea may be to periodically remove just portions of the clutter, with selective removal of older version; or sweep them to a wastebasket every now and then:

$ cd $SPEC/config
$ mkdir wastebasket
$ mv *cfg.2021* wastebasket/
$

VII.B. The log file and verbosity levels

$SPEC/result (Unix) or %SPEC%\result (Windows) contains reports and log files. When you are doing a build, you will probably find that you want to pay close attention to the log files such as hpc2021.001.log. Depending on the verbosity level that you have selected, it will contain detailed information about how your build went.

The SPEChpc 2021 tool suite provides for varying amounts of output about its actions during a run. These levels range from the bare minimum of output (level 0) to copious streams of information that are probably useful only to tools developers (level 99). Selecting one output level gives you the output from all lower levels, which may cause you to wade through more output than you might like.

VII.B.1. Useful Search Strings

When you are trying to find your way through a log file, you will probably find these (case-sensitive) search strings useful:

runhpc:The runhpc command for this log.
RunningPrinted at the top of a run of a benchmark.
#Printed at the top of a run of a benchmark for runs with multiple iterations. Useful for finding the ref workloads in reportable runs.
runtimePrinted at the end of a benchmark run.
BuildingPrinted at the beginning of a benchmark compile.
Elapsed compilePrinted at the end of a benchmark compile.

VII.B.2. About Temporary Debug Logs

There are also temporary debug logs, such as hpc2021.001.log.debug. A debug log contains very detailed debugging output from the SPEC tools, as if --verbose 99 had been specified.

For a successful run, the debug log will be removed automatically, unless you specify "--keeptmp" on the command line, or "keeptmp=yes" in your config file.

For a failed run, the debug log is kept. The debug log may seem overwhelmingly wordy, repetitive, detailed, redundant, repetitive, and long-winded, and therefore useless. Suggestion: after a failure, try looking in the regular log first, which has a default verbosity level of 5. If your regular log doesn't have as much detail as you wish, then you can examine the additional detail in the debug log.

If you file a support request, you may be asked to send in the debug log.

VII.B.3. Verbosity levels

The 'level' referred to in the table below is selected either in the config file verbose option or in the runhpc command as in 'runhpc --verbose n'.

Levels higher than 99 are special; they are always output to your log file. You can also see them on the screen if you set verbosity to the specified level minus 100. For example, the default log level is 5. This means that on your screen you will get messages at levels 0 through 5, and 100 through 105. In your log file, you'll find the same messages, plus the messages at levels 106 through 199.

Level What you get
0 Basic status information, and most errors. These messages can not be turned off.
1 List of the benchmarks which will be acted upon.
2 A list of possible output formats, as well as notification when beginning and ending each phase of operation (build, setup, run, reporting).
3 A list of each action performed during each phase of operation (e.g. "Building 605.lbm_s", "Setting up 635.weather_s")
4 Notification of benchmarks excluded
5 (default) Notification if a benchmark somehow was built but nevertheless is not executable.
6 Time spent doing automatic flag reporting.
7 Actions to update SPEC-supplied flags files.
10 Information on basepeak operation.
12 Errors during discovery of benchmarks and output formats.
15 Information about certain updates to stored config files
24 Notification of additions to and replacements in the list of benchmarks.
30 A list of options which are included in the hash of options used to determine whether or not a given binary needs to be recompiled.
35 A list of key=value pairs that can be used in command and notes substitutions, and results of env_var settings.
40 A list of 'submit' commands for each benchmark.
Note: If you would like to see all of the submit commands for every copy and every benchmark invocation, with all your variables (such as $BIND) resolved, try runhpc --verbose=40 --fake, or go to the run directory and use specinvoke -n (dry run).
70 Information on selection of median results.
89 Progress comparing run directory checksum for executables.
90 Time required for various internal functions in the tools.
95, 96, 97, 98 Flag parsing progress during flag reporting (progressively more detail)
99 Gruesome detail of comparing hashes of files being copied during run directory setup.
Messages at the following levels will always appear in your log files
100 Various config file errors, such as bad preprocessor directives, bad placement of certain options, illegal characters...
102 Information about output formats that could not be loaded.
103 A tally of successes and failures during the run broken down by benchmark.
106 A list of runtime and calculated ratio for each benchmark run.
107 Dividers to visually block each phase of the run.
110 Elapsed time for each portion of a workload (if an executable is invoked more than once).
120 Messages about which commands are being issued for which benchmarks.
125 A listing of each individual child processes' start, end, and elapsed times.
130 A nice header with the time of the runhpc invocation and the command line used. Information about what happened with your sysinfo program
140 General information about the settings for the current run.
145 Messages about file comparisons.
150 List of commands that will be run, and details about the settings used for comparing output files. Also the contents of the makefile written.
155 Start, end, and elapsed times for benchmark run.
160 Start, end, and elapsed times for benchmark compilation.
180 stdout and stderr from commands run
190 Start and stop of delays
191 Notification of command line used to run specinvoke.

VII.D. Help, I've got too many log files!

If you do many builds and runs, you may find that your result directory gets too cluttered. Within a result directory, all output formats other than .rsf can be regenerated from your .rsf files. Therefore, you could reduce clutter by deleting HTML, PDF, and other reports. You can delete old .debug logs unless you plan to submit a support request.

Still feel cluttered? A simple solution is to move your result directory aside, giving it a new name. Don't worry about creating a new directory; runhpc will do so automatically. You should be careful to ensure no surprises for any currently-running users. If you move result directories, it is a good idea to also clean temporary directories at the same time.
Example:

cd $SPEC
mv result old-result
rm -Rf tmp/
cd output_root     # (If you use an output_root)
rm -Rf tmp/

VII.E. Finding the build directory

As described under "About Disk Usage" in runhpc.html, the SPEChpc 2021 tools do the actual builds and runs in newly created directories. The benchmark sources are never modified in the src directory.

The build directories for a benchmark are located underneath that benchmarks' top-level directory, typically $SPEC/benchspec/HPC/nnn.benchmark/build (Unix)

(If you are using the output_root feature, then the first part of that path will change to be your requested root instead of SPEC.)

The build directories have logical names, typically of the form build_<tune>_<label>.0000.

If the directory build_<tune>_<label>.0000 already exists when a new build is attempted for the same tuning and label, the directory will be re-used, unless:

  • You have set the environment variable SPEC_NO_RUNDIR_DEL, or
  • The directory has been left in a "locked" state by an aborted previous build.

In such cases, the 0000 will be incremented until a name is generated that is available. You can find locked directories by searching for lock=yes in the file $SPEC/benchspec/HPC/<nnn.benchmark>/run/list (Unix)

When more than one build directory has been created for a given tuning and label, you may need to trace the directory back to the specific build attempt that created it. You can do so by searching for the directory name in the log files:

$ grep build_peak_blue271.0000 *log | grep Building
hpc2021.838.log:  Building 605.lbm_s peak blue271: (build_peak_blue271.0000) [2021-02-14 05:26:09]
$  

In the above example, the grep command locates log #838 as the log that corresponds to this run directory.

VII.F. Files in the build directory

A variety of files are output to the build directory. Here are some of the key files which can usefully be examined:

Makefile.spec The components for make that were generated for the current config file with the current set of runhpc options.
make.out For 1 pass compile: detailed commands generated.

VII.G. For more information

For more information about how the run directories work, see the descriptions of specinvoke, specmake, and specdiff in utility.html.

Appendix A. Troubleshooting

When something goes wrong, here are some things to check:

  1. Are there any obvious clues in the log file? Search for the word "Building". Keep searching until you hit the next benchmark AFTER the one that you are interested in. Now scroll backward one screen's worth of text.

  2. Did your desired switches get applied? Go to the build directory, and look at options*out.

  3. Did the tools or your compilers report any errors? Look in the build directory at *err.

  4. What happens if you try the build by hand? See the section on specmake in utility.html.

  5. If an actual run fails, what happens if you invoke the run by hand? See the information about "specinvoke -n" in utility.html

  6. Do you understand what is in your path, and why? Sometimes confusion can be greatly reduced by ensuring that you have only what is needed, avoiding, in particular, experimental and non-standard versions of standard utilities.

    Note: on Windows systems, SPEC recommends that Windows/Unix compatibility products should be removed from the %PATH% prior to invoking runhpc, in order to reduce the probability of certain difficult-to-diagnose error messages.

  7. Try asking the tools to leave more clues behind, with keeptmp.

Appendix B. Obsolete/removed items