Submitting a result to SPEC for publication on its website

Last updated: $Date: 2012-07-17 06:37:35 -0400 (Tue, 17 Jul 2012) $ by $Author: JohnHenning $

This document provides information about what needs to be done to get a benchmark run ready for submission to SPEC for publication on its website.

DRAFT this document is subject to change.

   Published results must follow rules in full detail
    - www.spec.org/cpu2006/Docs/runrules.html
    - all submitters should read the entire doc
    - note in particular section 4.2 about config disclosure
    - fields frequently in need of correction
       - describe the cpu caches correctly
       - if you adjusted bios (e.g. turn off hyper
         threading) you must say so
       - if compiler features require runtime settings,
         you must say what you did at runtime
       - if something was previously submitted with
         one spelling, please spell it the same way

    You need two flags files  
    - Ok to reference existing one for compiler
    - Need to write your own platform flags file

    Email address for submissions
    Email address for corrections and how to do them

    Requests from SPEC for field consistency
        Memory Descriptions (draft is below)
        Compiler versions (plaintext draft below)
        Phrasing of MHz and MHz changes (e.g. Turbo Boost)

Format for Memory Descriptions

Required unless tester asserts a reason not to use it

As an aid to customers, and to improve uniformity when searching SPEC's pages, the SPEC CPU Subcommittee expects the following format to be used for results published on SPEC's web site, unless the tester asserts to the subcommittee that there is a reason not to do so. Examples of such reasons may include:

Although the subcommittee will typically accept such reasoning by testers, SPEC intends to nevertheless encourage use of the following format, even when not technically required by the run rule.

The format

XX ss (YY x gg ss
   {eRxff PChv-wwwwwm-aa <from JEDEC standard [1]>}
   {, ECC <if supported>}
   {, per node <for clustered SUTs>}
   {, running at xxxx MHz <if memory is clocked at a speed other than the 
                          speed implied by the PChv-wwwwm-aa specification>}
      and CLy <if memory is clocked at a different speed than nominal>}
   {, other comments} [2]

{} indicates items that are included only if appropriate.
<> indicates commentary about this format. Do not include in submission.
Field Meaning
XX total memory in system
ss size unit specifier: e.g. MB, GB
YY number of DIMMs used
gg ss size of each DIMM, including unit specifier
eR number of ranks: e.g. 1R, 2R
xff device organization bit width: e.g. x4, x8
PCh DDR generation: PC3 for DDR3
v Module component supply voltage values: e.g. <blank> for 1.5v, L for 1.35V
wwwww module bandwidth in MB/s: e.g. 8500, 10600
m module type: e.g. R for registered
aa CAS latency in clocks at maximum operating frequency.
xxxx speed memory is actually running (data rate, in MT/s) if not at not nominal max
CLy CAS latency if the tester has changed the latency to something other than the defaults

Although the above string has many fields, in practice it is usually brief, as the following examples show:

Notice that the main string "gg ss eRxff PChv-wwwwwm-aa" can be read directly from the label on the memory itself for all vendors who use JEDEC standard labels.





Identification of compilers

   WHEREAS it is desirable to improve compiler naming for
   results published on the SPEC website, in order to:

     * 1. Put the most important information first.
     * 2.  Make  it easier for readers to understand what
       compilers were used.
     * 3.  Make it easier for reviewers to verify correct
     * 4. Make it possible to write tools that parse version
       strings and compare results.


   (Part A) The SPEC CPU
   submission  handler will reject submissions unless the
   compiler string matches the following:

       language: Version n.n.n of name [Build id] [; repeat pre vious]


      []         indicates optional content
      "language" is a compiler identification string
                 as defined below
      "Version"  is a literal string
      "n.n.n"    is a version identifier as defined below
      "of"       is a literal string
      "name"     is a name of up to 60 characters as defined below
      id         is optional; up to 20 characters, for example
                 "Build 20110304-patch"

   Several of the fields above are defined using Perl regular
   expression syntax below.

  $language = '(C|C\+\+|Fortran)(/C\+\+)?(/Fortran)?';
  $language = "($language|Libraries)";
  $version  = '([\d_-.]+)( Update [\d_-.]+)?';
  $name     = '([^;]){3,60}';
  $build    = '\sBuild [^;]{1,20}';
  $lvn      = "$language: Version $version of $name($build)?";

  $re       = qr/^$lvn(; $lvn)?\$/;

     * 1 The language field contains either a slash-separated
       list of languages, or the word "Libraries". If more than
       one language is mentioned, the order for them is C, C++,

     * 2 The version field contains digits, underscores, and
       dots, plus an optional string "Update" followed by more
       digits, underscores, and dots.

     * 3 The above definition is not intended to specify the
       exact  Perl  code  for the submission handler. The
       implementer may choose use other syntax to achieve the
       same effect.

     * 4 The checks requested by this proposal will be added to
       the checks already in the submission handler.

     * 5 The submission handler will not attempt to validate
       the name field for the compiler other than checking its
       length, and demanding that it not contain a semi-colon.
       See Part B.

     * 6 The Build field is, from a syntax point of view,
       optional. It may nevertheless be mandatory from a run
       rules point of view: see About Reproducibility, below.


   (Part  B) Starting 11-Oct-2011, during review of results,
   the compiler name will be adjusted if it 
   differs from an 'expected' spelling. The expected spelling
   for each compiler name and version will be determined by the
   CPU subcommittee, typically by following the recommendation
   of the compiler vendor, if the vendor is a member of the

   The expected spelling will take into account:

     * The run rule (4.1.2)
     * The naming intent of the brand
     * Clarity for readers of SPEC results
     * Reproducibility

About Reproducibility

   If  the  name  and version string as used for ordering
   purpopses are ambiguous with regard to performance-relevant
   updates, additional information shall be included to resolve
   such ambiguity. Examples:

     * The version string for ordering purposes is "12.2" but
       performance changes over time. The "cc -v" command
       prints "". The subcommittee may decide to
       require the version as printed by "cc -v" for this

     * The version string is always "12.2" (for both ordering
       and from "cc -v"); but performance changes with build
       id. The subcommittee may decide to require inclusion of
       the build id for this compiler.