SPEC/HPG Fair Use Rule

Consistency and fairness are guiding principles for SPEC. To assure these principles are sustained, the following guidelines have been created with the intent that they serve as specific guidance for any organization (or individual) who chooses to make public comparisons using SPEC benchmark results.

When any organization or any individual makes public claims using SPEC benchmark results, SPEC requires that the following guidelines be observed:

[1] Reference is made to the SPEC trademark. Such reference may be included in a notes section with other trademark references (see http://www.spec.org/spec/trademarks.html for all SPEC trademarks and service marks).

[2] The SPEC web site (http://www.spec.org) or a suitable sub page is noted as the source for more information.

[3] If any public claims or competitive comparisons are made, the results stated or compared must be compliant with that benchmark's run and reporting rules and must cite the following: SPEC metric, CPU description (number of chips and cores), and degree of parallelism (for example, the number of OpenMP threads and/or MPI ranks).

[4] If competitive comparisons are made the following rules apply:

a. the basis for comparison must be stated,

b. the source of the competitive data must be stated, and the licensee (tester) must be identified or be clearly identifiable from the source,

c. the date competitive data was retrieved must be stated,

d. all data used in comparisons must be publicly available (from SPEC or elsewhere)

e. the benchmark must be currently accepting new submissions if previously unpublished results are used in the comparison.

[5] Comparisons with or between non-compliant test results can only be made within academic or research documents or presentations where the deviations from the rules for any non-compliant results have been disclosed.

[6] For benchmarks that allow estimates, it is acceptable to compare estimates to rule-compliant results only if the estimates are clearly identified as estimates. Comparisons, including competitive comparisons, are allowed as an exception to rules [3] and [5] above. The cited information required by rule [3] must still be provided. Further details regarding clear identification of estimates are provided in the run rules for benchmarks that allow them.

The following paragraph is an example of acceptable language when publicly using SPEC benchmarks for competitive comparisons:

Example:

SPEC® and the benchmark name SPEComp® are registered trademarks of the Standard Performance Evaluation Corporation. Competitive benchmark results stated above reflect results published on www.spec.org as of Jan 12, 2002. The comparison presented above is based on the best performing 4-core servers currently shipping by Vendor 1 [SPECompMbase2001 1245, 4 cores, 2 chips, 8 threads], Vendor 2 [SPECompMbase2001 1243, 4 cores, 4 chips, 4 threads], and Vendor 3 [SPECompMbase2001 1000, 4 cores, 1 chip, 4 threads]. For the latest SPEC OMP benchmark results, visit http://www.spec.org/omp/results.

The following is the list of approved SPEC HPG metrics and submetrics that may be used for competitive comparisons within the guidelines of the Fair Use Rule:

SPEC MPI2007:

Metrics: SPECmpiM_base2007, SPECmpiM_peak2007, SPECmpiM_2007
Individual benchmark SPECratios with the inclusion of the metrics.
Median run times of the individual benchmarks with the inclusion of the metrics.
Clearly identified estimates of metrics may be used.

SPEC OMP (OMPM2001 and OMPL2001):

Metrics: SPECompMbase2001, SPECompMpeak2001, SPECompLbase2001, SPECompLpeak2001, SPECompM2001, SPECompL2001.
Individual benchmark SPECratios with the inclusion of the metrics.
Median run times of the individual benchmarks with the inclusion of the metrics.
Clearly identified estimates of metrics may be used.

SPEC HPC2002:

Metrics: SPECchemM2002, SPECchemS2002, SPECenvM2002, SPECenvS2002, SPECseisM2002, SPECseisS2002

SPEC HPC96:

Metrics: SPECseis96_SM, SPECseis96_MD, SPECseis96_LG, SPECseis96_XL, SPECclimate96_SM, SPECclimate_MD, SPECclimate_LG, SPECclimate_XL, SPECchem96_SM, SPECchem96_MD, SPECchem96_LG, SPECchem96_XL