SPEC subject: SPEC Reporting Rules for date: January 24, 1992 CFP92 from: SPEC Steering Committee ABSTRACT This memorandum describes the Rules for reporting the results of SPEC CFP92 suite. This is a companion document to SPEC Run Rules, which describes the process of running the benchmarks contained in the SPEC Suite. Memorandum to Performance Analysts SPEC subject: SPEC Reporting Rules for date: January 24, 1992 CFP92 from: SPEC Steering Committee 1. SPEC_CFP92_Introduction The SPEC CFP92 benchmark suite consists of CPU intensive benchmarks that are intended to be meaningful samples of applications which perform floating point logic and computations in a technical computing environment. SPEC CFP92 does not assess the ability of a system under test to handle disk, graphics, communication, or multi-user activity. Many of the SPEC CFP92 benchmarks have been derived from publicly-available application programs, and they are intended to be portable to as many current and future hardware platforms as possible. Consequently, the version of the programs in this distribution contains no hardware dependencies. Where possible, the programs have been structured to ensure that they do not unfairly favor one hardware architecture over another. For this reason, the application programs in this distribution should not be used to assess the probable performance of commercially available, tuned versions of the same application. Different degrees of tuning and availability of specialized hardware features may make large (and varying) differences between the performance of the programs in this distribution and that of commercial applications for production use. The individual benchmarks in this suite may be similar, but not necessarily identical to benchmarks with the same name which are available from sources other than SPEC. Therefore it may not be valid to compare the individual benchmark results of SPEC CFP92 to any benchmarks not obtained from SPEC. Reporting of results must follow the guidelines established in the "Reporting Guidelines" section of the distribution documentation. It is not intended that the SPEC CFP92 be used as a replacement for the benchmarking of actual customer applications to determine vendor or product selection. - 2 - 2. REPORTING_GUIDELINES This section describes the standard reporting format that must be used when reporting results using the SPEC CFP92 benchmarks to obtain a SPECfp92 for a system. The SPECfp92 rating is used to describe the speed of a system. EXAMPLES OF REPORTING RULES FOR SPEC RESULTS CAN BE SEEN IN THE QUARTERLY SPEC NEWSLETTER. 1 The intent of the SPECFP92 Reporting Guidelines is to describe a consistent method of reporting results to aid customers in interpreting and comparing results of different systems' performance. All measured and calculated values are reported to the nearest tenth. Unless noted to the contrary, the report must contain the following items of information: 2.1 Table_Of_Run_Times A table of 15 rows and 4 columns structured as follows: a. One row for each individual benchmark in SPEC CFP92 b. Four columns i. Name of benchmark (preferred order shown in example below) ii. SPEC Reference Time for each benchmark in the suite. A. 013.spice2g6: 24000 seconds B. 015.doduc: 1860 seconds C. 034.mdljdp2: 7090 seconds D. 039.wave5: 3700 seconds E. 047.tomcatv: 2650 seconds __________ 1. Articles describing the suite and metrics can be found in the December 1991 SPEC Newsletter. - 3 - F. 048.ora: 7420 seconds G. 052.alvinn: 7690 seconds H. 056.ear: 25500 seconds I. 077.mdljsp2: 3350 seconds J. 078.swm256: 12700 seconds K. 089.su2cor: 12900 seconds L. 090.hydro2d: 13700 seconds M. 093.nasa7: 16800 seconds N. 094.fpppp: 9200 seconds iii. For SPECFP92, Elapsed time (rounded to the nearest 0.1 second) for the System-Under-Test (SUT) taken to run each benchmark of SPEC CFP92 suite. The elapsed times reported are for each benchmark run one at a time. For the SPECFP92, the elapsed time value measures the execution of a single copy of each benchmark. The reported times are the best times that can be consistently reproduced. If any changes to an individual benchmark are made for portability purposes (e.g. to obtain a clean compile), the elapsed time should be followed by an "*", and a description of the change should be described in PART 7 of the report.2 iv. For SPECFP92, Column 4 has Ratio of SPEC Reference Time to SUT time (divide number in column 2 by number in column 3). The ratio is without dimension, and will be referred to as SPECratio. __________ 2. REFER to portability rules in SPEC run rules document. - 4 - c. A row with the Geometric mean for the numbers listed in column 4. This row is labeled as "Geom Mean: SPECFP92". The Geometric Mean is calculated as the nth root of the product of the individual values in the SPECratio column. For SPECFP92, the Geometric Mean of the SPECratios in column 4 is defined as the SPECFP92 of the SUT. Please note: if an "*" sign appears in any of the rows to indicate a portability change in a benchmark, the "*" must be displayed after the Geometric Mean calculation for elapsed time and SPECratio. The asterisk is dropped after the portability change has been approved by the Steering committee as a portability change and not a performance change and when the change is implemented in the tape. 2.2 Identification_Information This information includes the date (month/year) that the benchmarks were run, the name and location of the company that ran them and the license number of the SPEC tape used. This is typically listed under the table generated in Section 2.1. 2.3 Hardware_Configuration A description of the system (cpu/clock, fpu/clock), number of processors, relevant peripherals, etc. are included in this space. This information is typically listed next to the table generated in Section 2.1. The amount of information supplied should be sufficient to allow duplication of results by another party. The following checklist is provided to show examples of hardware features which may affect the performance of SPEC CFP92 benchmarks. The checklist is not intended to be all-inclusive, nor is each feature in the list required to be described. The rule of thumb is: "if it affects performance or the feature is required to duplicate the results, describe it". HARDWARE PLATFORM FEATURES o Manufacturer's model number o CPU component ID and clock speed o Floating Point Unit (FPU) ID and clock speed - 5 - o Number of CPUs and FPUs o Cache Size (per CPU), description and organization o Memory (Amount and description) o Disk subsystem configuration - Number of active connected disks - Disk controllers: ID, number and type - Disk: ID, number and type o Network Interface 2.4 Software_Configuration The description of the software configuration used should be detailed enough so that another party could duplicate the results reported. This information is typically reported after the hardware description. The following is a list of possible items that might be documented: SOFTWARE CONFIGURATION o Operating System Type and Release level (with revision/update level if appropriate) o Compiler release levels used. If multiple compilers are available, specify which ones were used. o Optional software required to reproduce results o File system type used o Firmware level 2.5 Systems_Environment This section should document the overall systems environment used to run the benchmark. It is typically listed after the software description. The following is a list of possible items that might be documented: SYSTEMS ENVIRONMENT CHECKLIST o Single or multi-user state - 6 - o System tuning parameters (e.g. MAKEFS and TUNEFS parameters) o Process tuning parameters (e.g. stack size, time slice) o Background load, if any 2.6 General_Availability_of_the_SUT This item indicates the availability (or projected availability) date for the system or components being benchmarked. If both hardware and software are not available at the same time, list them separately. SPEC requires that the availability of hardware and software should be within 6 months of the date of publication of the data. It should be listed in Month/Year format. 2.7 Individual_Benchmark_Notes This section documents ANY changes made to the individual benchmark source code or execution parameters for portability reasons. For example, if difficulty arises in compiling or executing an individual benchmark because of portability problems, the difficulty should be described, and the code changes made to resolve the portability problem documented. This should include module name, line number and actual source code changes. This section must also include any use of alternate source files, alternate input files, or alternate result files, as noted in "Runrules" section of the distribution documentation. 2.8 Graphical_Representation If desired, a graphical summary of the table from Section 2.1 may be included as an additional item of information. For SPECfp92, it should be shown as follows: a. The 14 benchmarks comprising SPEC CFP92 should appear as individual points on the x-axis ordered by benchmark numbers. b. The SPECratio of each benchmark (obtained from column 4 in the table described above), should be plotted as as a vertical bar graph. - 7 - c. A horizontal line should be drawn across the graph intersecting the y-axis at the SPECFP92 geometric mean value. This permits differentiation between each benchmark's SPECratio and the overall SPECCFP geometric mean. 3. SAMPLE_REPORTS 3.1 SPECFP92_-_Summary_of_SPEC_CFP92_Results _______________________________________________________________ | SPEC Benchmark | SPEC | System Under| SPECratio| | CFP92 | Reference | Test | | | | Time Elapsed| (Sample) | | | | Time | Elapsed Time| | | | (seconds) | (seconds) | | |____________________|______________|______________|___________| |____________________|______________|______________|___________| | 013.spice2g6 | 24000 | 456.6 | 52.6 | |____________________|______________|______________|___________| | 015.doduc | 1860 | 23.3 | 79.8 | |____________________|______________|______________|___________| | 034.mdljdp2 | 7090 | 567.9 | 12.5 | |____________________|______________|______________|___________| | 039.wave5 | 3700 | 123.0 | 30.1 | |____________________|______________|______________|___________| | 047.tomcatv | 2650 | 45.9 | 57.7 | |____________________|______________|______________|___________| | 048.ora | 7420 | 234.0 | 31.7 | |____________________|______________|______________|___________| | 052.alvinn | 7690 | 145.0 | 53.0 | |____________________|______________|______________|___________| | 056.ear | 25500 | 456.0 | 55.9 | |____________________|______________|______________|___________| | 077.mdljdp2 | 3350 | 78.6 | 42.6 | |____________________|______________|______________|___________| | 078.swm256 | 12700 | 345.6 | 36.7 | |____________________|______________|______________|___________| | 089.su2cor | 12900 | 267.4 | 48.2 | |____________________|______________|______________|___________| | 090.hydro2d | 13700 | 277.8 | 49.3 | |____________________|______________|______________|___________| | 093.nasa7 | 16800 | 411.4 | 40.8* | |____________________|______________|______________|___________| | 094.fppp | 9200 | 233.7 | 39.4 | |____________________|______________|______________|___________| | Geom Mean: SPECfp92| | | 41.9* | |____________________|______________|______________|___________| - 8 - *Changes made to benchmark for portability reasons, see PART 7 for details Note that the Geometric mean is the nth root of the product of n elements. Therefore in this case, it is the 14th root of the product of the 14 SPECratios in 4th column. 3.2 Identification_Information Tested on mm/yy, tested by xxxxx of (location), SPEC License No. yyyyy. 3.3 Hardware_Configuration Hardware Configuration Detail for "Sample" system o Vendor machine xxxxxxx, model number nnnnnnn o RISC processor xxxxxx; nn MHz clock speed o FPU xxxxxxxx, nn MHz clock speed o One processor/FPU o n MB main memory o n disk drives, each nnn megabytes capacity 3.4 Software_Configuration Software Configuration Detail for "Sample" system o Operating System name, Version nnn, update nnn applied o FORTRAN Version nnn o C Version nnn o Other software: (if applicable) o Firmware level: (if applicable) 3.5 General_Availability The hardware will be available mm/yy and the software will be available nn/yy. Note: When a beta version is used, the date of the final - 9 - release must be included which must be within 6 months of result publication. 3.6 Systems_Environment Systems Environment Detail for "Sample" system o Tuning Parameters: o Background load: o System state: (Mult-user or single user) 3.7 Notes_On_Individual_Benchmark_Runs - 093.nasa7: KAP was used to optimize the performance * In 093.nasa7, variable name xxxxx changed to yyyyy to avoid library conflict.