|
|
SPECpower_ssj2008 Run and Reporting RulesLast updated: May 05 2008(To check for possible updates to this document, please see http://www.spec.org/power_ssj2008/docs/SPECpower_ssj2008-Run_Reporting_Rules.html) OverviewSelecting one of the following will take you to the detailed table of contents for that section: 1. Introduction 2. Run Rules 4. Submission Requirements for SPECpower_ssj2008 5. SPECpower_ssj2008 Benchmark Kit Overview Detailed Contents1. Introduction 1.1 Philosophy 1.1.1 Applicability 1.1.2 Optimizations 1.2 Caveats 1.3 Research and Academic Usage 2. Run Rules 2.1 Measurement 2.2 Initializing and Running Benchmark 2.3 Workload 2.3.1 Manual Intervention 2.3.2 Sequence of Target Loads 2.3.2.1 The Active Idle Interval 2.4 SUT Configuration Parameters 2.5 Benchmark Control Parameters 2.5.1 Warehouse Count and Override 2.5.2 Validity Checks 2.6 Optimization Flags 2.7 Testbed Configuration 2.8 Line Voltage Source 2.9 Environmental Conditions 2.10 General Availability 2.10.1 SUT Availability for Historical Systems 2.11 System Under Test (SUT) 2.11.1 Electrical Equivalence 2.11.2 Hardware 2.11.2.1 Network Interfaces 2.11.3 Software 2.12 Java Specifications 2.12.1 Feedback Optimization and Precompilation 2.12.2 Benchmark Binaries and Recompilation 2.13 Power and Temperature Measurement 2.13.1 Power Analyzer Setup 2.13.2 Power Analyzer Specifications 2.13.3 Temperature Sensor Specifications 2.13.4 Supported and Compliant Devices 2.13.5 Acceptance Process for New Measurement Devices 3. Reporting Rules 3.1 Reporting Metric and Result 3.1.1 Publication 3.1.2 Estimates 3.1.3 Comparison to Other Benchmark Suites 3.1.4 Addendum to OSG Fair Use Policy 3.2 Reproducibility 3.3 Testbed Configuration Disclosure 3.3.1 General Availability Dates 3.3.2 Test Sponsor 3.3.3 Benchmark Results Summary 3.3.4 SUT 3.3.4.1 System Class - Component Source 3.3.4.2 SUT Hardware 3.3.4.3 SUT Software 3.3.4.4 System Under Test Notes 3.3.5 Controller System 3.3.5.1 Power Analyzer and Temperature Sensor 3.3.6 Disclosure Notes 3.3.7 Electrical and Environmental Data 4. Submission Requirements for SPECpower_ssj2008 5. SPECpower_ssj2008 Benchmark Kit Overview 5.1 Documents overview 1. IntroductionSPECpower_ssj2008 is the first generation SPEC benchmark for evaluating the power and performance of server class computers. This document specifies the guidelines on how SPECpower_ssj2008 V1.00 is to be run for measuring and publicly reporting power and performance results of servers. These rules abide by the norms laid down by SPEC in order to ensure that results generated with this benchmark are meaningful, comparable to other generated results, and repeatable, with documentation covering factors pertinent to reproducing the results. Per the SPEC license agreement, all results publicly disclosed must adhere to these Run and Reporting Rules. 1.1 PhilosophySPEC believes the user community will benefit from an objective series of benchmark results, which can serve as a common reference and be considered as part of an evaluation process. SPEC expects that any public use of results from this benchmark suite shall be for Systems Under Test (SUTs) and configurations that are appropriate for public consumption and comparison. For results to be publishable, SPEC expects:
1.1.1 ApplicabilitySPEC intends that this benchmark measures the power and performance of systems providing environments for running server-side Java applications. It is not a J2EE benchmark and therefore it does not measure Enterprise Java Beans (EJBs), servlets, Java Server Pages (JSPs), etc. Power consumption measured by this benchmark should not be assumed to represent the power consumption of other applications on the same hardware. 1.1.2 OptimizationsSPEC is aware of the importance of optimizations in producing the best system power and performance. SPEC is also aware that it is sometimes difficult to draw an exact line between legitimate optimizations that happen to benefit SPEC benchmarks and optimizations that specifically target a SPEC benchmark. However, with the rules below, SPEC wants to increase the awareness of implementers and end users of issues of unwanted benchmark-specific optimizations that would be incompatible with SPEC's goal of fair benchmarking.
1.2 CaveatsSPEC reserves the right to investigate any case where it appears that these guidelines and the associated benchmark run and reporting rules have not been followed for a published SPEC benchmark result. SPEC may request that the result be withdrawn from the public forum in which it appears and that the benchmarker correct any deficiency in product or process before submitting or publishing future results. SPEC reserves the right to adapt the benchmark codes, workloads, and rules of SPECpower_ssj2008 as deemed necessary to preserve the goal of fair benchmarking. SPEC will notify members and licensees whenever it makes changes to the benchmark and may rename the metrics. In the event that the workload or metric is changed, SPEC reserves the right to republish in summary form "adapted" results for previously published systems, converted to the new metric. In the case of other changes, a republication may necessitate retesting and may require support from the original test sponsor. Relevant standards are cited in these run rules as URL references, and are current as of the date of publication. Changes or updates to these referenced documents or URL's may necessitate repairs to the links and/or amendment of the run rules. The most current run rules will be available at the SPEC web site at httpp://www.spec.org. SPEC will notify members and licensees whenever it makes changes to the suite. 1.3 Research and Academic UsageSPEC encourages use of the SPECpower_ssj2008 benchmark in academic and research environments. SPEC encourages researchers to obey as many of the run rules as practical. However, it is understood that experiments in such environments may be conducted in a less formal fashion than that demanded of licensees disclosing results which are intended to comply with the run rules. For example, a research environment may use early prototype hardware that simply cannot be expected to remain operational for the length of time required to run the required number of points, or may use research software versions that are unsupported and are not generally available. Adhering to the SPEC run rules improves the clarity, reproducibility, and comparability of research results. When the rules cannot be followed the run is non-compliant and cannot be represented as compliant and the benchmark metric cannot be used. Such results must not be compared to compliant results. SPEC requires research results be clearly distinguished from compliant results, and the disclosure of any and all deviations from the rules. 2 Run Rules2.1 MeasurementThe provided SPECpower_ssj2008 tools must be used to run and produce measured SPECpower_ssj2008 results. The SPECpower_ssj2008 metric is a function of the SPECpower_ssj2008 workload (see section 2.3), and the defined benchmark control parameters (see section 2.5). SPECpower_ssj2008 results are not comparable to any other application area power and performance metric. 2.2 Initializing and Running BenchmarkFor guidance, please consult the User Guide and Hardware Setup Guide (http://www.spec.org/power_ssj2008/). 2.3 WorkloadSPECpower_ssj2008 exercises a Java application workload. A detailed description can be found in the design document (http://www.spec.org/power_ssj2008/). 2.3.1 Manual InterventionNo manual intervention or optimization to the SUT or its internal and external environment is allowed during the benchmark run. 2.3.2 Sequence of Target LoadsThe benchmark runs at multiple target loads to determine the power consumption of the SUT under varying processing loads. First, the maximum throughput achievable by the SUT is determined by running the workload unconstrained for at least 3 calibration intervals. The maximum is set as the arithmetic average of the throughputs achieved during the final two calibration interval runs. The workload is then run in a constrained manner, with delays inserted into the workload stream, to obtain total throughputs of 100%, 90%, 80%, 70%, 60%, 50%, 40%, 30%, 20%, and 10% of the maximum throughput. The delays inserted into workload streams are exponentially random with a fixed maximum of 10 seconds. During each of these target loads, the power characteristics of the SUT as well as the temperature are recorded. Finally, the power characteristics and temperature are measured and recorded during an idle interval during which the SUT processes no Java transactions. The preceding sequence is automatically implemented by the benchmark harness and must not be changed for a compliant run. 2.3.2.1 The Active Idle IntervalDuring active idle, the SUT must be in a state in which it is capable of completing workload transactions. The active idle measurement interval is treated in a manner consistent with all other target load levels, with the exception that no transactions occur during the active idle interval. The intent in defining and automating active idle power measurement within the SPECpower_ssj2008 benchmark is to prevent manipulation of idle power measurements. The benchmark workload and the JVM process in which it is running are to remain active without interruption for the duration of this phase. 2.4 SUT Configuration ParametersThe "SPECpower_ssj_config.props" file contains configuration information used to generate the final report, and values must be populated appropriately by the tester to reflect the SUT for a compliant run. 2.5 Benchmark Control ParametersThere are a number of parameters which control the operation of SPECpower_ssj2008. The "SPECpower_ssj.props" file is used to control the parameters of the benchmark run. The properties in the "Changeable Input Parameters" section of the benchmark parameters properties file may be set to values other than default for a compliant run. These are marked gray in the table below. For a compliant run, the properties in the "Fixed Input Parameters" section of the properties file being used must not be changed from the values as provided by SPEC. All workload JVM instances must use the same parameters in a multi-JVM environment.
2.5.1 Warehouse Count and OverrideThe SPECpower_ssj2008 benchmark runs a fixed number of warehouses, N, equal to the number of logical processors in the system under test. This number is, by default, the value returned by the java.lang.Runtime.getRuntime.availableProcessors API. The value may be overridden by setting the input.load_level.number_warehouses property provided that the result is submitted to SPEC for review and an acceptable reason is given in the config.sw.notes section of the disclosure report. An example of an acceptable reason to override the default value would be if System.availableProcessors() does not return an accurate or valid value for the hardware architecture of the SUT. An example of an unacceptable reason would be to decrease the value of N from the default to hide scalability problems and artificially obtain a higher score. 2.5.2 Validity ChecksAt the beginning of each run, the benchmark parameters are checked for conformance to the run rules. Warnings are displayed for non-compliant properties and printed in the final report; however, the benchmark will run to completion producing a report that is not valid for publication. The following are required for a valid run and are automatically checked:
2.6 Optimization FlagsBoth JVMs and native compilers are capable of modifying their behavior based on flags. Flags which do not break conformance to section 2.12 are allowed. All command-line flags used must be reported. All flags used must be documented and supported within the time frame specified in this document for general availability. At the time a result is submitted to SPEC, descriptions of all flags used but not currently publicly documented must be available to SPEC for the review process. When the result is published, all flags used must be publicly documented, either in the vendor's public documentation, in the disclosure, or in a separate flags file. 2.7 Testbed ConfigurationThese requirements apply to all hardware and software components used in producing the benchmark result, including the System under Test (SUT), network, and controller.
2.8 Line Voltage SourceThe preferred Line Voltage source used for measurements is the main AC power as provided by local utility companies. Power generated from other sources often has unwanted harmonics which are incapable of being measured correctly by many power analyzers, and thus would generate inaccurate results.
The usage of an uninterruptible power source (UPS) as the line voltage source is allowed, but the voltage output must be a pure sine-wave. For placement of the UPS, see 2.13.1. This usage must be specified in the Note-section of the FDR. If an unlisted line voltage source is used, a reference to the standard is necessary. For situations in which the appropriate voltages are not provided by local utility companies (e.g. measuring a server in the United States which is configured for European markets, or measuring a server in a location where the local utility line voltage does not meet the required characteristics), an AC power source may be used, and the power source must be specified in the notes section of the disclosure report. In such situation the following requirements must be met, and the relevant measurements or power source specifications disclosed in the notes section of the disclosure report:
The intent is that the AC power source not interferes with measurements such as power factor by trying to adjust its output power to improve the power factor of the load. 2.9 Environmental ConditionsSPEC requires that power measurements be taken in an environment representative of the majority of usage environments. The intent is to discourage extreme environments that may artificially impact power consumption or performance of the server. SPECpower_ssj2008 requires the following environmental conditions to be met:
2.10 General AvailabilityThe entire testbed must be comprised of components that are generally available on or before date of publication, or shall be generally available within three months of the first publication of these results. Products are considered generally available if they are orderable by ordinary customers and ship within a reasonable time frame. This time frame is a function of the product size and classification and common practice. Some limited quantity of the product must have shipped on or before the close of the stated availability window. Shipped products do not have to match the tested configuration in terms of CPU count, memory size, and disk count or size, but the tested configuration must be available to ordinary customers. The availability of support and documentation of the products must be coincident with the release of the products. Hardware products that are still supported by their original or primary vendor may be used if their original general availability date was within the last five years. The five-year limit is waived for hardware used in client systems. Software products that are still supported by their original or primary vendor may be used if their original general availability date was within the last three years. Information must be provided in the disclosure to identify any component that is no longer orderable by ordinary customers. See http://www.spec.org/osg/policy.html - Appendix C. Guidelines for General Availability 2.10.1 SUT Availability for Historical SystemsIn the interest of providing some historical perspective on the power consumption and performance for older or obsolete systems, SPEC will consider review and publication of otherwise compliant benchmark results. This is expressly for systems where the general availability date is/was beyond the current general availability limits and may no longer be supported by their original vendor. Submissions will be reviewed and accepted with the stipulation that the hardware availability date reflect the first availability of the platform (as opposed to the date of purchase). If the SUT is "parts built", the general availability date shall reflect that of the "core components", i.e. the chipset and processor combination. For these historic results, this entry must be included in the SUT
Notes section: For "historic results", it is expected that the benchmark will be run with software of similar vintage to that of the hardware where possible (the benchmark may require a later JVM). 2.11 System Under Test (SUT)The SPECpower_ssj2008 benchmark is intended to measure the power and performance attributes of a computer server under the specific benchmark workload. In the sense of this benchmark the server or system under test (SUT) is defined by the following properties:
2.11.1 Electrical EquivalenceMany other SPEC benchmarks allow duplicate submissions for a single system sold under various names. Each SPECpower_ssj2008 result submitted to SPEC or made public must be for an actual run of the benchmark on the SUT named in the result. Electrically equivalent submissions are not allowed. 2.11.2 HardwareAny hardware configuration sufficient to install, start, and run the benchmark to completion in compliance with these run rules (including the availability guidelines in 2.10) shall be considered a compliant configuration. Any device configured at the time the benchmark is started must remain configured for the duration of the benchmark run. Devices which are configured but not needed for the benchmark (e.g. additional on-board NICs) may be disabled prior to the start of the benchmark run. Manual intervention to change the configuration state of components after the benchmark run has begun is not allowed. External devices required for initial setup or maintenance of the SUT, but not required for normal operation or for running the benchmark (e.g. an external optical drive used for OS installation) may be removed prior to the benchmark being started. If the model name or product number implies a specific hardware configuration, these specific components can not be removed from the hardware configuration but may be upgraded. Any upgrades are subject to the support, availability and reporting requirements of this document. For example, if the SUT is available from the vendor only with dual power supplies, both supplies must be configured and measured during the benchmark run. The power supplies may be upgraded if the vendor offers and supports such an upgrade, and the upgrade must be documented in the benchmark disclosure report. A video monitor, if configured, may be powered by a separate power source and need not be included in the power measurement of the SUT. All other configured devices must receive their power from the measured power source. The components are required to be:
Any tuning or deviation from the default installation or configuration of hardware components is allowed by available tools only and must be reported. This includes BIOS settings, power saving options in the system board management, or upgrade of default components. Only modifications that are documented and supported by the vendor(s) are allowed. 2.11.2.1 Network InterfacesAt least one port of the SUT´s fastest network interface controller must be connected and operating at its full rated speed or 1Gb. Automatically reducing network speed and power consumption in response to traffic levels is allowed for network interface controllers with such capabilities, as long as they are also capable of increasing to their full rated speed automatically. 2.11.3 SoftwareAll software required to run the SPECpower_ssj2008 benchmark must be installed on and executed from a stable storage device which is considered part of the SUT. Required software components are
Optional power management software, when installed, must be reported. The operating system must be in a state sufficient to execute a class of server applications larger than the benchmark alone. The majority of operating system services should remain enabled. Disabling operating system services may subject disclosures to additional scrutiny by the benchmark subcommittee and may cause the result to be found non-compliant. Any changes from the default state of the installed software must be disclosed in sufficient detail to enable the results to be reproduced. Examples of tuning information which must be documented include, but are not limited to:
These changes must be "generally available", i.e., available, supported and documented. For example, if a special tool is needed to change the OS state, it must be available to users and documented by the vendor. The tester is expected to exercise due diligence regarding the reporting of tuning changes, to ensure that the disclosure correctly records the intended final product. The software environment on the SUT is intended to be in a state where applications other than the benchmark could be supported. Disabling of operating system services is therefore discouraged but not explicitly prohibited. Disabled services must be disclosed. The submitter/sponsor will be responsible for justifying the disabling of service(s). Services that must not be disabled include but are not limited to logging services such as cron or event logger. A list of active operating system services may be required to be provided for SPEC's results review. The submitter is required to generate and keep this list for the duration of the review period. Such a list may be obtained, for example, by:
2.12 Java SpecificationsTested systems must provide an environment suitable for running typical server-side J2SE 5.0 (or higher) applications. Any tested system must include an implementation of the Java (tm) Virtual Machine as described by the following references, or as amended by SPEC for later Java versions:
The system must include a complete implementation of those classes that are referenced by this benchmark as in the J2SE 5.0 specification (http://java.sun.com/j2se/1.5.0/). SPEC does not intend to check for implementation of APIs not used in the benchmark. For example, the benchmark does not use AWT (Abstract Window Toolkit, http://java.sun.com/j2se/1.5.0/docs/guide/awt/index.html, and SPEC does not intend to check for implementation of AWT. Note that the reporter does use AWT, however it is not necessary to run the reporter on the SUT. 2.12.1 Feedback Optimization and PrecompilationFeedback directed optimization and precompilation from the Java bytecodes are allowed, subject to the restrictions regarding benchmark-specific optimizations in section 1.1.2. Precompilation and feedback-optimization before the measured invocation of the benchmark are also allowed. Such optimizations must be fully disclosed. 2.12.2 Benchmark Binaries and RecompilationThe SPECpower_ssj2008 benchmark binaries are provided in jar files containing the Java classes. Valid runs must use the provided jar files and these files must not be updated or modified in any way. While the source code of the benchmark is provided for reference, the benchmarker must not recompile any of the provided .java files. Any runs that use recompiled class files are marked not valid and can not be reported or published. 2.13 Power and Temperature MeasurementThe SPECpower_ssj2008 benchmark tool set provides the ability to automatically gather measurement data from supported power analyzers and temperature sensors and integrate that data into the benchmark result. SPEC requires that the analyzers and sensors used in a submission be supported by the measurement framework, and be compliant with the specifications in the following sections. SPEC also encourages licensees to submit code supporting additional devices that meet the benchmark requirements, which can be incorporated into later releases. 2.13.1 Power Analyzer SetupThe power analyzer must be located between the AC Line Voltage Source and the SUT. No other active components are allowed between the AC Line Voltage Source and the SUT. Power analyzer configuration settings that are set by PTDaemon must not be manually overridden.2.13.2 Power Analyzer SpecificationsTo ensure comparability and repeatability of power measurements, SPEC requires the following attributes for the power measurement device used during the benchmark. Please note that a power analyzer may meet the requirements when used in some power ranges but not in others, due to the dynamic nature of power analyzer Accuracy and Crest Factor.
An analyzer with a vendor-specified accuracy of +/- 0.5% of reading +/- 4 digits, used in a test with a maximum wattage value of 200W, would have "overall" accuracy of (((0.5%*200W)+0.4W)=1.4W/200W) or 0.7% at 200W. An analyzer with a wattage range 20-400W, with a vendor-specified accuracy of +/- 0.25% of range +/- 4 digits, used in a test with a maximum wattage value of 200W, would have "overall" accuracy of (((0.25%*400W)+0.4W)=1.4W/200W) or 0.7% at 200W. 2.13.3 Temperature Sensor SpecificationsTemperature must be measured no more than 50mm in front of (upwind of) the main airflow inlet of the SUT. To ensure comparability and repeatability of temperature measurements, SPEC requires the following attributes for the temperature measurement device used during the benchmark:
2.13.4 Supported and Compliant DevicesSee Device List (http://www.spec.org/power_ssj2008/) for a list of currently supported (by the benchmark software) and compliant (in specifications) power analyzers and temperature sensors. 2.13.5 Acceptance Process for New Measurement DevicesAdding a new measurement device to the SPEC power measurement framework includes three components:
Documentation to prove compliance with all required attributes must be provided. Publicly available documentation is preferred, but in special cases where a device vendor does not wish to disclose information perceived as proprietary, the device vendor may request its documentation remain SPEC Confidential. For new device modules, all source code submitted to SPEC must include a signed SPEC Permission to Use Form ( http://www.spec.org/spec/docs/permission_to_use.pdf) and must be freely available for use by other members and licensees of the benchmark. Supporting documentation must be provided as needed for the review. Once the code has been submitted, SPEC will then review the code. Barring any issues, SPEC will then incorporate the device module into a new version of the benchmark. Compliant runs must be done with SPEC provided binaries only. The final step is testing of the device to verify that it meets the run rules requirements of section 2.13. The intent of this testing is to ensure that results obtained with the device are comparable to results obtained with other measurement devices. SPEC provides a series of tests (see SPEC´s Power Analyzer Acceptance Testing) that must be performed to determine power analyzer behavior under dynamic benchmark conditions. The preferred method of running these tests is to connect the new measurement device in series with another power analyzer that has already been accepted as compliant with the run rules requirements. These tests should be run by the submitter. In cases where the submitter does not have a currently-accepted power analyzer, a member of SPEC may volunteer to run those tests if a device is provided to them. SPEC will review the test results against a set of criteria specified (see SPEC´s Power Analyzer Acceptance Testing). If questions arise, SPEC may ask that additional testing be performed. Once a set of satisfactory results is produced, the device will be accepted as compliant and incorporated into the next release of the benchmark software. Note: Since only SPEC-provided binaries may be used for compliant results, it is recommended that the device acceptance process be started well in advance of any benchmark use of a new device. 3 Reporting RulesIn order to publicly disclose SPECpower_ssj2008 results, the tester must adhere to these reporting rules in addition to having followed the run rules above. The goal of the reporting rules is to ensure the system under test is sufficiently documented so that someone could reproduce the test and its results and to ensure that the tester has complied with the run rules. 3.1 Reporting Metric and ResultSPECpower_ssj2008 expresses power and performance in the terms of "overall ssj_ops/watt". Overall ssj_ops/watt represents the sum of the performance measured at each target load level (in ssj_ops) divided by the sum of the average power (in W) at each target load including active idle. The report of results is an HTML file generated by the tools provided by SPEC. These tools must not be changed, except for portability reasons with prior SPEC approval. The tools perform error checking and will flag some error conditions as resulting in an "invalid run". However, these automatic checks are only there for debugging convenience, and do not relieve the benchmarker of the responsibility to check the results and follow the run and reporting rules. The section of the ssj.wxyz.raw file that contains actual test measurement must not be altered. Corrections to the SUT descriptions may be made as needed to produce a properly documented disclosure. 3.1.1 PublicationAny benchmark result not in full compliance with the run and reporting rules must not be represented using the SPECpower_ssj2008 metrics. SPEC requires that each licensee test location (city, state/province and country) submit a single compliant result for review, and have that result accepted, before publicly disclosing or representing as valid any compliant SPECpower_ssj2008 result. Only after acceptance of a compliant result by the subcommittee may the licensee publicly disclose any SPECpower_ssj2008 result produced in compliance with these run and reporting rules, without acceptance by the SPECpower subcommittee.The intent of this requirement is that the licensee demonstrates the ability to produce a compliant result before publicly disclosing additional results without review by the subcommittee. 3.1.1.1 Disclosure RequirementIf a SPECpower_ssj2008 licensee publicly discloses a SPECpower_ssj2008 result (for example in a press release, academic paper, magazine article, or public web site), any SPEC member may request that the FDR(s) from the run(s) be sent to SPEC. Such results must be made available to all interested members no later than 10 working days after the request. Any SPEC member may request that the result and its FDR be reviewed by SPEC. If the tester does not wish to have the result posted on the SPEC web pages, the result will not be posted. When public claims are made about SPECpower_ssj2008 results, whether by vendors or by academic researchers, SPEC reserves the right to take actions, for example if it should occur that the FDR is not made available, or shows substantially different performance from the tester's claim, or shows obvious violations of the run rules. 3.1.2 EstimatesEstimated results are not allowed to be publicly disclosed. 3.1.3 Comparison to Other Benchmark SuitesPower and performance results from SPECpower_ssj2008 must not be compared to the results from any other benchmark. 3.1.4 Addendum to OSG Fair Use PolicyAny entity choosing to make public statements using SPECpower_ssj2008 must follow the OSG Fair Use guidelines (http://www.spec.org/osg/fair_use-policy.html ) and following addendum to this policy: For comparisons,
Since the Active Idle measurement point does not have a performance load level, comparisons of Active Idle points must include the primary metric and the performance at the 100% target load level in close proximity. The calibration throughputs must not be used (use the 100% target load level instead) "Close proximity" as used above is defined to mean in the same paragraph, in the same font style and size, and either within 100 words or on the same presentation slide. The following paragraphs are examples of acceptable language when publicly using SPECpower_ssj2008 for comparisons for the three cases listed above, including a 4th example for Active Idle comparisons:
Note that, for each of these examples, the following is also required, but could be included in a footnote: "SPEC and the benchmark name SPECpower_ssj are trademarks of the Standard Performance Evaluation Corporation. Benchmark results stated above reflect results published on http://www.spec.org as of November 30, 2007. For the latest SPECpower_ssj2008 benchmark results, visit http://www.spec.org/power_ssj2008." 3.2 ReproducibilitySPEC is aware that power or performance results for pre-production systems may sometimes be subject to change, for example when a last-minute bugfix reduces the final performance. If the sponsor becomes aware that the SPECpower_ssj2008 metric of a typical released system is more than 5% lower than that reported for the pre-release system, the tester is required to submit a new result for the production system, and the original result shall be marked non-compliant (NC). By submitting or publishing a benchmark disclosure (report) to SPEC, the test sponsor implicitly states that the system performance and power measured is representative of such systems. Power consumption is dependent on many factors that may vary over time within a specific vendor model. It can also vary from system to system due to well-known variability in electronic component fabrication processes. 3.3 Testbed Configuration DisclosureThe system configuration information that is required to reproduce published power and performance results must be reported. The principle is that if anything affects power or performance or is required to duplicate the results, it must be described. Any deviations from the standard, default configuration for the SUT must be documented so an independent party would be able to reproduce the result without any further assistance. For most of the following configuration details, there is an entry in the configuration file, and a corresponding entry in the tool-generated HTML result page. If information needs to be included that does not fit into these entries, the Notes sections must be used. 3.3.1 General Availability DatesThe dates of general customer availability must be listed for the major components: hardware, server software, and operating system, by month and year. All the system, hardware and software features are required to be available within three months of the first publication of these results. With multiple components having different availability dates, the latest availability date must be listed. 3.3.2 Test SponsorThe reporting page must list:
3.3.3 Benchmark Results SummaryThe reporter automatically populates the Benchmark Result Summary. For each Target Load:
Also a graphical representation of these values is automatically rendered. 3.3.4 SUT3.3.4.1 System Class - Component SourceThe component source, either "Single Supplier" or "Parts Built", needs to be disclosed
3.3.4.2 SUT HardwareThe following SUT hardware components must be reported:
3.3.4.3 SUT SoftwareThe following SUT software components must be reported:
3.3.4.4 System Under Test NotesThe System Under Test Notes section is used to document:
3.3.5 Controller SystemThe following properties must be reported:
3.3.5.1 Power Analyzer and Temperature SensorThe following properties must be reported:
3.3.6 Disclosure NotesThe Notes section is used to document:
3.3.7 Electrical and Environmental DataThe reporter automatically populates (values from measurements) the following table entries. For each Target Load:
The reporter also automatically populates measured values for the average Power Factor and the Minimum Temperature (°C).
4 Submission Requirements for SPECpower_ssj2008When a potentially-compliant run is completed and acceptance by SPEC is desired, the raw results file must be submitted. The required file(s) should be e-mailed to SPEC as an attachment. The committee may request additional benchmark output files from the submitter as well. The submitter should be prepared to participate in discussion during the review cycle and at the subcommittee meeting in which the result is voted on for final acceptance, to answer any questions raised about the result. The submitter is also required to keep the log files for the SUT and Controller from the run for the duration of the review cycle and make them available upon request. Licensees of the benchmark wishing to submit results for acceptance may be required to pay a fee. The complete submission process is documented in Submitting OSG Benchmark Results to SPEC. (http://www.spec.org/osg/submitting_results.html). 5 SPECpower_ssj2008 Benchmark Kit OverviewThe benchmark kit includes tools for running the benchmark and reporting its results. The workload and CCS components are written in Java; precompiled class files are included with the kit, so no build step is necessary. This software implements various checks for conformance with these run and reporting rules, therefore the SPEC software must be used. The kit also includes C code for the PTDaemon and its device modules. Any new device modules will be evaluated by the sub-committee according to the acceptance process (see section 2.13.5). Once the code is accepted by the sub-committee, it will be made available for any licensee to use in their measurements and submissions. 5.1 Documents overviewThe benchmark related documents (Run and Reporting Rules, User Guide, Hardware Setup Guide, Design Document, Methodology, and FAQ) can be found in the doc directory of the distribution. For the latest versions, please consult SPEC´s website ( http://www.spec.org/power_ssj2008/). Copyright © 2007-2008 Standard Performance Evaluation Corporation |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||