This document describes the process by which a SPEC member or licensee
can add software support for a power analyzer to the SPECpower_ssj2008
benchmark, and submit tests for review and possible inclusion by SPEC
as an accepted device. To accomplish the tasks required for this
process, prior knowledge of the usage of the SPECpower_ssj2008
benchmark, Power Analyzer setup and usage, and programming skills are
required.
The first step in adding device support must be to compare the new
device?s specifications to those of the
SPECpower_ssj2008 Run Rules Section 2.13.2-Power Analyzer Specifications.
If it is desired that the device be included as an accepted measurement
device for SPECpower_ssj2008, suitable documentation must be provided
showing compliance with that section.
Devices that do not meet SPEC?s requirements may still have software
support added for the SPECpower_ssj2008 harness; however, any benchmark
testing done using such devices will be marked as non-compliant and may
not be published outside of SPEC or submitted to SPEC for review.
A SPECpower_ssj2008 license is required to obtain the source code
necessary to make additions to SPEC?s PTDaemon (SPEC?s Power and
Temperature Daemon). This benchmark license may be purchased from SPEC
or obtained through SPEC membership. PTDaemon source code is found in
the ptd directory of the benchmark kit, while scripts and executables
needed for PTDaemon testing will be provided on request to licensees.
A unique device number must be obtained from SPEC prior to beginning
the addition of a new device. This number may be acquired by
non-SPEC-members by contacting the SPEC office.
Further information on the coding necessary to add a new device to the
PTDaemon may be found in the
PTDaemon
Design Document.
Basic PTDaemon functionality controlling a device can easily be tested
by connecting to the PTDaemon with telnet and using the commands found
in the PTDaemon Design Document. Commands such as ?Identify?, ?Go,0,0?,
?Stop?, ?Watts?, ?Volts?, ?Amps? and ?PF? should all be tested for
correct responses.
More extensive standalone testing can be performed using the Perl
script ptd-test.pl. The values returned by the modified PTDaemon should
be compared against the actual device readings to ensure that the
software is correctly reading values. Ideally the error rate during
testing should be zero to guarantee valid benchmark tests.
Finally, a standard SPECpower_ssj2008 benchmark run should be performed
using the modified PTDaemon binary. If successful, you are ready for
the testing required for SPEC acceptance.
The following sections describe the two sets of tests required for
SPEC review and possible acceptance. These tests are designed to verify
that a power analyzer is in fact behaving as its specifications
indicate and is compliant with the run rules requirements. All results
must meet the acceptance criteria specified below before an analyzer
can be accepted by SPEC.
This pair of tests compares the results from an accepted power analyzer
with the new analyzer under benchmark conditions. Both analyzers are
run simultaneously, connected in series and measuring the same SUT.
Connect the new power analyzer in series with another power
analyzer already accepted by SPEC. Ideally both analyzers should be
connected to the same controller system, so the time stamps in the
PTDaemon output files will be in sync. If not, the time must be
synchronized within a few milliseconds through a service like ntp or
the "net time" command.
The power analyzer voltage and current ranges should be manually
set to values appropriate for the supply voltage and maximum system
current draw. These values need to be documented in the spreadsheet
used for the acceptance process. Use of appropriate settings will
provide the most accurate power readings and avoid errors caused by
auto-ranging during tests.
Modify ccs.props as needed to run both power
analyzers. The second instance of PTDaemon will need a unique port
number such as 8890 to not conflict with the default PTDaemon port of
8888 and the PTDaemon temperature device port of 8889. Make sure to
correctly identify each analyzer in the ccs.props file.
Run both instances of the PTDaemon with the -l logfile
parameter so that all measurements will be logged for later use (e.g.
for the first analyzer, ptd-windows32 ?l device1.log 6 com1
and for the second analyzer, ptd-windows32 ?l device2.log ?p
8890 7 com2 )
Execute a single SPECpower_ssj2008 run with the default benchmark
parameters and save all results files, raw and detail files, and the
PTDaemon log files.
Reverse the order of the power analyzers in the chain between the
AC power source and the SUT, and run another SPECpower_ssj2008 test.
The intent of running the second test with the power analyzers reversed
is to minimize or eliminate any slight measurement differences caused
by loading or voltage drops of the daisy-chained analyzers.
Postprocess the ccs log files ssj.xxxx.ccs-log.csv
from both runs with the Perl script ssj-parse.pl
Usage: perl ssj-parse.pl input file output file to extract the
measured values from the benchmark run. Import that data into a
spreadsheet program as comma-separated records. Paste the data into the
two tabs of the provided spreadsheet ssj2008-comparison.xls
to perform the overall comparison between devices. The spreadsheet will
sum the results from the two tests and divide by two to obtain the
averaged power at each benchmark load level, then compare those results
from each device to determine the difference.
Acceptance criteria for this set of tests: the averaged power values
from the two devices found in the calculation tab of the spreadsheet
must not differ by more than the sum of the tolerances of each
individual device. However, a preferable result would be differences
much less than the sum of tolerances. It should be expected that the
analyzer connected closer to the AC power should have slightly higher
power readings in each test.
Tolerance formulas for some power analyzers are very complex, so some
assistance from SPEC members may be required to complete the
calculations for the acceptance criteria.
Install the PAT binary on the SUT. This binary is only currently
available for Windows operating systems. It is recommended that a dual
socket server-class system with a server operating system, with no
applications active, be used as the SUT for this test. Power management
features must be set to provide the maximum difference between idle and
full load power. Typically, more consistent results are measured with
OS power management disabled.
Set up two analyzers for measurements of a SUT as in section 3.1.
Both power analyzers should be connected to the same controller system,
using ports 8888 and 8890. Enable output logging using the -l
filename parameter. Use unique names such as analyzer1-pat1-results.csv
and analyzer2-pat1-results.csv for each test.
On the controller, use the SPEC perl script start2ptd.pl
to start both analyzers simultaneously. Both the start2ptd.pl and
stop2ptd.pl scripts may need to be modified if the two PTDaemon
instances are using ports other than 8888 and 8890.
Start the PAT test on the SUT using the first set of PAT
parameters : pat ?l pat1.csv (number of threads) 10 10 5000
After the load test completes, stop the analyzers with the stop2ptd.pl
script, and collect the analyzer and load log files.
The three log files should be imported into a spreadsheet program
as comma-separated data, then pasted into the "PAT Predictor"
spreadsheet (pat_test1_predictor.xls for the first set of
parameters, pat_test2_predictor.xls for the second set) to
examine the captured data. Follow the instructions in the
?instructions? tab of the spreadsheets. Adjust the Y axes and time
scales of the graphs to view the entire test duration. The predictor
spreadsheet will plot the measured values from the tests along with the
predicted results and the acceptance criteria ranges.
Repeat the last 5 steps for the second set of PAT parameters
pat ?l pat2.csv (number of threads) 2 100 5010
Acceptance Criteria:
Pulse width test: the measured power values should correspond to
the pulse widths generated by the AC load, within a tolerance of +/-
25%. All pulses must be captured by the device.
Sliding window test: A maximum of 2 duplicate samples and 2 missing
samples of the 200 short power pulses generated by the AC load are
allowable in the analyzer output. It is possible that a pulse may be
missed or duplicated as the time window shifts versus the analyzer's
internal clock.
In order for SPEC to review a device?s compliance to the Run Rules
requirements, the following must be submitted to SPEC:
Documentation to prove compliance with Run Rules section
2.13.2-Power Analyzer Specifications
Source code changes and new files necessary to add the new
device. All source code submitted to SPEC must include a signed SPEC
Permission to Use Form and must be freely available for use by
other members and licensees of the benchmark.
Spreadsheet along with the PTDaemon log files and
SPECpower_ssj2008 results files from the tests in section 3.1
Spreadsheets along with PTDaemon and PAT log files from the tests
in section 3.2
SPEC will review the information for compliance with run rules and
acceptance criteria above. Further information and/or testing may be
requested of the submitter. If all criteria are met, SPEC will accept
the device for compliant benchmark testing and add the support to a
later release of the benchmark kit.
Questions regarding this process should be sent to the standard
SPECpower_ssj2008 support alias. See your benchmark documentation for
further information.
