Standard Performance Evaluation Corporation
SPEC CPU Benchmark Search Program
The SPEC CPU Benchmark Search Program
SPEC holds to the principle that better benchmarks can be developed from actual applications. With this in mind, SPEC is once again seeking to encourage those outside of SPEC to assist us in locating applications that could be used in the next CPU-intensive benchmark suite, currently planned to be SPEC CPU2004.
The CPU2004 search program is now closed. No further entries will be accepted.
Rules, Categories, & Criteria
What is the SPEC CPU Search Program?
The CPU Search Program is sponsored by the Open System Group (OSG) of the Standard Performance Evaluation Corporation (SPEC, a non-profit consortium). It was designed to encourage and recognize achievements of the academic and industrial community who provide and develop application code and datasets that could be used as components of the next revision of the SPEC CPU benchmark suites.
For each step of the process that a submission passes, SPEC will compensate the Program Submitter (in recognition of the submitter's effort and skill). A submission that passes all of the steps and is included in the next SPEC CPU benchmark suite will receive $5000 overall and a license for the new benchmark suite when released.
What is SPEC looking for?
Overall, the SPEC CPU benchmarks have provided performance metrics for comparing systems on a known compute-intensive workload, with emphasis on the system's processor, memory hierarchy and compiler. SPEC is expecting the next CPU benchmark suite to continue this trend.
SPEC has held to the principle that better benchmarks are based on applications, and these applications can come from any area of work. For example, the current SPEC CPU2000 suite includes applications from the following areas:
SPEC is not limiting consideration to just these areas. A proposed application may come from any area or field of work. The submission must contain a description of the utility of the program.
Criteria SPEC considers important for the next CPU benchmark suite:
The SPEC CPU Search Program Official Rules
Who May Submit Benchmarks?
VOID WHERE PROHIBITED BY LAW. Not open to any employees or contractors of any of the SPEC member companies and their immediate families.
Entries may be submitted on an individual entrant basis (defined as a natural person), or on a group or team basis, in which case the registration and submission will identify a leader of such group or team.
Corporate entities are not eligible.
The award program is sponsored by Standard Performance Evaluation Corporation, 6585 Merchant Place; Suite 100, Warrenton, Virginia, 20187, U.S.A.
Participation Constitutes Agreement to Official Rules
Participation in the Search Program constitutes Submitter's full and unconditional agreement to and acceptance of these Official Rules.
Submissions for the first step in the search program will be accepted by SPEC beginning December 26, 2001 and ending March 31, 2004 at midnight, United States Pacific Standard Time.
Status updates through the rest of the process will be provided to the submitters at least every three months until the release of the next SPEC CPU suite.
How To Enter
Step 1: Complete the legal paperwork
Submit a CPU Search Program Proposal to SPEC.
All submitters must provide SPEC with the following information:
This should be done through the entry form found at: http://www.spec.org/osg/cpu/CPU2004/entry_form.html
Upon receipt of the proposal, the SPEC CPU Subcommittee will acknowledge its receipt within seven (7) working days. Within four (4) weeks, the SPEC CPU subcommittee will notify the submitter as to whether or not their entry has been accepted for Step 2 of the program. If accepted for Step 2, SPEC will provide the Submitter with a copy of the SPEC benchmark framework and information on how to prepare the application.
Step 2: Code Review
If the proposal is accepted for Step 2 by the SPEC CPU subcommittee, the submitter must now prepare their code for submission to SPEC and submit it to SPEC. The requirements are:
If, in SPEC's sole judgment, the submission does not meet these criteria, the Submitter will be notified as to what expectations were not met. If the submitted code meets these criteria, the Submitter is awarded $2000 and is accepted for Step 3. Note that SPEC, in its sole judgment, may decide at any time to discontinue work on a program.
Step 3: Complete Code Testing
In Step 3, two things occur. The first is that the SPEC CPU Subcommittee decides in its sole discretion to continue the program as an official candidate for the next SPEC CPU benchmark suite. If the subcommittee continues the program as an official candidate, the Submitter must then work with SPEC to complete the following steps:
If the program is continued as an official candidate and the additional criteria are met, the submitter receives $500 and the program moves to Step 4, where it is evaluated and tested by the SPEC CPU Subcommittee members. Again, SPEC may decide at any time to discontinue work on a program.
Step 4: Recommendation to the Open System Group
If the program is accepted by the SPEC CPU Subcommittee and meets all criteria, including that stated in Step 3 and is recommended by the SPEC CPU Subcommittee and the SPEC Open Systems Group, in their sole discretion, to the OSG membership, the submitter receives $500 and the program moves to Step 5. Otherwise, the SPEC CPU Subcommittee will notify the Submitter that the program was removed from consideration.
Step 5: Acceptance into the CPU Suite
If the program is accepted by the OSG membership in its sole discretion, then the program is included in the suite and the submitter will receive $2000 and a license for the suite when it is released.
Number of Entries and Return of Entries
Submitters may enter as many times as they wish. However, each entry must be accompanied by a separate registration. Sponsor will not return any entries to the submitters.
Sponsor will make its best effort to use personal data about submitter only for the administration of the award program, in accordance with these Official Rules and for internal marketing purposes. Submitter agrees to such use of the personal data.
Sponsor is not responsible for lost, late, incomplete, illegible or misdirected mail or e-mail, for failed, partial or garbled computer transmissions, or for technical failures of any kind. Sponsor reserves the right to cancel or modify the awards, if fraud, technical failures or other events or circumstances impair the integrity of the award program as determined by Sponsor in its sole discretion.
Submitter agrees that Sponsor's only obligations are to submit entries from eligible submissions to the SPEC CPU Subcommittee, according to the procedures and criteria set forth in these Official Rules, and to award the compensation set forth herein, to submitters as criteria in SPEC's sole judgment are met, subject to the terms and conditions herein.
Submitters will be held harmless from any liability incurred from the intended or unintended use of the application as a benchmark or application.
Submitter agrees that Sponsor's obligations are fair and adequate consideration for any entry submitted, and that Submitter is not entitled to and shall not seek any further compensation.
Submitter hereby grants Sponsor a perpetual, non-exclusive license to copy, modify, display, and sub-license the entry without geographical limitations or further compensation to Submitter of any kind, provided that Sponser's sole use of the entry will be for the purposes of inclusion of the entry in Sponser's CPU benchmark suite. Submitter agrees that e-mail shall satisfy any writing requirement, which may apply to intellectual property licenses. Upon request of Sponsor, Submitter agrees to obtain written consent from the owner of the copyright in the program, if that person is not Submitter, and to execute any documents required to effectuate the terms of these Official Rules.
Submitter warrants that it has the right to submit this program and information to SPEC and to grant the license rights granted herein, and that to Submitter's knowledge use of the program will not infringe or violate any intellectual property rights (including but not limited to patent, trademark, trade secret, and copyright) or license rights of any third party. Submitter agrees to notify of any claim of such infringement or violation within fifteen (15) days of its receipt of the claim. Submitter shall defend, indemnify and hold harmless SPEC, its successors, assigns and licensees for all costs (including attorneys' fees) arising out of any breach of these warranties and representations.
All issues and questions concerning the construction, validity, interpretation and enforceability of these Official Rules, or the rights and obligations of Submitter and Sponsor in connection with the Award Program, shall be governed by, and construed in accordance with, the laws of the State of California, U.S.A., without giving effect to any choice of law or conflict of law rules or provisions (whether of the State of California, U.S.A., or any other jurisdiction), which would cause the application of the laws of any jurisdiction other than the State of California, U.S.A.
Taxes, customs duties, fees, freight charges and other related charges, on prizes are the sole responsibility of the winners. Winners may waive the right to receive compensation. Sponsor will reimburse winners up to U.S. $1,000 for payments made for customs duties, fees, freight charges (but not taxes, including taxes on income, goods and services, and value added taxes), upon presentation by submittal of original and official receipts for such payments.
Submitters will be notified by registered mail of their entry's successful completion of each step and may be required to complete an affidavit of eligibility and publicity release (except where prohibited by law), which must be returned within 20 days of receipt by Submitter. Failure to sign and return the affidavit or release or to comply with any term or condition in these Official Rules may result in disqualification and forfeiture of compensation.
This Program is subject to all applicable laws. Void where prohibited.
All questions about the search program can be directed to SPEC at firstname.lastname@example.org.