Skip navigation

Standard Performance Evaluation Corporation

 
 

SPEC Frequently Asked Questions

Contents:


SPEC General

What Is SPEC?

SPEC, the Standard Performance Evaluation Corporation, is a non-profit corporation formed to "establish, maintain and endorse a standardized set of relevant benchmarks that can be applied to the newest generation of high-performance computers" (quoted from SPEC's bylaws). The founders of this organization believe that the user community will benefit greatly from an objective series of applications-oriented tests, which can serve as common reference points and be considered during the evaluation process. While no one benchmark can fully characterize overall system performance, the results of a variety of realistic benchmarks can give valuable insight into expected real performance.

Legally, SPEC is a non-profit corporation registered in California.

SPEC basically performs two functions:

  • SPEC develops suites of benchmarks intended to measure computer performance. These suites are packaged with source code and tools and are extensively tested for portability before release. They are available to the public for a fee covering development and administrative costs. By license agreement, SPEC members and customers agree to run and report results as specified in each benchmark suite's documentation.
  • SPEC publishes news and benchmark results at http://www.spec.org/. This provides a centralized source of information for SPEC benchmark results. Both SPEC member and non-SPEC member companies may publish on the SPEC website, though there is a fee for non-members. (Note that results may be published elsewhere, as long as the format specified in the SPEC Run Rules and Reporting Rules is followed.)

For more in depth inquiry, we have more information online about SPEC, including its background, the philosophy, and organizational structure.

How to Contact SPEC?

For SPEC contact information, see the SPEC Contact Info Page.

What Groups Make Up SPEC?

SPEC is comprised of four groups:

  • Graphics and Workstation Performance Group (GWPG): Develops consistent, repeatable graphics and workstation performance benchmarks in a way that reflects user experiences with popular applications.
  • High Performance Group (HPG): Develops benchmarks to represent high-performance computing applications for standardized, cross-platform performance evaluation.
  • Open Systems Group (OSG): Develops component- and systems-level benchmarks for workstations and servers running open operating system environments.
  • Research Group (RG): Promotes research on benchmarking methodologies and tools used to deliver benchmark suites and performance analysis frameworks for established and newly emerging technologies.

Why Should I Join SPEC?

Membership in SPEC is open to any interested company or entity that is willing to commit to SPEC's standards and is able to provide for the support of the organization.

Membership is group-based. One can be a member of one or more groups depending upon one's interests. Each group (GWPG, HPG, OSG, RG) is run independently; each member is responsible for paying the full dues and other responsibilities for each group they join -- there are no "bulk discounts".

SPEC does make an effort to accommodate the participation of academic institutions and other non-profit organizations with reduced (or waived) dues and other arrangements. SPEC has "associates", which pay little or no dues, but still get full access to the benchmark development process.

The dues go towards offsetting SPEC's operating costs. Typically these dues are a few thousand dollars per year with a similar initiation fee to cover starting costs; however, the exact values vary from group to group (and potentially year to year) depending upon the specific costs.

Membership allows for full access to existing benchmarks (this included company-wide licenses for current benchmark products) and full participation in ongoing development of new benchmarks. Additionally, members get unlimited result publication through SPEC (as membership dues already cover much of the cost). The benefits for associates are the same as for members except for having voting rights. The intention for associates is that they can act in an advisory capacity to SPEC, getting first-hand experience in an area that is widely neglected in academia but nonetheless very important in the "real world", and providing technical input to SPEC's tasks.

All members and associates of each group are welcome at all meetings for that group. Meetings are usually held four to six times a year, with subcommittee teleconferences held on an as-needed basis (often weekly or bi-weekly during benchmark development). Final decisions are usually made by a "steering committee" elected by the membership of the group, with benchmark standards needing ratification by the groups full membership.

For further information about membership in any of SPEC's groups, contact SPEC at info@spec.org.

SPEC Products

What Are the SPEC Products and Services?

SPEC's main product is the benchmark suites. The code in the suites are developed by SPEC from code donated by various sources. SPEC works on portability and creates tools and meaningful workloads for the codes chosen as benchmarks. Therefore, the SPEC benchmarks are not the same as public domain programs that might exist under a similar name or the same name, and their execution times will, in general, be different.

The SPEC benchmark sources are generally available; though, they are licensed software products. Each SPEC group is charging separately for its benchmark suites. The income from the benchmark source sales is intended to support the administrative costs of the corporation. Buyers of the benchmarks receive a shrink-wrapped site license with their first order of any of SPEC's products. SPEC uses CD-ROMs as the distribution media for its benchmarks.

SPEC Benchmarks and Year 2000 Compliance

SPEC, the Standard Performance Evaluation Corporation, produces performance evaluation software [benchmarks] for modern computer systems. The various types software used in the SPEC benchmarks do not require nor use dates in any computations of a business critical or performance sensitive manner.

SPEC does not anticipate any issues with its products due to Year 2000 date effects. Should specific problems be found, SPEC will address them on a case by case basis.

The software products that SPEC licenses are not warranted to be Y2K compliant. SPEC makes no warrantees, express or implied, related to the sale, use or performance of its products, which are supplied on an "as is" basis. SPEC has no plans to implement any Y2K compliance programs for any of its products.

What does "retired" mean in relation to SPEC benchmarks?

SPEC will often retire a benchmark when there is a more recent replacement or sometimes when the original benchmark has become too out-of-date to continue to support. "Retired" means that SPEC no longer provides technical support or accepts result submissions for posting on the SPEC web site. Recognizing that there may be historical interest in these older benchmarks, in many cases it may still be possible to obtain copies of such a benchmark by contacting the SPEC office, but the buyer should be aware that additional support will not be provided. Notice of planned retirements will be posted on the SPEC web site, and such news is usually reported months in advance of the retirement date.

Are there trial or evaluation versions of the SPEC benchmarks available?

We do not offer trial versions of the SPEC benchmarks. If you purchase software and are dissatisfied for any reason, you may return the software and license for a full refund of the software purchase price within thirty days. Contact the SPEC office for an RMA number.

What kind of license is included with the software?

SPEC benchmark suites are provided with a site license which allows the licensee to install and run the software on as many systems within their organization as they wish. You do not need to purchase multiple licenses. A sample license agreement (PDF) is available for review prior to purchase.

Who qualifies for the non-profit and educational pricing?

Educational discounts are available only when the software is purchased by and licensed to an educational institution; the end user must be a faculty member or a full-time employee of the institution, not a student.

Non-profit pricing is available to recognized tax-exempt organizations and government agencies. Individuals do not qualify for the discounted pricing.

How Do I Order from SPEC?

SPEC products can be obtained by using our online order form. For other methods or additional questions about ordering, contact the SPEC staff.

SPEC Misc

Which System/CPU/Component Would Be Better for Me?

Unfortunately, SPEC does not make recommendations on computer purchases. SPEC will provide what data it can (all available on the Web site http://www.spec.org/) but can not offer opinions on it.

Without knowing a lot about your exact workload, it would be difficult to provide any useful specific advice; and providing general advice can get us all in legal trouble.

What About SPEC 1170?

SPEC 1170 does not have anything to do with SPEC, the Standard Performance Evaluation Corporation. It's a different spec.

SPEC 1170 is an older name for what is now called UNIX 98, which is an X/Open brand for compliance with a "Single UNIX Specification" -- basically a standardized UNIX. More information about UNIX 98 can be obtained from: http://www.unix-systems.org.

SPEC More

Where Can I Go If I Have More Questions?

Most questions can be answered by browsing the appropriate areas:

  • OSG - SPEC Open Systems Group
  • HPG - SPEC High Performance Group
  • GPC - SPEC Graphics Performance Characterization