|
|
SPEC High Performance GroupPolicies & Procedures
1.0 General Policies and PrinciplesThe intent of this section is to provide an overview of the general policies and principles of the SPEC organization, as context for the policies of the High Performance Group (HPG). The controlling documents for governance of SPEC are the Articles of Incorporation and Bylaws, filed with the State of California, and available from the SPEC Administrator.1.1 SPEC's Purpose The primary purpose of the Standard Performance Evaluation Corporation (SPEC) is to develop suites of benchmark programs that are effective and fair in comparing the performance of high performance computing systems, and to assure that they are readily available to manufacturers and users of such systems. 1.2 SPEC's Formation SPEC was formed from the instigation and sponsorship of Electronic Engineering Times (E.E. Times), and by the cooperative development work of Hewlett-Packard Co., Sun Microsystems Inc., Apollo Computer Inc., and MIPS Computer Systems Inc. 1.3 The Corporation Upon approval by the representatives of the founding companies, their effort to develop a benchmark standardizing activity was converted into a non-profit Corporation of the state of California, on November 14, 1988. The Corporation has a Board of Directors, President, and a staff to carry out the business of SPEC. The board has established three technical committees:
Each group has a steering committee to manage and supervise the development of applicable benchmarks. Each steering committee may sponsor a number of subcommittees or working groups to facilitate benchmark development. The Board may establish additional technical committees. 1.4 Meetings The annual meeting of SPEC members will be held in January. Election of the Board of Directors and members of the steering committee(s) and a financial report by the Board will be a necessary part of that meeting. The Board of Directors will meet at least quarterly. 1.5 The Board To facilitate a fair and balanced representation on the Board, SPEC's policy is that the Board of Directors has staggered two-year terms, with half of the Board elected each year. Decisions of the board require agreement of the majority. Elections to the Board will be held at the annual meeting to be held at the beginning of the calendar year. Board positions are held by individuals, not by companies. Currently there are seven elected directors on the board. Each member of each SPEC technical steering committee (OSG, GPC, and HPG) may cast one vote per open board seat. E.g., if a company is a member of two steering committees, it may cast two votes per open board seat. 1.5.1 Officers The board elects a president, who generally serves for a term of one year, with eligibility for re-election, from among candidates proposed by the board members. The board also elects a secretary and a treasurer of the Corporation, who are not required to be members of the board. The secretary and treasurer serve terms of one year and are eligible for re-election. 1.5.2 Administrative Duties SPEC has an administrative organization that carries out the publicity, accounting, organizational and other necessary work of the Corporation. The administrative staff reports to the Board, through the president. The budget of the staff is developed by the president and the treasurer and approved by the board. 1.6 Membership Any company or organization that wishes to join SPEC may join provided they pay the initiation fee and annual dues as determined by the Board of Directors. All members are eligible to stand for election to the steering committee seats, to nominate candidates for offices and the Board of Directors, and to champion proposed benchmarks. All members have the opportunity to review and comment on all benchmarks developed by the HPG. This can be done through active participation in the subcommittee developing the suite or during the benchmark's final call for comments. The fees and dues for membership must not discriminate among prospective members. 1.7 Copyrights and Trademarks The standards and documents authorized by the Corporation may be copyrighted or trademarked, as appropriate. The purpose of the copyrights and trademarks is to assure the computer industry that programs and documents developed by SPEC are genuine. The intent of protecting by trademarks and copyrights is not to restrict the use or availability or make any significant revenue from them, but rather to assure their general availability, promote confidence in them, and prevent fraud. 1.8 Openness Information about the activities and developments of SPEC are intended to be readily available to all parties with legitimate interests in the performance of advanced computer systems. The board is responsible to carry out this policy. SPEC meetings must be open to representatives of each member company. The Board may invite interested parties who are not SPEC members, but who have a legitimate interest in its work, to the meetings. Meetings of SPEC's technical steering committees and their subcommittees are generally closed to the press. A meeting may be opened to the press by agreement of the steering committee or by agreement of the Board and the meeting's chair. The annual meeting of the SPEC membership must
be open to any interested party with a legitimate interest in the work
of the Corporation, including the press. 2.0 High Performance Group The High Performance Group focuses on the development of benchmarks for high performance computers, including but not limited to, symmetric multiprocessor systems, clusters, distributed memory parallel systems, and vector parallel supercomputers. 2.1 High Performance Group's FormationSPEC's High Performance Group was formed in 1994, with the cooperation of Convex Computer Corp., Cray Research Inc., Dartmouth College, Digital Equipment Corp., Fujitsu America, Hewlett-Packard, IBM, Intel Scalable Systems Division, Kuck & Associates, NEC/HNSX Supercomputers, Silicon Graphics Inc., the University of Illinois, the University of Michigan and the University of Minnesota.2.2 High Performance Group and SPECThe High Performance Group (HPG) is chartered to manage and supervise the technical development of the SPEC/HPG benchmarks suites. In case of any conflict between provisions within this Charter and Resolutions enacted by the HPG, the Resolutions shall prevail. In case of any conflict between HPG's Resolutions and SPEC's Articles of Incorporation and Bylaws, the Articles of Incorporation and Bylaws shall prevail.2.3 High Performance Steering CommitteeThe voting membership of the High Performance Group constitutes the High Performance Steering Committee (HPSC). HPSC is formally charged with managing and supervising the development of SPEC/HPG benchmarks.2.4 Membership Any company or organization
interested in the technical development of HPG benchmarks is welcome to
participate. The requirements are
the Fair Use Guidelines, the Guidelines for Handling
SPEC-confidential Material, and
All
members are eligible to nominate candidates for offices and the Board
of Directors, and to champion proposed benchmarks. All members have the
opportunity to review and comment on all benchmarks developed by the
HPG. This can be done through active participation in the subcommittee
developing the suite or during the benchmark's final call for comments.
HPG provides
for three types of participation: Sustaining Membership, Not-for-profit
Associates and Supporting Roles. 2.4.1 Sustaining Memberships Sustaining members are dues-paying members
and they accrue all the rights, privileges and responsibilities that
full membership entails. Sustaining memberships are available to
commercial organizations and individuals. Sustaining members may
become active participants in any working group or subcommittee of the
HPG. All sustaining members are eligible to
nominate candidates for offices and the Board of Directors, and to
champion proposed benchmarks. All members have the opportunity to
review and comment on all benchmarks developed by the HPG either
through active participation in the subcommittee developing the suite
or during the benchmark's final call for comments. The HPG’s fees and dues for membership can
not discriminate among prospective members. 2.4.2 Non-Membership Roles SPEC does make an effort
to accommodate the participation of academic institutions and other
non-profit organizations that might be unable to meet the full
requirements for Sustaining Membership. These
accommodations are made by special arrangement of the HPG and can
include reduced (or waived) dues and other arrangements. These
accommodations generally fall into two categories:
To encourage
participation from universities and other non-profit institutions
interested in its work, SPEC maintains a SPEC Associate role that
enables non-profit organizations to share in the SPEC process, but
(since they are not full members) does not allow for voting on
benchmarks. Associates must be willing to abide by SPEC's Bylaws, the Fair
Use Guidelines, the Guidelines
for Handling SPEC-confidential Material, HPG’s Policies
and Procedures and pay (potentially
reduced) initiation fees and annual dues as determined by the Board of
Directors. 2.4.2.2 Supporting
Role Supporting Role is available to any
commercial organization, academic institution, or individual by
invitation only. A subcommittee may issue an
invitation to an interested party to apply for this form of membership.
The application must be approved by the High Performance Group. Supporting persons may participate in only
one subcommittee and have no voting rights and pay no dues.
Supporting persons may help develop new benchmark suites,
provide expertise to the HPG or help provide other support within the
HPG.
2.5 HPG Officers, Elections, Voting, & Meetings The HPG officers include a Chairperson, Vice Chairperson, Secretary and Release Manager(s) elected from the membership. Elections for officers will be held at the beginning of each calendar year to be set by the Chair. HPG officers serve for one year and are eligible for re-election. If any elected office becomes vacant for any reason, the HPG shall duly elect a replacement to serve the remaining term. Each HPG Sustaining Member shall have one vote in all matters. Groups of Five Associates may combine to cast one vote; the single vote cast by each group of Associates is equal to a single Sustaining Member vote. When there are fewer than five Associates available to form a combinatorial group, those Associates present will be deemed a sufficient combinatorial group to cast one vote. Supporting Members are not entitled to voting rights. To conduct business there must be a quorum. The group's quorum is a simple majority of the voting membership. All HPG votes shall be open and public votes, except the election of HPG officers, which is by secret ballot. Exceptions to the secret ballot may be made if there is only one candidate per office. A voting member may vote 'yes', 'no', 'abstain', or 'pass' when the vote is first called. If the member votes 'pass', the chair will return to that member to finalize their vote as either 'yes', no', or 'abstain' after the remaining members have voted. The motion will pass if the number of 'yes' votes exceeds the number of 'no' votes. Votes of 'abstain' count only toward establishing quorum. It is the responsibility of the chair to make sure that member's reasons for voting 'no' or 'abstain' are recorded in the minutes. Absent Members will be marked as absent, unless proxies have been received. A voting member not attending two consecutive duly noticed meetings, including face to face meetings and conference calls, will have their voting rights suspended. The suspension of voting rights will take place at the beginning of the next scheduled meeting. Voting rights are reinstated at the beginning of the second of two consecutive meetings that the member attends. HPG Face-to-Face meetings shall be held no less than twice a year. One meeting shall be held in conjunction with SPEC's annual meeting. One of the other meeting will accommodate the HPG membership. Teleconference are usually held every two weeks, for no more than one hour. The schedule for these meetings are posted on the private page. 2.6 HPG Guidelines for Result Submission and Review Each benchmark suite produced by the HPG includes a set of specific run and reporting rules that the licensee must follow to produce a publishable result. These basic tenets of these rules require:
2.6.1 Results Review The HPG's results review process is detailed on the HPG private page (reviewprocess.html).The standard review cycle is two weeks. In cases where the submitter has made source code changes the review cycle is a minimum of four weeks. At the beginning of the review cycle a primary reviewer is assigned by the Chair to review the results. The primary reviewer is encouraged to offer his preliminary comments on the results within one week. At the end of the review cycle the results are accepted or rejected by a majority vote of the committee. Once a result is accepted the full disclosure report is posted on the SPEC web site, and once posted on the SPEC web site (http://www.spec.org/), the information contained in the result disclosure is considered public information. If the HPG votes that the submission is not compliant and should not be published on SPEC's web site, the submitter may acquiesce to the majority position and withdraw the submission. Alternately, the submitter may make an appeal to the SPEC Board of Directors. The submitter may withdraw, correct, or replace the questioned submission at any time during this process. If minor corrections are made to a submission, e.g. documentation of tuning options or product names, which do not affect performance, then the corrected result remains on the same publication schedule as the original submission. If major corrections are made, e.g. a new run, different performance results, or different system configuration, then the corrected result starts review over as a new submission. To assure fairness to the submitter, the HPG has established the following guidelines regarding the proper use of review materials:
Members participating in the review of results submissions are expected to be familiar with the Guidelines for Handling SPEC Information, particularly section A.7.0 dealing with Results Review (see Appendix A.) 2.6.2 Handling HPG Run Rule Violations in Published Results The HPG may also [vote to] undertake a re-review of a previously published result, if significant questions as to its adherence to the applicable run and reporting rules arise. The HPG reserves the ability to denote a publish result as non compliant if it has been found in violation of the run and reporting rules. The following process has been established to handle instances where run and reporting rule violations are discovered in results that have been published on SPEC's web site. Once HPG identifies a result, published on the public web site is in violation of the benchmark's run and reporting rules, the result may re-enter a review process unless the submitter agrees with the finding of non-compliance. If the subcommittee's review returns a finding that the result is not in compliance with the benchmark's run and reporting rules, the following should occur. The results are modified:
2.6.3 Handling Continued Availability in Published Results The various SPEC benchmarks' Run and Reporting Rules have clauses that at a certain time after the first publication (on SPEC's web site or otherwise), all components of the System Under Test (SUT) must be available for general customer shipment. The purpose of this requirement is:
For the case that this availability status cannot be maintained over a minimum time length, the following rules apply: Either at the request of the test sponsor or as a result of a resolution, the HPG can undertake a re-review of a previously published result if a performance relevant component ceases to be available for 30 days or more within 90 days after initial overall availability (i.e. initial availability of all components of the SUT). If the committee's review returns a finding that this non availability condition holds for a performance relevant component, the result is marked "Not Available" (NA) in the following way:
"Performance relevant" is defined in the various benchmarks' Run and Reporting Rules' reproducibility clauses. Replacement by a new similar component with equivalent or better performance is considered acceptable and not a reason for an NA marking. In cases where the Run Rules allow use of a beta product, this beta product or a later equivalent or better performing regular product (see clause above) must remain available. Note: Effective for results published on or after July 1, 2002. 2.6.4 Results for Electronically Equivalent Systems HPG encourages result submitters to run the benchmark on all systems. Submitters are responsible for the accuracy of their results. There may be cases where a submitter may want to submit the same results file for multiple submissions. This is acceptable if that performance is representative of those systems (e.g., just the power supply or chassis is different between the systems). A definition of what is acceptable as "electronically equivalent" systems is provided in the next section. As part of the SPEC HPG review process, the submitter should expect to be asked to justify why the systems should have the same performance. It would be within the purview of the HPG committee to ask for a rerun on the exact system if the technical criteria are not satisfied. In cases where HPG accepts the submitter's claim of electronic equivalence the submitter would have to add a line in the notes section of each of the result stating that the systems are electronically equivalent. For example, if a submitter submits the same result file for Model A and Model, the following note needs to be added to both the results: Example: "Vendor's Model A and Vendor's Model B are electronically equivalent." 2.6.5 Definition of Electronic Equivalence There are no noticeable differences in the behavior of the systems down to the level of electronic signals. Examples of when a system is considered electronically equivalent include:
Examples of when a system is not considered electronically equivalent include:
2.7 HPG Technical Support Guidelines The members of HPG are responsible for providing adequate resources to handle technical support questions regarding current benchmark suites. It is the goal of technical support to provide timely response to questions regarding the nature and usage of these benchmark suites to help facilitate testing by other SPEC benchmark licensees. The purpose of SPEC's technical support is NOT to provide OS, compiler or other vendor tool support or assistance. At the technical support person's discretion, the user may be referred to a third party, who may charge a fee for such support. For each released benchmark, the SPEC administrator will maintain an email alias (i.e. ompm2001support@spec.org). HPG may establish a quarterly rotation schedule among its active members (based on license number) to serve as technical support person. In special cases a member may be assigned to provide support for specific subset of technical support questions for an extended period or to handle support for a specific geography. It is the chair's responsibility is to see that questions are answered promptly and provide backup when the vendor on rotation becomes unavailable. The current support representative or chairs may assign a technical support call to another member, if they believe vendor specific knowledge may be required to resolve the call. The SPEC administrator remains the initial point of contact for technical support questions. The administrator will make the determination as to which email alias the problem is reported to. 2.8 HPG General Member Guidelines The HPG is open to any organizations that have an interest in the use and development of standardized performance benchmarks for high performance computer systems. By joining the HPG, members may:
2.9 Fair Use of SPEC/HPG Benchmark Results Consistency and fairness are guiding principles for SPEC. To assure these principles are sustained, the following guidelines have been created with the intent that they serve as specific guidance for any organization (or individual) who chooses to make public comparisons using SPEC benchmark results. When any organization, including vendor or research oriented organizations, or any individual makes public claims using SPEC benchmark results, SPEC requires that the following guidelines be observed:
The following paragraph is an example of acceptable language when publicly using SPEC benchmarks for competitive comparisons: Example: SPEC® and the benchmark name SPEComp® are registered trademarks of the Standard Performance Evaluation Corporation. Competitive benchmark results stated above reflect results published on www.spec.org as of Jan 12, 2002. The comparison presented above is based on the best performing 4-cpu servers currently shipping by Vendor 1 [SPECompMbase2001 1245, 4 cores, 2 chips, 8 threads], Vendor 2 [SPECompMbase2001 1243, 4 cores, 4 chips, 4 threads], and Vendor 3 [SPECompMbase2001 1000, 4 cores, 1 chip, 4 threads]. For the latest SPEC OMP benchmark results, visit http://www.spec.org/omp/results/. The following is the list of approved SPEC HPG metrics and submetrics that may be used for competitive comparisons within the guidelines of the Fair Use Rule:
2.10 HPG Other Resources The HPG can, after notification in writing to
the Board of Directors, enlist the aid of advisors and consultants from
the industry and academia. 2.11 Benchmark Release Approval The process to approve
the release of new benchmark suites consists of several steps.
The HPG must finalize the benchmark and prepare a beta version that can
be recommended for general membership review. General membership
refers to all the members of the HPG and not just the members who
actively worked on developing and testing the benchmark suite. When
this beta is ready, the committee must send out a final call for
comments to the At the conclusion of the final call for comments period, the voting membership of the HPG will take part in a final vote to approve the benchmark for release. Appendix A. Guidelines for Handling SPEC InformationA.1.0 Introduction SPEC is an industry consortium whose membership includes many competing for-profit corporations. Membership in SPEC is open to all persons who are directly and materially affected by SPEC's activities, and participation in SPEC activities is generally open to all members. However, in order to protect the interests of SPEC and its members, and to encourage free and open participation and discussion to further SPEC's objectives, public release of SPEC internal information may be prohibited without explicit approval from the SPEC organization concerned. Violators of this policy will be subject to the full force of the SPEC penalties process. A.2.0 Anti-Trust Compliance Since all SPEC meetings are to be conducted in compliance with all applicable laws, including anti-trust laws, the following policies shall be followed in the course of SPEC meetings, telephone meetings, and electronic exchanges: Any discussions that relate to the validity or cost of patent use should be avoided. Any discussions or any material relating to an ongoing litigation shall be avoided, except for the Board in executive session with advice of counsel to discuss matters affecting SPEC. Any discussions of pricing or issues that would violate US antitrust laws shall be avoided. A.3.0 How Information is Classified and Unclassified SPEC officers and the SPEC Administrator may explicitly designate certain information classified and mark it appropriately. Any member that wishes to be sure that its information is treated as SPEC Confidential should clearly label each page (electronic or hard copy) as such. It is the member's responsibility to assure that such information is appropriately labeled. Information not explicitly marked but which meets the descriptions in this policy must be taken as though it were so marked. Classified information may become public through the authorized public release of the information by SPEC or by a member. If classified information is disclosed without authorization (leaked), that does not automatically remove its classification and relieve participants of the obligation to protect that information from further disclosure. However, if an unauthorized disclosure is widespread, then a SPEC officer may redesignate that information as public, stating the reason for so doing. A.4.0 Classifications SPEC identifies two levels of internal information which must be protected by SPEC members and associates, employees, and contractors, in accordance with these policies. "SPEC Confidential" is the ordinary classification of information which should not be disclosed outside SPEC. "SPEC Confidential Need-to-Know" refers to the most sensitive information described below. A.4.1 SPEC Confidential Need-to-Know This is information that must be restricted to only those individuals who must have it in order to carry out their SPEC responsibilities. In meetings, this information is discussed only when the board adjourns to executive session. Only the author of such a document, or the SPEC President, can change the classification or the approved reader list. Examples of Confidential Need-to-Know information includes: information concerning litigation certain matters that relate to the formation of contracts with third parties employee salaries, performance appraisals, and other personnel matters administrative (root) passwords of SPEC business computers A.4.1.1 Document Handling Guidelines Documents of this sensitivity class will always be clearly labeled "SPEC Confidential Need-to-Know" on each page. The documents may be numbered for control purposes. SPEC Private documents must have a cover page which serves to: 1. hide the first page of the document, 2. document the approved reader list, and 3. document the date the information is no longer SPEC Confidential, if any, Such documents should always be carefully transported. Documents should be kept inside an envelope, clearly labeled with the approved reader's name, when not in active use. Only persons on the approved reader list are privy to the document contents. Email should be used sparingly for communicating such information. Messages should be sent directly to the intended recipient(s) rather than to any mail alias. Messages should be retained no longer than absolutely necessary, and should be kept separate from other messages. Encryption should be employed where possible. A.4.2 SPEC Confidential The following information is "SPEC Confidential":
Note that non-members of SPEC should not be on mailing aliases. Please note that SPEC offers various levels of membership to fill different needs for information access. A.5.0 Who May Have Access to SPEC Confidential Information Confidential information is provided to SPEC members through their designated primary SPEC representatives, and those representatives of SPEC subcommittees designated by the member's primary SPEC representative. It is the responsibility of those representatives to determine who within a member company should have access to the information in order to carry out the member's SPEC activities; and it is their responsibility to ensure that anyone who does have access to SPEC Confidential information understands and agrees to abide by these guidelines. A member's SPEC representatives should use whatever internal controls are appropriate in their company to ensure that the member complies with SPEC's policies on confidentiality. A.6.0 Meetings Information about the activities and developments of SPEC are intended to be readily available to all parties with legitimate interests in the performance of advanced computer systems. The Board is responsible to carry out this policy. The annual meeting of the SPEC membership must be open to any interested party with a legitimate interest in the work of the Corporation, including the press. Other meetings of each SPEC group and committee must be open to representatives of every member of that group. It is expected that meetings of any SPEC group will also be open to members of any other SPEC group, to encourage wider participation and exchange of ideas. However, a SPEC group may set a more restrictive policy for some or all meetings, subject to Board review. Interested parties who are not SPEC members, but who have a legitimate interest in its work, may be invited to its meetings by the SPEC group holding the meeting. A.6.1 Press Participation Press representatives who are not SPEC members are generally excluded from SPEC meetings other than the annual meeting, unless the chair of the SPEC group or subcommittee, invites the press to attend. If the press is in attendance, the chair should announce press attendance. The press is encouraged to use such attendance opportunities to gain background information in technical areas of performance benchmarking. SPEC is primarily an engineering organization which depends on frank and open exchange and debate of views. Views expressed in SPEC meetings are not expected necessarily to reflect official positions of SPEC member companies and organizations. SPEC asks that the press show restraint in any coverage based on SPEC meetings, so as not to jeopardize our open exchange of views. A.6.2 Audio/Visual Recording SPEC committees may apply restrictions on the use of audio recording, video recording, or photography equipment where they may impede free discussion, or where they are disruptive. Such restrictions should be clearly identified, in advance, to attendees. A.7.0 Results Review Benchmark results submitted for SPEC review, as for publication on the SPEC web site, are considered SPEC Confidential. This includes the benchmark result, the configuration, availability dates, and tuning notes. Within a member company this information should be shared with only those people necessary to provide a thorough review of the results, per paragraph 5.0. Information which the submitting company makes public is not considered confidential. E.g., a company may issue a press release or briefing listing performance of new systems and, around the same time, submit those benchmark results for SPEC publication. In this case details included in the full SPEC disclosure under review, which were not included in the company's public announcement are not considered SPEC Confidential, because the run rules of each benchmark require the full SPEC disclosure to be publicly available. If results are later published, then the result is public. If results are withdrawn or not accepted during SPEC review and are not published, then they remain SPEC Confidential. E.g., if a company submits results that are not accepted for SPEC publication, you may not make public use of that fact. New and unique optimization techniques that appear in a result disclosure should not be utilized by another member until the original submission has completed its review cycle. The subcommittee may decide to suspend a result in review should this guideline not be followed. Trying to surpass a result currently under review is considered common practice and is not prohibited. Published information found in press releases, public web sites, manuals, etc. can be used at any time. A.7.1 New Benchmark Announcements More restrictive confidentiality rules may be applied by the SPEC groups on the occasion of results review for SPEC's announcement of a new benchmark suite, in addition to all of the preceding rules. A.8.0 SPEC Press Relations Press releases from SPEC must be approved by the SPEC group concerned, and by the Board. SPEC members may occasionally be interviewed
by the press, e.g. on the subject of newly released benchmarks.
Normally SPEC's PR consultant will participate in such interviews in
order to see that SPEC is treated fairly, and to assure all SPEC
members that information is being represented in an unbiased manner.
The Board Press Committee may on a case by case basis, waive this
requirement, based on such factors as: degree of controversy of the
subject, potential for appearance of conflict of interest by the
spokesman, legal advice, advice from SPEC's PR consultant,
recommendation of steering committee and technical committees and their
chairs, experience of spokesman, experiences with the reporter, etc. Appendix B. Guidelines for General Availability One of the principal tenets of SPEC benchmarks is that the implementation of the System under Test (SUT) must be generally available, documented and supported by the providing vendor. The run and reporting rules for each benchmark includes this statement. The available, documented, and supported trio is frequently referred to with the single term: general availability. The purpose of including general availability requirements is to ensure that the systems and their hardware and software components actually represent real products that solve real business and computational problems. It is also intended to ensure that the benchmarked system does not contain so-called benchmark specials which improve benchmark performance but fail one or more of the tests of general availability. These tests include: 1. Has the system/component shipped or has the vendor committed to shipping the finished product within a time specified by the specific benchmark's run rules, usually 3 to 6 months from the date the result was made public? A finished product will have the vendor's endorsement that the product is suitable for production use in an environment comparable to the one represented by the benchmark. Typically this version will be designated as the final released version and not designated as a prototype or a preliminary release (i.e. alpha, beta, field test, etc.). Shipping the product refers to the vendor's standard process for getting said finished product into the hands of the end-user base. This can include putting it in a box and physically transporting it to end-user sites or simply advising the customer base that the product can be obtained on demand, for example by having an announced URL where the product can be downloaded or purchase information can be found. The commitment to ship a product within a specific time frame provided in a results disclosure must be taken as a public statement of intent. Any potential end-user should be able to confirm the vendor's intent to ship the finished product within the stated time frame and that information should not be considered secret once the result has been publicly disclosed. Shipping a product generally requires that one or more instances of the final product be in end-user hands or enroute to end-users with delivery imminent and based on shipping method used. 2. Is the product documented such that there are written descriptions on the product, how to install it, how to configure it and how and when to use it? The product documentation should be consistent with the deployment of the product in the benchmark such that there are no undocumented features employed solely for the benchmark. Availability of documentation should be concurrent with product availability. 3. Is the product supported such that the vendor has established clear expectations and procedures with its end-users with regards to problem resolution? Also, is the level of support reasonable for the type of product? Levels of support can clearly have a wide range depending on the type of product. An end-user acquiring a supported product will have access to some method of problem resolution consistent with the product's importance within the system. An inadequately supported product would be one that most consumers of that type of product could not afford either due to excessive costs imposed by the vendor for support or due to the risk involved in using a product with insufficient recourse to problem resolution. The requirement for support does not preclude the use of open source software products or other products where support is typically acquired separately. The requirement for support is present to prevent vendors from deploying benchmark specials that may seemingly meet availability and documentation tests but which are subtly discouraged from practical and common use by imposing unusually high costs and/or lengthy or inadequate methods for problem resolution. 4. Does the product meet the vendor's own definition of general availability, in addition to all the other tests specified by SPEC? Many vendors have their own internal processes for product development that set specific requirements for products to be considered generally available. If so, those requirements must be met for the product to be considered generally available by SPEC, in addition to the tests listed here. If the vendor has no such definition, then all other tests specified by SPEC must still be met. A vendor is not required to discuss or disclose its internal processes to SPEC. However the SPEC representative is responsible to identify the internal definition most closely matching SPEC's intent of general availability, and applying that definition to any statements to SPEC regarding general availability. Note that the vendor may use different terms, e.g. full customer availability, open shipping status, phase 4 exit, etc. 5. Does the vendor represent the product to customers as being generally available? This is not a clear cut test, as products may have several different distribution channels, and a product may not become generally available in all channels simultaneously. However, it is an indication that a product may not be generally available if the vendor specifically says in some venue that it is not. E.g., Web page saying product
is not available. It is an indication that a product may be generally available if the vendor specifically says so in some venue. E.g.,
In case of conflicting information, SPEC will weigh the preponderance of the evidence with the standard that it show the product to be generally available in some channel. The rules for each benchmark may have slightly different requirements related to general availability and may specify other types of benchmark specials to avoid. The overall goal of ensuring fair benchmarking for the vendors submitting results and the end-users reviewing those results remains the same. Note: The term "benchmark special" refers to any optimization which only improves benchmark performance. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||