Last updated: $Date: 2011-09-07 11:08:21 -0400 (Wed, 07 Sep 2011) $ by $Author: CloyceS $
(To check for possible updates to this document, please see http://www.spec.org/cpu2006/Docs/ )
Creating a suite of source code benchmarks to provide a consistent, comparative measure of CPU-intensive performance is not simple. The challenges are numerous, for example:
And in the end, you hope you have something that is fair, relevant, and understandable, that will address technology for a period of years.
Such is the case with SPEC CPU2006, which is a worthy update and replacement for CPU2000. The new benchmark suite:
All of this is due to the efforts of a large team of people from around the world, that SPEC would like to thank (with apologies and thanks to those who may not be listed).
SPEC CPU benchmarks are based on real-world applications (rather than using loop kernels or artificial stress tests). We therefore thank the authors of those real-world applications.
Thank you to the many participants of the "SPEC CPU Benchmark Search Program" and, in particular, the authors that contributed programs that were made part of the suite:
|456.hmmer||Sean R. Eddy|
|434.zeusmp||Michael L. Norman|
|437.leslie3d||Christopher Stone and Suresh Menon|
|444.namd||James Christopher Phillips|
|453.povray||Thorsten Fröhlich, Chris Cason, Ville Saari, Dan Farmer|
A special thank-you goes to Mark Kremenetsky, who contributed 410.bwaves. Dr. Kremenetsky would be in the list just above were it not for the fact that he is an employee of a SPEC member company (SGI), and therefore not eligible for the Search Program.
Thank you to the original authors of, and contributors to, open-source programs that were adapted by SPEC for inclusion in the suite. These programs include:
|400.perlbench||based on Perl by Larry Wall and a cast of thousands, along with many plug-in modules, including SpamAssassin by Theo Van Dinter and many others, MIME::Base64, Digest::MD5 and HTML::Parser by Gisle Aas, MHonArc by Ed Hood, IO::stringy by Eryq, TimeDate by Graham Barr, and MailTools by Mark Overmeer.|
|401.bzip2||based on bzip2 by Julian Seward|
|403.gcc||based on the GNU C Compiler by Richard Stallman and a large cast of helpers.|
|416.gamess||based on GAMESS from the Gordon Resarch Group at Iowa State University|
|447.dealII||uses the Boost C++ libraries, with contributors numbering in the thousands|
|454.calculix||uses the SPOOLES package by Cleve Ashcraft|
|471.omnetpp||based on OMNeT++ by András Varga|
|481.wrf||based on the Weather Research and Forecasting (WRF)
Model, which has 5 development teams
and a cooperative effort spanning multiple agencies, labs, and universities.
Also, 481.wrf uses NetCDF, which has many contributors.
|482.sphinx3||based on CMU Sphinx 3. The Carnegie Mellon Speech projects have had many contributors.|
|483.xalancbmk||based on Xalan-C++ and Xerces-C++ with contributions from the Apache development commmunity, IBM, and others. This product includes software developed by the Apache Software Foundation (http://www.apache.org/).|
|Tools||SPEC's toolset makes use of many open source tools, including perl and numerous CPAN modules, bzip2, GNU make, GNU tar, filepp, rxp, and zsh. A complete list may be found in the file redistributable_sources/licenses/README.|
For more information regarding the SPEC benchmarks, including information about other contributors, see the documentation in the individual benchmarks directories.
We would also like to thank:
SPEC management, who provided us with resources, and kept the SPEC CPU Subcommittee on their toes:
The SPEC office, who were the ones who really provided us with the resources:
Most importantly, thank you to the people who sweated through technical details and porting issues, debated their view points vigorously, and were willing to drive issues to conclusion. Without these people this benchmark suite would truly not exist.
A benchmark "Project Leader" serves as the focal point for porting a program to all the tested platforms, improves the stability of the code, removes code that SPEC does not want to test (such as platform-specific extensions), writes documentation, consults with both the author and the users, and generally tries to give each benchmark its best chance at making it through SPEC's testing process. Thank you to these project leaders:
In addition to the above, many other people from the member companies assisted with benchmark development, including but not limited to Louis Barton, Chris Cambly, Mat Colgrove, Aaron Cragin, Khushboo Goel, Darryl Gove, Linda Hsi, Andreas Jaeger, Carol LePage, Jim McInnes, Wendy Korn, Willi Kratzer, Dave Raddatz, Jeff Reilly, David Reiner, Hideki Saito, Alexey Starovoytov, Wilfried Stehling, Brian Whitney, and Joel Williamson.
Analysis of the candidate benchmarks was a substantial task. Contributors to profiling and other forms of analysis included: Yusuf Abdulghani, Tom Elken, Darryl Gove, John Henning, Lizy John, Ajay Joshi, Jim Koehler, Bob Larson, Alan MacKay, Aashish Phansalkar, Rahul Rahatekar, Joydeep Ray, Cloyce Spradling, Wilfried Stehling, and Reinhold Weicker
Cloyce Spradling is the primary author of the SPEC CPU2006 toolset.
Thank you to all who submitted bug reports and RFEs for the tools, notably including Carl Ponder (5 bugs/RFEs), Joydeep Ray (5), Yusuf Abdulghani (6), Alan MacKay (7), Rahul Rahatekar (9), Manas Mandal (21), and John Henning (86). Cloyce himself submitted the largest number (198).
Benchmark testing was coordinated by Yusuf Abdulghani.
There were numerous bug reports and RFEs suggested for the benchmarks themselves, but there's not an easy way to give credit for vigor by the testers, since much of that effort happened in difficult-to-summarize email and in person at benchathons. You who attended benchathons, and stayed up late working on bugs, know who you are.
Thank you to Julie Reilly for installation testing on Windows.
John Henning is the primary author of the documentation. Cloyce Spradling wrote first drafts for most of the new features in CPU2006. Many others contributed updates and did reviews, including Yusuf Abdulghani, Kevin Ball, Miriam Blatt, Mat Colgrove, Mark Debbage, Rick Jones, Bob Larson, Rahul Rahatekar, Jeff Reilly, Cathy Sandifer, David Schmidt, Myke Smith, Wilfried Stehling.
Finally, we thank all of the people behind the scenes in the compiler groups, architecture groups and performance groups who supported their work for SPEC at their respective companies.
Cloyce D. Spradling, Release Manager, SPEC CPU2006
John L. Henning, Vice-Chair/Secretary, SPEC CPU Subcommittee
Jeffrey W. Reilly, Chair, SPEC CPU Subcommittee
Copyright 1999-2011 Standard Performance Evaluation Corporation
All Rights Reserved