SPEC/GPC's Application Performance Characterization (APC) project group is also nearing completion on a Pro/ENGINEERTM benchmark, scheduled for release by the end of this year. Additional CAD/CAM application benchmarks are planned for the future, along with Digital Content Creation (DCC) and visualization application benchmarks.
Filling a Void
Benchmarks from the APC group address what its members think is a benchmarking void: Although there are a few ad hoc application-level benchmarks that focus on graphics and visualization, there has been no standardization to date, and a wider range of industry benchmarks are needed. The group develops its benchmarks by working with user groups, testing labs, application developers, existing data sets, and other groups that have already developed general-purpose benchmarks.
"We are working to provide benchmarks that measure system performance based on how users typically interact with graphics-intensive applications," says Paul Martin of Hewlett-Packard, chair of the APC project group. "We bring to the table the proven SPEC/GPC methodology, which helps ensure that the resulting benchmarks contain appropriate graphics workloads and standardized measurement criteria, review processes, and reporting formats."
"These benchmarks extend SPEC/GPC's value to the graphics community," says Kaivalya Dixit, SPEC's president. "We are grateful to SolidWorks and id software for their cooperation in developing the first APC test suites."
A Day in the Life
SPEC/GPC's SolidWorks 98 benchmark is designed to represent a day in the life of a typical SolidWorks 98 user. The benchmark was developed by SolidWorks and enhanced by the APC project group. It is written in Visual Basic and C, and runs on Windows NT and Windows 95 platforms. The benchmark uses different-sized CAD/CAM solid models, the largest of which is a line assembly model with 276,000 polygons.
Five tests are included within the benchmark. I/O-intensive operations, CPU-intensive operations, and three different types of graphics operations are timed based on common user interaction with the models. A single number is derived from a weighted geometric mean of the normalized score for all five tests. Scores are also reported for each of the five individual tests and for the geometric mean of the three graphics tests. Results are normalized to a reference machine (300-MHz Pentium II processor; PERMEDIA 2 graphics processor) chosen by the APC group's membership.
Real-World Quake II Sessions
SPEC/GPC's Quake II benchmark is designed to measure performance and image quality based on real-world game sessions. Quake II runs under platforms that support the OpenGL application programming interface (API). It is one of the most graphically intensive games on the market, challenging the texture-mapping capabilities of most current hardware.
SPEC/GPC's benchmark uses Quake II's "TIMEDEMO" function to time the playback of a number of different pre-recorded game sessions, which include both single and multi-player matches. Results are reported in frames per second. The benchmark requires that results submissions include screen captures from the game, so that visual quality can be assessed. Results can be reported for two different tests at four different screen resolutions: 640x480, 800x600, 1024x768 and 1280x960. Vendors must report results for both tests under a given resolution, but have the option to report numbers for whatever screen resolutions they choose.
SPEC/GPC's Quake II benchmark is fully endorsed by id software, the developer of Quake II.
Questions regarding membership and application benchmarking can be addressed to the APC project group at gpcapc-info@spec.org.
For more detailed information on the APC, go to the APC FAQ document.