What is This Thing Called "SPECviewperf®"?


SPECviewperf® is a portable OpenGL performance benchmark program written in C. It was developed by IBM. Later updates and significant contributions were made by SGI, Digital (Compaq, HP), 3Dlabs (Creative Labs) and other SPECopcSM project group members. SPECviewperf provides a vast amount of flexibility in benchmarking OpenGL performance. Currently, the program runs on most implementations of UNIX, Windows XP, Windows 2000, and Linux.

The OpenGL Performance Characterization (SPECopc) project group has endorsed SPECviewperf as its first OpenGL benchmark. Performance numbers based on SPECviewperf were first published in the Q4 1994 issue of The GPC Quarterly, a printed publication.

SPECopc group member companies have ported the SPECviewperf code to their operating systems and window environments. The SPECopc project group maintains a single source code version of the SPECviewperf code that is available to the public.

Benchmarking with SPECviewperf

SPECviewperf parses command lines and data files, sets the rendering state, and converts data sets to a format that can be traversed using OpenGL rendering calls. It renders the data set for a pre-specified amount of time or number of frames with animation between frames. Finally, it outputs the results.

SPECviewperf reports performance in frames per second. Other information about the system under test -- all the rendering states, the time to build display lists (if applicable), and the data set used -- are also output in a standardized report.

A "benchmark" using SPECviewperf is really a single invocation of SPECviewperf with command-line options telling the SPECviewperf program which data set to read in, which texture file to use, what OpenGL primitive to use to render the data set, which attributes to apply and how frequently, whether or not to use display lists, and so on. One quickly realizes that there are an infinite number of SPECviewperf "benchmarks" (an infinite number of data sets multiplied by an almost infinite number of command-line states).

Real-World Benchmarking

SPECopc project group members recognize the importance of real-world benchmarks. From the beginning, the group has sought benchmarks representative of the OpenGL rendering portion of independent software vendor (ISV) applications. Along these lines, the project group has come up with what it calls a viewset. A viewset is a group of individual runs of SPECviewperf that attempt to characterize the graphics rendering portion of an ISV's application.

Viewsets are generally not developed by the SPECopc project group; they come from the ISVs themselves. Members of the SPECopc project group often "sponsor" the ISV. Sponsorship entails helping the ISV in several areas, including how to obtain the SPECviewperf code, how to convert data sets to a SPECviewperf format, how to use SPECviewperf, how to create SPECviewperf tests to characterize the application, how to determine weights for each of the individual SPECviewperf tests based on application usage, and finally to help offer the viewset to the SPECopc project group for consideration as a standard SPECopc viewset. Any ISV wishing to develop a viewset should contact an SPECopc representative listed on the GPC Organization page.

Currently, there are six standard SPECopc application viewsets:

These type of applications typically render large data sets. All six viewsets represent relatively high-end applications.

The input to SPECviewperf is the command line. Here are the current command line options for SPECviewperf 7.1:

Program options:

Viewperf v7.1
 
Program options:
-polygon -pg   <file>                 : Viewpoint object to be used in the tests
-triangle -tr  <file>                 : Viewpoint object to be used in the tests
-quad -qd      <file>                 : Viewpoint object to be used in the tests
-mesh -mh      <file>                 : Mesh object to be used in the tests
-frameattr -fa <file>                 : Mesh3 attributes for each frame
-rendermode -rm <mode>                : POINT, VECTOR, LINE, POLYGON, TMESH, TFAN,
                                        TRIANGLE, or QUAD - default LINE
-vcriteria -vcrit                     : AUX Visual selection criteria - EXACT, MIN
                                        - default MIN
-vid <id>                             : Ask AUX for visual with ID = <id>
-vaccum -vac                          : Ask AUX for an accumulation buffer visual
-valpha -val                          : Ask AUX for an alpha buffer visual
-vdepthbuffer -vz                     : Ask AUX for a depth buffer visual
-vstencil -vst                        : Ask AUX for a stencil buffer visual
-indirectrender -ir                   : Render indirect - default direct
-nodither -ndi                        : Disable dithering
-ortho -or                            : Parallel/Orthographic projection - default Perspective
-displaylist -dl                      : Render with display list mode
-vertexarray -va                      : Render with vertex array mode (OpenGL1.1)
-interleaved -il                      : Packing vertex data interleaved in memory
-colorper -cp  <mode>                 : FRAME = Color per Frame,
                                        PRIMITIVE = Color per Primitive,
                                        VERTEX = Color per Vertex - default FRAME
-texture -tx   <file>                 : Image for texturing
-texgen -txg <file> <mode>            : <file> is image for enviroment mapping
                                        <mode> is SPHERE_MAP, OBJECT_LINEAR, EYE_LINEAR
                                        - default EYE_LINEAR
-texobj -txo <file>                    : Texture object mode <file> is list of images
-magfilter -magf <flt>                 : NEAREST, LINEAR - default NEAREST
-minfilter -minf <flt>                : NEAREST, LINEAR, NEAREST_MIPMAP_NEAREST,
                                        LINEAR_MIPMAP_NEAREST, NEAREST_MIPMAP_LINEAR,
                                        LINEAR_MIPMAP_LINEAR - default NEAREST
-texenv -te <env>                     : Texture enviroment, MODULATE, DECAL, BLEND
                                        - default DECAL
-texcomp -tc <num>                    : Texture components where <num> is 1,2,3, or 4
                                        -default 3
-blend -bl                            : Enable Blending
-srcblendfunc -sbf                    : ZERO, ONE, DST_COLOR, ONE_MINUS_DST_COLOR, SRC_ALPHA,
                                        ONE_MINUS_SRC_ALPHA, DST_ALPHA, ONE_MINUS_DST_ALPHA,
                                        SRC_ALPHA_SATURATE - default SRC_ALPHA
-dstblendfunc -dbf                    : ZERO, ONE, SRC_COLOR, ONE_MINUS_SRC_COLOR, SRC_ALPHA,
                                        ONE_MINUS_SRC_ALPHA, DST_ALPHA, ONE_MINUS_DST_ALPHA,
                                        - default ONE_MINUS_SRC_ALPHA
-linewidth -lw <width>                : Linewidth for wire/vector tests - default 1.0
-xwinsize -xws <side>                 : Size of test windows X dimension - default 700
-ywinsize -yws <side>                 : Size of test windows Y dimension - default 700
-numframes -nf <num>                  : Number of frames to be rendered during measurement
                                        Takes priority over -mp
-numilights -nil <num>                : Turns on <num> infinite lights - default 0
-numllights -nll <num>                : Turns on <num> local lights - default 0
-colormaterial -cm <side> <mode>      : <side> is FRONT, BACK, FRONT_AND_BACK 
                                        - default FRONT
                                        <mode> is AMBIENT, DIFFUSE, EMISSION, SPECULAR,
                                        AMBIENT_AND_DIFFUSE 
                                        - default AMBIENT_AND_DIFFUSE
-backface -bf                         : Cull Backfacing primitives - default off
-frontface -ff                        : Cull Frontfacing primitives - default off
-singlebuffer -sb                     : Single buffer mode
-fog -fg                              : Enable fog
-linesmooth -ls                       : Enable line antialiasing
-polysmooth -ps                       : Enable polygon antialiasing
-facetnormal -fn                      : Use facet normals when lighting
-linestipple -lp                      : Enable line stipple
-polystipple -pp                       : Enable polygon stipple
-batch -bt <num>                      : Batch <num> primitives together per glBegin/glEnd
                                       Valid with POINT, VECTOR, TRIANGLE, and QUADS
-polymodefront -pmf                   : POINT, LINE, or FILL - default FILL
-polymodeback  -pmb                   : POINT, LINE, or FILL - default FILL
-flat -f                              : Set shademodel to FLAT - default GOURAUD
-zbuffer -zb                          : Enable zbuffer for tests - default off
-clip -c                              : Align object on 3D clip boundary
-lighttwoside -l2s                    : Light both sides of model
-localview -lv                        : Define local viewer for lit tests
-normalize -nm                        : Enable normal normalization
-minperiod -mp <num>                  : Set minimum testing period in seconds
-mblur <num>                          : Use motion blur with num being amount of decay
-aa_multi <x> <r>                     : Full scene antialiasing rendered x times at an
                                        offset of r.  r should be tuned to the viewset
-walkthru -wt                          : Walkthru mode
-geomrepeat -gr                        : Geometry repeat
-grab <script>                        : Screen capture
-grabonly -go                         : Don't run benchmark, just do screen capture
-swap <true/false>                    : Force byte swapping
-notitle -nt                          : Disable title screen
-startframe -sf <num>                 : Set starting frame of a walkthru
-framepause -fp                       : Pause after each frame waiting for enter key
-newview -nv                          : Enable new view frustum calculation
-fov <angle>                          : Set field of view in degrees
-spin x y z dx dy dz                  : Set rotation initial values and speeds (degrees)
-clipplane                            : Enables a user clip plane
-nomaterial                           : Disables material changes if present
-nolight                               : Disables viewperf lighting
-forcecalibrate                       : Forces viewperf calibration
-tracelights -tl                      : Use lights generated in scene
-norejection                          : Turns off trivial rejection for walk through.
-noframecounter                       : Turn off display of frame counter 
-staticFrameCount -sfc <num>          : Count of frames between data edits (default = 10)
-staticBlockCount -sbc <num>          : Number of memory blocks changed per edit (default = 3)
-staticEditSize   -ses <num>          : Amount of frame data changed per edit (default = 0.001)
-staticEditsVisible -sev              : Attempt to make edits visible in the frame during execution.
-staticEditsDisabled                  : Allow all data to be static, no frame edits for entire test.
 

The -threads option is applicable only if SPECviewperf has been compiled with the C Preprocessor variable MP defined. SPECviewperf should only be compiled this way if the vendor supplied makefile has defined the MP flag (-DMP).

Here is an example of the command line from test #1 of the proe-02 viewset:
viewperf -mh WorldCar6 -fa gradient -tracelights -va -zb -mp 12 -forcecalibrate -xws 960 -yws 960 -grab grab.scr

This command line is providing the following directions:

Upon entering the command line, the system under test will run the benchmark for at least 12 seconds. This is to insure a valid test result. During the test, the SPECviewperf code renders the data set, WorldCar6.mh3, which is the WorldCar model used in the SPECapc for Pro/ENGINEER 2001™, rotating it about the center of the window with the above specified characteristics.

What follows is the output of SPECviewperf 7.1 running the first test from proe-02:

-------------------- 
Viewset: ProEngineer 
Test Number 01: WorldCar6 : Weight 13.50 
-------------------- 
    Viewperf Version                                   7.1
    Viewperf Arguments                                 -mh WorldCar6 -fa gradient -tracelights -va -zb -mp 12 -forcecalibrate -xws 960 -yws 960 -grab grab.scr 
    Month                                              3
    Day                                                4
    Year                                               2003
    Host                                               system HOSTNAME
    Operating System                                   Microsoft Windows NT
    Operating System Version                           Version 5.1 (Build 2600: Service Pack 1)
    Host Vendor                                        unknown
    Host Model                                         AT/AT COMPATIBLE
    Host CPU                                           x86 Family 15 Model 2 Stepping 7
    Host CPU Count                                     unknown
    Host Memory Size (MB)                              1023
    Host Primary Cache Size (KB)                       unknown
    Host Secondary Cache Size (KB)                     unknown
    Window System                                      Win32
    Driver Version                                     Version of GFX driver
    OpenGL Vendor                                      Name of company making GFX card
    OpenGL Version                                     Version of OpenGL supported by GFX card
    OpenGL Extensions                                  List of all supported OpenGL Client extensions
    OpenGL Renderer                                    Name of GFX card
    OpenGL Client Vendor                               GFX Card OpenGL client vendor string
    OpenGL Client Version                              Version of OpenGL client supported by GFX card
    OpenGL Client Extensions                           List of all supported OpenGLClient extensions
    GLU Version                                        1.2.2.0 Microsoft Corporation
    GLU Extensions                                     List of all supported GLU extensions
    Direct Rendering                                   False
    Double Buffer                                      True
    Stereo                                             False
    RGBA                                               True
    Color Index Size                                   -1
    Red Size                                           8
    Green Size                                         8
    Blue Size                                          8
    Alpha Size                                         0
    Accum Red Size                                     16
    Accum Green Size                                   16
    Accum Blue Size                                    16
    Accum Alpha Size                                   16
    Depth Size                                         24
    Stencil Size                                       0
    Auxiliary Buffer Count                             0
    Frame BufferLevel                                  0
    Window Width (pixels)                              960
    Window Height (pixels)                             960
    Screen Width (pixels)                              1280
    Screen Height (pixels)                             1024
    Pixel Format ID                                    4
    Visual Selection Criteria                          MINIMUM
    Number of Execution Threads                        1
    Geometry File                                      WorldCar6
    Input Mode                                         -mh 
    Minimum Test Period                                12.000000
    Number of Frames                                   0
    Number of Primitives                               319413
    Number of Vertices per Frame                       3917495
    Number of Vertices per Primitive                   12.264670
    Estimated bytes per frame                          0.000000
    Static frame count                                 0.000000
    Model edited every                                 10 frames
    Amount model changed                               0.001000
    Toggle Mode                                        NONE
    Batching Count                                     0
    Render Mode                                        LINE
    Color per                                          COLOR_PER_FRAME
    Orthographic Projection                            FALSE
    Display List                                       FALSE
    Vertex Array (OpenGL1.1)                           TRUE
    Interleaved Data                                   FALSE
    Clip Geometry                                      FALSE
    Walkthrough Mode                                   FALSE
    Geom Repeat Mode                                   FALSE
    Back Face Cull                                     FALSE
    Front Face Cull                                    FALSE
    Front Polygon Mode                                 FILL
    Back Polygon Mode                                  FILL
    Polygon Stipple Enable                             FALSE
    Polygon Antialiasing Enable                        FALSE
    Line Width                                         1.000000
    Line Stipple Enable                                FALSE
    Line Antialiasing Enable                           FALSE
    Number of Infinite Lights                          0
    Number of Local Lights                             0
    Color Material Enable                              FALSE
    Color Material Face                                FRONT
    Color Material Mode                                AMBIENT_AND_DIFFUSE
    Facet Normals                                      FALSE
    Two Sided Lighting Enable                          FALSE
    Local Viewer Enable                                FALSE
    Flat Shading                                       FALSE
    Fog Enable                                         FALSE
    Normal Normalization                               FALSE
    User Clip Plane                                    FALSE
    Material Changes Disabled                          FALSE
    Attribue Changes Disabled                          FALSE
    Trace Lights Enabled                               TRUE
    Texture Enable                                     FALSE
    TexObj Enable                                      FALSE
    Texture Generation Mode                            NO_TEXTURE_GENERATION
    Texture File                                       NONE
    Texture Minification Filter                        NEAREST
    Texture Magnification Filter                       NEAREST
    Texture Environment Mode                           DECAL
    Texture Components                                 3
    Depth Test Enable                                  TRUE
    Blend Enable                                       FALSE
    Source Blend Function                              SRC_ALPHA
    Destination Blend Function                         ONE_MINUS_SRC_ALPHA
    Dithering Enable                                   TRUE
    Motion Blur Amount                                 0.000000
    Full Scene Antialiasing Redraws                    0
    Full Scene Antialiasing Jitter Amount              0.000000
 
               Number of frames run: 280, Test period: 12.187000 (sec)
    23.0       frames/sec     --      -mh WorldCar6 -rm LINE -mp 12 -cp FRAME -ir -va -zb -tracelights -vz
 

 

During the calibration period SPECviewperf tests to see how many frames need to be rendered to meet the minimum time period and then runs the test.

The last line is the number of frames per second, which is the fundamental metric for each test. This is the result that is printed in the summary and individual report pages within the GPC News SPECopc section. Also printed in the summary and individual report pages is a single composite number for each viewset. This composite number is derived using a weighted geometric mean methodology (essentially the product of the results of each test raised to the power of the test weightings).

SPECviewperf 7.1

SPECviewperf 7.1 introduces new data changes to discourage optimizations that benefit the benchmark, but might not improve real-world application performance.  SPECviewperf 7.1 inserts a small amount of variation at regular intervals within its viewsets.  This ensures that the test system examines and processes each frame individually and is more aligned with the behavior of real-world applications. A second frame counter that moves around within the frame has also been added to the benchmark.  This provides visual verification of whether frame changes are being rendered. For more information, please read the documentation provided with the SPECviewperf source code.

Rob Putney / Simon Green / Chris Watts / Bill Licea-Kane / Allen Jensen / Ian Williams