Skip navigation

Standard Performance Evaluation Corporation

Facebook logo LinkedIn logo Twitter logo Google+ logo
 
 

Open Forum

Profile of a SPEC95 Benchmark Candidate

By Peter R. Homan
Norwalk, Calif.

Published March, 1995; see disclaimer.


The VORTEx benchmark was derived from a full Object Oriented Database Management System (OODBMS) and conforms to SPEC CINT95 (CPU component benchmark) guidelines. The VORTEx benchmark is a single-user object-oriented database transaction benchmark which contains a highly portable system kernel coded in integer ANSI C. The 39,000 Non-Comment Source Statements (NCSS) that comprise the benchmark source are distributed across 69 files. The VORTEx benchmark exercises a rich repertoire of database transaction activities, object class configurations, and entity relationships. The benchmark includes all object database functionality except for disk caching, and permanent storage on disk of the database (commit). VORTEx is an acronym for "Virtual Object Run Time Expository."

This article presents background information on the VORTEx benchmark and on the experience of converting an existing application into a SPEC95 CPU component benchmark candidate.

Initial Screening Criteria

Before selecting VORTEx to provide a foundation for a CINT95 benchmark candidate, a number of issues had to be addressed in the screening process.

Other Alternatives?

While not an exhaustive search, a number of public domain database offerings were examined as potential benchmark candidates. When considered in terms of the SPEC95 development schedule, many of the database offerings exhibited potential difficulties meeting the requirements for multiple architecture compliance, compiler portability, simple build process, and workload derivation.

What makes it Unique?

A database benchmark opens up a new application area for the SPEC95 CPU integer benchmarks. Finding a benchmark in this application area that is also object-oriented further increases its appeal.

A somewhat uncommon feature of the VORTEx OODBMS is that it supports multiple databases in a single environment. Each database in the environment is physically segregated and is a logical clustering of objects that are contextually related. Objects may be globally related between databases (tuples, sets).

Does it Look Portable?

VORTEx in its full implementation supports object oriented language interfaces for C, C++, and FORTRAN. Under the hood, the database engine itself was written in ANSI C and has proved to be highly portable. Even before the development efforts for SPEC95, VORTEx had already been ported to a variety of platforms and compilers: UNIX (RISC), three compilers under DOS (x86), WIN32(x86), and MAC(68K).

Does the Code Look Reasonable?

Code quality affects benchmark quality. The source code found in VORTEx exhibits high quality from many aspects. A rigorous semantic style has been adhered to throughout the development process of VORTEx. The program is written in a formalized syntax, and the format of the programming style is such that elements of the functional specification document are extracted directly from the source code. The code contains extensive runtime diagnostic routines.

Is the Code Well Designed?

Good application design helps make a good benchmark. VORTEx conforms to the "Golden Rules" of The Object-Oriented Database System Manifesto [Atkinson89] which describes the main features and characteristics a system must have to qualify for the designation of an object-oriented database system. The thirteen rules of conformance provide support for: complex objects, object identity, encapsulation, types and classes, class or type hierarchies, late binding, computational completeness, extensibility, persistence, secondary storage, concurrency, recovery, and an ad-hoc query facility.

Is it Hard to Build and Run?

Simplified build and runtime procedures are considered an asset for a benchmark. No special language preprocessor is needed to build the VORTEx benchmark. In addition, VORTEx supports using a pre-built schema, eliminating the need to generate database translation mappings at measurement time.

Workloads that are similar to existing standards help a benchmark to gain acceptance. The workload of VORTEx has been modeled after common object-oriented database benchmarks with enhancements to vary the mix of transactions. (More details on this topic in the next section.)

CINT95 Conversion Requirements

Like the CINT92 suite, CINT95 focuses on testing a triad of system components: processor, cache, and compiler. SPEC95 brings with it an updated set of conversion criteria. To gain acceptance as a SPEC95 CPU component benchmark, a candidate must exhibit a number of desirable attributes after conversion. Here are some of the conversion criteria, and how VORTEx was modified to meet them.

Floating Point Content

One (usually minor) consideration for CINT95 is that the instruction mix for an integer benchmark contains less than one percent of floating point operations. No modifications to the VORTEx source had to be made to meet this criteria.

Workloads

At present writing, a CINT95 candidate needs at least three workloads. A "reference" workload for the runtime measurement, a shorter "test" workload to validate the build, and a very short "train" workload to be used as input for feedback-directed compiler optimization. VORTEx, like the other SPEC95 component candidates, was modified to accept workload configuration data from an input file. The two smaller workloads are derived from the reference workload required for measurement runs.

Obtaining "realistic" workloads is also desirable for SPEC95 CPU component benchmarks. The VORTEx benchmark is somewhat compromised at the user interface. (SPEC95, like its predecessors is a batch benchmark suite.) Does using a batch workload provide for a realistic approximation of the full application? If it is assumed that the sequence of generated events is sufficiently random in nature to simulate a series of a database transactions (via an interactive interface), or even to simulate a single long transaction, then the VORTEx benchmark can be considered representative of typical OODBMS activities.

Run time

The run time of a SPEC95 CPU component benchmark for the reference workload must be in the range of 4 to 20 minutes on a "high-end" system of today. The run time range is an attempt to compensate for increasing system performance over the lifetime of the SPEC95 suite. By varying the number and type of database transactions, VORTEx can be adjusted for virtually any run time length.

Dataset Size

The SPEC95 CPU component benchmarks were not intended to measure the effects of virtual memory paging. The VORTEx input files can also be used to adjust dataset size. Limiting the dataset to a maximum near 40 MB prevents paging on most systems with 64 MB of memory.

I/O Activity

SPEC95 CPU component benchmarks are not designed to measure I/O activity. Database applications typically exhibit significant I/O rates, and the complete VORTEx OODBMS is no exception. In the benchmark version of VORTEx, a goal was set to minimize I/O as much as possible. Two major features of the VORTEx OODBMS were deactivated to reach this goal: the commit function (which establishes persistence of objects in a database repository), and the memory caching function (which utilizes a temporary disk file to access resident data). There is some I/O activity involved in the initial loading of the schema database, (about 3 MB).

Profile Criteria

It is desirable for a SPEC95 CPU component benchmark candidate to exhibit a runtime profile spread evenly over the main procedures. (An even profile increases the program's resistance to benchmark-specific optimizations.) VORTEx has an even distribution of time spent in major procedures. The top six procedures of VORTEx all deal with memory management of objects, (a distribution representative of non-paging database applications.)

Performance Neutrality

For SPEC95 the attempt has been made, (through the active participation of the many vendors), to keep the code base from being biased towards a particular hardware system. Here are two examples from the development of the VORTEx benchmark.

It was noted that different compilers made distinctly different decisions about what procedures to inline. The procedures in question were located at the end of a chain of macro definitions. One of those procedures was called MemGetWord which makes up about 10% to 15% of the VORTEx benchmark runtime. There are sixteen procedures defined by hm.c and hm.h (MemGetWord is one) that are considered "semantic wrappers," and are used to provide isolation from the lower-level memory management (Mem) procedures located at the next level down. When the Hm procedures are "stripped" (to meet SPEC95 component CPU benchmark guidelines), they are essentially reduced to a direct call to Mem. Given that the primary use of the macros in the complete VORTEx OODBMS was to serve as development scaffolding, the decision was made to convert the Hm procedures into macros in the source code so that the compilers would not have to go through this particular optimization effort.

A second example involves workload definition. The database transaction series is determined by a sequence of random numbers. A portable random number generator was incorporated in the source to ensure that all implementations would be experiencing the same transaction sequence.

Cache Effects

An attempt has been made in SPEC95 to select benchmarks that both reflect real applications and to provide stress on the underlying hardware. The VORTEx benchmark models a cyclic sequence of random events that exhibits code variation and data access across a wide range of memory addresses. To enhance cache stress, two modifications were made to the VORTEx benchmark. First, the DeCache DBs event relocates six dynamic arrays that are the core directories of every database instantiation. The movement simulates memory movement that would normally occur when a database is committed or freed. This was found to be a significant factor in increasing the data cache miss rate. If the blocks are not moved, they become resident in the cache due to frequent references.

Second, the insertion of new parts on every iteration of the inner loop sequence results in attaining a better distribution of memory access. Insertions tend to create breaks in data access locality.

Portability

Since development began on the VORTEx benchmark for the CINT95 suite, the DBMS kernel and administrative modules have been ported to the wide range of hardware configurations found at SPEC benchathons and in vendor collections.

The VORTEX OODBMS was originally developed on the Intel x86 architecture under MS-DOS. When VORTEx was in its early phase of development, all of the MS-DOS C compilers supported only 16 bit data types. (These compilers were restricted to one byte member and struct alignments.) Even though developed in this restrictive environment, VORTEx was designed from the start for 16 and 32 bit interchangeability (using typesdefs for supported data types).

The initial port to a 32-bit architecture (with 32-bit pointers) exposed design aspects not considered before. Hardware dependencies, (revealed in the generation of class topologies), were found to impact the internal database representation and behavior of object transactions. Conflicts with coordinating the internal and external class topology maps were due to byte and struct alignment variations, and compiler dependent sizing of enumerated types.

Language compilation problems across various platforms have been minimized due to the adherence to standard C/ C++ coding conventions. Several runtime errors were encountered, caused by compiler optimization (mostly loop variable ordering), but these problems were easily located and resolved in the source code. Several ANSI preprocessor commands were found to be order dependent and were interpreted inconsistently across platforms. Some macros had to be reconfigured for conformance.

One of the requirements of SPEC95 benchmarks is to be portable to architectures that support 64-bit data types. For the VORTEx benchmark, this port revealed internalized database dependencies on the size of an address type (now eight bytes; was four bytes), and database schema inconsistencies. Built-in debugging tools in the VORTEx kernel were used to compare the internal database topology image and the external topology created by the compiler. Each major object topology was checked, corrected, and validated using this approach. These tools proved to be invaluable in resolving the remaining 64-bit port problems.

The VORTEx OODBMS is "mandated" to support both a Data Repository Interface (DRI) and an Object Management Interface (OMI), (of which the DRI is an encapsulated subset). A major design issue was to maintain code modules that handle both schema and non-schema driven implementations of certain container class objects (such as a B+ tree and 2D dynamic arrays). The OODBMS kernel and API modules were subsequently revised to handle architecture dependent issues which affected these class objects.

The multiple ports to 16, 32 and 64 (big and little endian) architectures, and to various operating systems, proved to be a rigorous test of the VORTEx OODBMS integrity. The basic requirement is to maintain a persistent database image decipherable by each of the target systems within a distributed client-server environment. A modification of the VORTEx integer benchmark (enhanced to add back complete database functionality) has proved to be a valuable tool in the VORTEx OODBMS verification and validation process.

Evolution of the Benchmark

In the beginning it was felt that existing class libraries and corresponding class methods, (already constructed and tested in the VORTEx environment), could be combined to produce a viable benchmark. A time constraint of two months to project completion provided an additional (and compelling) reason for leveraging existing class modules.

At the time SPEC95 development was initiated, three application programs had been written to validate and verify behavior of the VORTEx OODBMS. Each program exercised and accessed a unique library of class objects. The intent of the combined applications was to model each of the supported entity-relationships, data member types (attributes), "iterators", query commands, and various class inheritance schemes of the VORTEx Object Definition Language (ODL). Names of the three class modules developed are the Draw, Parts, and Persons libraries.

The first version of the benchmark was based on a single database of instantiated objects found in the Draw object library. Each non-Abstract Data Type (non-ADT) class had a separate code module to rigorously test unique aspects modeled by that class type. (Aspects which included attribute type and relationships.) A new driver program was designed that accessed the sub-modules in a sequence of random selection. This driver formed a primitive benchmark of sorts.

Measurements of cache activity, instruction mix, and procedure profiles were collected from this test setup. For this first setup, the cache miss rate was disappointingly low, and the run time profile did not show sufficient variation.

The second attempt to design a benchmark was to create a VORTEx-based implementation of the Engineering Database Benchmark (EDB0), [Cattell92]. EDB0 was designed to predict DBMS performance for engineering design applications. Its primary features include object creation, object navigation, and memory caching. For VORTEx, the Parts library already existed and conveniently implemented the needed procedures: create, connect, lookup, traverse, reverse, and insert.

Hardware measurements from the second attempt still did not exhibit significant data cache activity or Cycles Per Instruction (CPI) variation. Why was this? The workload in EDB0 only performed navigation operations: lookups, traverses, and reverses. Objects were only touched in the database, but not modified. Also there was no connectivity to other object types -- the schema was simplistic. As a result, data was frequently referenced in the cache.

A third version was now considered. This version was to combine elements of the Draw library with that of the Parts library, and to integrate the combined activities as a modified subset of the EDB0 event sequences. (The derived EDB0 benchmark, "The 007 Benchmark" [Carey94], was considered as a candidate, but time constraints dictated by the SPEC95 schedule abrogated its implementation.)

The current version of the integer benchmark "VORTEx" is based on the EDB0 event sequence mentioned above, however noteworthy modifications were made. Transactions were added for: deletion of objects, query of objects, traversal over different set types, and multiple databases. The Part object of EDB0 was enhanced to 'link' with objects in both the Draw and Persons libraries. The very complex class type, TestObj was included in the database schema. These modifications helped to enhance the benchmark's behavior. Hardware measurements of cache miss rates indicated that this benchmark could be interesting in the single-user environment as defined by SPEC95.

An important attribute of the VORTEx benchmark is that there is a good mix of transactions, data types, and memory accesses that do not compromise the model of "real-life" activities. Although a bit contrived, (as all benchmarks tend to be), the integration of the three libraries gives one the semblance of reality. Each person owns a set of parts; each part has a paired draw object; each object is drawn on a 2D coordinate system.

Overview of the Benchmark

General Overview

Transactions to and from the database are translated though a schema. A schema is a machine-readable translation that maps an internally stored data block to a model viewable in the context of the application. All objects declared in the database schema of this benchmark are derived from a common base object, called "Image01".

The database schema of class descriptors and entity relationships is processed and added to the VORTEx benchmark environment during a separate administration run. For a measurement run, the environment is loaded from a local disk file.

Those readers with access to the source code can obtain a complete perspective of the integrated benchmark application by looking at the files: draw*.h, rect*.h, emp*.h, and bmt01.h. The class descriptor for the base object "Image01" and its member methods is found in the header file obj01.h. ("Image01" is a pure virtual object, and is never instantiated.)

The VORTEx OODBMS was designed to be extensible for compliance with the Common Object Request Broker (CORBA) of the Object Management Group (OMG). The VORTEx Object Definition Language (ODL) is used to describe the interfaces that client objects call, and object implementations provide. All attributes and data types of the VORTEx ODL resolve to the basic data types defined in the OMG IDL [CORBA91]. The interface to the ORB is incomplete, however the required topological mapping is embedded in and accessible from the database schema of the system environment. The Object Definition Language (ODL) processor accepts modified ANSI C or C++ header files as input.

Library Overview

The schema as provided with the benchmark is pre-configured to manipulate three different databases: mailing list, parts list, and geometric data. Little-endian and big-endian versions of the schema are provided.

Three databases are instantiated (created) during the program execution: a Parts database, Draw database, and a Persons database. The Persons library is string intensive, whereas the Draw library is dominated by various types of dynamic arrays and reference attributes. The Parts library models embedded objects (3 connectors for each part), global relationships (links to objects in another database), and dictionaries (Generic Region Packets). Two additional databases are loaded at initialization of the program, providing the schema and database information of the VORTEx environment.

The Draw Library

The first library to be examined is the Draw library. A collection of objects are derived from a base object (ADT) DrawObj. Each of the subtypes models a particular base attribute type supported by the VORTEx environment. One class type TestObj is comprised of 32 attributes and exercises the complete repertoire of the data member functionality in the VORTEx API. Three class types are used to model relationships (a B+ tree and a doubly linked keyed set, and a doubly linked list) in this library. Members (direct and indirect) and embedded objects are referenced by strings (embedded and variable length), doubly dimensioned arrays, and dynamic arrays. Each of the draw objects has three virtual functions defined as methods of that class. Virtual functions behave according to the C++ language rules and provide some variation in code type.

The Parts Library

The next library models the Part-Connector objects (as defined as in the EDB0 benchmark database schema). "Each part has five data fields: a part id, a type, an (x,y) coordinate pair, and a build date. Each part has exactly three out-going ('to') connections to other parts plus a variable number of incoming 'from' connections, and each connection has a type and length" [Carey94].

The PartLib object owns a container class VpartsDir that is a dictionary of C-structs of TypePartToken, indexed by the unique part-id of a Part. The TypePartToken structure contains the database identifier to the Part instance, and the handle to the virtual array of incoming parts. The dictionary is implemented as a Generic Region Packet (GRP), that clusters the C structs in packets (320 per packet), to facilitate memory management at run time.

The Persons Library

The last of the three libraries is the Persons library. It was developed to compare run times and behavior with other database implementations, and is typical of class types presented in database literature. The unique objects in this library are a 'primal object' (database anchor), a Person object, and an Address object. A list of name and address information is the input for instantiation of the objects. For each unique name in this list, a Person and Address object is instantiated and paired with one another. The Person object is a member of B+ tree set associated with a primary key (last name) and a secondary key (first name).

Cross-Database Links

The VORTEx benchmark builds and manipulates three separate, but inter-related databases based on the schema.

The three base libraries are interrelated via 'global' sets and tuples (paired objects). A Part is globally paired (Tuple) with a DrawObj instance. The part object is a member of the global set (spans databases), PersonObjs.

The Part, DrawObj link is the 'tuple' PartDrawObj (in this case an 'independent pair'). in this implementation of the benchmark, when a >Part object is created, a randomly selected DrawObj is created and populated with randomly generated data member values. During the execution of the application, a DrawObj may be independently deleted and disassociated from its paired Part object during the "Look up parts" command. (See benchmark outline below.) When a Part instance is deleted, the paired DrawObj is also deleted.

The Part, Person link is via a one-to-many keyed set. The Person object is the set (i.e PersonParts) owner. When a Part is created, it is added into the global set PersonParts of a randomly selected Person object. If a Part object is deleted, it is also removed from the PersonParts set.

VORTEx Benchmark Event Sequence

A general outline of the macro commands of the event sequence:

  1. Invoke the environment.
  2. Load the VORTEx schema.
  3. Configure block sizes and create DBs.
  4. Read persons list and instantiate the Persons DB.
  5. Build queries.
  6. BEGIN OUTER LOOP
  7. Create parts.
  8. Traverse sets.
  9. Commit parts. [About 7% of runtime to this point.]
  10. BEGIN INNER LOOP
  11. Validate NamedDrawObjs set.
  12. Look up parts.
  13. Traverse parts.
  14. Delete parts. [54% of runtime]
  15. Reverse traverse parts.
  16. Create parts. [17% of runtime]
  17. Commit parts. [11% of runtime]
  18. DeCache DBs.
  19. END INNER LOOP
  20. Delete all TestObj objects.
  21. END OUTER LOOP
  22. Clean up and exit.

Summary

We believe that the VORTEx SPEC95 candidate will make a strong addition to the SPEC95 CPU component benchmark suite. It is derived from a sizable full-scale OODBMS application and meets the conversion criteria for SPEC95 component benchmarks.

Selected References

[Atkinson89] "The Object-Oriented Database System Manifesto," Atkinson, M., F. Bancilhon, D. DeWitt, K. Dittrich, D. Maier, and S. Zdonik, Proceedings of the Deductive Object-Oriented Database Conference, Kyoto, Japan, pp. 40-57, December 1989.

[Carey94] "The 007 Benchmark", Carey, M., D. DeWitt, J. Naughton., SIGMOD Record, 22(2):12-21, 1993. Also available as Technical Report No. 1140, Computer Sciences Dept., U. of Wisconsin-Madison, April 12, 1993, revised January 1994.

[Cattell92] "Object Operations Benchmark", R. Cattell and J. Skeen, ACM Transactions on Database Systems, 17(1), March 1922.

[CORBA91] The Common Object Request Broker: Architecture and Specification, published by Object Management Group and X/Open, December 1991.

[Homan93] The VORTEx Object Database Documentation, P. R. Homan, copyright 1993.

Acknowledgments

I would like to thank Terry Jackson and Dick Fowles of the Hewlett-Packard (HP) Systems Technology Division (STD) for sponsorship of the VORTEx CINT95 benchmark candidate and for supplying the essential resources for cross platform portability. Dan Homan (HP-STD) was instrumental in the debugging and modification process to both the application and system kernel code, and in resolving issues of portability and benchmark performance. Phil Vitale (HP-STD) lent encouragement, guidance, and editorial assistance in benchmark compliance and presentation. Thanks also to Alex Carlton of the HP General Systems Division for consultation on the design of database benchmarks.

Peter Homan is a Senior Systems Analyst/Architect with over 20 years software engineering experience including analysis, design and development.