1. Introduction

The SPECjEnterprise®2018 Web Profile benchmark measures the end-to-end performance of a Java Enterprise Edition (Java EE) Web Profile application in a global enterprise. The Web Profile benchmark targets the Java EE 7 Web Profile specification. It provides a standard for users of web applications to compare the performance of the software and hardware for all levels of the application stack, from the Java EE Web Profile Application Server, Java Runtime Environment, network and operating system to the Database Server and storage.

This document specifies how the SPECjEnterprise®2018 Web Profile benchmark is to be run for measuring and publicly reporting performance results. These rules abide by the norms laid down by SPEC®. This ensures that results generated with this benchmark are meaningful, comparable to other generated results, and are repeatable (with documentation covering factors pertinent to duplicating the results).

Per the SPEC® license agreement, all results publicly disclosed must adhere to these Run and Reporting Rules.

The general philosophy behind the rules for running the SPECjEnterprise®2018 Web Profile benchmark is to ensure that an independent party can reproduce the reported results.

For results to be publishable, SPEC® expects:

  • Proper use of the SPEC® benchmark tools as provided.

  • Availability of all required Full Disclosure files.

  • Support for all of the appropriate standards specified in this document.

SPEC® is aware of the importance of optimizations in producing the best system performance. SPEC® is also aware that it is sometimes hard to draw an exact line between legitimate optimizations that happen to benefit SPEC® benchmarks and optimizations that specifically target the SPEC® benchmarks. However, with the rules below, SPEC® wants to increase the awareness by implementers and end users of issues of unwanted benchmark-specific optimizations that would be incompatible with SPEC®'s goal of fair benchmarking.

  • Hardware and software used to run the SPECjEnterprise®2018 Web Profile benchmark must provide a suitable environment for running typical server-side Java programs. Note that this may be different from a typical environment for client Java programs.

  • Optimizations must generate correct code for a class of programs, where the class of programs must be larger than a single SPEC® benchmark.

  • Optimizations must improve performance for a class of programs, where the class of programs must be larger than a single SPEC® benchmark.

  • The implementation is generally available, documented and supported by the providing vendor.

Results must be reviewed and accepted by SPEC® prior to public disclosure. The submitter must have a valid SPEC® license for this benchmark to submit results. Furthermore, SPEC® expects that any public use of results from this benchmark shall follow the SPEC® Fair Use Rules and those specific to this benchmark (see the Fair Use section below).

In the case where it appears that these guidelines have been violated, SPEC® may investigate and take action in accordance with current policies.

SPEC® reserves the right to modify the benchmark codes, workloads, and rules of the SPECjEnterprise®2018 Web Profile benchmark as deemed necessary to preserve the goal of fair benchmarking. SPEC® will notify members and licensees whenever it makes changes to the benchmark and may rename the metrics. In the event that the workload or metric is changed, SPEC® reserves the right to republish in summary form adapted results for previously published systems.

2. Running the SPECjEnterprise®2018 Web Profile benchmark

2.1. Definition of Terms

ACID Properties refers to the Atomicity, Consistency, Isolation and Durability (ACID) properties defined in Transaction Property Requirements

A Business Transaction is a unit of work initiated by a User and may involve any combination of REST Interactions and Web Interactions.

The Components refers to Java EE elements such as EJBs, JSPs, WebSockets, REST APIs, JSFs, and Servlets.

The Configuration Diagram is the picture in a common graphics format that depicts the entire configuration (including the SUT and load drivers). The Configuration Diagram is part of a Full Disclosure. See Configuration Diagram

The Concurrent User Load refers to the rate business transactions are injected into the SUT.

The Cycle Time refers to the time elapsed from the first byte sent by the Driver to request a Business Transaction until the first byte sent by the Driver to request the next Business Transaction. The Cycle Time is the sum of the Response Time and Delay Time.

The Delay Time refers to the time elapsed from the last byte received by the Driver to complete a Business Transaction until the first byte sent by the Driver to request the next Business Transaction. The Delay Time is a function of the Response Time and the Injection Rate. For a required Injection Rate, the Delay Time will be smaller for when Response Times are longer. The driver will adjust the delay to ensure that transactions are delivered in a steady rate.

A Database Transaction is a unit of work on the database with full ACID properties. A Database Transaction is initiated by an enterprise bean as part of a Business Transaction.

The Deployment Unit refers to a Java EE Application Server or set of Application Server instances in which the components from the Insurance application are deployed.

The Driver or Insurance Driver refers to the provided client software used to generate the benchmark workload.

Full Disclosure refers to the information that must be provided when a benchmark result is reported. For the SPECjEnterprise®2018 Web Profile benchmark, Full Disclosure includes the Submission File, Full Disclosure Archive ( FDA ), and Configuration Diagram. See Full Disclosure Archive.

The Full Disclosure Archive ( FDA ) is an archive of files that is part of a Full Disclosure. It contains the benchmark run results and configuration information necessary to reproduce the results.

The Full Disclosure Report ( FDR ) is the formatted benchmark result generated from the Submission File. It includes links to the Full Disclosure Archive ( FDA ) and Configuration Diagram.

The Injection Rate (IR) refers to the rate at which Business Transaction requests are injected into the SUT.

The Measurement Interval refers to a steady state period during the execution of the benchmark for which the test submitter is reporting a performance metric.

A node is the hardware system that runs one or more Java EE Application Servers or Database Server instances. For example, a blade enclosure would generally consist of multiple nodes.

Non volatile storage refers a storage device whose contents are preserved when its power is off.

A Persistent Storage Driver is software that handles the connection between the application server and a persistent storage.

A Relational Database Management System (RDBMS) is a database management system (DBMS) that is based on the relational model.

The Resource Manager is the software that manages a database and is the same as a Database Manager.

The Response Time refers to the time elapsed from when the first transaction in the Business Transaction is sent from the Driver to the SUT until the response from the last transaction in the Business Transaction is received by the Driver from the SUT.

A REST Interaction is a JAX-RS request to the REST portion of the Application.

The SPECjEnterprise®2018 Web Profile Application (or just " Application ") refers to the implementation of the Components provided for the SPECjEnterprise®2018 Web Profile workload.

The SPECjEnterprise®2018 Web Profile Kit (or just " Kit ") refers to the complete kit provided for the SPECjEnterprise®2018 Web Profile benchmark. This includes the SPECjEnterprise®2018 Web Profile Application, Driver, load programs, documentation and other supporting files.

"SPECjEnterprise®2018 WebjOPS" is the primary SPECjEnterprise®2018 Web Profile metric and denotes the average number of successful jEnterprise Web Operations Per Second completed in the Insurance domain during the Measurement Interval. "SPECjEnterprise®2018 WebjOPS" is composed of the total number of Business Transactions completed in the Insurance Domain per second.

The Submission File is a text file containing the Full Disclosure information specified in Submission File. The Submission File is part of a Full Disclosure.

The System Under Test (SUT) comprises all components which are being tested. This includes the Java EE Web Profile Application Servers, Database Servers, network switch(s), etc. It does not include the Driver.

The Test Submitter (or just " Submitter ") refers to the organization that is submitting a benchmark result and is responsible for the accuracy of the submission.

A User represents an individual customer to the System Under Test (SUT)

A Web Interaction is an HTTP request to the Web-based portion of the Application.

2.2. Product Requirements

2.2.1. Hardware Availability

All hardware required to run the SPECjEnterprise®2018 Web Profile Application must be generally available, supported and documented (see the General Availability section for details on general availability rules).

2.2.2. Software Availability

All software required to run the benchmark in the System Under Test (SUT) must be implemented by products that are generally available, supported and documented (see the General Availability section for details on general availability rules). These include but are not limited to:

  • Operating System

  • Java EE Web Profile Application Server

  • Java Runtime Environment (JRE)

  • Database Server

  • Database Driver

  • Persistent Storage Driver

2.2.3. Java EE Compatibility

The Java EE Web Profile Application Server must provide a runtime environment that meets the requirements of the Java Enterprise Edition Version 7 (Java EE 7) Web Profile or later specifications during the benchmark run.

A major new version (i.e. 1.0, 2.0, etc.) of a Java EE Web Profile Application Server must have passed the Java EE 7 Web Profile or later Compatibility Test Suite (CTS) by the product’s general availability date.

A Java EE Web Profile Application Server that has passed the Java EE Web Profile Compatibility Test Suite (CTS) satisfies the Java EE Web Profile compatibility requirements for this benchmark regardless of the underlying hardware and other software used to run the benchmark on a specific configuration, provided that the runtime configuration options result in behavior consistent with the Java EE Web Profile specification.

Comment : The intent of this requirement is to ensure that the Java EE Web Profile Application Server is a complete implementation satisfying all requirements of the applicable Java EE Web Profile specification and to prevent any advantage gained by a server that implements only an incomplete or incompatible subset of the Java EE Web Profile specification.

2.3. Scaling the Benchmark

The throughput of the SPECjEnterprise®2018 Web Profile benchmark is driven by the activity of the Insurance driver. The throughput of the driver is directly related to the chosen Injection Rate (IR). To increase the throughput, the IR needs to be increased. The benchmark also requires a number of rows to be populated in various database tables. The scaling requirements are used to maintain the ratio between the Business Transaction load presented to the SUT, the cardinality of the tables accessed by the Business Transactions, the IR, and the number of users generating the load.

2.3.1. Database Scaling Rules

Database scaling is defined by the Load Injection Rate( LIR ), which is a step function of the Injection Rate( IR ). The Load Injection Rate is defined to be:

LIR = (CEILING( IR / step)) * step

where the step is:

step = 10 ^ (INT (LOG(IR))) {log is base 10 }

For example:

IR Step LIR

1-10

1

1,2,3…​10

11-100

10

20,30,40…​100

101-1000

100

200,300,400…​1000

etc.

Comment : The Load Injection Rate is calculated automatically by the database load program from the Injection Rate.

2.3.2. Database Scaling Requirements

The following scaling requirements represent the initial configuration of the database tables.

TABLE 1 : Database Scaling Rules

Table Name Cardinality (in rows) Comments

V_VEHICLEDESCRIPTION

200*LIR (minimum of 3984, maximum of 231,072)

Used by the Vehicle service.

I_POLICYHOLDER

40*LIR

Used by the Insurance service

I_VEHICLE

(I_POLICYHOLDER * 80%)

Used by the Insurance service. Contains approximately 80% of the number of PolicyHolders. The size may vary depending on actual random numbers generated.

P_ACTUARIAL_CLASS

5

Used by the Insurance Provider service.

P_VEHICLE_CLASS

29435

Used by the Insurance Provider service.

P_COVERAGE_CLASS

96

Used by the Insurance Provider service.

P_DRIVER_CLASS

1300

Used by the Insurance Provider service.

2.3.3. Scaling the Insurance Driver

To stress the ability of the Java EE Web Profile Application Server to handle concurrent user load, the benchmark implements a fixed number of users equal to 10 * IR. The number does not change over the course of a benchmark run.

2.4. Database Requirements

To satisfy the requirements of a wide variety of customers, the SPECjEnterprise®2018 Web Profile SUT consists of a number of databases and Java EE Web Profile Application Servers that can be mapped to nodes as required. The implementation must not, however, take special advantage of the co-location of databases and Java EE Servers, other than the inherent elimination of WAN/LAN traffic.

The application modules may be combined and deployed on a single Application Server instance. This means that the benchmark implementer can choose to run a single Deployment Unit that accesses a single database that contains the tables of all the modules. However, a benchmark implementer is free to separate the modules into their own Deployment Units, with one or more database instances.

For the Insurance workload the Transaction property requirements described in Transaction Property Requirements must be compatible with the Java EE 7 Web Profile or later specification.

All tables must have the properly scaled number of rows as defined by the database population requirements.

The database schema must be derived directly from the reference schema scripts provided in the databases/default directory of the SPECjEnterprise®2018 Web Profile kit. Derived means that the database schema used must match the schema that the scripts describe, not that the scripts must be used to physically generate the database schema. Modifications to the database schema are allowed only as documented below.

Additional database objects may be added to the reference schema and DDL modifications may be made to the reference schema, however all additions and/or modifications must be disclosed along with the specific reason for the addition/modification. The base tables and indexes in the reference schema cannot be deleted. Views are not allowed. The data types of fields can be modified provided they are semantically equivalent to the standard types specified in the scripts.

Comment : Replacing CHAR with VARCHAR would be considered semantically equivalent. Changing the size of a field (for example: increasing the size of a char field from 8 to 10) would not be considered semantically equivalent. Replacing CHAR with INTEGER (for example: zip code) would not be considered semantically equivalent.

Modifications that a customer may make for compatibility with a particular Database Server are allowed. Changes may also be necessary to allow the benchmark to run without the database becoming a bottleneck, subject to approval by SPEC®. Examples of such changes include:

  • additional indexes on fields used in query predicates,

  • additional fields to support optimistic concurrency control,

  • additional constraints (i.e. unique and foreign key),

  • additional DDL statement options,

  • implementing primary keys as indexes,

  • implementing unique constraints as unique indexes,

  • specifying fields as 'NOT NULL', and

  • horizontally partitioning tables.

Scripts or any other files for schema generation provided by the vendors are for convenience only. They do not constitute the reference or the baseline script in the databases/default directory. Deviations from the scripts must be disclosed in the submission file even though the vendor-provided scripts or any other files for schema generation were used directly.

In any committed state the primary key values must be unique within each table. For example, in the case of a horizontally partitioned table, primary key values of rows across all partitions must be unique.

The databases must be populated using the supplied load programs prior to the start of each benchmark run. That is, after running the benchmark, the databases must be reloaded, or table data restored from a backup of a previous load state, prior to a subsequent run. Modifications to the load programs are permitted for porting purposes. All such modifications made must be disclosed in the Submission File.

The database must be accessible at all times for external updates and maintain data access transparency.

Data Access Transparency is the property of the system which removes from the application program any knowledge of the location and access mechanisms of partitioned data. An implementation that uses vertical and/or horizontal partitioning must meet the requirements for transparent data access. The system must prevent any data manipulation operation, including external applications, which would result in a violation of the consistency and isolation requirements. An external application must be able to manipulate any set of rows or columns transparently using the specified table names in the schema script.

External updates may be assumed to update the database version columns.

2.5. Application Deployment Requirements

The submitter must run the SPECjEnterprise®2018 Web Profile Application provided in the kit without modification, except as permitted by these run rules.

Changes necessary to configure and deploy the benchmark in a specific configuration are permitted to the Java Enterprise Edition metadata (configuration files), subject to the requirements of this section. Vendor specific metadata (configuration files) are permitted.

Modifications to persistence.xml and orm.xml consistent with these Run Rules are permitted, but must be provided in the FDR and are subject to review for compliance.

Named query annotations may be overridden in orm.xml provided the new query remains an EJBQL query. Overridden EJBQL queries must return the same data set in the same format as the original query. Replacing EJBQL queries with native queries is not allowed. Exceptions to allow native queries are permitted for queries including "select count(a) from …​" and used exclusively during auditing prior to the ramp-up period, provided they are semantically equivalent.

Comment : The intent of the exception to allow native queries is to reduce the time to perform the auditing counts prior to the run start. There should be no impact to the outcome of the audit tests or the reported benchmark results.

The deployment must assume that the database could be modified by external applications.

2.6. Driver Requirements

The Insurance Driver is responsible for driving load to the Insurance service. For a compliant benchmark run, all of the workload must be driven using the https protocol. When the https protocol is configured in the driver, the driver will use the TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 cipher suite.

2.6.1. Business Transaction Mix Requirements

Each Business Transaction that is run by the Insurance Driver is a sequence of operations to the Insurance service. Business Transaction sequences are selected by the Driver based on the mix shown in Table 2.

TABLE 2 : Business Transaction Mix Requirements

Business Transaction Sequence Percent Mix

Register Invalid

2

Register

8

Login / Unregister

5

Login / View User / Logout

5

Login / View User / Update User / Logout

10

Login / Add Vehicle / View Quote / Logout

25

Login / Delete Vehicle / Logout

10

Login / Accept Quote / View Vehicle / Logout

20

Login / Accept Quote + WebSocket / View Vehicle / Logout

5

Login / View Vehicle / View Insurance / Logout

10

The Driver sends 85% of the load to the JAX-RS API and 15% of the load to the JSF generated HTML pages. Each sequence in the above table has both REST and JSF variations. For example, if a Business Transaction sequence in the table above has a mix of 10%, there is a 8.5% mix for the REST version of the Business Transaction sequence and a 1.5% mix for the JSF version of the Business Transaction sequence.

The actual mix achieved in the benchmark must be within a 4% range (+/- 2%) of the targeted mix for each type of the Business Transaction operation. The Driver checks and reports on whether the mix requirements were met.

2.6.2. Response Time Requirements

The Driver measures and records the Response Time for each of the different operations in the Business Transaction sequences. Only successfully completed operations for a Business Transaction sequence in the Measurement Interval are included. At least 90% of each operation of the Business Transactions must have a Response Time of less than 2 seconds. The average Response Time of each Business Transaction operation must not be greater than 0.1 seconds more than the recorded 90% Response Time. This requirement ensures that all users will see reasonable response times. For example, if the 90% Response Time of register transactions is 1 second, then the average cannot be greater than 1.1 seconds. The Driver checks and reports on whether the response time requirements were met.

2.6.3. Cycle Time Requirements

For each Business Transaction sequence, the Driver selects cycle times from a negative exponential distribution, computed from the following equation:

Tc = -ln(x) * 10

where:

Tc = Cycle Time ln = natural log (base e) x = random number with at least 31 bits of precision, from a uniform distribution such that (0 < x ⇐ 1)

The distribution is truncated at 5 times the mean. For each Business Transaction sequence, the Driver measures the Response Time Tr and computes the Delay Time Td as Td = Tc - Tr. If Td > 0, the Driver will sleep for this time before beginning the next Business Transaction sequence. If the chosen cycle time Tc is smaller than Tr, then the actual cycle time ( Ta ) is larger than the chosen one.

The average actual cycle time is allowed to deviate from the targeted one by 5%. The Driver checks and reports on whether the cycle time requirements were met.

2.6.4. Miscellaneous Requirements

During a benchmark run, 99% or more of each operation of the Business Transaction sequences must succeed during steady state. This allows for up to 1% of an operation type to fail. Additionally, the amount of rows in the database tables must contain no more than a +/- 1% deviation from the expected size. The Driver checks and reports on whether these requirements were met.

2.6.5. Performance Metric

The metric is Insurance Transactions/sec, composed of the total count of all Business Transaction sequences successfully completed during the measurement interval divided by the length of the measurement interval in seconds.

2.7. Driver Rules

The Driver is provided as part of the SPECjEnterprise®2018 Web Profile kit.

The class files provided in the specjwebprofile.jar file of the SPECjEnterprise®2018 Web Profile kit must be used as is. No source code recompilation is allowed.

Faban is used as the infrastructure on which the driver is built. The version included in the SPECjEnterprise®2018 Web Profile benchmark kit must be used as is. No source code recompilation is allowed.

The Driver must reside on one or more systems that are not part of the SUT.

Comment : The intent of this requirement is that the communication between the Driver and the SUT be accomplished over the network.

The Insurance Driver communicates with the SUT using HTTPS. The Insurance Driver uses a single URL to establish a connection with the web tier. If more than one Driver system is used all Driver systems must have the same URL for the Insurance Driver.

Comment : The purpose of the identical URL requirement is to ensure that any load balancing is done by the SUT and is transparent to the Driver systems.

As part of the run, the driver checks many statistics and audits that the run has been properly executed. The driver tests the statistics and audit results against the requirements specified in this document and marks each criteria as "PASS" or "FAIL" in the summary reports. A compliant run must not report failure of any criteria. Only results from compliant runs may be submitted for review and published. Non-compliant runs are not allowed to be published.

Pre-configured Driver decisions, based on specific knowledge of SPECjEnterprise®2018 Web Profile and/or the benchmark configuration, are disallowed.

The Driver systems may not perform any processing ordinarily performed by the SUT, as defined in System Under Test (SUT) Requirements. This includes, but is not limited to:

  • Executing part or all of the SPECjEnterprise®2018 Web Profile Application

  • Caching database or Java EE Web Profile Application Server specific data

  • Communicating information to the SUT regarding upcoming transactions

The Driver records all exceptions in the Driver log. Any errors that appear in the Driver log must be explained in the Submission File. The logging level for a compliant run a minimum logging level of INFO.

2.8. Measurement Requirements

The Measurement Interval must be preceded by a ramp-up period of at least 10 minutes at the end of which a steady state throughput level must be reached. At the end of the Measurement Interval, the steady state throughput level must be maintained for at least 5 minutes, after which the run can terminate.

2.8.1. Steady State

The reported metric must be computed over a Measurement Interval during which the throughput level is in a steady state condition that represents the true sustainable performance of the SUT. Each measurement interval must be at least 60 minutes long and should be representative of a 24 hour run.

Memory usage must be in a steady state during the Measurement Interval.

At least two database checkpoints or continuous checkpointing must take place during the Measurement Interval. The checkpoint interval must be disclosed in the FDR.

Comment : The intent of this requirement is that any periodic fluctuations in the throughput or any cyclical activities, e.g. JVM garbage collection, database checkpoints, etc. be included as part of the Measurement Interval.

2.8.2. Reproducibility

To demonstrate the reproducibility of the steady state condition during the Measurement Interval, a minimum of one additional (and non-overlapping) Measurement Interval of the same duration as the reported Measurement Interval must be measured and its "SPECjEnterprise®2018 WebjOPS" must be equal to or greater than the reported metric. This reproducibility run’s metric is required to be within 5% of the reported metric.

2.9. Transaction Property Requirements

The Atomicity, Consistency, Isolation and Durability (ACID) properties of transaction processing systems must be supported by the SUT during the running of this benchmark.

2.9.1. Atomicity Requirements

The System Under Test must guarantee that all Transactions are atomic; the system will either perform all individual operations on the data, or will assure that no partially-completed operations leave any effects on the data. The tests described below are used to determine if the System Under Test meets all the transactional atomicity requirements. If any of the tests have a result of "FAILED" then the SUT does not comply with the transaction atomicity requirements of the benchmark.

2.9.1.1. Atomicity Tests
2.9.1.1.1. Atomicity Test 1

This test checks if the proper transaction atomicity levels are upheld in transactions associated with the benchmark. This test will add a Vehicle to a PolicyHolder and then cause a rollback which should cause the transaction changes to be removed from the database. It will then again check that the number of Vehicles associated with the PolicyHolder is identical to the number before the injection.

2.9.1.1.2. Atomicity Test 2

Similar to atomicity test 1, but here we do not cause a rollback and we expect that the number of Vehicles associated with the PolicyHolder has increased.

2.9.2. Consistency and Isolation Requirements

This section describes the consistency and isolation requirements for Transactional Resources (currently this consists of but is not limited to the database). Submitters can choose to implement the requirements in this section by any mechanism supported by the SUT. The isolation levels Strict READ_COMMITTED and Strict REPEATABLE_READ as used in this benchmark are defined in Isolation Level Definitions.

The Java EE 7 Web Profile specification defines the transactional requirements for Java EE 7 Web Profile compatible application servers. The transactional consistency and isolation requirements for the application represented by this benchmark are defined in this section of the run rules. Compliant results must satisfy the requirements of both the Java EE 7 Web Profile specification and these run rules.

2.9.2.1. JPA Entities

The SPECjEnterprise®2018 Web Profile benchmark application requires a minimum isolation level of Strict READ_COMMITTED.

The benchmark application is designed to preserve the referential integrity between entities. For example to maintain integrity between PolicyHolder and Vehicle the application maintains the references to the PolicyHolder from the Vehicle. The underlying Java EE 7 product implementation used for the benchmark must fully support this referential integrity per the JPA 2.1 specification.

VehicleDescription entities are assumed to be infrequently updated by an external application only and not the benchmark application itself. VehicleDescription entity query results and/or row state may be cached, provided that stale information is refreshed with a time-out interval of no more than 20 minutes using a logical isolation level of Strict READ_COMMITTED. In other words, no transaction may commit if it has used data from a stale read and/or row state that was obtained from the database more than 20 minutes previously. The effects of any vehicle description insertion, vehicle description deletion, or update of any vehicle description’s details are thereby ensured to be visible to all transactions that commit 20 minutes later, or thereafter.

Entities that are used in the Insurance Provider service are assumed to be updated less frequently than VehicleDescription entities. Any updates to the entities are assumed to be done less frequently than the time of a benchmark run. Entity query results and/or row state may be cached, provided that stale information is refreshed with a time-out interval that may be greater than the length of a benchmark run using a logical isolation level of Strict READ_COMMITTED.

A stale read is defined as the reading of entities or entity state that is older than the point at which the JPA persistence context was started.

For entities in the Insurance Provider and Vehicle services, it is possible to provide the applications a reference to an entity avoiding copying of the entity data.

Optimistic verification must always be performed against the database.

If a transaction is terminated by successful execution of a commit statement then all changes made to entity data are accessible to all concurrent and subsequent transactions on the database.

2.9.2.1.1. JPA Write Through Test

The JPA write through test is run by the benchmark driver to ensure that data is written from the JPA Entities to the database by the Java EE Application server. The test is run by the driver immediately prior to ramp-up.

Description of the test :

  1. The driver selects a policy holder entity using JPA.

  2. The "PHONE" field of this policy holder is then changed to a new number via JPA.

  3. In a new transaction and with a new database connection the same policy holder is read from the database via JPA to check that the update to the "PHONE" field is reflected in the database.

2.9.2.1.2. Phantom Delete Test

The phantom delete test is run by the benchmark driver to test if the phantom delete phenomena as defined in Isolation Level Definitions could occur. The test is run by the driver immediately prior to ramp-up.

Description of the test :

  1. A policy holder entity is selected using JPA.

  2. The Insurance Driver starts in parallel to the main thread an additional thread. This thread executes the following steps :

    1. The policy holder is deleted using JPA.

    2. The changes are flushed to the database using JPA.

    3. Wait a certain amount of time (10 sec) until the Insurance Driver’s main thread has finished step 4.

    4. Transaction is rolled back to preserve the policy holder in the database.

  3. Wait a certain amount of time (5 sec) until the additional thread has reached step 2c.

  4. The existence of the policy holder is verified, ensuring that phantom deletes are prevented.

Due to the chronological dependencies, passing this test is necessary but not sufficient to guarantee that no Phantom Deletes could occur.

2.9.2.1.3. Database

All transactions must take the database from one consistent state to another. All transactions must have an isolation level of Strict READ_COMMITTED or higher; i.e., dirty reads and phantom deletes are not allowed.

If an entity is deployed with a logical isolation of Strict READ_COMMITTED, and if that entity is not changed in a given Database Transaction, then the Java EE Server must not issue a database update that would have the effect of losing any external updates that are applied while the Database Transaction is executing. If the Java EE Web Profile Application Server does not have the ability to suppress unnecessary updates that could interfere with external updates, then all entities must be deployed using the Strict REPEATABLE_READ isolation level (or higher).

On an entity with an isolation level of Strict REPEATABLE_READ, optimizations to avoid database updates to an entity that has not been changed in a given transaction are not valid if the suppression of updates result in an effective isolation level lower than Strict REPEATABLE_READ. Additionally, if the Java EE Server pre-loads entity state while executing finder methods (to avoid re-selecting the data), the mechanism normally used to ensure Strict REPEATABLE_READ must still be effective, unless another mechanism is provided to ensure Strict REPEATABLE_READ in this case. For example, if SELECT FOR UPDATE would normally be used at select time, then SELECT FOR UPDATE should be used when executing those finder methods which pre-load entity state.

2.9.3. Durability Requirements

Transactions must be durable from any single point of failure on the SUT.

Durability from a single point of failure can be achieved by ensuring that the database and application server transactional logs can withstand failure at a single point. This is typically achieved by mirroring the logs onto separate non volatile storage.

Using cached disk for transactional logs is allowed as long as the disk device has a battery backup capable of meeting the 24 hour run requirement consistent with section 2.8.1.

Configurations where the logs and mirror devices are housed in the same disk array are accepted as being durable as long as all other durability criteria are met.

The word "disk" is used to represent non volatile storage, these run rules apply equally to other non volatile storage such as flash devices.

2.10. System Under Test (SUT) Requirements

The SUT comprises all components which are being tested. This includes network connections, Java EE Web Profile Application Servers, Database Servers, etc.

2.10.1. SUT Components

The SUT consists of:

  • The host system(s) (including hardware and software) required to support the Workload.

  • All network components (hardware and software) on the host system(s) required to run the Workload are considered part of the SUT.

  • Components which provide load balancing within the SUT.

  • Any software that is required to build and deploy the SPECjEnterprise®2018 Web Profile Application.

Comment : A basic configuration consisting of one or more switches between the Driver and the SUT is not considered part of the SUT. However, if any software/hardware is used to influence the flow of traffic beyond basic IP routing and switching, it is considered part of the SUT. For example, if DNS Round Robin is used to implement load balancing, the DNS server is considered part of the SUT and therefore it must not run on a driver client.

2.10.2. Database Services

The SUT services HTTP and REST requests from the Driver and returns results generated by the SPECjEnterprise®2018 Web Profile Application which may involve information retrieval from a database. The database must be accessible via external applications.

2.10.3. Storage

The SUT must have sufficient on-line disk storage to support any expanding system files and the durable database population resulting from executing the SPECjEnterprise®2018 Web Profile Business Transaction mix for 24 (twenty four) hours at the reported "SPECjEnterprise®2018 WebjOPS".

2.11. Auditing

To ensure that SPECjEnterprise®2018 Web Profile results are correctly obtained and requirements are met, the driver will make explicit audit checks by calling auditing components on the SUT. These audit checks are necessary and all tests must pass but they are not sufficient to ensure that the benchmark run is in compliance with these run and reporting rules. Passing the audit checks does not guarantee compliance with all of the run and reporting rules.

These tests are designed and intended to run both prior to ramp-up and also to run immediately following the end of the ramp-down period and it is a requirement that no mechanism is used to delay the start or end of the auditing checks. The driver will include audit results with the run results. The table below lists the individual auditing activities the driver performs, the pass/fail criteria, and the specific purpose of a particular audit:

TABLE 3 : Audit List

Audit Purpose Criteria

Benchmark setup

Ensures valid run properties and secured connection.

Run properties and connections are set according to run rules for a compliant run

Initial auditing counts

Ensures that all the database tables have been loaded with the expected amount of data

See Database Scaling Requirements

Atomicity Tests

Ensures atomicity requirements as documented in Atomicity Requirements

See Atomicity Tests

Persistence and Correctness Tests

Ensure that entities are stored correctly

See JPA Write Through Test and Phantom Delete Test

Final auditing counts

Ensures that the database tables contain all the new entities, updates and deletes performed by the driver

The amount of data in the database tables is within +/- 1% of what is expected.

Operations audit

Ensures that there is only a small amount of requests that fail

Operations have a success rate of at least 99%


3. Reporting Results

3.1. SPECjEnterprise®2018 Performance Metric

The primary metric of the SPECjEnterprise®2018 Web Profile benchmark is jEnterprise Web Operations Per Second ("SPECjEnterprise®2018 WebjOPS").

The primary metric for the SPECjEnterprise®2018 Web Profile benchmark is calculated by the number of Insurance Transactions / sec:

SPECjEnterprise(R)2018 WebjOPS = Insurance Transactions/sec

All reported "SPECjEnterprise®2018 WebjOPS" must be measured, rather than estimated, and can be rounded to the nearest whole number. For example, if a measurement yielded "123.55 SPECjEnterprise®2018 WebjOPS", this can be rounded to "124 SPECjEnterprise®2018 WebjOPS".

3.2. Required Reporting

A graph of the transaction throughput versus elapsed time must be reported for Insurance applications for the entire test run. The x-axis represents the elapsed time from the start of the run. The y-axis represents the throughput in Business Transactions. At least 150 different intervals must be used with a maximum interval size of 30 seconds. The start and end of the Measurement Interval must also be reported. An example of such a graph is shown below.

image

3.3. Benchmark Optimization Rules

Benchmark specific optimization is not allowed. Any optimization of either the configuration or products used on the SUT must improve performance for a larger class of workloads than that defined by this benchmark and must be supported by the provider. Optimizations must have the vendor’s endorsement, be supported and should be suitable for production use in an environment comparable to the one represented by the benchmark. Optimizations that take advantage of the benchmark’s specific features are forbidden.

An example of an inappropriate optimization is one that requires access to the source code of the application.

Comment : The intent of this section is to encourage optimizations to be done automatically by the products.

3.4. General Availability

All hardware and software used must be possible to order by customers. For any product not already generally released, the Submission File must include a committed general delivery date. That date must be within 3 months of the result’s publication date. However, if Java and/or Java EE Web Profile related licensing issues cause a change in software availability date after publication date, the change will be allowed to be made without penalty, subject to SPEC® review.

The purpose of including general availability requirements is to ensure that the systems and their hardware and software components actually represent real products that solve real business and computational problems. Detailed Guidelines for General Availability are provided in SPEC® Open Systems Group Policy Appendix C.

All products used must be the proposed final versions and not prototypes. When the product is finally released, the product performance must not decrease by more than 5% of the published "SPECjEnterprise®2018 WebjOPS". If the submitter later finds the performance of the released system to be more than 5% lower than that reported for the pre-release system, then the submitter is required to report a corrected test result.

The intent is to test products that customers will use, not prototypes. Beta versions of products can be used, provided that General Availability (GA) of the final product is within 3 months. If a beta version is used, the date reported in the results must be the GA date.

The 5% degradation limit only applies to a difference in performance between the tested product and the GA product. Subsequent GA releases (to fix bugs, etc.) are not subject to this restriction.

3.5. Fair Use of Results

In order to publicly disclose SPECjEnterprise®2018 results, the submitter must adhere to these fair use and reporting rules in addition to having followed the run rules described in this document.

SPECjEnterprise®2018 Web Profile Results must be reviewed and accepted by SPEC® prior to public disclosure.

Any public use of results from this benchmark must also follow the SPEC® Fair Use Rules and those specific to this benchmark.

3.5.1. Result Disclosure and Submission

Compliant runs need to be submitted to SPEC® for review and must be accepted prior to public disclosure. Submissions must include the Submission File, a Configuration Diagram, and the Full Disclosure Archive for the run (see Full Disclosure).

The goal of the reporting rules is to ensure the system under test is sufficiently documented such that someone could reproduce the test and its results.

See section 5 of the SPECjEnterprise®2018 Web Profile User’s Guide for details on submitting results to SPEC®.

Public disclosure of compliant SPECjEnterprise®2018 results reviewed and accepted for publication by SPEC® must always be quoted using the performance metric "SPECjEnterprise®2018 WebjOPS".

Test results that have not been accepted and published by SPEC® must not be publicly disclosed except as noted in Section 3.6, Research and Academic Usage. Research and academic usage test results that have not been accepted and published by SPEC® must not use the SPECjEnterprise®2018 Web Profile metric ("SPECjEnterprise®2018 WebjOPS").

3.5.2. Estimates

Estimates are not allowed.

3.5.3. Comparison to Other Benchmarks

SPECjEnterprise®2018 Web Profile results must not be publicly compared to results from any other benchmark.

3.6. Research and Academic Usage

SPEC® encourages use of the SPECjEnterprise®2018 Web Profile benchmark in academic and research environments. The researcher is responsible for compliance with the terms of any underlying licenses (Java EE Web Profile Application Server, Database Server, hardware, etc.).

3.6.1. Research and Academic Usage - Restrictions

It is understood that experiments in such environments may be conducted in a less formal fashion than that demanded of licensees submitting to the SPEC® web site. SPEC® encourages researchers to obey as many of the run rules as practical, even for informal research.

If research results are being published, SPEC® requires:

  • The publication must not use the SPECjEnterprise®2018 Web Profile metrics ("SPECjEnterprise®2018 WebjOPS") unless the result has been reviewed and accepted by SPEC®. Renaming or re-computation of the primary metrics is also not allowed.

  • The publication must clearly document any deviations from the SPECjEnterprise®2018 Web Profile Run and Reporting Rules.

  • The research results must not be compared with the results published on the SPEC® web site.

  • The research results must not be compared with any other benchmark results.

  • In any publication where results will be compared between two competing products, SPEC® expects that the Fair Use Guidelines be followed with regard to any claims made (see Fair Use).

  • If this project is sponsored/paid for by a third party, this must be disclosed.

SPEC® reserves the right to require a full disclosure of any published results.

3.6.2. Research and Academic Usage - Disclosure

Public use of SPECjEnterprise®2018 Web Profile benchmark results are bound by the SPEC® Fair Use Rules and the SPECjEnterprise®2018 Web Profile specific Run and Reporting Rules (this document). All publications must clearly state that these results have not been reviewed or accepted by SPEC® using text equivalent to this:

SPECjEnterprise® is a registered trademark of the Standard Performance Evaluation Corp. (SPEC®). The SPECjEnterprise®2018 Web Profile results or findings in this publication have not been reviewed or accepted by SPEC®, therefore no comparison nor performance inference can be made against any published SPEC® result. The official web site for the SPECjEnterprise®2018 Web Profile benchmark is located at http://www.spec.org/jEnterprise2018web.

This disclosure must precede any results quoted from the tests. It must be displayed in the same font as the results being quoted.


4. Full Disclosure

A Full Disclosure is required in order for results to be compliant with the SPECjEnterprise®2018 Run and Reporting Rules. For the SPECjEnterprise®2018 Web Profile benchmark, Full Disclosure includes the Submission File, Full Disclosure Archive ( FDA ), and Configuration Diagram. The requirements for each of these are described in this section.

A Full Disclosure Report ( FDR ) is generated from the Submission File. The FDR is the format used to display results on the SPEC® web site. The FDR in HTML format includes links to the Configuration Diagram and the Full Disclosure Archive ( FDA ).

Comment : The intent of this disclosure is to be able to replicate the results of a submission of this benchmark given the equivalent hardware, software, and documentation.

In the sections below, when there is no specific reference to where the disclosure must occur, it must occur in the Submission File. Disclosures in the Archive are explicitly called out.

4.1. Submission File

The Submission File is a text file containing the information specified in this section.

An example of a Submission File is included in the kit, see "reporter/sample/webprofile/output.submission.properties". The Submission File is generated by the Reporter using the Submission Template and the results of a compliant run. An example of the Submission Template is included in the kit, see "reporter/sample/jEnterprise.webprofile.properties".

The Results Section of the compliant run included in the Submission File must not be modified.

4.1.1. Software

4.1.1.1. Software Products

All commercially available software products must be identified in the system.sw.* sections of the Submission File, along with the product name and version, vendor, and availability date. These products include, but are not limited to:

  • Java EE Web Profile Application Servers

  • JVM products

  • Database management systems

  • Database driver products

Additional information required for specific product types are specified in the subsections below.

4.1.1.1.1. Java EE Web Profile Application Servers

In addition to the standard information required for all software products, additional information on the Java EE servers is required in the system.sw.JEE.product[#].* sections as follows:

  • Date the Java EE application server passed the Compatibility Test Suite ( CTS ) must be disclosed in the system.sw.JEE.product[#].date_passed_CTS field.

  • The CTS version this product is certified on must be disclosed in the system.sw.JEE.product[#].CTS_version field. Allowed versions for the SPECjEnterprise®2018 Web Profile benchmark are 7.0 or later. Minor versions of these CTS tests are allowed.

4.1.1.1.2. JVMs

The submitter must disclose the Operating System and hardware architecture this JVM is built for. In addition, 64-bit or 32-bit version for the JVM must be indicated. If the same binary image is used for many Operating Systems, one product may be declared with the system.sw.JVM.product[#].os field set to the architecture the binary image is built for.

4.1.1.1.3. Other software

All other software must provide a description how it is used in the benchmark. The description must be provided in the system.sw.other.product[#].description field.

4.1.1.2. Software Instances

All information on software instance configurations must be disclosed in system.sw sections of the Submission File. If multiple instances use exactly the same configuration and are used for exactly the same purpose, they may be listed as one single configuration. The hardware type running this instance configuration and the total number of instances must be disclosed in the system.sw.<type>.config[#].hw_type and system.sw.<type>.config[#].instances sections.

4.1.1.2.1. Java EE Web Profile Application Server Instances in the SUT

Product references and configuration and tuning information must be disclosed in the system.sw.JEE.config[#] sections of the Submission File. These include references to the product sections for the Java EE Web Profile Application Server itself and all other products used inside the Java EE Server, including, but not limited to the JVM, the Database driver(s), and any other products that are not part of the Java EE Web Profile Application Server product.

Moreover, the modules that this instance configuration deploys and runs must be disclosed in the system.sw.JEE[#].web.* and system.sw.JEE[#].rest.* sections by marking the modules as true or false.

4.1.1.2.2. Database Server Instances

The Database Server products, instance configurations, and database tuning information necessary to recreate the results must be disclosed in the system.sw.DB.config[#] sections of the Submission File.

4.1.1.2.3. Driver Instances

Each driver agent must be listed in the system.sw.driver.config[#] sections of the Submission File and all tuning on each agent must be documented in the tuning fields of this section.

JVM instances and tuning used for the drivers, the RMI registry, the controller, and the driver must be disclosed in the system.sw.notes section of the Submission File.

4.1.1.2.4. Other Software Instances

Software not used as part of the Java EE Web Profile Application Server instances in the SUT but needed on the system, except the operating system itself, must be listed in the sections system.sw.other.config[#].* . A reference to the software product and instance tuning information must be disclosed as part of these sections.

4.1.1.3. Miscellaneous Disclosures

In addition to product information and instance configuration information, certain information about the software deployment, configuration, and tuning must be provided as listed in the following subsections.

4.1.1.3.1. Database Character Set

The character set used to store character strings in the database must be disclosed. For example, ASCII, ISOLATIN1 (ISO8859-1), UTF-8, UTF-16.

4.1.1.3.2. Injection Rate

The Injection Rate used to load the database(s) must be disclosed in the benchmark.load.injection_rate section of the Submission File.

4.1.1.3.3. Schema Modifications

If the schema was changed from the reference one provided in the Kit (see the Database Requirements), the reason for the modifications must be disclosed in the benchmark.schema_modifications section of the Submission File.

4.1.1.3.4. Load Program Modifications

If the load program was changed from the reference on provided in the Kit, the reason for the modifications must be disclosed in the benchmark.load_program_modifications section of the Submission File.

4.1.1.3.5. Isolation Requirements

The method used to meet the isolation requirements in Consistency and Isolation Requirements must be disclosed in the benchmark.isolation_requirement_info section of the Submission File.

4.1.1.3.6. Durability Requirements

The method used to meet the Durability Requirements must be disclosed in the benchmark.durability_requirement_info section of the Submission File.

4.1.1.3.7. Explanation of Errors

Any errors that appear in the Driver error logs must be explained in the notes section of the Submission File.

4.1.2. Hardware

4.1.2.1. Hardware Description

The number and types of systems used must be disclosed in the system.hw[#] section of the Submission File. The following information is required for each system configuration:

  • Label (a free text description of the purpose of the hardware)

  • Whether this hardware configuration is part of the SUT or not; driver configuration must have this field set to "false."

  • Vendor and model number

  • System availability date

  • Operating system (product name, vendor, and availability date)

  • CPU (processor type, number and speed (MHz/GHz) of the CPUs)

  • Cache (L1, L2, and "other")

  • Memory Amount (GB)

  • # and size of DIMMS

  • Memory Details - any configuration details that may affect performance, e.g. interleaving and access time.

  • Disks and file system used

  • Network interface

  • All software configurations and instances running on this hardware

  • Number of systems with this exact same configuration

Note: the system availability date, # and size of DIMMS, and memory details are not required for systems which are not part of the SUT.

4.1.2.2. Storage Requirements

The method used to meet the storage requirements of Storage must be disclosed in the benchmark.storage_requirement_info section of the Submission File.

4.1.3. Network

4.1.3.1. Network Optimization

If any software/hardware is used to influence the flow of network traffic beyond basic IP routing and switching, the additional software/hardware and settings must be disclosed in the benchmark.other section of the Submission File.

4.1.3.2. Network Bandwidth

The bandwidth of the network used in the tested configuration must be disclosed in the benchmark.other section of the Submission File.

4.1.3.3. Load Balancing

The hardware and software used to perform load balancing must be disclosed in the benchmark.other section of the Submission File. If the driver systems perform any load-balancing functions as defined in the Driver Rules, the details of these functions must also be disclosed.

4.1.4. Benchmark Run Results

4.1.4.1. Benchmark Version

The version number of the SPECjEnterprise®2018 Web Profile Kit used to run the benchmark must be included in the Submission File.

4.1.4.2. Reproducibility Run "SPECjEnterprise®2018 WebjOPS"

The "SPECjEnterprise®2018 WebjOPS" from the reproducibility run (see Reproducibility) must be disclosed in the result.reproducibility_run.metric field of the Submission File.

4.1.5. Bill of Materials (BOM)

The Bill of Materials, which contains the hardware and software used in the SUT, must be disclosed in the benchmark.bom section of the Submission File.

The intent of the BOM is to enable a reviewer to confirm that the tested configuration satisfies the run rule requirements and to document the components used with sufficient detail to enable a customer to reproduce the tested configuration and obtain pricing information from the supplying vendors for each component of the SUT.

4.1.5.1. BOM Suppliers

The suppliers for all components must be disclosed. All items supplied by a third party (i.e. not the Test Submitter) must be explicitly stated. Each third party supplier’s items must be listed separately.

4.1.5.2. BOM Level of Detail

The Bill of Materials must reflect the level of detail a customer would see on an itemized bill (that is, it should list individual items in the SUT that are not part of a standard package).

For each item, the BOM should include the item’s supplier, description, the item’s ID (the code used by the vendor when ordering the item), and the quantity of that item in the SUT.

4.1.5.3. BOM Hardware Component Substitution

For ease of benchmarking, the BOM may include hardware components that are different from the tested system, as long as the substituted components perform equivalently or better in the benchmark. Any substitutions must be disclosed in the BOM. For example, disk drives with lower capacity or speed in the tested system can be replaced by faster ones in the BOM. However, it is not permissible to replace key components such as CPU, memory or any software.

4.1.5.4. BOM SUT

All components of the SUT (see SUT Components) must be included, including all hardware, software, and support for a three year period.

All hardware components included must be new and not reconditioned or previously owned. The software may use term limited licenses (i.e., software leasing), provided there are no additional conditions associated with the term limited licensing. If term limited licensing is used, the licensing must be for a minimum of three years. The three year support period must cover both hardware maintenance and software support.

The number of users for SPECjEnterprise®2018 Web Profile is 10 * IR Internet users. Any usage based licensing for the above number of users should be based on the licensing policy of the company supplying the licensed component.

4.1.5.5. BOM Additional Components

Additional components such as operator consoles and backup devices must also be included if explicitly required for the installation, operation, administration, or maintenance, of the SUT.

If software needs to be loaded from a particular device either during installation or for updates, the device must be included.

4.1.5.6. BOM Support

Hardware maintenance and software support must include 7 days/week, 24 hours/day coverage, either on-site, or if available as standard offering, via a central support facility.

If a central support facility is utilized, then all hardware and software required to connect to the central support must be installed and functional on the SUT during the measurement run and included.

The response time for hardware maintenance requests must not exceed 4 hours on any component whose replacement is necessary for the SUT to return to the tested configuration.

The use of spares in lieu of the hardware maintenance requirements is allowed if the part to be replaced can be identified as having failed by the customer within 4 hours. An additional 10% of the designated part, with a minimum of 2, must be included. A support service for the spares which provides replacement on-site within 7 days must also be included for the support period.

Comment : The use of spares is intended to assist in complying with the 4-hour maximum hardware maintenance response requirement. It cannot be a substitute for maintenance support, as the configuration documented in the BOM must maintain the same quantities of components, including spares, for 3 years.

Software support requests must include problem acknowledgment within 4 hours.

Comment : Customers may have to flag the request with the appropriate severity to ensure acknowledgement within 4 hours.

No additional charges will be incurred for the resolution of software defects. Problem resolution for more than 10 non-defect problems per year is permitted to incur additional charges. Software support must include all available maintenance updates over the support period.

4.2. Full Disclosure Archive

The Full Disclosure Archive (FDA) must be in ZIP format.

4.2.1. Database Configuration

4.2.1.1. Database Logical Volume Configuration Files and Scripts

All scripts/programs and configuration files used to create any logical volumes for the database devices must be included as part of the FDA. The distribution of tables and transaction logs across all media must be explicitly depicted.

4.2.1.2. Database Table/Index Configuration Files and Scripts

All table definition statements and all other statements used to set-up the database must be disclosed as part of the FDA. The configuration files or scripts used to create the database should be included in the "Schema" sub-directory.

4.2.1.3. Database Load Program

If the load programs in the SPECjEnterprise®2018 Web Profile kit were modified (see the Database Requirements), all such modifications must be disclosed in the benchmark.load_program_modifications section of the Submission File and the modified programs must be included in the FDA.

The steps and supporting files (configuration files, scripts, etc) required to configure and populate the database to the state necessary to reproduce the results must be included in the Full Disclosure Archive (FDA). To keep the size of the FDA manageable, data originally loaded by the supplied load program does not need to be included in the FDA.

4.2.2. Benchmark Configuration

4.2.2.1. Deployment Descriptors

All deployment descriptors used must be included in the FDA under the "Deploy" sub-directory. The deployed WAR files must also be in the "Deploy" directory.

4.2.2.2. Driver

The input parameters to the Driver must be disclosed by including the following files used to run the benchmark in the FDA:

  • faban/driver.properties file

  • gradle.properties file

If the Driver package was modified, its source must be included in the FDA.

4.2.3. Benchmark Run Results

4.2.3.1. Final Run Output Directory

The entire output directory from the run must be included in the FDA under the "FinalRun" sub-directory.

4.2.3.2. Reproducibility Run Output Directory

The entire output directory from the reproducibility run (see Reproducibility) must be included in the FDA under the "RepeatRun" sub-directory.

4.2.3.3. Insurance Throughput Graph

A graph, in PNG, JPEG or GIF format, of the insurance throughput versus elapsed time (see Required Reporting) must be included in the FDA under the "FinalRun" sub-directory.

4.3. Configuration Diagram

A Configuration Diagram of the entire configuration (including the SUT and load drivers) must be provided in PNG, JPEG or GIF format. The diagram should include, but is not limited to:

  • Number and type of processors.

  • Number of LAN (e.g., Ethernet) connections, including routers, etc., that were physically used in the benchmark run.

  • The type and the run-time execution location of software components (e.g., Java EE Web Profile Application Server, DBMS, benchmark driver processes, software load balancers, etc.).

  • A clear indication of which components are part of the SUT.


Appendix A: Isolation Level Definitions

A.1. Isolation Level Phenomena

The various isolation levels are described in ANSI SQL and Java SE documentation for java.sql.Connection. ANSI SQL defines isolation levels in terms of phenomena (P1, P2, P3) that are or are not possible under a specific isolation level. The interpretations of P1, P2, P3 and PD used in this benchmark are as follows:

P1 (Dirty read)

Transaction T1 modifies a row. Another transaction T2 then reads that row and obtains the modified value, before T1 has completed a COMMIT or ROLLBACK. Transaction T2 eventually commits successfully; it does not matter whether T1 commits or rolls back and whether it does so before or after T2 commits.
Note : a transaction "obtains the modified value" of a row if the modified value is used inside the Database Server during "selection" (predicate evaluation, i.e. evaluation of the "where" clause) or if the value is used during "projection" (i.e. determining the column list to be returned to the application).

P2 (Non-repeatable read)

Transaction T1 reads a row. Another transaction T2 then modifies or deletes that row, before T1 has completed a COMMIT. Both transactions eventually commit successfully.

P3 (Phantom read)

Transaction T1 reads the set of rows N that satisfy some search condition. Transaction T2 then generates one or more rows that satisfy the search condition used by T1, before T1 has completed a COMMIT. Both transactions eventually commit successfully.

PD (Phantom delete)

Transaction T1 deletes one or more rows. Before T1 commits or rolls back, another transaction T2 then reads the set of rows N that satisfy some search condition, but the deleted rows are excluded from N even if they would have satisfied the search condition. Transaction T2 eventually commits successfully; it does not matter whether T1 commits or rolls back and whether it does so before or after T2 commits.

A.2. Strict READ_COMMITTED

An isolation level of Strict READ_COMMITTED is defined to disallow P1 and PD but allow P2 and P3.

A.3. Strict REPEATABLE_READ

An isolation level of Strict REPEATABLE_READ is defined to disallow P1, PD and P2 but allow P3.

The ANSI SQL definition for REPEATABLE_READ disallows P2. Disallowing P2 also disallows the anomaly known as Read Skew. Read Skew arises in situations such as the following:

  • there is an integrity constraint between X and Y

  • transaction T1 reads X

  • transaction T2 modifies X and Y to new values and commits successfully before T1 has completed

  • T1 now reads Y and sees a state that violates the constraint between X and Y

  • T1 eventually commits successfully

If P2 is disallowed, transaction T2 upon trying to modify X, would either be blocked until T1 completes OR it would be allowed to proceed but transaction T1 would eventually have to be rolled back.

Disallowing P2 also disallows the anomaly known as Write Skew. Write Skew arises in situations such as the following:

  • there is an integrity constraint between X and Y

  • transaction T1 reads X and Y

  • transaction T2 reads X and Y, writes X, and commits successfully

  • transaction T1 then writes Y and also commits successfully

If P2 is disallowed, either T1 or T2 would have to be rolled back.


Product and service names mentioned herein may be the trademarks of their respective owners.

Copyright © 2001-2018 Standard Performance Evaluation Corporation
All Rights Reserved