Table of Contents


Section 1 - Introduction

1.1 Goals of SPECjms2007

The major goal of the SPECjms2007 benchmark is to provide a standard workload and metrics for measuring and evaluating the performance and scalability of JMS-based Message-Oriented Middleware (MOM) platforms. In addition, the benchmark should provide a flexible framework for JMS performance analysis. To achieve this goal, the SPECjms2007 workload must meet several important requirements:

Representativeness

No matter how well a benchmark is designed, it would be of little value if the workload it is based on does not reflect the way platform services are exercised in real-life systems. Therefore, the most important requirement for the SPECjms2007 benchmark is that it is based on a representative workload scenario including a representative set of interactions, message types, message sizes and message delivery modes. The communication style and the types of messages sent and received by the different parties in the scenario should represent a typical transaction mix. The goal is to allow users to relate the observed behavior to their own applications and environments.

Comprehensiveness

The second important requirement is that the workload is comprehensive in that it should exercise all platform features typically used in the major classes of JMS applications including both point-to-point (P2P) and publish/subscribe (pub/sub) messaging. The features and services stressed should be weighted according to their usage in real-life systems. There is no need to cover features of MOM platforms that are used very rarely in practice.

The following dimensions have to be considered when defining the workload transaction mix:

Focus

The workload should be focused on measuring the performance and scalability of the JMS Server's software and hardware components. It should minimize the impact of other components and services that are typically used in the chosen application scenario. For example, if a database would be used to store business data and manage the application state, it could easily become the limiting factor of the benchmark. This is especially true in the case of SPECjms2007, since JMS Servers, in their role as mediators in interactions, are typically less loaded than database or application servers. Another potential concern is the Client side of the benchmark where messages are sent and received. The impact of Client-side operations, such as XML parsing, on the overall performance of the benchmark should be minimized.

Configurability

As mentioned above, in addition to providing standard workload and metrics for JMS performance, SPECjms2007 aims to provide a flexible performance analysis framework which allows users to configure and customize the workload according to their requirements. Producing and publishing standard results for marketing purposes is just one usage scenario for SPECjms2007. Many users would be interested in using the benchmark to tune and optimize their platforms or to analyze the performance of certain specific MOM features. Others could use the benchmark for research purposes in academic environments where, for example, one might be interested in evaluating the performance and scalability of novel methods and techniques for building high-performance MOM servers. All these usage scenarios require that the benchmark framework allows the user to precisely configure the workload and transaction mix to be generated. Providing this configurability is a great challenge because it requires that interactions are designed and implemented in such a way that one could run them in different combinations depending on the desired transaction mix. The ability to switch interactions off implies that interactions should be decoupled from one another. On the other hand, it should be ensured that the benchmark, when run in its standard mode, behaves as if the interactions were interrelated according to their dependencies in the real-life application scenario.

Scalability

The SPECjms2007 application scenario must provide a way to scale the workload along the following two dimensions:

In the Horizontal scaling, the workload is scaled by increasing the number of Destinations (queues and topics) while keeping the traffic per Destination constant. In the Vertical scaling, the traffic (in terms of message count) pushed through a Destination is increased while keeping the number of Destinations fixed. Both types of scaling should be supported in a manner that preserves the relation to the real-life business scenario modeled. In addition, the user should be offered the possibility to scale the workload in an arbitrary manner by increasing the number of Destinations (queues and topics) and/or the message traffic pushed through a Destination.

1.2 Business and Application Scenario

The application scenario chosen for SPECjms2007 models the supply chain of a supermarket company. The participants involved are the supermarket company, its stores, its distribution centers and its suppliers. The scenario was defined based on the requirements discussed in the previous section. It offers an excellent basis for defining interactions that stress different subsets of the functionality offered by JMS Servers, e.g. different message types as well as both P2P and pub/sub communication. The scenario also offers a natural way to scale the workload, e.g. by scaling the number of Supermarkets (Horizontal) or by scaling the amount of products sold per Supermarket (Vertical).

Overview
FIGURE 1: SPECjms2007 Supermarket Supply Chain

The participants involved in the scenario can be grouped into the following four roles:

  1. Headquarters
  2. Distribution Centers
  3. Supermarkets
  4. Suppliers

The first three roles are owned by the supermarket company and therefore all communication among them is intracompany. The Suppliers are external companies and therefore their communication with the roles of the supermarket company is intercompany. The interactions among the different roles are illustrated in Figure 1.

1.2.1 Scenario Roles

Company Headquarters (HQ)

The company's corporate Headquarters are responsible for managing the accounting of the company, managing information about the goods and products offered in the supermarket stores, managing selling prices and monitoring the flow of goods and money in the supply chain.

Distribution Centers (DCs)

The Distribution Centers supply the Supermarket stores which sell goods to end customers. Every Distribution Center is responsible for a set of stores in a given area. The Distribution Centers in turn are supplied by external Suppliers.

The Distribution Centers are involved in the following activities:

Supermarkets (SMs)

The Supermarkets sell goods to end customers. The scenario focuses on the management of the inventory of Supermarkets including their warehouses. Some Supermarkets are smaller than others, so that they do not have enough room for all products, others may be specialized for some product groups like certain types of food. We assume that every Supermarket is supplied by exactly one of the Distribution Centers.

Suppliers (SPs)

The Suppliers deliver goods to Distribution Centers of the Supermarket company. Not every Supplier offers the same products. Instead, the Suppliers have their own product catalogues. They deliver goods on demand, i.e. they must receive an order from the Supermarket company to send a shipment. To keep things simple, it is assumed that each SP offers either all products of a given product family or none of them.

1.2.2 Interactions Modeled

The following Interactions are part of the scenario:

  1. Order/shipment handling between Supermarkets and Distribution Centers.
  2. Purchase order/shipment handling between Distribution Centers and Suppliers.
  3. Price updates sent by Headquarters to the Supermarkets.
  4. Supermarket inventory management.
  5. Sales statistics collection.
  6. New product announcements.
  7. Distribution of credit card hot lists.

In the next section, we take an in-depth look at these Interactions and show how they are implemented in SPECjms2007.

Section 2 - Application Design

2.1 Definition of Terms

The SPECjms2007 Kit refers to the complete kit provided for running the SPECjms2007 benchmark. The SPECjms2007 Kit includes all documentation, source and compiled binaries for the benchmark.

The Provider Module refers to an implementation of the org.spec.perfharness.jms.providers.JMSProvider interface. A default, provider-independent implementation that uses JNDI is included in the SPECjms2007 Kit. A product-specific Provider Module may be required to run the benchmark on some JMS products.

The JMS Server or Server refers to the pieces of hardware and software that provide the JMS facilities to JMS Clients. It may be comprised of multiple hardware and/or software components and is viewed as a single logical entity by JMS Clients. The Server also includes all the stable storage for persistence as required by the JMS Specification.

The JMS Clients or Clients refer to Java application components that use the JMS API. The SPECjms2007 benchmark is a collection of Clients in addition to other components required for benchmark operation.

Destination refers to a JMS destination, which is either a queue or a topic.

Location refers to a single logical entity in the benchmark application scenario. The four entities defined in the SPECjms2007 benchmark are Supermarket (SM), Supplier (SP), Distribution Center (DC), and Headquarters (HQ).

A Topology is the configuration of Locations being used by a particular benchmark run. The SPECjms2007 benchmark has two controlled topologies, the Vertical Topology and the Horizontal Topology, that are used for Result Submissions. The Freeform Topology allows the user complete control over the benchmark configuration.

The BASE parameter is the fundamental measure of performance in the SPECjms2007 benchmark. It represents the throughput performance of a benchmark run in a particular Topology. Each benchmark Topology uses a different metric to report the result and the value of the metric, which is a measure of the SUT performance in that Topology, is the BASE parameter. In the Vertical Topology, the metric is called SPECjms2007@Vertical and in the Horizontal Topology, the metric is called SPECjms2007@Horizontal. In other words, the BASE parameter is the result achieved for a particular Topology. The BASE parameter can not be compared across the different Topologies, i.e. a result achieved in the Horizontal Topology can not be compared with a result achieved in the Vertical Topology.

An Interaction is a defined flow of messages between one or more Locations. The Interaction is a complete message exchange that accomplishes a business operation. SPECjms2007 defines seven (7) Interactions of varying complexity and length.

A Flow Step is a single step of an Interaction. An Interaction therefore is comprised of multiple Flow Steps.

An Event Handler (EH) is a Java Thread that performs the messaging logic of a single Flow Step. A Flow Step may use multiple Event Handlers to accomplish all of its messaging as required by the benchmark. In relation to the SPECjms2007 benchmark, the Clients are the Event Handlers, which are the components in which all JMS operations are performed.

A Driver (DR) is an Event Handler that does not receive any JMS messages. Drivers are those Event Handlers that initiate Interactions by producing JMS messages. Although they are only the initiators in the JMS message flow and not JMS message receivers, they are included collectively under Event Handlers as they are modelling the handling of other, non-JMS business events (e.g. RFID events) in the benchmark application scenario.

The System Under Test (SUT) is comprised of all hardware and software components that are being tested. The SUT includes all the Server machines and Client machines as well as all hardware and software needed by the Server and Clients.

An Agent is a collection of Event Handlers (EHs which includes DRs) that are associated with a Location. A Location can be represented by multiple Agents.

An AgentJVM is a collection of Agents that run in a single Java Virtual Machine (JVM).

The Controller (also referred to as the ControlDriver) is the benchmark component that drives the SPECjms2007 benchmark. There is exactly one Controller when the benchmark is run. The Controller reads in all of the configuration, instantiates the Topology (the Locations to use and connections between them), monitors progress, coordinates phase changes and collects statistics from all the components.

A Satellite (also referred to as a SatelliteDriver) is the benchmark component that runs AgentJVMs and is controlled by the Controller.

The Framework is the collective term used for the Controller, Satellite and benchmark coordination classes.

A Node is a machine in the SUT. The four kinds of nodes are, server nodes, client nodes, db nodes, and other nodes. A server-node is any node that runs the JMS Provider's server software. The client-nodes run the benchmark components and must each run exactly one Satellite. If the JMS product is configured with a database which runs on machines that are separate from the ones that the JMS server software runs on, then these machines are referred to as database nodes. Any other machines that are needed for the JMS product operation that are not covered by the three nodes described above are included in other nodes.

The Delivery Time refers to the elapsed time measured between sending a specific message and that message being received.

Normal Operation refers to any time the product is running, or could reasonably be expected to run, without failure.

A Typical Failure is defined as a failure of an individual element (software or hardware) in the SUT. Some examples to qualify this include Operating System failure, interruption to electricity or networking or death of a single machine component (network, power, RAM, CPU, disk controller, or an individual disk, etc). It includes failure of the Server as well as Clients.

A Single-Point-of-Failure is defined as a single Typical Failure. It is not extended to cover simultaneous failures or larger scale destruction of resources.

Non-Volatile Storage refers to the mechanism by which persistent data is stored. Non-Volatile Storage must be online and immediately available for random access read/write. Archive storage (e.g. tape archives or other backups) does not qualify as it is not considered as online and immediately available.

The Measurement Period refers to length of time during which measurement of the performance of the SUT is made.

The Warmup Period refers to the period from the commencement of the benchmark run up to the start of the Measurement Period.

The Drain Period refers to the period from the end of the Measurement Period up to the end of the benchmark run.

The Final Version refers to the final version of a product that is made available to customers.

The Proposed Final Version refers to a version of a product, that while not a final version as defined above, is final enough to use to submit a benchmark result. Proposed final versions can include Beta versions of the product.

The General Availability (GA) date for a product is the date that customers can place an order for, or otherwise acquire the product.

The Run Result refers to the HTML file that is generated at the end of a benchmark run that indicates the pass/fail status of all the Interactions in the benchmark.

A Result Submission refers to a single JAR file containing a set of files describing a SPEC benchmark result that is submitted to SPEC for review and publication. This includes a Submission File, a Configuration Diagram and a Full Disclosure Archive.

The Submission File refers to an XML-based document providing detailed information on the SUT components and benchmark configuration, as well as some selected information from the benchmark report files generated when running SPECjms2007. The Submission File is created using the SPECjms2007 Reporter.

The Configuration Diagram refers to a diagram in common graphics format that depicts the topology and configuration of the SUT for a benchmark result.

The Full Disclosure Archive (FDA) refers to a soft-copy archive of all relevant information and configuration files needed for reproducing the benchmark result.

The Full Disclosure Report (FDR) refers to the complete report that is generated from the Submission File. The FDR is the format in which an official benchmark result is reported on the SPEC Web site.

The SPECjms2007 Reporter refers to a standalone utility provided as part of the SPECjms2007 Kit that is used to prepare a Result Submission. The SPECjms2007 Reporter generates the Submission File and gathers all information pertaining to the Result Submission into a directory to be packaged and submitted to SPEC for review and publication. The SPECjms2007 Reporter also generates a preview of the Full Disclosure Report to be verified by the submitter before submitting the result.

2.2 Interaction Flows

2.2.1 Interaction 1 - Order/Shipment Handling between SM and DC

This Interaction models order and shipment handling between SMs and DCs. It exercises persistent P2P messaging.

The Interaction includes the following steps:

  1. A SM sends an order to its DC (order message).
  2. Step 2 contains three parts:
  3. The DC sends information about the transaction to the HQ (statInfoOrderDC message).
  4. A confirmation is sent to the DC (shipConf message).
Interaction 1
FIGURE 2: Interaction 1 - Communication between SM and DC

2.2.2 Interaction 2 - Purchase Order/Shipment Handling between DC and SP

When goods in a DC are depleted, the DC has to order from a SP to refill stock. First, a SP has to be selected. The DC sends a call for offers to all SPs that are supplying the respective types of goods. The SPs send offers to the DC, the DC selects one of them and places a purchase order based on the offer. The selected SP receives the order, delivers the goods to the DC and sends an invoice to the Supermarket HQ. After receiving the shipment, the DC acknowledges receipt and reports it to the HQ. The Interaction uses both P2P and pub/sub communication and is illustrated in Figure 3.

Interaction 2
FIGURE 3: Interaction 2 - Communication between SP and DC

Interaction 2 includes the following steps:

  1. A DC sends a call for offers to all SPs that supply the types of goods that need to be ordered (callForOffers message).
  2. SPs that can deliver the goods send offers to the DC (offer messages).
  3. Based on the offers, the DC selects a SP and sends a purchase order to it (pOrder message).
  4. Two parts:
  5. The DC sends a delivery confirmation to the SP (pShipConf message).
  6. The DC sends information about the transaction to the HQ (statInfoShipDC message).

Every Supplier offers products of exactly m product families. For every prodcut family of the n product families exists one topic.

2.2.3 Interaction 3 - Price Updates

Selling prices are changed by the company administration from time to time. To communicate this, the company HQ sends messages (priceUpdate) with pricing information to the Supermarkets. The communication here is one-to-many and is based on pub/sub messaging.

2.2.4 Interaction 4 - SM Inventory Management

This Interaction exercises persistent P2P messaging inside the SMs. It is triggered when goods leave the warehouse of a Supermarket to refill a shelf. This is registered by RFID readers and a message (inventoryInfo) is sent to the local information system to update inventory. Note that since incoming goods are part of another Interaction (i.e., Interaction 1), they are not considered here.

2.2.5 Interaction 5 - Sales Statistics Collection

This Interaction exercises non-persistent P2P messaging between the SMs and the HQ. It is triggered when SMs send sales statistics (statInfoSM messages) to the HQ. HQ can use this data as a basis for data mining in order to study customer behavior and provide useful information to marketing. For example, based on such information, special offers or product discounts could be made.

2.2.6 Interaction 6 - New Product Announcements

New products are announced by the company administration from time to time. To communicate this, the company HQ send messages (productAnnouncement) with product information to the Supermarkets offering the respective product types. This communication is based on non-persistent, non-durable pub/sub messaging.

2.2.7 Interaction 7 - Credit Card Hot Lists

HQ send credit card hot lists (creditCardHL messages) to SMs. A complete list is sent once every hour and incremental updates as required. This Interaction is used to exercise non-durable, non-persistent pub/sub messaging.

2.3 Message Types and Destinations

2.3.1 Message Destinations

Table 1 shows the Destinations (queues and topics) used in the implementation of the seven Interactions. Destination names are prefixed with the Location type they belong to (SP, SM, HQ or DC) and suffixed with 'T' or 'Q' depending on the Destination type (topic or queue).

TABLE 1: Queues and topics used by Interactions
Interaction Location Destination JNDI Name Message
1 DC DC_OrderQ order
1 DC DC_ShipDepQ shipDep
1 DC DC_ShipConfQ shipConf
2 DC DC_IncomingOffersQ offers
2 DC DC_POrderConfQ pOrderConf
2 DC DC_PShipArrQ pShipment
1 SM SM_ShipArrQ shipInfo
1 SM SM_OrderConfQ orderConf
4 SM SM_InvMovementQ inventoryInfo
2 SP SP_POrderQ pOrder
2 SP SP_PShipConfQ pShipConf
1 HQ HQ_OrderDCStatsQ statInfoOrderDC
2 HQ HQ_ProductFamily<n>T callForOffers
2 HQ HQ_ShipDCStatsQ statInfoShipDC
2 HQ HQ_InvoiceQ invoice
3 HQ HQ_PriceUpdateT priceUpdate
5 HQ HQ_SMStatsQ statInfoSM
6 HQ HQ_ProductAnnouncementT productAnnouncement
7 HQ HQ_CreditCardHLT creditCardHL

Note 1: Topic "HQ_ProductFamily<n>T" represents a given product family and is used for distributing callForOffers messages in Interaction 2.

Note 2: Since the Company Headquarters are responsible for centrally managing the single topic hierarchy used, all topic names are prefixed with HQ.

2.3.2 Message Types

The types of messages used are detailed in Table 2. Precise details on the message contents are given in Appendix B.

TABLE 2: Messages Types Used in The Interactions
Intr. Message Destination Type Message Type Size Properties Description
1 order Queue ObjectMessage small to medium Persistent, Transacted Incoming orders from SMs including ordered products and other order information.
1 orderConf Queue ObjectMessage small to medium Persistent, Transacted Confirmation sent by DC to SM confirming receipt of an order.
1 shipInfo Queue TextMessage small to large Persistent, Transacted EPCs representing items in a shipment (registered by RFID readers as shipment leaves DC / enters SM).
1 shipDep Queue TextMessage small to large Persistent, Transacted EPCs representing items in a shipment (registered by RFID readers as shipment leaves DC / enters SM).
1 shipConf Queue ObjectMessage small to medium Persistent, Transacted Confirmation sent to DC to confirm shipment arrival at SM.
2 callForOffers Topic TextMessage small to medium Persistent, Transacted, Durable Call for Offers send to all SP offering a product family (XML document).
2 offer Queue TextMessage small to medium Persistent, Transacted Offer created by SPs based on callForOffers (XML document).
2 pOrder Queue TextMessage small to medium Persistent, Transacted Incoming purchase order based from DC on an offer (XML document).
2 pOrderConf Queue TextMessage small to medium Persistent, Transacted Confirmation sent by SP to DC confirming receipt of a purchase order (XML document).
2 pShipment Queue TextMessage small to large Persistent, Transacted XML document describing an arriving shipment from SP (registered by RFID readers as they enter DC).
2 pShipConf Queue TextMessage small to medium Persistent, Transacted Confirmation sent to SP to confirm shipment arrival at DC (XML document).
2 invoice Queue TextMessage small to medium Persistent, Transacted Purchase order invoice sent by SP (XML document).
1 statInfoOrderDC Queue StreamMessage small Non-Persistent, Non-Transacted Statistical information sent to HQ containing information on the transactions and flow of goods between SMs and DCs.
2 statInfoShipDC Queue StreamMessage small Non-Persistent, Non-Transacted Statistical information sent to HQ containing information on the transactions and flow of goods between SPs and DCs.
3 priceUpdate Topic MapMessage small Persistent, Transacted, Durable Price update sent to SM by HQ.
4 inventoryInfo Queue TextMessage small to large Persistent, Transacted EPCs representing observed item movements in the inventory of a SM (registered by RFID readers).
5 statInfoSM Queue ObjectMessage small to very large Non-Persistent, Non-Transacted, AutoAck Statistical information sent to HQ containing sales statistics.
6 productAnnouncement Topic StreamMessage small to large Non-Persistent, Non-Transacted, Non-Durable Product announcements sent to SM by HQ.
7 creditCardHL Topic StreamMessage small to large Non-Persistent, Non-Transacted, Non-Durable Credit card hotlist sent to SMs by HQ.

2.3.3 Message Acknowledgement

In SPECjms2007, AUTO_ACKNOWLEDGMENT is used as acknowledgment mode for all non-transactional sessions. The reason is that most real-world applications do not use the other acknowledgment modes (CLIENT_ACKNOWLEDGMENT or DUPS_OK_ACKNOWLEDGMENT). However, SPECjms2007 still offers the possibility to change the acknowledgment mode for some of the messages (see org.spec.jms.Interaction7.autoAck property in <topology>.properties), although this is not allowed for published benchmark results.

2.4 Event Handlers

A separate Event Handler is defined for each queue and topic. For maximal performance, multiple instances of an Event Handler could exist to enable parallel processing. The Event Handlers at the different Locations are part of the various applications emulated by the benchmark, e.g. the four Event Handlers of a DC represent the DC-App used in the diagrams in Figures 4 and 5. An Event Handler has a similar name to its respective Destination but is distinguished by an 'EH' suffix in place of 'Q' or 'T'.

Sequence 1
FIGURE 4: Sequence Diagram for Interaction 1


Sequence 2
FIGURE 5: Sequence Diagram for Interaction 2

Section 3 - Application Implementation

3.1 Controller and SPECj Framework

The SPECjms2007 scenario includes many Locations represented by many Event Handlers. In order to drive the JMS Server to its capacity, Event Handlers may well be distributed across many Nodes. The reusable Framework designed to control SPECjms2007 aims to co-ordinate these distributed activities without any inherent scalability limitations. Key design decisions were that

Framework topology
FIGURE 6: SPECjms2007 SUT layout

In addition to controlling the benchmark cycle as described below the Controller also provides the following

The stages of the benchmark are as follows (corresponding text can be observed in output from the Controller).

Starting ControlDriver Framework

The Controller component reads in all of the configuration and topological layout preferences given by the user. This will include items such as the number of different types of Location and lists of the Nodes across which they may be run. In the two submittable topologies, many of these values are either fixed or are automatically calculated based upon the scaling factor.

With this knowledge, the Controller instantiates the software components of the SUT. It begins this by starting an RMI server and connecting to a Satellite process on each Node machine identified as part of this test to give it specific instructions. In all places (throughout this benchmark) where lists are given, work is distributed equally using a simple round-robin algorithm.

Starting Agents

The Satellite is a simple part of the Framework that knows how to build the correct environment to start the required JVM processes. It takes the Controller's configuration and starts the Agents relevant to that Node. Although each Agent is logically discrete from its peers, the Satellite will, based upon the user configuration, combine many Agents into a single AgentJVM for reasons of scalability. There is an architectural limit of one AgentJVM per class of Location (SP, SM, DC, HQ), meaning a minumum of four AgentJVMs in the SUT. Each Agent connects back to the Controller to signal their readiness.

Starting Event Handlers

The Controller signals all Agents to initialise their Event Handler threads and connect to the JMS resources they will be using (this includes both incoming and outgoing Destinations). Each Event Handler is implemented as a Java thread.

Graph showing measurement points during SPECjms2007 run
FIGURE 7: Formal measurement points during SPECjms2007 run
Warmup Period

Load-generating threads (Drivers) ramp up their throughput from zero to their configured rate over the Warmup Period. This helps ensure the SUT is not swamped by an initial rush when many of its constituent elements may not yet be fully prepared.

The Agents are the only parts of the SUT which perform JMS operations (i.e. talk directly to the JMS Server).

Measurement Period

The Measurement Period is also known as the steady-state period. All Agents are running at their configured workload and no changes are made. The Controller will periodically (thirty seconds by default) check there are no errors and may also collect periodic realtime performance statistics.

Drain Period

In order to make sure all produced messages have an opportunity to be consumed, the Controller signals Agents to pause their load-generating threads (Drivers). This period is not expected to be long in duration as a noticeable backlog in messages would invalidate audit requirements on throughput anyway.

Measurement Points

The Controller takes formal measurements at three points during the run (see Figure 7). The first two, the beginning and end of the Measurement Period, are used to audit the messaging throughput. The final, at the end of the Drain Period, is used to audit the final message counts.

The Controller will also periodic measurements (via the realtime reporter) to display progress to users. These are not used as part of the auditting.

Stopping Event Handlers

Agents will terminate all Event Handlers but remain present themselves so that the Controller can collect final statistics.

Post-processing results

Having collected statistics from all parties, the Controller begins post-processing them into different reports. This process can take some time where the duration is related to total the number of Event Handlers involved in the benchmark. Large Horizontal Topologies may also require a significant Java heap size to post-process.

3.2 Agent and Event Handler Framework

Framework AgentJVM
FIGURE 8: SPECjms2007 AgentJVM Detail

The Agents are logically isolated from each other, each intended to represent a geographically distinct Location. For purposes of heap management and simplicity, the Framework allows multiple Agents to be hosted within the same AgentJVM. Agents communicate with the SPECj Framework via the RMI-enabled org.spec.specj.Agent interface but are otherwise self-contained (it is possible to run them without any Controller present). The system for controlling the lifecycle of the Event Handler threads is an extension of the reusable Performance Harness for JMS Framework defined in org.spec.perfharness. The structure of these classes is demonstrated in Appendices A.3 and A.4 whilst the Performance Harness for JMS Framework can be found in Appendix A.7.

3.2.1 Isolating Agents within an AgentJVM

The logical isolation referred to above is enforced by the use of an IsolationClassLoader which separates the class structure of each Agent within a single JVM. This means static data cannot be shared between co-located Agents. If the JMS Client code makes use of JNI only one Agent in a JVM can be run successfully; the binary library cannot be loaded twice and there is no parent ClassLoader to hold a shared instance. In this case the configuration must set *.jvms_per_node=0 to ensure there is only a single Agent in each JVM.

3.3 Message Injection Rates

Each Interaction is operated independently by its associated Driver. The input rate is scaled during the Warmup Period and fixed during the Measurement Period. For each Interaction Driver thread, cycle times (the delay between two message injections) are selected from a negative exponential distribution, computed from the following equation.

Tc = -ln(x) * Tm

where:

Tc = Cycle Time
Tm = Mean desired cycle time
ln = natural log (base e)
x = random number, with at least 31 bits of precision, from a uniform distribution such that (0 < x <= 1)

The distribution is truncated at 5 times the mean. For each Interaction Driver, the current time is compared to the target time and, if this is greater than zero, the Driver will sleep for for difference. By this method, inaccuracies from the precision of getting the current time, sleeping for a fixed time and the time taken to compute cycles are removed from the system.

Section 4 - Workload Description

In addition to providing a standard workload and metrics for JMS performance, SPECjms2007 aims to provide a flexible framework for performance analysis of JMS Servers that allows users to configure and customize the workload according to their requirements. To achieve this goal, the Interactions are implemented in such a way that one could run them in different combinations depending on the desired transaction mix. SPECjms2007 provides three different workload topologies which correspond to three different modes in which the benchmark can be run:

  1. Horizontal
  2. Vertical
  3. Freeform

The Horizontal Topology allows the workload to be scaled horizontally by increasing the number of Locations (SMs, DCs, SPs, HQs) while keeping the traffic per Location constant. In the Vertical Topology, a fixed set of Locations is used and the workload is scaled by increasing the rate at which goods are sold at Supermarkets. Finally, the Freeform Topology allows the user to define his own Topology and transaction mix and scale the workload in an arbitrary manner by increasing the number of Destinations and/or increasing the Interaction rates. Note that while in the Horizontal and Vertical Topology scaling is supported in a manner that preserves the relation to the modeled real-life business scenario, this does not necessarily apply to the Freeform Topology since the goal there is to give the user maximum flexibility.

Note: Although the modeled scenario has a single HQ Location, the benchmark allows multiple HQ instances to exist each with its own set of queues. The goal is to avoid the HQ queues becoming a bottleneck when scaling the number of SMs, DCs and SPs. It is assumed that messages sent to the HQ are distributed evenly among the HQ instances. Multiple HQ instances are considered as separate servers within the same Location.

SPECjms2007 provides the following workload configuration parameters that can be set by the user:

While in the Horizontal and Vertical topologies there are some restrictions as to which parameters can be changed by the user, these restrictions are relaxed in the Freeform Topology. This allows the user to precisely configure the workload and transaction mix to be generated and the way it is distributed among Client processes. The user can selectively turn off Interactions or change the rate at which they are run to shape the workload according to his requirements. At the same time, when running the Horizontal or Vertical Topology, the benchmark behaves as if the Interactions were interrelated according to their dependencies in the real-life application scenario.

4.1 Horizontal Topology

The goal of scaling the Topology in a "horizontal" direction is to increase the number of JMS Destinations and associated Event Handlers whilst maintaining a fixed throughput per Destination. This is accomplished by locking the Driver rates and defining the number of Locations in terms of the Horizontal scaling value (org.spec.jms.horizontal.BASE). This is the single property which controls the Horizontal Topology. Attempting to override the Driver rates or change other properties which affect the Topology is possible but the situation is detected and results cannot be submitted. Tuning properties are still allowed.

This Topology would allow vendors to make use of horizontally scaled (e.g. clustered) hardware and software (within bounds of the Run Rules) to best implement the Topology.

The following table shows, how the different parameters are defined for the Horizontal Topology. Some of them are based on the BASE parameter, some are fixed.

TABLE 3: Parameters for Horizontal Topology
Parameter Fixed value
No. of SMs BASE
No. of SPs RoundUp(BASE*0.4)
SMs Per DC 5
No. of Product Families (Topics) BASE
Interaction 1 Injection Rate 1.539201540
Interaction 2 Injection Rate 2.133333334
Interaction 3 Injection Rate 6.000000000
Interaction 4 Injection Rate 3.378378378
Interaction 5 Injection Rate 11.544011544
Interaction 6 Injection Rate 11.385199240
Interaction 7 Injection Rate 9.230769230
Number Cash Desks 5
ProductFamilies offered per SP 5
CRC frequency 1000
Interaction 7 Auto Ack true
SM_Interaction1DR.msg.prob.1 95
SM_Interaction1DR.msg.prob.2 4
SM_Interaction1DR.msg.prob.3 1
SM_Interaction1DR.msg.size.1 5
SM_Interaction1DR.msg.size.2 100
SM_Interaction1DR.msg.size.3 700
DC_Interaction2DR.msg.prob.1 95
DC_Interaction2DR.msg.prob.2 4
DC_Interaction2DR.msg.prob.3 1
DC_Interaction2DR.msg.size.1 3
DC_Interaction2DR.msg.size.2 35
DC_Interaction2DR.msg.size.3 200
SM_Interaction4DR.msg.prob.1 95
SM_Interaction4DR.msg.prob.2 4
SM_Interaction4DR.msg.prob.3 1
SM_Interaction4DR.msg.size.1 10
SM_Interaction4DR.msg.size.2 100
SM_Interaction4DR.msg.size.3 500
SM_Interaction5DR.msg.prob.1 95
SM_Interaction5DR.msg.prob.2 4
SM_Interaction5DR.msg.prob.3 1
SM_Interaction5DR.msg.size.1 2
SM_Interaction5DR.msg.size.2 7
SM_Interaction5DR.msg.size.3 13
HQ_Interaction6DR.msg.prob.1 95
HQ_Interaction6DR.msg.prob.2 4
HQ_Interaction6DR.msg.prob.3 1
HQ_Interaction6DR.msg.size.1 100
HQ_Interaction6DR.msg.size.2 10
HQ_Interaction6DR.msg.size.3 1000
HQ_Interaction7DR.msg.prob.1 95
HQ_Interaction7DR.msg.prob.2 4
HQ_Interaction7DR.msg.prob.3 1
HQ_Interaction7DR.msg.size.1 50
HQ_Interaction7DR.msg.size.2 500
HQ_Interaction7DR.msg.size.3 3000

The ratios between transactional and non-transactional messages as well as the proportions between Pub/Sub and P2P are shown below.


TABLE 4: Ratios between Transactional vs. Non-Transactional and Pub/Sub vs. P2P for Horizontal Topology
Messaging Type Number of Messages Number of Bytes
    Transactional - P2P 25.55% 23.74%
    Transactional - Pub/Sub 12.45% 2.41%
    Non-Transactional - P2P 24.55% 49.19%
    Non-Transactional - Pub/Sub 37.46% 24.66%


Ratio Horizontal
FIGURE 9: Ratios Transactional vs. Non-Transactional, Pub/Sub vs. P2P

No of Msg. per Interaction
FIGURE 10: Proportions of the different Interactions based on message throughput.    

Kybte per Interaction
FIGURE 11: Proportions of the different Interactions based on message traffic in KBytes.

4.2 Vertical Topology

The goal of scaling the Topology in a "Vertical" direction is to increase the throughput per JMS Destination whilst maintaining a fixed number of those Destinations. This is accomplished by locking the number of Locations to a small set and defining rates in terms of the Vertical scaling value (org.spec.jms.vertical.BASE). This is the single property which controls the Vertical Topology. Attempting to override the number of Locations or change other properties which affect the Topology is possible but the situation is detected and results cannot be submitted. Tuning properties are still allowed.

This Topology would allow vendors to make use of larger individual pieces of hardware or greater software multithreading to best implement the Topology.

The following table shows, how the different parameters are defined for the Vertical scenario. Some of them are based on the BASE parameter, some are fixed.

TABLE 5: Parameters for Vertical Topology
Parameter Fixed value
No. of SMs 10
No. of SPs 5
SMs Per DC 5
No. of HQs 2
No. of Product Families (Topics) 100
Interaction 1 Injection Rate 0.076190476*BASE
Interaction 2 Injection Rate 0.106666667*BASE
Interaction 3 Injection Rate 0.05*BASE
Interaction 4 Injection Rate 0.162162162*BASE
Interaction 5 Injection Rate 0.577200577*BASE
Interaction 6 Injection Rate 0.142314991*BASE
Interaction 7 Injection Rate 0.102564103*BASE
Number Cash Desks 5
ProductFamilies offered per SP 50%
CRC frequency 200*BASE
Interaction 7 Auto Ack true
SM_Interaction1DR.msg.prob.1 95
SM_Interaction1DR.msg.prob.2 4
SM_Interaction1DR.msg.prob.3 1
SM_Interaction1DR.msg.size.1 5
SM_Interaction1DR.msg.size.2 100
SM_Interaction1DR.msg.size.3 700
DC_Interaction2DR.msg.prob.1 95
DC_Interaction2DR.msg.prob.2 4
DC_Interaction2DR.msg.prob.3 1
DC_Interaction2DR.msg.size.1 3
DC_Interaction2DR.msg.size.2 35
DC_Interaction2DR.msg.size.3 200
SM_Interaction4DR.msg.prob.1 95
SM_Interaction4DR.msg.prob.2 4
SM_Interaction4DR.msg.prob.3 1
SM_Interaction4DR.msg.size.1 10
SM_Interaction4DR.msg.size.2 100
SM_Interaction4DR.msg.size.3 500
SM_Interaction5DR.msg.prob.1 95
SM_Interaction5DR.msg.prob.2 4
SM_Interaction5DR.msg.prob.3 1
SM_Interaction5DR.msg.size.1 2
SM_Interaction5DR.msg.size.2 7
SM_Interaction5DR.msg.size.3 13
HQ_Interaction6DR.msg.prob.1 95
HQ_Interaction6DR.msg.prob.2 4
HQ_Interaction6DR.msg.prob.3 1
HQ_Interaction6DR.msg.size.1 100
HQ_Interaction6DR.msg.size.2 10
HQ_Interaction6DR.msg.size.3 1000
HQ_Interaction7DR.msg.prob.1 95
HQ_Interaction7DR.msg.prob.2 4
HQ_Interaction7DR.msg.prob.3 1
HQ_Interaction7DR.msg.size.1 50
HQ_Interaction7DR.msg.size.2 500
HQ_Interaction7DR.msg.size.3 3000

The ratios between transactional and non-transactional messages as well as the proportions between Pub/Sub and P2P are shown below.

TABLE 6: Ratios between Transactional vs. Non-Transactional and Pub/Sub vs. P2P for verticalTopology
Messaging Type Number of Messages Number of Bytes
    Transactional - P2P  40.74%  29.52%
    Transactional - Pub/Sub  5.99% 2.25%
    Non-Transactional - P2P 39.09%   61.03%
    Non-Transactional - Pub/Sub 14.19% 7.19%
Ratio Vertical
FIGURE 12: Ratios Transactional vs. Non-Transactional, Pub/Sub vs. P2P

Interactions No. of Msg
FIGURE 13: Proportions of the different Interactions based on message throughput.

Interactions in KByte
FIGURE 14: Proportions of the different Interactions based on message traffic in KBytes.

4.3 Freeform Topology

Freeform topologies are intended to allow experimentation with the benchmark. The two submittable topologies (Horizontal and Vertical) are controlled by fixing certain properties into values or ratios. In all other respects they are the same as Freeform Topology which contains those same variables but provides no restrictions on their use. When in freeform mode no checking is done on the validity of the defined Topology.

This allows users (and developers!) to define their own workloads using the full set of configurable parameters given by SPECjms2007.

Appendix A - UML Diagrams

A.1 Package Structure

The following is a diagram demonstrating the interaction of the major packages which comprise the benchmark. Not visualised but of note are:

org.spec.jms.preauditor This contains a helper application which checks Destinations are clear and ready for a benchmark run. It utilises the location package and will therefore only audit Locations currently defined. Due to limitations of the JMS specification, this tool only checks that Queues are empty and performs no operation on Topics.
org.spec.perfharness.jms.providers This package contains an interface, JMSProvider, which can be implemented to provide product-specific operations for the creation or lookup of JMS objects. Such implementations are packaged separately to the main benchmark.
org.spec.jms.setup This contains an application which attempts to create the relevant JMS Destinations for the current configuration. This is only possible if a product-specific JMS provider class is configured which can actually perform the creation. In the default case, it simply lists the names of required Destinations.

FIGURE 15: Package Structure

A.2 Location Package

The following serves to highlight aggregation and inheritance within the org.spec.jms.location package.


FIGURE 16: Location Package UML

A.3 Event Handler Package

The following serves to highlight aggregation and inheritance within the org.spec.jms.eventhandler package. Only the eventhandler.hq package is used to demonstrate the relationships.


FIGURE 17: Event Handler Package UML

A.4 Agent Package

The following serves to highlight aggregation and inheritance within the org.spec.jms.agent package.


FIGURE 18: Agent Package UML

A.5 Stats and Results Packages

The following serves to highlight aggregation and inheritance within org.spec.jms.stats and org.spec.jms.results. Not all results reporters are shown. Statistics are stored following the Composite pattern and the resulting structure is parsed (by the results classes) using a Visitor pattern.


FIGURE 19: Stats and Results Package UML

A.6 Framework and SPECj Packages

The following serves to highlight aggregation and inheritance within org.spec.jms.framework and org.spec.specj. The SPECjms2007 control classes and methodology are extended from the SPECj Framework.


FIGURE 20: Framework and SPECj Package UML

A.7 Perfharness Package

The following serves to highlight aggregation and inheritance within org.spec.perfharness. The SPECjms2007 Agent classes and methodology are extended from ControlThread and WorkerThread whilst product-specific plugins are extended from JNDI.


FIGURE 21: Perfharness Package UML

Appendix B - Message Schemas

B.1 Messages between SPs and DCs

All communication between SPs and DCs/HQ is done via TextMessages containing XML (inter-company communication).

B.1.1 pOrder (Purchase Order)

Purchase Order
FIGURE 22: pOrder Message Schema

pOrder is a TextMessage (in XML format) containing a purchase order sent from DC to SP. The pOrder message includes some header information (about the order, SP and DC) and the pOrderLines (information about the ordered items). The size of a message depends on the number of pOrderLines.

B.1.2 pOrderConf (Purchase Order Confirmation)

Purchase Order Confirmation
FIGURE 23: pOrderConf Message Schema

pOrderConf is a confirmation message sent from SP to DC upon receival of an order from the DC. It header information and a list of confirmed pOrderLines. The size of pOrderConf is similar to the size of the confirmed pOrder.

B.1.3 pShipInfo (Purchase Order Shipment)

pShipment
FIGURE 24: pShipInfo Message Schema

This message emulates the observation of an incoming shipment by RFID readers at a DC's warehouse. An observation can contain different items as well as multiple instances of the same item. Multiple instances of an item are represented as a list of EPCs (tags) or as a single record containing an itemID (similar to an EPC) of a product and number of observed instances. A combination of both representations for a shipment is possible. Which representation is selected depends on the types of observed items. For example, if a pallet of cheese is delivered there is no need for a list containing tags of single cheese packages, however, if a shipment of notebooks arrives, the serial numbers of individual items are of interest.

B.1.3 pShipConf (Purchase Order Shipment Confirmation)

pShipConf
FIGURE 25: pShipConf Message Schema

This message emulates a confirmation sent from DC to SP to confirm the receipt of a shipment. It contains confirmation header information (data about the shipment, the DC and the SP) and for every observed item type a pShipConfLine element containing the quantity received. Every shipment is associated with a pOrder and for every pOrderLine of the latter, a pShipConfLine is included in the pShipConf message. The size of the message is small to medium, depending on the number of different items ordered.

B.1.4 callForOffers

Call for offers
FIGURE 26: callForOffers Message Schema

This message is sent when goods in a DC are depleted and the DC has to order from a SP to refill stock. The DC sends a callForOffers message to the HQ_ProductFamilynT topic corresponding to the respective product family. SPs that can deliver the goods send offers to the DC.

B.1.5 offer

Offer
FIGURE 27: offer Message Schema

An offer message is sent by a SP in response to a received callForOffers message. It contains a list of the products with pricing information.

B.2 Message between SMs and DCs

The data structures of the messages exchanged between SMs and DCs are similar to the respective messages exchanged between DCs and SPs as discussed in the previous section (e.g. orders and purchase orders). However, since the communication between SMs and DCs is intra-company, there is no need to use XML-based TextMessages and ObjectMessages can be used instead. The only exception is for the shipInfo message, which is implemented as a TextMessage similar to pShipment.

B.2.1 order

Order
FIGURE 28: order Message Schema

An order message is an ObjectMessage sent by a SM to a DC containing an order. It is defined using three Java classes:

The size of an order message depends on the number of orderLines.

B.2.2 orderConf (Order Confirmation)

Order Confirmation
FIGURE 29: orderConf Message Schema

OrderConf is a confirmation sent by a DC to a SM upon receival of an order from the SM. Like pOrderConf it contains some header information and a list of confirmed orderLines. The size of an orderConf message depends on the number of orderLines of the respective order.

OrderConf is defined using three Java classes:

B.2.3 shipInfo (Shipment Information)

Shipment
FIGURE 30: shipInfo Message Schema

Since shipment and pShipment represent nearly the same information (EPC tag data observed by RFID readers), they have a similar format.

B.2.4 shipConf (Shipment Confirmation)

Shipment Confirmation
FIGURE 31: shipConf Message Schema

This message is generated by SM to confirm shipment arrival. The data structure is mapped to two Java classes:

The size of a shipConf message depends on the size of the respective order.

B.3 Messages between DCs and HQ

B.3.1 statInfoOrderDC (Statistical Information about Orders)

StatinfoOrderDC
FIGURE 32: statInfoOrderDC Message Schema

statInfoOrderDC is a StreamMessage that contains statistical information on SM orders received by a DC.

B.3.2 statInfoShipDC (Statistical Information about Shipments)

This message contains a stream of items shipped to a DC by a SP (purchase order statistics). For every arriving shipment a separate message is generated. The size depends on the number of items in the shipment. The schema is identical to that of the shipConf message defined in section B.2.4.

B.4 Messages between SMs and HQ

B.4.1 statInfoSM (Statistical Information)

Statistical Information (SM) Sales

FIGURE 33: statInfoSM Message Schema

StatInfoSM contains sales statistics (e.g. for every cash desk in a SM all sales made in a day or a sample of the sales). The data structure is mapped to four Java classes:

  1. statInfoSM (sales data for an entire SM, contains cashDeskData objects)
  2. cashDeskData (sales data for a single cash desk, contains sales objects)
  3. sales (a single sale, contains salesLine objects)
  4. salesLine (details of a sale)

The size of statInfoSM messages would typically be very large. Therefore they could be used to test JMS performance with large messages.

B.4.2 priceUpdate

Price Udpate
FIGURE 34: priceUpdate Message Schema

priceUpdate is a MapMessage sent by HQ to the SMs. It contains pricing information for a single product.

B.4.3 productAnnouncement

Product Announcement
FIGURE 35: productAnnouncement Message Schema

The productAnnouncement StreamMessage is produced by HQ and contains a list of product announcements. All products announced in one message belong to the same product family (e.g. food, electronics, ...).

B.4.4 creditCardHL (Credit Card Hotlist)

Creditcard hotlist
FIGURE 36: creditCardHL Message Schema

This message is sent by the HQ to SMs periodically. It contains a list of revoked credit cards.

B.5 Messages between SPs and HQ

All communication between SPs and HQ is done via TextMessages containing XML (inter-company communication).

B.5.1 invoice

Invoice
FIGURE 37: invoice Message Schema

This message represents an invoice issued by a SP for a pOrder that has been processed. For every pOrderLine of the pOrder, the invoice contains an invoiceLine.

B.6 Internal Messages within DCs

B.6.1 shipDepature

The shipDepature message has the same structure as the shipment message.

B.7 Internal Messages within SMs

B.7.1 inventoryInfo

Inventory movement
FIGURE 38: inventoryInfo Message Schema

The message is sent when goods leave the warehouse of a SM (to refill a shelf). Goods are registered by RFID readers and the local warehouse application is notified so that inventory can be updated.