SPEC OSG SPECmail2009 Benchmark
Workload Characterization for SPECmail_Ent2009 Metric

Mike Abbott, Yun-seng Chao

December 2008

 

Summary

This document summarizes the studies on mail server workload collected from multiple university and corporate sources, using a variety of IMAP4 clients. The analyzed workloads consist of both SMTP and IMAP4 requests. Each request is described by parameters which fully characterize its behavior. The proposed models, which are obtained by analyzing these parameters, are able to reproduce the behavior of the mail server workloads.

Document Organization

The report is organized as follows. We start with a description of the measurements and of the parameters considered in our studies. We then present the models characterizing the mail server workloads and we briefly describe how to use these models.

 

SPECmail2009 Additions/Changes

Much of the document discusses the workload changes between new SPECmail2009 and the original SPECmail2008 benchmark workload. Many of the internal distributions were updated with complete message and folder profiles provided by Apple, Inc in 2008. Most of this data replaces the original message and mailbox composition distributions. The SMTP traffic levels have been incorporated into the recipient and message size distributions.

One workload addition not discussed in this document is the ability to test using encrypted TCP connections. The reason lies in where this encryption incurs its cost. The e-mail clients issue commands according to user or programatic directives, regardless of the network connection's encryption mode. Empirical data shows both SUT and e-mail clients require extra computing and/or memory resources if encryption exists. Therefore, the benchmark's Secure metric influences the number of concurrent network sessions and interarrival times but not the actual command sequences. The two SPECmail2009 metrics show the effects of encrypted network connections on the SUT.

Measurements and Parameters

The measurements analyzed in our studies come from different sources. The measurements related to SMTP and IMAP4 have been provided by four companies and by two universities.? The collected sessions were divided into five IMAP4 and two SMTP groups.? The sessions within each group form the basis for all of the parameters that define the Enterprise User Profile, emulated by the SPECmail2009 benchmark.

 

Mailbox and Message Structures

The IMAP4 protocol allows email clients to create and maintain any number of folders and subfolders, in addition to the standard Inbox folder used in the SPECmail2001 POP3 user profile.? The IMAP4 command set also allows email clients to ask the server to describe these structures.? This information is independent of the delivery or retrieval protocols and so is treated outside of specific protocol and/or server context.

Multipurpose Internet Mail Extension (MIME) Profile

MIME is an internet attachment scheme, defined as a formal standard by RFCs 1521, 1522, and 1523.? The Sun and Apple data sets provided detailed information about mailbox and message structure.? Thus they form the basis for the following probability distribution tables used in the benchmark.?

The initial processing of all message sizes distinguished between single part sizes and multipart sizes.? The IMAP4 benchmark prioritizes individual MIME part size over the global message size distribution.

Single Part messages (Sun: 76% of total, Apple: 47% of total)

1. Use “Content-type: text/plain” or no content-type at all in message headers
2. Use subpart content size distribution

 

Multipart Message (Sun: 24% of total, Apple: 53% of total)

1. Use “Content Type: multipart/mixed; boundary=”xxxxxxxxx-counter” or “Content Type: multipart/alternative; boundary=”xxxxxxxxx-counter” in message headers
2. Use distributions for message part width and depth to help establish the set of multipart message bodies.
3. Categorize MIME messages to fall into one of these pre-defined multipart buckets.
4. Use subpart content size distribution to define the sub-part sizes in the fixed pool of pre-defined multipart messages.

Below are the distributions used in constructing messages in compliant with the MIME standard.

 

 

The following tables show the distribution of the number of MIME parts at the top level (without regard to nesting). It reflects the count of multipart/mixed parts immediately “attached” to the main message. It does not reflect any counting of multipart/alternative parts (i.e. text/plain and text/html, alternative formats of the same text). Nor does it reflect the MIME attachment depths (“attachments” to “attachments” or forwarded messages).

 

 

 

 

The next tables show the distribution of the nested MIME Part Levels that occur within a given message from the sample of MIME parts. It generally reflects messages or attachments which are forwarded multiple times, each time adding another depth level to the resulting message.

 

 

 

The following tables show the distribution of primary MIME Content Type (not including subtype) of all the parts in the entire sample.

 

 

After Sun's values were reviewed, a former employee noted that the Unix company that provided MIME distributions tended to use more text messages. Other companies have more and larger MIME parts that have richer, non-textual, content such as word processor documents, presentations, spreadsheets, web pages, calendar events, images, audio, and both rich and simple alternate MIME structures. The major effect of this shift is a tendency to increase the overall message sizes, and decreasing the Text content type in favor of the other categories.

However, increased Alternate structures does not eliminate the Text portion's counts. It just increases the other content types counters. Also, the IMAP server is not required to interpret the actual MIME parts content. It must extract the MIME part(s) and send the content, as is, to the IMAP4 client, which performs the interpretation. Therefore, the shift in Content Type distribution affects the benchmark's MIME structure of the message delivered to the SUT. The SUT still must deconstruct these MIME structures, but not the actual content.

 

Messages Per Folder

The following tables show the distribution of messages in folders at the first five levels.

 

Here is the same data from Apple bucketed such that each contains roughly five percentage points. These are the actual values used in the benchmark.

Mailbox Distribution Profile

 

A mail server that supports IMAP is likely to support a hierarchy of several mailboxes (also known folders) in addition to the default INBOX mailbox for each user.? Below are several distributions to construct the structure of mailboxes contained within a mailstore supported by IMAP.? The data used is extracted from the four enterprise data samples (Mirapoint, Openwave, Sun, Apple).

The following tables show the probably of an individual user having a certain number of mailboxes (aka folders) at each level (depth).? The data reflects the probability distributions for the first five (5) levels, even though the actual samples went many levels deeper than that.?

 

 

 

 

The following tables show the percent of folders at each level containing any subfolders.

 

 

 

Mailbox Structure Example

Below is a walk through of the construction of a folder tree with a diagram to illustrate the use of the above distribution tables in creating a folder tree for user “U1”. The probability values used are only examples, not actual distribution table entries.

 

The diagram below shows the mailbox structure for user U1.

Figure 1: Mailbox Structure Diagram

Peak Hour Determination

The overall peak traffic hour must be based on both SMTP and the corresponding IMAP activity over the same period of time.? Therefore, only two data samples were used to determine the relative workloads – Mirapoint and Openwave.? The other data samples did not provide corresponding SMTP logs for this purpose.

Peak Hour Traffic Volumes and Active Users

The following table shows the overall traffic volumes and users over the course of the peak day (determined by total number of message activity from the data samples).

 

 

The Peak Hour Percent of Active Users is computed by first using the larger of the two unique Sending or Recipient users and dividing that value by the total number of provisioned users.? The percentages are then pro-rated based on the relative number of actual users to compute the actual Percent of Active Users used in the benchmark.

 

 

PEAK_PCT_USERS = "78"

The Peak Hour Activity Percentage can be derived by using the traffic volume from the peak hour and the daily total for each protocol.? Again the benchmark value is computed by pro-rating each data sample within the overall user counts.

 

 

 

 

SMTP log files

SMTP Traffic Analysis

The SMTP log files reflect mail transfer agent workloads from the four enterprises, three collected over the course of fourteen (14) to thirty (30) days of operation and one from one day. The workload refers to all the requests processed by the mail server for delivering incoming and outgoing messages. These enterprises ranged from approximately 120 to 40,000 users.? The data logs cover the full 24-hour day, over the course of the data collection period.

The parameters used to describe the requests processed by the mail server are:

• time stamp of the request
• size [byte]
• number of recipients

The table below shows the statistics for SMTP traffic flows and message sizes for the four enterprises.?? The ISP user profile statistics are included to illustrate the difference with the original user model.

 

The following two tables contain the profile of the number of recipients per message, based on the Mirapoint and Openwave SMTP data from the busiest day of the week. The Apple SMTP sample did not distinguish between remote and local recipients, nor the actual recipient count from the RCPT TO step. So this version reuses the original recipient information. This data was extracted from both recipients named in the RCPT TO lines, as well as, recipient counts based on the mailing list expansions. The benchmark uses the probably distributions in the second table to generate the actual SMTP traffic.