Standard Performance Evaluation Corporation

SPEC SIP Committee

The SPEC SIP Committee is developing a new industry standard benchmark for evaluating servers using the Session Initiation Protocol (SIP).

Interest in SIP has grown over the last few years with uses beyond just Voice over IP. SIP based applications include instant messaging, presence, video conferencing, and IPTV. IT professionals and decision makers considering SIP intrastructures will benefit from the availability of a standard benchmark from SPEC to compare SIP server solutions.

The objective of the SPEC SIP subcommittee is to create server benchmarks in the converging area of multimedia communications over IP that use the IETF Session Initiation Protocol (SIP). These benchmarks will permit fair and accurate analysis and evaluation of platform performance, at the Service Provider (ISP, NSP and Carriers) and Enterprise level. All SPEC-SIP benchmarks will have realistic workloads derived from standard profiles, and measure (at a minumum, but not limited to) throughputs, response time and QoS metrics.

Benchmark Goal

The primary requirement and goal for SPEC SIP is to evaluate SIP server systems as a comparative benchmark for aiding customer evaluations. The main features are:

• The SUT is the SIP server software and underlying hardware.
• Core SIP operations are exercised (e.g., packet parsing, user lookup, message routing, stateful proxying, registration).
• Generated traffic is based on best available estimates of user behavior.
• Configurations match requirements of SIP application providers (e.g., to require authentication).
Logging for accounting and management purposes is required.
• The standards defined by core IETF SIP RFCs (e.g., RFC 3261) are obeyed.
• Assumptions are made explicit so as to allow modifying the benchmark for custom client engagements (but not for publishing numbers)

The last feature illustrates a secondary additional goal, which is to aid in capacity planning and provisioning of systems. SPEC SIP explicitly defines a standard workload, which consists of standard SIP scenarios (what SIP call flows happen) and the frequency with which they happen (the traffic profile). Different environments (e.g., ISP, Telco, University, Enterprise) clearly have different behaviors and different frequencies of events. SPEC SIP is designed to be configurable so as to allow easy modification of the benchmark for use in customer engagements so that the traffic profile and resulting generated workload can be adapted to more closely match a particular customer's environment. Towards this end, we make traffic profile variables explicit to allow customization. However, for publishing numbers with SPEC, the standard SPEC SIP benchmark with the standard traffic profile must be used.

Current Status

The SPEC SIP Subcommittee was officially chartered in March 2007 and has been working together to define the benchmark workload. The first version of the design document has been ratified, and implementation, testing, and documentation is under way.

To encourage participation in the SubCommittee and solicit feedback about the benchmark, we are making our design document available below:

• Design Document
• Design FAQ

Future Efforts

Once SPEC has released its first SIP benchmark, we plan to refine the original benchmark to include additional features such as RTP media traffic, more detailed workload studies, etc. The committee also plans to look at additional SIP applications such as Instant Messaging and Presence.

Committee Membership

Current SPEC member companies that have participated in the SPEC SIP Subcommittee include AMD, BEA, Communigate Systems, HP, IBM, Intel, and Sun Microsystems. Columbia University is an academic member.

SPEC welcomes organizations to join and participate in our work, please contact info@spec.org for additional information.