Development Update on Our New Benchmark for Cloud Workloads

By Sundar Iyengar, Chair, and Neha Pathapati, Release Manager

Historically, benchmarking using standardized tests that simulate real-world scenarios has served as a foundational tool for comparing systems, significantly guiding purchasing and deployment decisions for server systems. Beyond evaluation, objective benchmarking is instrumental in shaping the design of the next generation of both software and hardware architectures by uncovering system bottlenecks.

Over the years, software architectures have shifted from monolithic designs to microservices, where applications are decomposed into independently deployable services that run in containerized environments and communicate through standardized interfaces and protocols. At the same time, hardware has become increasingly heterogeneous, integrating a mix of CPUs, IPUs, and specialized accelerators.

While traditional single-node, CPU-centric benchmarks have long been the standard for evaluating system performance, they fall short in capturing the complexities of modern cloud computing environments and keeping pace with the rapid transformation and modernization of datacenters. Hence, there is a pressing need for a next-generation, system-level benchmark that accurately reflects the performance of the modern infrastructure. This benchmark must represent a diverse set of real-world, cloud-enabled workloads and evaluate the performance of the full system—including CPUs, IPUs, networking, storage subsystems, and specialized hardware accelerators—to provide a holistic view of the system’s capabilities.

To address these challenges, the SPEC Cloud Committee is currently developing a novel system-level benchmark for datacenters that mimics emerging cloud use-cases and exercises all the relevant components of the modern infrastructure. The primary objective of the benchmark is to establish a comprehensive suite of real-world, cloud-enabled workloads and key performance indicators that adequately capture the performance of modern datacenter systems at scale. The benchmark will also provide valuable insights that will help in the design of future generations of computing platforms. Beyond performance characterization, the benchmark is expected to model portability, reproducibility, transparency and observability, with the help of integrated telemetry and detailed reporting, thereby fostering broad industry-wide adoption.

Collaboration is the Key to a Successful Benchmark

The new cloud benchmark is currently being developed through a collaborative effort involving leading members of the technology industry and academia, drawing on the rich expertise of specialists from 13 organizations. The participating organizations are drawn from prominent cloud service providers (CSPs), original equipment manufacturers (OEMs), and CPU and software designers. The benchmark consists of a robust suite with well-defined metrics and a benchmarking methodology to fairly measure the performance of modern datacenter systems. It represents a consensus on modern datacenter benchmarking methodologies developed by leading experts in architecture, systems, and cloud-native technologies.

While the development effort is ongoing, SPEC welcomes additional participation. If you have specialized knowledge related to solving the challenges this new type of benchmark presents and would like to contribute to its development, I strongly encourage you to contact the SPEC Cloud Committee at info@spec.org.