Standard Performance Evaluation Corporation
SPEC Cloud® IaaS 2018
The SPEC Cloud® IaaS 2018 benchmark is SPEC's second benchmark suite to measure cloud performance. The benchmark suite's use is targeted at cloud providers, cloud consumers, hardware vendors, virtualization software vendors, application software vendors, and academic researchers.
The SPEC Cloud® IaaS 2018 benchmark addresses the performance of infrastructure-as-a-service (IaaS) cloud platforms. IaaS cloud platforms can either be public or private.
The current version of the benchmark is version 1.1,
released on December 12, 2019. This update includes
bug fixes and usability improvements. SPEC Cloud IaaS 2018
builds on the original 2016 release with a variety of
enhancements and new primary metrics.
Beginning in March 2020, all result submissions must be made using version 1.1.
This new release includes usability improvements to make it easier to set up the cbtool harness and run an initial simulated test run and generate an example FDR report. Cbtool has made it easier to setup workload images on the tester's cloud platform. New and updated adapters are included in the release allowing users to test a variety of public and private cloud platforms. Depending on the cloud platform, instances may utilize a physical machine, a virtual machine or a container.
The benchmark is designed to stress provisioning as
well as runtime aspects of a cloud using I/O and CPU
intensive cloud computing workloads. SPEC selected the
social media NoSQL database transaction and K-Means
clustering using map/reduce as two
significant and representative workload types within
Each workload runs in multiple instances, referred to as an application instance. The benchmark instantiates multiple application instances during a run. The application instances and the load they generate stress the provisioning as well as run-time aspects of a cloud. The run-time aspects include CPU, memory, disk I/O, and network I/O of these instances running in a cloud. The benchmark runs the workloads until quality of service (QoS) conditions are reached. The tester can also limit the maximum number of application instances that are instantiated during a run.
The key benchmark metrics are:
For more detail on the SPEC Cloud IaaS 2018 benchmark please review the benchmark documentation listed below.
Press release material, documents, and announcements:
Benchmark Technical Support