SPEC Cloud IaaS 2016 Benchmark
Copyright © 2016 Standard Performance Evaluation Corporation
Cloud Vendor: Dell Inc.
Cloud Type / SUT Type: private/whitebox
Hardware Platform: x86_64
Hypervisor: VMware vSphere 6.5
Cloud Infrastructure: VMware Integrated Openstack 3.1 (Mitaka)
Scalability: 78.5 @ 72 Application Instances
Elasticity: 87.4%
Mean Instance Provisioning Time: 25s
Tested by: Dell Inc. SPEC Licence Number : 55 Test Date : May-2017
Performance Sections
Performance Summary
Performance Details
Validation, Errors, and Issues
Glossary of Terms
SUT Configuration Sections
Instance Configuration
Cloud Configuration
Network Configuration
Storage Configuration
Elasticity + Scalability Phase Date/Time and Test Region
Elasticity + Scalability Start Time: 2017-04-14_04:28:24_UTC
Elasticity + Scalability End Time: 2017-04-14_09:02:48_UTC
Test Region: US Central Time Zone
Cloud Informational Metrics
AI Provisioning Success: 100.00%
AI Run Success: 94.52
Total Instances: 468
Reference ID: ALLRUN04140320

Performance Summary

Baseline Summary Results for YCSB
Throughput (ops/s) Insert Latency 99% (ms) Read Latency 99% (ms) AI Provisioning Time (s)
Average 11887.9 1.830 2.963 121.2

Baseline Summary Results for KMeans
Completion Time (s) AI Provisioning Time (s)
Average 151.9 102.0

Scalability/Elasticity Summary Results for YCSB for 36 Valid AIs
Av. Throughput (ops/s) Av. Insert Latency 99% (ms) Av. Read Latency 99% (ms) Av. Provisioning Time (s) Scalability Elasticity (%)
9588.1 1.984 3.257 113.5 42.24 89.59

Scalability/Elasticity Summary Results for KMeans for 36 Valid AIs
Av. Completion Time (s) Av. Provisioning Time (s) Scalability Elasticity (%)
189.0 91.6 36.30 85.26

YCSB Summary Results by AI
AI name Run count Run code Av. Throughput (ops/s) Av. Insert Latency 99% (ms) Av. Read Latency 99% (ms) AI. Provisioning time (s) AI. Prov. Initiated from Elasticity Start (s)
ai_11 66 1 10782.8 1.795 2.929 104.0 3.0
ai_14 66 1 10464.3 1.686 2.929 109.0 570.0
ai_16 67 1 10799.2 1.699 2.962 109.0 931.0
ai_17 65 0 10207.1 1.733 2.727 108.0 1410.0
ai_19 64 1 10479.6 1.637 2.842 119.0 1766.0
ai_21 65 0 10885.7 1.626 2.483 110.0 2095.0
ai_24 64 0 10338.0 1.558 2.700 110.0 2671.0
ai_26 63 0 10361.9 1.883 2.897 111.0 3174.0
ai_28 60 0 10107.4 1.844 3.052 106.0 3572.0
ai_30 60 0 10012.6 1.778 3.138 111.0 3886.0
ai_32 55 0 9651.2 1.776 3.098 107.0 4479.0
ai_34 62 0 11681.5 1.567 2.962 109.0 4905.0
ai_35 49 0 9453.7 2.049 3.291 110.0 5397.0
ai_38 46 0 9500.3 1.875 2.982 105.0 5935.0
ai_40 44 0 9514.3 2.050 3.403 111.0 6506.0
ai_42 41 0 9374.8 2.065 3.309 118.0 7042.0
ai_44 40 0 9748.9 1.867 2.923 116.0 7408.0
ai_46 37 0 9159.4 2.114 3.641 108.0 7954.0
ai_48 35 0 9262.5 2.004 3.386 123.0 8313.0
ai_50 34 0 9748.0 1.918 3.240 113.0 8884.0
ai_52 30 0 8982.5 2.100 3.261 110.0 9275.0
ai_54 30 0 9408.6 1.956 3.358 108.0 9674.0
ai_56 27 0 9331.5 1.939 3.283 117.0 10056.0
ai_58 26 0 9236.7 1.968 3.385 120.0 10440.0
ai_60 23 0 8900.9 2.205 3.701 120.0 10861.0
ai_62 25 0 11174.6 1.513 2.752 113.0 11447.0
ai_64 18 0 9301.0 1.903 3.194 115.0 12034.0
ai_66 15 0 8857.7 2.162 3.691 119.0 12608.0
ai_68 13 0 8708.7 2.171 3.507 114.0 13102.0
ai_70 11 0 8650.7 2.225 3.571 111.0 13513.0
ai_72 9 0 8827.4 2.311 3.403 121.0 13994.0
ai_74 8 0 8661.6 2.440 3.673 116.0 14412.0
ai_76 6 0 9332.0 2.118 3.436 114.0 14781.0
ai_78 4 0 8020.6 2.547 4.052 128.0 15190.0
ai_80 2 0 8333.4 2.487 3.848 120.0 15739.0
ai_82 1 0 7911.5 2.871 4.239 122.0 16063.0

KMeans Summary Results by AI
AI name Run count Run code Av. Completion Time (s) Av. Provisioning time (s) AI. Prov. Initiated from Elasticity Start (s)
ai_12 72 0 176.6 85.0 10.0
ai_13 71 0 174.3 84.0 416.0
ai_15 71 0 167.9 80.0 898.0
ai_18 67 0 171.2 90.0 1447.0
ai_20 61 0 182.5 81.0 1908.0
ai_22 59 0 189.3 81.0 2209.0
ai_23 57 0 187.7 87.0 2576.0
ai_25 57 0 180.9 82.0 2904.0
ai_27 55 0 180.1 79.0 3447.0
ai_29 53 0 176.3 88.0 3816.0
ai_31 52 0 175.5 86.0 4307.0
ai_33 49 0 179.3 87.0 4886.0
ai_36 44 0 189.7 88.0 5442.0
ai_37 45 0 179.1 85.0 5755.0
ai_39 40 0 195.0 84.0 6353.0
ai_41 42 0 173.4 91.0 6672.0
ai_43 37 0 188.6 104.0 7254.0
ai_45 34 0 195.2 90.0 7651.0
ai_47 32 0 197.7 85.0 8203.0
ai_49 31 0 188.7 100.0 8587.0
ai_51 28 0 199.2 92.0 9140.0
ai_53 27 0 192.8 92.0 9612.0
ai_55 25 0 194.0 93.0 9931.0
ai_57 22 0 205.9 90.0 10340.0
ai_59 21 0 197.2 93.0 10839.0
ai_61 19 0 199.2 107.0 11403.0
ai_63 17 0 194.9 102.0 11999.0
ai_65 15 0 184.6 95.0 12471.0
ai_67 13 0 212.4 100.0 12798.0
ai_69 11 0 188.5 106.0 13374.0
ai_71 8 0 225.5 94.0 13863.0
ai_73 8 0 211.1 96.0 14211.0
ai_75 7 0 168.1 103.0 14613.0
ai_77 5 0 174.2 96.0 15099.0
ai_79 3 0 250.3 94.0 15438.0
ai_81 1 0 157.0 108.0 15949.0
ai_83 0 2 0.0 99.0 16310.0


Validation, Errors, and Issues

Glossary of "Run code": 0 no error, 3 if an AI failed to provision, 2 if an AI that was provisioned but did not report any runs when the benchmark terminated - the AIs that are provisioning towards the end of elasticity + scalability phase may have zero runs, 1 for all other errors including AIs that were provisioned but did not report any result or reported results after QoS limits.

AIs that were issued but their provisioning was inprogress when the benchmark terminated are not shown.


Instance Configuration

Instance Configuration Hadoop Master
Parameter Value
instance_type_name m1.medium
cpu 2
memory 4GB
disk_size 43GB
disk_backed block storage
image_name cb_speccloud_kmeans_271
image_OS Ubuntu 14.04
metadata__hw_vif_model virtualVmxnet3
metadata__vmware_adaptertype lsiLogic
metadata__vmware_disktype streamOptimized
metadata__vmware_ostype ubuntu64Guest
Notes Flavor disk size modified to match image size of 40GiB

Instance Configuration Hadoop Slave
Parameter Value
instance_type_name m1.medium
cpu 2
memory 4GB
disk_size 43GB
disk_backed block storage
image_name cb_speccloud_kmeans_271
image_OS Ubuntu 14.04
metadata__hw_vif_model virtualVmxnet3
metadata__vmware_adaptertype lsiLogic
metadata__vmware_disktype streamOptimized
metadata__vmware_ostype ubuntu64Guest
Notes Flavor disk size modified to match image size of 40GiB

Instance Configuration YCSB
Parameter Value
instance_type_name m1.medium
cpu 2
memory 4GB
disk_size 43GB
disk_backed block storage
image_name cb_speccloud_cassandra_2111
image_OS Ubuntu 14.04
metadata__hw_vif_model virtualVmxnet3
metadata__vmware_adaptertype lsiLogic
metadata__vmware_disktype streamOptimized
metadata__vmware_ostype ubuntu64Guest
Notes Flavor disk size modified to match image size of 40GiB

Instance Configuration Cassandra
Parameter Value
instance_type_name m1.medium
cpu 2
memory 4GB
disk_size 43GB
image_name cb_speccloud_cassandra_2111
image_OS Ubuntu 14.04
metadata__hw_vif_model virtualVmxnet3
metadata__vmware_adaptertype lsiLogic
metadata__vmware_disktype streamOptimized
metadata__vmware_ostype ubuntu64Guest
Notes Flavor disk size modified to match image size of 40GiB


Cloud Configuration

Cloud High Level Information and Availability
Parameter Value
Name VMware Integrated Openstack 3.1
Type private
Sut_Type whitebox
Virtualized VMware vSphere 6.5
Manager_Protocol VMware Integrated Openstack 3.1
Manager_Version (Mitaka)
Hardware_Availability Jun-2016
Software/Service_Availability Feb-2017
Geographic_Distribution 1
Notes The SUT consists of - 1) VIO_Mgmt_Cluster where VIO 3.1 is installed across 3 of 4 FC6030's. 2) Two Nova Compute clusters of 4 FC630's each, the first set of nodes has 512GB RAM and the second set has 384GB RAM. DRS is enabled and set to Partially Automated. 3) A cluster of 4 C6220's, where vCenter Server Appliance runs on one host, Cloudbench Harness VM runs on another host, and a third VM runs on another host is use to access Horizon GUI. All VMs on all hosts use one or more Portgroups on a Distributed Virtual Switch (DVS) for communications. Network consist of Two PC8024 stacked via ports 2, one Switch is PC8024 (Ethernet) the other PC8024F (Fiber).

Cloud Under Test Software Levels
Parameter Value
Python_version 2.7.6
JVM_version 1.8.0_85
Hadoop_version 2.7.1
Cassandra_version 2.1.11
Notes None

Benchmark Harness Node
Parameter Value
Vendor Dell Inc.
Platform PowerEdge FX2 Chassis with 4 FC630
CPU_Description Intel Xeon, E5-2698v4
CPU_Frequency_MHz 2200
RAM_Memory 256 GB, ECC, DDR4, 2133MHz RDIMM
Local_Storage 2x 8GB, IDSDM, RAID1
Quantity 4
Hw_Availability_Date June-2016
OS_Version ESXi 6.5.0-4564106.x86_64_Dell_Customized_A00
OS_Availability_Date DEC-2016
Network_Adapter_1 1 x Qlogic QLE2562 Fiber Channel 8GB 2P
Network_Adapter_2 1 x Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet.quad-port
Network_port_type_1 1000BASE-T
Network_port_type_2 10GBASE-T
Network_speed_port_1 1 Gb/s
Network_speed_port_2 10 Gb/s
Notes Cloud infrastructure and management nodes all reside in a single datacenter located in Austin, TX. Cloud installation followed standard VIO 3.1 installation guide.

Cloud Manager Node
Parameter Value
Hw_Vendor Dell Inc.
Hw_Platform PowerEdge FX2 Chassis with 4 FC630
CPU_Type Intel Xeon, E5-2698v4
CPU_Count 2
CPU_Cores_Per 20
CPU_Frequency_MHz 2200
RAM_Memory_GB 256 GB
Primary_Cache_KB 64 KB I + 32 KB D per core, on chip
Secondary_Cache_KB 256 KB I+D per core, on chip
RAM_Memory_Type ECC, DDR4, 2133MHz RDIMM
Hw_Availability_Date June-2016
Local_Storage 2x 8GB, IDSDM, RAID1
Quantity 4
Virtualized VMware ESXi 6.5
Network_Adapter_1 1 x Qlogic QLE2562 Fiber Channel 8GB 2P
Network_Adapter_2 1 x Broadcom Corporation NetXtreme BCM57810 Gigabit Ethernet.Dual-port
Network_port_type_1 1000BASE-T
Network_port_type_2 10GBASE-SR
Network_speed_port_1 1 Gb/s
Network_speed_port_2 10 Gb/s
OS_Version ESXi 6.5.0-4564106.x86_64_Dell_Customized_A00
OS_Availability_Date Dec-2016
Notes None

Cloud Computes Nodes
Parameter Value
type 1_Compute_nodes 4
type 1_Hw_Vendor Dell Inc.
type 1_Hw_Platform PowerEdge FX2 Chassis with 4 FC630
type 1_Quantity 4
type 1_CPU_Type Intel Xeon, E5-2698v4 (Nodes 1-3), E5-2699v4 (Node 4)
type 1_CPU_Count 8
type 1_CPU_Cores_Per 20 (Nodes 1-3), 22 (Node 4)
type 1_CPU_Frequency_MHz 2200
type 1_RAM_Memory_GB 512
type 1_Primary_Cache_KB 64 KB I + 32 KB D per core, on chip
type 1_Secondary_Cache_KB 256 KB I+D per core, on chip
type 1_RAM_Memory_Type ECC, DDR4, 2400MHz RDIMM
type 1_Hw_Availability_Date June-2016
type 1_Local_Storage 2x 8GB, IDSDM, RAID1
type 1_Virtualized VMware ESXi 6.5
type 1_Network_Adapter_1 1 x Qlogic QLE2562 Fiber Channel 8GB 2Port
type 1_Network_Adapter_2 1 x Broadcom Corporation NetXtreme BCM57810 Gigabit Ethernet.Dual-port
type 1_Network_port_type_1 1000BASE-T
type 1_Network_port_type_2 10GBASE-SR
type 1_Network_speed_port_1 1 Gb/s
type 1_Network_speed_port_2 10 Gb/s
type 1_OS_Version ESXi 6.5.0-4564106.x86_64_Dell_Customized_A00
type 1_OS_Availability_Date Dec-2016
type 1_Notes Each server has a single 3.2TB NVMe (DPN# 61DHT) in PCIe slots attached to each server. Xeon E5-2698v4 used in 3 nodes and E5-2699v4 used in 1 node.
type 2_Compute_nodes 4
type 2_Hw_Vendor Dell Inc.
type 2_Hw_Platform PowerEdge FX2 Chassis with 4 FC630
type 2_Quantity 4
type 2_CPU_Type Intel Xeon, E5-2698v4
type 2_CPU_Count 8
type 2_CPU_Cores_Per 20
type 2_CPU_Frequency_MHz 2200
type 2_RAM_Memory_GB 384
type 2_Primary_Cache_KB 64 KB I + 32 KB D per core, on chip
type 2_Secondary_Cache_KB 256 KB I+D per core, on chip
type 2_RAM_Memory_Type ECC, DDR4, 2400MHz RDIMM
type 2_Hw_Availability_Date June-2016
type 2_Local_Storage 2x 8GB, IDSDM, RAID1
type 2_Virtualized VMware ESXi 6.5
type 2_Network_Adapter_1 1 x Qlogic QLE2562 Fiber Channel 8GB 2Port
type 2_Network_Adapter_2 1 x Broadcom Corporation NetXtreme BCM57810 Gigabit Ethernet.Dual-port
type 2_Network_port_type_1 1000BASE-T
type 2_Network_port_type_2 10GBASE-SR
type 2_Network_speed_port_1 1 Gb/s
type 2_Network_speed_port_2 10 Gb/s
type 2_OS_Version ESXi 6.5.0-4564106.x86_64_Dell_Customized_A00
type 2_OS_Availability_Date Dec-2016
type 2_Notes Each server has a single 3.2TB NVMe (DPN# 61DHT) in PCIe slots attached each server.

Cloud Network Information

Cloud Network Information
Parameter Value
network 1_Technology ethernet
network 1_Protocol TCP/IP
network 1_Protocol_version IPV4
network 1_Speed_Mbps 10000
network 1_Function data
network 1_Notes network_1 related notes
network 2_Technology ethernet
network 2_Protocol TCP/IP
network 2_Protocol_version IPV4
network 2_Speed_Mbps 1000
network 2_Function management
network 2_Notes network_2 related notes

Cloud Schematic