Performance Summary:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• Firmware settings used:
• Memory.MemoryMode=Independent
• Memory.PatrolScrub=Disable
• OperatingModes.ChooseOperatingMode=Maximum Performance
• Processors.EarlySnoopPreference=Disable

Storage Notes

• The System Under Test used 16 x 800GB SSDs. Detailed configuration is as follows:
• Create 4 x RAID10 arrays, each with 4 x SSD drives
• Create 8 x logical disks for each RAID array
• The first logical disk has the OS installed. The other logical disks were partitioned for the VMs of 1 tile

Virtualization Software Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline clocksource=tsc cgroup_disable=memory,cpu,cpuacct,blkio,hugetlb pci=realloc nosoftlockup mce=ignore_ce ipv6.disable=1 intel_iommu=on intel_idle.max_cstate=0 console=tty0 console=ttyS0,115200n8 console=ttyS1,115200n8
• The following services were disabled:
• abrt-oops, abrt-pstoreoops, abrtd, arp-ethers, atd, autofs, autovt@, avahi-daemon, blk-availability, cgconfig, cgdcbxd, cgred, chrony-wait, chronyd, console-getty, console-shell, cpupower, crond, debug-shell, dm-event, dnsmasq, ebtables, fcoe, firewalld, gssproxy, ip6tables, ipmievd, iprdump, iprinit, iprupdate, iptables, irqbalance, iscsid, iscsiuio, kdump, ksmtuned, libstoragemgmt, libvirt-guests, lldpad, lvm2-lvmetad, lvm2-lvmpolld, NetworkManager-dispatcher, NetworkManager-wait-online, NetworkManager, nfs-blkmap, nfs-server, nfs, ntpdate, numad, plymouth-halt, plymouth-kexec, plymouth-poweroff, plymouth-quit-wait, plymouth-quit, plymouth-read-write, plymouth-reboot, plymouth-start, postfix, psacct, radvd, rdisc, rdma, rhel-dmesg, rhel-domainname, rhsmcertd, rngd, rsyncd, saslauthd, serial-getty@, smartd, svnserve, systemd-bootchart, systemd-nspawn@, target, tcsd, virt-who, wpa_supplicant
• /usr/lib/sysctl.d/specvirt.conf was updated with the following lines:
• kernel.sched_min_granularity_ns = 10000000
• kernel.sched_wakeup_granularity_ns = 15000000
• vm.dirty_background_ratio = 5
• vm.dirty_ratio = 10
• vm.swappiness = 0
• kernel.sched_latency_ns = 20000000
• kernel.sched_migration_cost_ns = 500000
• kernel.numa_balancing = 0
• Network configuration is as follows:
• One 331FLR NIC port(eno1) was used for host access
• The 4 x x520-DA2 SFP+ ports were assigned to bridges (br0,br1,br2,br3)
• The x520-DA2 SFP+ were configured in /etc/modprobe.d/ixgbe.conf as follows,
• options ixgbe max_vfs=63,63,63,63
• blacklist ixgbevf
• All the webservers, infraservers, mailservers, appservers and dbservers were assigned with a VF
• Batch VMs did not use VFs, instead virtio-net via the bridge was used
• VM configurations were as follows:
• All VMs were managed with libvirt
• All VMs were created with the following XML template:
• <domain type='kvm'>
• <name>$vmname</name>
• <memory>$vmem</memory>
• <currentMemory>$vmem</currentMemory>
• <vcpu placement='static' cpuset='$host_cpu_list'>$vcpus</vcpu>
• <os>
• <type arch='x86_64' machine='pc'>hvm</type>
• <boot dev='hd'/>
• </os>
• <features>
• <acpi/>
• <apic/>
• <pae/>
• </features>
• <cpu mode='custom' match='exact'>
• <model>SandyBridge</model>
• <vendor>Intel/>vendor>
• <topology sockets='1' cores='$vcpus' threads='1'/>
• </cpu>
• <clock offset='utc'>
• <timer name='pit' tickpolicy='delay'/>
• </clock>
• <on_poweroff>destroy</on_poweroff>
• <on_reboot>restart</on_reboot>
• <on_crash>restart</on_crash>
• <devices>
• <emulator>/usr/libexec/qemu-kvm</emulator>
• <disk type='block' device='disk'>
• <driver name='qemu' type='raw' cache='none' io='native'/>
• <source dev='$vdisk'/>
• <target dev='vda' bus='virtio'/>
• <address type='pci' domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </disk>
• <interface type='hostdev' managed='yes'/>
• <mac address='$vmac'/>
• <source>
• <address type='pci' domain='0x0000' bus='0x$bus' slot='0x$slot' function='0x$func'/>
• </source>
• </interface>
• <memballoon model='none'>
• </devices>
• </domain>
• Batch VMs had the following declaration instead of 'hostdev' block above:
• <interface type='bridge'>
• <mac address='$vmac'/>
• <source bridge='$bridge'/>
• <model type='virtio'/>
• <address type='pci' domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </interface>
• VMs' nice value was set after startup via the renice command to the following settings:
• appservers = -19
• dbserver = -19
• webserver = 19
• infraserver = 19
• mailserver = 19
• All VMs of tiles 1-12 were bound to NUMA node 0
• All VMs of tiles 13-24 were bound to NUMA node 1
• infraserver25 mailserver25 webserver25 were bound to NUMA node 0
• appserver25 dbserver25 batchserver25 were bound to NUMA node 1

Web Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio console=ttyS0,115200
• ixgbevf module option used: InterruptThrottleRate=2000
• The following services were disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, cgconfig, cgred, cups, dovecot, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, php-fpm, portreserve, postfix, psacct, qpidd, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, winbind
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• vm.swappiness=0
• root file-system options: barrier=0,noatime
• Contents of nginx configuration file /etc/nginx/nginx.conf
• user nginx;
• worker_processes 4;
• worker_cpu_affinity 00000001 00000001 00000010 00000010;
• worker_rlimit_nofile 65535;
• pid /var/run/nginx.pid;
• events {
• use epoll;
• worker_connections 65535;
• multi_accept on;
• accept_mutex off;
• }
• http {
• include /etc/nginx/mime.types;
• default_type application/octet-stream;
• log_format common '$remote_addr - $remote_user [$time_local] $request" $status " $body_bytes_sent ';
• access_log /var/log/access.log common gzip buffer=4m flush=1m;
• sendfile on;
• sendfile_max_chunk 1m;
• tcp_nopush on;
• tcp_nodelay on;
• keepalive_timeout 120;
• open_file_cache max=204800 inactive=20s;
• open_file_cache_min_uses 1;
• open_file_cache_valid 30s;
• charset utf-8;
• server_names_hash_bucket_size 128;
• client_header_buffer_size 2k;
• large_client_header_buffers 4 4k;
• include /etc/nginx/conf.d/*.conf;
• }
• Contents of nginx configuration file /etc/nginx/conf.d/default.conf
• server {
• listen 80 backlog=8192;
• server_name localhost;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• error_page 500 502 503 504 /50x.html;
• location = /50x.html {
• root /usr/share/nginx/html;
• }
• location ~ .*.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$
• {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ .php$ {
• root html;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• Contents of nginx configuration file /etc/nginx/conf.d/example_ssl.conf
• server {
• listen 443 ssl;
• server_name localhost;
• ssl on;
• ssl_certificate /etc/pki/tls/certs/localhost.crt;
• ssl_certificate_key /etc/pki/tls/private/localhost.key;
• ssl_session_cache shared:SSL:1m;
• ssl_session_timeout 5m;
• ssl_ciphers ALL:!kEDH!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
• ssl_prefer_server_ciphers on;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• location ~ .*.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$
• {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ .php$ {
• root html;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 800000
• * hard nofile 800000
• * soft nproc 102400
• * hard nproc 102400
• apache soft nproc 102400
• apache hard nproc 102400

Application Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio console=ttyS0,115200
• The following services were disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, cgconfig, cgred, crond, cups, dovecot, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, mdmonitor, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, portreserve, postfix, psacct, qpidd, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, winbind
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness = 0
• root file-system options: barrier=0
• Following items were set in /opt/common_scripts/ostune_misc.sh and executed by /etc/rc.d/rc.local
• echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource
• There are two WebSphere profiles, AppSrv01 for SPECjAppServer2004 and AppSrv02 for SPEC Emulator
• Heap size for AppSrv01 has been changed to 2048 MB
• The following change was made to the defaults in run.properties
• runLO=4
• warmUpStyle = 1
• ixgbevf module option used: InterruptThrottleRate=2400
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 130172
• * hard nofile 130172
• * soft nproc 130172
• * hard nproc 130172
• spec soft nproc 130172
• spec hard nproc 130172
• * soft memlock 8388608
• * hard memlock 8388608

Mail Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio console=ttyS0,115200
• ixgbevf module option used: InterruptThrottleRate=1000
• The following services were disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, cgconfig, cgred, cups, dovecot, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, portreserve, postfix, psacct, qpidd, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, winbind
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• root file-system options: barrier=0
• mail store file-system options: noatime,barrier=0,data=writeback
• Contents of dovecot configuration file /etc/dovecot/dovecot.conf:
• dict {
• }
• !include conf.d/*.conf
• base_dir = /var/run/dovecot/
• disable_plaintext_auth = no
• log_path = /var/log/dovecot.log
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• mbox_write_locks = fcntl
• mail_fsync=never
• maildir_very_dirty_syncs = yes
• mbox_very_dirty_syncs = yes
• passdb {
• args = scheme=plain /etc/dovecot/passwd.dovecot
• driver = passwd-file
• }
• postmaster_address = postmaster@example.com
• service auth {
• user = root
• }
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• service_count = 0
• }
• service imap {
• process_limit = 1024
• }
• ssl = no
• ssl_cert = </etc/pki/dovecot/certs/dovecot.pem
• ssl_key = </etc/pki/dovecot/private/dovecot.pem
• userdb {
• driver = passwd
• }
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 102400
• * hard nofile 102400
• * soft nproc 102400
• * hard nproc 102400
• dovecot hard nproc 102400
• dovecot soft nproc 102400
• dovenull hard nproc 102400
• dovenull soft nproc 102400

Database Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio console=ttyS0,115200
• The following services were disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, cgconfig, cgred, cups, dovecot, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, portreserve, postfix, postgresql-9.2, psacct, qpidd, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, winbind
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 32384
• net.ipv4.tcp_max_tw_buckets = 1000000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness = 0
• kernel.sem = 500 64000 400 512
• kernel.shmmax = 137438953472
• kernel.shmmni = 11118096
• kernel.shmall = 8294967296
• root file-system options: barrier=0
• database store file-system options: noatime,barrier=0,data=writeback
• Following items were set in /opt/common_scripts/ostune_misc.sh and executed by /etc/rc.d/rc.local
• echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource
• sysctl -w kernel.randomize_va_space=0
• sysctl -w vm.nr_hugepages=0
• sysctl -w vm.nr_hugepages=12288
• sysctl -w vm.nr_overcommit_hugepages=512
• ixgbevf module option used: InterruptThrottleRate=1500
• Database tuning options:
• db2 update db cfg for specdb using DATABASE_MEMORY 6100000
• db2set DB2_LARGE_PAGE_MEM=DB; db2set DB2_PINNED_BP=ON
• alter bufferpool IBMDEFAULTBP size 5500000
• alter tablespace roll_space NO FILE SYSTEM CACHING
• alter tablespace C_cust_space NO FILE SYSTEM CACHING
• alter tablespace C_supp_space NO FILE SYSTEM CACHING
• alter tablespace C_site_space NO FILE SYSTEM CACHING
• alter tablespace C_parts_space NO FILE SYSTEM CACHING
• alter tablespace C_custinv_space NO FILE SYSTEM CACHING
• alter tablespace M_parts_space NO FILE SYSTEM CACHING
• alter tablespace M_bom_space NO FILE SYSTEM CACHING
• alter tablespace M_wo_space NO FILE SYSTEM CACHING
• alter tablespace M_lo_space NO FILE SYSTEM CACHING
• alter tablespace M_inv_space NO FILE SYSTEM CACHING
• alter tablespace O_cust_space NO FILE SYSTEM CACHING
• alter tablespace O_ords_space NO FILE SYSTEM CACHING
• alter tablespace O_ordl_space NO FILE SYSTEM CACHING
• alter tablespace O_item_space NO FILE SYSTEM CACHING
• alter tablespace S_comp_space NO FILE SYSTEM CACHING
• alter tablespace S_sc_space NO FILE SYSTEM CACHING
• alter tablespace S_po_space NO FILE SYSTEM CACHING
• alter tablespace S_site_space NO FILE SYSTEM CACHING
• alter tablespace S_supp_space NO FILE SYSTEM CACHING
• alter tablespace U_space NO FILE SYSTEM CACHING
• update database config for specdb using LOGFILSIZ 65536
• update db cfg for specdb using logprimary 54 logsecond 4
• The following lines were added to /etc/security/limits.conf:
• * soft nproc 578528
• * hard nproc 578528
• * soft nofile 331072
• * hard nofile 331072
• db2inst1 soft nproc 578528
• db2inst1 hard nproc 578528
• postgres soft nproc 578528
• postgres hard nproc 578528
• * soft memlock 33554432
• * hard memlock 33554432

Infraserver VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio console=ttyS0,115200
• The following services were disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, cgconfig, cgred, cups, dovecot, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, messagebus, netconsole, nfslock, ntpd, ntpdate, portreserve, postfix, psacct, qpidd, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, winbind
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_syn_retries = 4
• net.ipv4.tcp_retries2 = 7
• vm.swappiness=0
• root file-system options: barrier=0,noatime
• backend store file-system options: noatime,barrier=0,data=writeback
• httpd.conf changes:
• Timeout 20
• #KeepAliveTimeout 15
• #Include conf.d/*.conf
• #MaxKeepAliveRequests 100
• CustomLog logs/access_log common
• Listen 81
• ServerName besim:80
• <IfModule prefork.c>
• StartServers 10
• MinSpareServers 10
• MaxSpareServers 10
• ServerLimit 4096
• MaxClients 4096
• MaxRequestsPerChild 640000
• </IfModule>
• #IndexOptions FancyIndexing VersionSort NameWidth=* HTMLTable
• #LoadModule file_cache_module modules/mod_file_cache.so
• #LoadModule file_cache_module modules/mod_mem_cache.so
• LoadModule fastcgi_module modules/mod_fastcgi.so
• ScriptAlias /fcgi-bin/ "/var/www/fcgi-bin/"
• FastCgiIpcDir /etc/httpd/fastcgi
• <Directory "/var/www/fcgi-bin ">
• AllowOverride None
• Options +ExecCGI -Includes
• SetHandler fastcgi-script
• Order allow,deny
• Allow from all
• </Directory>
• AddHandler fastcgi-script fcgi
• FastCgiServer /var/www/fcgi-bin/besim_fcgi.fcgi -processes 50
• NFS export options:
• rw,async,no_subtree_check,no_root_squash
• ixgbevf module option used: InterruptThrottleRate=3000
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 102400
• * hard nofile 102400
• * soft nproc 102400
• * hard nproc 102400
• apache soft nproc 102400
• apache hard nproc 102400

Batch Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio console=ttyS0,115200
• The following services were disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, cgconfig, cgred, cups, dovecot, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, portreserve, postfix, psacct, qpidd, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, winbind
• tuned virtual-guest profile applied to VM
• root file-system options: barrier=0
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 102400
• * hard nofile 102400
• * soft nproc 102400
• * hard nproc 102400

Client Driver Notes

• All prime clients were VMs running on Red Hat Enterprise Linux Server 6.5 KVM
• The prime controller ran on the first vClient
• Two vClients were used for each tile. One vClient per tile ran the webserver workload, the other vClient per tile ran the other workloads
• Each vClient has 4 vCPUs, 6GB memory, 16GB disk, 1 virtio_blk device and 1 virtio_net device
• The first vClient also acted as prime controller and therefore had one more virtio_net device for management
• All physical clients have 4 x 10 Gbps ports and were configured as software bridges for the vClients
• Create one raid10 array with 4 x 800GB SSDs. It contains one logical volume with all VM images deployed on it

Power Configuration Notes

• The Yokogawa WT310 power meter was connected to the SUT only.
• The Digi Watchport/H temperature sensor was connected to the SUT only.

Other Notes

• 1 x Extreme Network Summit X670-48x-SFP+ switch was used to connect the clients and the SUT

Performance Details: