Performance Summary:

Performance Details:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• BIOS settings used:
•   HP Power Profile: Maximum performance
•   HP Power Regulator: HP Static High Performance Mode
•   Thermal configuration: Increased Cooling
•   Processor Power and Utilization Montoring: Disabled
•   Memory Refresh Rate: 1x Refresh

Storage Notes

• All storage accessed with 1 HP Smart Array P420i controller and 2 dual port HP Smart Array P421 controllers.
• 2 x 300GB 15K SFF 6Gb internal SAS drives were connected to the P420i controller, built as RAID1 and used for OS.
• 4 x D2700 disk enclosures are connected via 2 dual port HP Smart Array P421 controllers.
• Each D2700 contains 10 x 400 GB MLC SSD drives. Total drive count is 40.
• Each enclosure was partitioned and used as follows:
•   Create 5 x RAID1 array and each has 2 x SSD drives.
•   Create 2 x logic disks for each RAID array.
•   Each logic disk was partitioned for the VMs of 1 tile.

Virtualization Software Notes

• Boot parameters set: intel_iommu=on elevator=deadline selinux=0 audit=0 clocksource=tsc idle=halt console=ttyS0,115200 intremap=no_x2apic_optout cgroup_disable=memory
• The following services were disabled:
•   abrtd, atd, auditd, autofs, cgconfig, iptables, ip6tables, irqbalance, iscsi, iscsid, ksm, ksmtuned, nfslock, postfix, rhnsd, rpcbind, rpcgssd, cpuspeed
• /etc/sysctl.conf was updated with the following lines:
•   net.core.somaxconn = 8192
•   net.ipv4.tcp_max_tw_buckets = 500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   net.core.netdev_max_backlog = 500000
•   kernel.sched_min_granularity_ns = 10000000
•   kernel.sched_wakeup_granularity_ns = 15000000
•   fs.aio-max-nr = 1048576
•   vm.dirty_ratio = 5
•   vm.dirty_background_ratio = 2
• In /etc/security/limits.conf
•   hard and soft nofile set to 102400,
•   hard and soft nproc set to 102400
• Network configuration is as follows:
•   The first integrated 331FLR NIC port(eth0) is for host access.
•   The 4 Solarflare NIC ports were assigned to bridges (eth4 to br0, eth5 to br1, eth6 to br2, eth7 to br3).
•   All the public VNICs of the tiles use the 4 bridges,
•   the odd tiles use br0 & br1, the even tiles use br2 & br3.
•   The Solarflare NICs were configured with
•   sfboot sriov=enabled vf-count=127 vf-msix-limit=1
•   All the webservers, infraservers, mailservers dbservers and appservers were assigned with a VF.
• All VMs are managed with libvirt and created using virsh utility
• All VMs are created with the following XML template (exceptions noted below template)
•   <domain type='kvm'>
•    <name>$vm_name</name>
•    <memory unit='MiB'>$vm_memory</memory>
•    <currentMemory unit='MiB'>$vm_memory</currentMemory>
•    <vcpu cpuset='$vm_cpu_set'>1</vcpu>
•    <os>
•     <type arch='x86_64' machine='rhel6.4.0'>hvm</type>
•     <boot dev='hd'/>
•    </os>
•    <features>
•     <acpi/>
•     <apic/>
•     <pae/>
•    </features>
•    <cpu match='exact'>
•     <model>SandyBridge</model>
•     <topology sockets='1' cores='1' threads='1'/>
•    </cpu>
•    <clock offset='utc'>
•     <timer name='pit' tickpolicy='delay'/>
•    </clock>
•    <on_poweroff>destroy</on_poweroff>
•    <on_reboot>restart</on_reboot>
•    <on_crash>restart</on_crash>
•    <devices>
•    <emulator>/usr/libexec/qemu-kvm</emulator>
•    <disk type='block' device='disk'>
•    <driver name='qemu' type='raw' cache='none' io='native'/>
•    <source dev='$vm_disk'/>
•    <target dev='vda' bus='virtio'/>
•    <address type='pci' domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
•    </disk>
•    <interface type='bridge'>
•    <mac address='$mac address'/>
•    <source bridge='$bridge'/>
•    <model type='virtio/>
•    <address type='pci' domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
•    </interface>
•    <serial type='pty'>
•    <target port='0'/>
•    </serial>
•    <console type='pty'>
•    <target type='serial' port='0'/>
•    </console>
•    <input type='tablet' bus='usb'/>
•    <input type='mouse' bus='ps2'/>
•    <graphics type='vnc' port='-1' autoport='yes'/>
•    <video>
•    <model type='cirrus' vram='9216' heads='1'/>
•    <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
•    </video>
•    <memballoon model='none'/>
•    </devices>
•    </domain>
• After VMs are started, virtual functions are utilized by using command:
• 'virsh attach $vm_name $device.xml' to hot-plug the interface into VM, where $device.xml has the form:
• <hostdev mode='subsystem' type='pci' managed='no'>
• <source>
• <address domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </source>
• </hostdev>
• Idleserver VMs do not use SR-IOV for network adapter, and therefore do not have hot-plug device added.
• VMs are bound to host CPUs via their XML configuration in the following manner:
• All VMs of tiles 1-14,30,32 are bound to NUMA node 0
• All VMs of tiles 15-28,29,31 are bound to NUMA node 1
• App, Mail, Infra and Idle VMs of 33 are bound to NUMA node 0
• Web and DB VMs of 33 are bound to NUMA node 1
• VMs' nice value was set after startup via the renice command to the following settings:
•   appservers = -20
•   dbserver = -12
•   idleserver = 19
•   infraserver = 2
•   mailserver = 6
•   webserver = 3

Web Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
•   abrtd, atd, auditd, autofs, cpuspeed, cups, dovecot, ip6tables, iptables, irqbalance, kdump, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcbind, rpcgssd, rpcidmapd
• Following items were added in /etc/sysctl.conf
•   fs.file-max = 500000
•   net.core.somaxconn = 8192
•   net.ipv4.tcp_max_tw_buckets = 500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   net.ipv4.neigh.default.gc_thresh1 = 4096
•   net.ipv4.neigh.default.gc_thresh2 = 8192
•   net.ipv4.neigh.default.gc_thresh3 = 8192
•   net.ipv4.neigh.default.base_reachable_time = 86400
•   net.ipv4.neigh.default.gc_stale_time = 86400
•   net.ipv6.neigh.default.gc_thresh1 = 4096
•   net.ipv6.neigh.default.gc_thresh2 = 8192
•   net.ipv6.neigh.default.gc_thresh3 = 8192
•   net.ipv6.neigh.default.base_reachable_time_ms = 86400
•   net.ipv4.tcp_tw_reuse = 1
• In /etc/security/limits.conf
•   hard and soft nofile set to 102400,
•   hard and soft nproc set to 102400
• root file-system options: barrier=0,noatime
• The following items were set in /opt/common_scripts/ostune_misc.sh which was executed after boot up
•   net.ipv4.tcp_syn_retries=4
•   net.ipv4.tcp_retries2=7
•   net.ipv4.tcp_keepalive_time=120
•   vm.swappiness=0
•   vm.overcommit_memory=1
• httpd.conf changes
•   Timeout 20
•   KeepAlive on
•   MaxKeepAliveRequest 0
•   KeepAliveTimeout 60
•   StartServers 1024
•   MinSpareServers 1
•   MaxSpareServers 1024
•   ServerLimit 1024
•   MaxClients 1024
•   MaxRequestsPerChild 0
•   CustomLog logs/access_log common
•   The following modules were disabled: auth_basic_module, auth_digest_module, authn_file_module, authn_alias_module, authn_anon_module, authn_dbm_module, authn_default_module, authz_user_module, authz_owner_module, authz_groupfile_module, authz_dbm_module, authz_default_module, ldap_module, authnz_ldap_module, ext_filter_module, mime_magic_module, dav_module, status_module, dav_fs_module, userdir_module, rewrite_module, proxy_module, proxy_balancer, proxy_ftp_module, proxy_http_module, proxy_ajp_module, proxy_connect_module, cache_module, disk_cache_module
• NFS mount options:
•   noatime,actimeo=14400,nocto

Application Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
•   abrtd, atd, auditd, autofs, cpuspeed, cups, dovecot, httpd, ip6tables, iptables, irqbalance, kdump, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcbind, rpcgssd, rpcidmapd
• Following items were added in /etc/sysctl.conf
•   fs.file-max = 500000
•   net.core.somaxconn = 8192
•   net.ipv4.tcp_max_tw_buckets = 500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   net.ipv4.neigh.default.gc_thresh1 = 4096
•   net.ipv4.neigh.default.gc_thresh2 = 8192
•   net.ipv4.neigh.default.gc_thresh3 = 8192
•   net.ipv4.neigh.default.base_reachable_time = 86400
•   net.ipv4.neigh.default.gc_stale_time = 86400
•   net.ipv6.neigh.default.gc_thresh1 = 4096
•   net.ipv6.neigh.default.gc_thresh2 = 8192
•   net.ipv6.neigh.default.gc_thresh3 = 8192
•   net.ipv6.neigh.default.base_reachable_time_ms = 86400
•   net.ipv4.tcp_tw_reuse = 1
• In /etc/security/limits.conf
•   hard and soft nofile set to 102400,
•   hard and soft nproc set to 102400
• root file-system options: barrier=0
• The following items were set in /opt/common_scripts/ostune_misc.sh which was executed after boot up
•   sysctl -w vm.swappiness=0
•   sysctl -w vm.overcommit_memory=1
•   echo 658 > /proc/sys/vm/nr_hugepages
•   id -g spec > /proc/sys/vm/hugetlb_shm_group
•   mount -t hugetlbfs hugetlbfs /dev/hugepages
• The following lines were added to glassfish domain.xml:
•   To jdbc-connection-pool line parameter 'max-pool-size=128' was set
•   To jdbc-connection-pool line parameter 'steady-pool-size=64' was set
•   To ejb-container line, parameter 'max-pool-size=256' was set
•   To mdb-container line, parameter 'max-pool-size=256' was set
•   In http-thread-pool section: parameter 'max-thread-pool-size=32' was set
•   In jvm-options section: set -XX:+UseLargePages, set -Xmn600m, set -Xms1024m, set -Xmx1024m
•   set -XX:-UseBiasedLocking, -XX:CompileThreshold=1000, -XX:+AggressiveOpts, -XX:+UseCompressedOops
•   set -XX:+UseFastAccessorMethods, -XX:+UseStringCache, -XX:+OptimizeStringConcat

Mail Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
•   abrtd, atd, auditd, autofs, cpuspeed, cups, httpd, ip6tables, iptables, irqbalance, kdump, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcbind, rpcgssd, rpcidmapd
• Following items were added in /etc/sysctl.conf
•   fs.file-max = 500000
•   net.core.somaxconn = 8192
•   net.ipv4.tcp_max_tw_buckets = 500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   net.ipv4.neigh.default.gc_thresh1 = 4096
•   net.ipv4.neigh.default.gc_thresh2 = 8192
•   net.ipv4.neigh.default.gc_thresh3 = 8192
•   net.ipv4.neigh.default.base_reachable_time = 86400
•   net.ipv4.neigh.default.gc_stale_time = 86400
•   net.ipv6.neigh.default.gc_thresh1 = 4096
•   net.ipv6.neigh.default.gc_thresh2 = 8192
•   net.ipv6.neigh.default.gc_thresh3 = 8192
•   net.ipv6.neigh.default.base_reachable_time_ms = 86400
•   net.ipv4.tcp_tw_reuse = 1
• In /etc/security/limits.conf
•   hard and soft nofile set to 102400,
•   hard and soft nproc set to 102400
•   root file-system options: barrier=0
• datastore file-system options: noatime,barrier=0,data=writeback
• Contents of dovecot configuration file /etc/dovecot/dovecot.conf:
• dict {
• }
• !include conf.d/*.conf
• base_dir = /var/run/dovecot/
• disable_plaintext_auth = no
• log_path = /var/log/dovecot
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• mbox_write_locks = fcntl
• passdb {
• args = scheme=plain /etc/dovecot/passwd.dovecot
• driver = passwd-file
• }
• postmaster_address = postmaster@example.com
• service auth {
• user = root
• }
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• service_count = 0
• }
• service imap {
• process_limit = 1024
• }
• ssl = no
• ssl_cert = </etc/pki/dovecot/certs/dovecot.pem
• ssl_key = </etc/pki/dovecot/private/dovecot.pem
• userdb {
• driver = passwd
• }

Database Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
•   abrtd, atd, auditd, autofs, cpuspeed, cups, dovecot, httpd, ip6tables, iptables, irqbalance, kdump, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcbind, rpcgssd, rpcidmapd
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
•   net.core.somaxconn = 8192
•   net.ipv4.tcp_max_tw_buckets = 500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   net.ipv4.neigh.default.gc_thresh1 = 4096
•   net.ipv4.neigh.default.gc_thresh2 = 8192
•   net.ipv4.neigh.default.gc_thresh3 = 8192
•   net.ipv4.neigh.default.base_reachable_time = 86400
•   net.ipv4.neigh.default.gc_stale_time = 86400
•   net.ipv6.neigh.default.gc_thresh1 = 4096
•   net.ipv6.neigh.default.gc_thresh2 = 8192
•   net.ipv6.neigh.default.gc_thresh3 = 8192
•   net.ipv6.neigh.default.base_reachable_time_ms = 86400
•   net.ipv4.tcp_tw_reuse = 1
• In /etc/security/limits.conf
•   hard and soft nofile set to 102400,
•   hard and soft nproc set to 102400
•   root file-system options: barrier=0
•   datastore file-system options: noatime,barrier=0,data=writeback
• Database tuning options in /var/lib/pgsql/9.0/data/postgresql.conf
•   wal_buffers = 20MB
•   checkpoint_completion_target = 0.9
•   fsync = off
•   synchronous_commit = off
•   wal_sync_method = fdatasync
•   effective_cache_size = 1200MB
•   shared_buffers = 1000MB
•   work_mem = 64MB
•   maintenance_work_mem = 1MB
•   max_connections = 250
•   ssl = off
•   full_page_writes = off
•   debug_assertions = false
•   log_min_duration_statement = 1000
•   default_statistics_target = 100
•   log_rotation_size = 16MB
•   log_destination = syslog
• The following items were set in /opt/common_scripts/ostune_misc.sh which was executed after boot up
•   id -g postgres > /proc/sys/vm/hugetlb_shm_group
•   echo never > /sys/kernel/mm/transparent_hugepage/enabled
•   sysctl -w vm.nr_hugepages=0
•   sysctl -w vm.nr_hugepages=550
•   sysctl -w vm.nr_overcommit_hugepages=200

Infraserver VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
•   abrtd, atd, auditd, autofs, cpuspeed, cups, dovecot, httpd, ip6tables, iptables, irqbalance, kdump, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcgssd
• Following items were added in /etc/sysctl.conf
•   fs.file-max = 500000
•   net.core.somaxconn = 8192
•   net.ipv4.tcp_max_tw_buckets = 500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   net.ipv4.neigh.default.gc_thresh1 = 4096
•   net.ipv4.neigh.default.gc_thresh2 = 8192
•   net.ipv4.neigh.default.gc_thresh3 = 8192
•   net.ipv4.neigh.default.base_reachable_time = 86400
•   net.ipv4.neigh.default.gc_stale_time = 86400
•   net.ipv6.neigh.default.gc_thresh1 = 4096
•   net.ipv6.neigh.default.gc_thresh2 = 8192
•   net.ipv6.neigh.default.gc_thresh3 = 8192
•   net.ipv6.neigh.default.base_reachable_time_ms = 86400
•   net.ipv4.tcp_tw_reuse = 1
• In /etc/security/limits.conf
•   hard and soft nofile set to 102400,
•   hard and soft nproc set to 102400
• root file-system options: barrier=0,noatime
• datastore file-system options: noatime,barrier=0,data=writeback
• The following items were set in /opt/common_scripts/ostune_misc.sh which was executed after boot up
•   net.ipv4.tcp_syn_retries=4
•   net.ipv4.tcp_retries2=7
•   net.ipv4.tcp_keepalive_time=120
• httpd.conf changes
•   Timeout 20
•   KeepAlive On
•   MaxKeepAliveRequests 0
•   KeepAliveTimeout 60
•   StartServers 10
•   MinSpareServers 10
•   MaxSpareServers 10
•   ServerLimit 1536
•   MaxClients 1536
•   MaxRequestsPerChild 0
•   CustomLog logs/access_log common
• NFS export options:
•   rw,async,no_subtree_check,no_root_squash

Idle Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
•   abrtd, atd, auditd, autofs, cpuspeed, cups, dovecot, httpd, ip6tables, iptables, irqbalance, kdump, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcbind, rpcgssd, rpcidmapd
• Following items were added in /etc/sysctl.conf
•   fs.file-max = 500000
•   net.core.somaxconn = 8192
•   net.ipv4.tcp_max_tw_buckets = 500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   net.ipv4.neigh.default.gc_thresh1 = 4096
•   net.ipv4.neigh.default.gc_thresh2 = 8192
•   net.ipv4.neigh.default.gc_thresh3 = 8192
•   net.ipv4.neigh.default.base_reachable_time = 86400
•   net.ipv4.neigh.default.gc_stale_time = 86400
•   net.ipv6.neigh.default.gc_thresh1 = 4096
•   net.ipv6.neigh.default.gc_thresh2 = 8192
•   net.ipv6.neigh.default.gc_thresh3 = 8192
•   net.ipv6.neigh.default.base_reachable_time_ms = 86400
•   net.ipv4.tcp_tw_reuse = 1
• In /etc/security/limits.conf
•   hard and soft nofile set to 102400,
•   hard and soft nproc set to 102400
• root file-system options: barrier=0
• The following items were set in /opt/common_scripts/ostune_misc.sh which was executed after boot up
•   vm.swappiness=0
•   vm.overcommit_memory=1

Client Driver Notes

• All prime clients were VMs running on Red Hat Enterprise Linux Server 6.5 KVM.
• The prime controller runs on the first vClient.
• Each vClient has 2 vCPUs, 8GB memory, 1 virtio_blk device and 1 virtio_net device.
• All physical clients have 2 x 10 Gbps ports, all ports are connected to the private network,
• and are configured as 2 software bridges (br0 and br1), each vClient use one bridge,
• the odd vClient use br0 and the even vClient use br1.
• 8 physical clients host 32 vClients, 4 vClients per physical system,
• the 9th physical client hosts 33rd vClients.

Other Notes

• A single HP H3C S5820-24XG-SFP+ switch was used to connect SUT to clients.