HyperTransport Technology (Default=HT 3)

This BIOS option allows setting the speed of the HyperTransport (HT) interconnect. HT 3 capable systems/processors ship with "HT 3" by default, but this may be set to "HT 1". HT 3 increases the interconnect rate from 2GT/s with HT 1 up to a maximum 4.8GT/s with HT 3 which may improve performance on non-NUMA aware workloads. Although HT 3 runs the interconnect at a higher speed, there is a possibility of a slight performance overhead on some platforms and workloads due to enabling CRC checking on the interconnect and to increased pipelines to handle the higher interconnect speed.

For most workloads the default setting should be used, but there may be cases where performance is improved by changing the default setting. However, users should verify improved performance before modifying this option.

Node Interleaving Enabled (Default = Disabled):

This BIOS option allows the enabling/disabling of memory interleaving across CPU nodes. When disabled, each CPU chip can only access memory within its own node.

Linux Huge Page settings

In order to take full advantage of using PGI's huge page runtime library, your system must be configured to use huge pages. It is safe to run binaries compiled with "-Msmartalloc=huge" on systems not configured to use huge pages, however, you will not benefit from the performance improvements huge pages offer. To configure your system for huge pages perform the following steps:

• Create a mount point for the huge pages: "mkdir /mnt/hugepages"
• The huge page file system needs to be mounted when the systems reboots. Add the following to a system boot configuration file before any services are started: "mount -t hugetlbfs nodev /mnt/hugepages"
• Set vm/nr_hugepages=N in your /etc/sysctl.conf file where N is the maximum number of pages the system may allocate.
• Reboot to have the changes take effect.

Note that further information about huge pages may be found in your Linux documentation file: /usr/src/linux/Documentation/vm/hugetlbpage.txt

PGI_HUGE_PAGES

The maximum number of huge pages an application is allowed to use can be set at run time via the environment variable PGI_HUGE_PAGES. If not set, then the process may use all available huge pages when compiled with "-Msmartalloc=huge" or a maximum of n pages where the value of n is set via the compile time flag "-Msmartalloc=huge:n.

Using numactl to bind processes and memory to cores

For multi-copy runs or single copy runs on systems with multiple sockets, it is advantageous to bind a process to a particular core. Otherwise, the OS may arbitrarily move your process from one core to another. This can effect performance. To help, SPEC allows the use of a "submit" command where users can specify a utility to use to bind processes. We have found the utility 'numactl' to be the best choice.

numactl runs processes with a specific NUMA scheduling or memory placement policy. The policy is set for a command and inherited by all of its children. The numactl flag "--physcpubind" specifies which core(s) to bind the process. "-l" instructs numactl to keep a process memory on the local node while "-m" specifies which node(s) to place a process memory. For full details on using numactl, please refer to your Linux documentation, 'man numactl'

Note that some versions of numactl, particularly the version found on SLES 10, we have found that the utility incorrectly interprets application arguments as it's own. For example, with the command "numactl --physcpubind=0 -l a.out -m a", numactl will interpret a.out's "-m" option as it's own "-m" option. To work around this problem, a user can put the command to be run in a shell script and then run the shell script using numactl. For example: "echo 'a.out -m a' > run.sh ; numactl --physcpubind=0 bash run.sh"

submit = echo "$command" > run.sh ; $BIND bash run.sh

When running multiple copies of benchmarks, the SPEC config file feature submit is sometimes used to cause individual jobs to be bound to specific processors. This specific submit command is used for Linux. The description of the elements of the command are:

• echo "$command" > run.sh : $command is the program to be started, in this case, the benchmark instance to be started. $command is redirected into the file run.sh
• $BIND bash run.sh:
  The run.sh script is executed within a bash shell. $BIND is replaced by the appropriate bind value. In this case the bind statement is:
  
  • bindn = numactl -m nodes --physcpubind=cpus
• numactl (-m nodes | -l) --physcpubind=cpus
  numactl runs processes with a specific NUMA scheduling or memory placement policy. The policy is set for command and inherited by all of its children. In addition it can set persistent policy for shared memory segments or files. The switches used above are defined as follows:
  • --physcpubind=cpus, -C cpus
    Only execute process on cpus. This accepts physical cpu numbers as shown in the processor fields of /proc/cpuinfo.
  • --membind=nodes, -m nodes
    Only allocate memory from nodes. Allocation will fail when there is not enough memory available on these nodes.
  • --localalloc, -l
    Do always local allocation on the current node.

ulimit -s <n>

Sets the stack size to n kbytes, or unlimited to allow the stack size to grow without limit.

ulimit -l <n>

Sets the maximum size of memory that may be locked into physical memory.

NCPUS

Sets the maximum number of OpenMP parallel threads auto-parallelized (-Mconcur) applications may use.

powersave -f

Makes the powersave daemon set the CPUs to the highest supported frequency.

HUGETLB_MORECORE

Setting this to yes instructs libhugetlbfs to override libc's normal morecore() function with a hugepage version and use it for malloc().

HUGETLB_LIMIT

For the x86 Open64 compiler, the maximum number of huge pages an application is allowed to use can be set at run time via the environment variable HUGETLB_LIMIT. If not set, then the process may use all available huge pages when compiled with "-HP (or -HUGEPAGE)" or a maximum of n pages where the value of n is set via the compile time flag "-HP:limit=n".