CPU2006 Flag Description
Hewlett Packard Enterprise ProLiant DL385 Gen10 (2.10 GHz, AMD EPYC 7281)

Test sponsored by HPE

Compilers: x86 Open64 Compiler Suite

Implicitly Included Flags

This section contains descriptions of flags that were included implicitly by other flags, but which do not have a permanent home at SPEC.

• • -O3
  • 

  • Perform all the optimizations at the -O2 level as well as many more aggressive optimizations. Examples of such aggressive optimizations are loop nest optimizations and generation of prefetch instructions. Although these more aggressive optimizations can significantly speed up the run time execution of the compiled program, in rare cases they may not be profitable and may instead lead to a slow down. Also, some of these more aggressive optimizations may affect the accuracy of some floating point computations.

    If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective. Level 2 is assumed if no value is specified (i.e. "-O". The default is "-O2".

  • Includes:
    • -O2
      • -O1
• • -O2
  • 

  • Perform extensive global optimizations. Examples of such optimizations are control flow optimizations, partial redundancy elimination, and strength reduction. These optimizations can very often reduce the execution time of the compiled program significantly, but they may do so at the expense of increased compilation time. This is the default level of optimization.

    If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective. Level 2 is assumed if no value is specified (i.e. "-O". The default is "-O2".

  • Includes:
    • -O1
• • -O1
  • 

  • Perform minimal local optimizations on sections of straight-line code (basic blocks) only. Examples of such optimizations are instruction scheduling and some peephole optimizations. These optimizations do not usually have any noticeable impact on compilation time.

    If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective. Level 2 is assumed if no value is specified (i.e. "-O". The default is "-O2".

• • -ipa
  • 

  • Instructs the compiler to invoke inter-procedural analysis. Specifying "-ipa" is equivalent to "-IPA" and "-IPA:" with no suboptions, thus the default settings for the individual IPA suboptions are used.

• • -OPT:Ofast
  • 

  • -OPT:Ofast
    Maximizes performance for a given platform using the selected optimizations. "-OPT:Ofast" specifies four optimizations; "-OPT:ro=2", "-OPT:Olimit=0", "-OPT:div_split=ON", and "-OPT:alias=typed". Note the specified optimizations are ordinarily safe but floating point accuracy due to transformations may be diminished.

  • Includes:
    • -OPT:ro
    • -OPT:Olimit
    • -OPT:div_split
    • -OPT:alias
• • -OPT:ro
  • 

  • -OPT:roundoff,ro=(0|1|2|3)
    "-OPT:roundoff" specifies acceptable levels of divergence for both accuracy and overflow/underflow behavior of floating-point results relative to the source language rules. The roundoff value is in the range 0-3 with each value described as follows:
    0 Do no transformations which could affect floating-point results. The default for optimization levels "-O0", "-O1", and "-O2".
    1 Allow all transformations which have a limited affect on floating-point results. For roundoff, limited is defined as only the last bit or two of the mantissa is affected. For overflow or underflow, limited is defined as intermediate results of the transformed calculation may overflow or underflow within a factor of two of where the original expression may have overflowed or underflowed. Note that effects may be less limited when compounded by multiple transformations. This is the default when "-O3" is specified.
    2 Specifies transformations with extensive effects on floating-point results. For example, allow associative rearrangement (i.e. even across loop iterations) and the distribution of multiplication over addition or subtraction. Do not specify transformations known to cause: a. cumulative roundoff errors, or b. overflow/underflow of operands in a large range of valid floating-point values. This is the default when specifying "-OPT:Ofast".
    3 Specify any mathematically valid transformation of floating-point expressions. For example, floating point induction variables in loops are permitted (even if known to cause cumulative roundoff errors). Also permitted are fast algorithms for complex absolute value and divide (which will overflow/underflow for operands beyond the square root of the representable extremes).

• • -OPT:Olimit
  • 

  • -OPT:Olimit=N
    Controls the size of procedures to be optimized. Procedures above the specified cutoff limit, N, are not optimized. N=0 means "infinite Olimit", which causes all procedures to be optimized with no consideration regarding compilation times. Note if "-OPT:Ofast" is enabled then "-OPT:Olimit=0" or when "-O3" is enabled "-OPT:Olimit=9000". The default is "-OPT:Olimit=6000".

• • -OPT:div_split
  • 

  • -OPT:div_split=(on|off|0|1)
    Instruct the compiler to transform x/y into x*(recip(y)). Flags -OPT:Ofast or -OPT:IEEE_arithmetic=3 will enable this optimization. Note this transform generates fairly accurate code. The default is "-OPT:div_split=OFF".

• • -OPT:alias
  • 

  • The "-OPT:" option group controls various optimizations. The "-OPT:" options supersede the defaults that are based on the main optimization level.

    -OPT:alias=<model>
    Identify which pointer aliasing model to use. The compiler will make assumptions during compilation when one or more of the following <model> is specified:

    typed
    Assumes that two pointers of different types will not point to the same location in memory (i.e. the code adheres to the ANSI/ISO C standards). Note when specifying "-OPT:Ofast" turns this option ON.

    (restricted|restrict)
    Assumes that distinct pointers are pointing to distinct non-overlapping objects. The default is that this optimization is disabled.

    disjoint
    Assumes that any two pointer expressions are pointing to distinct non-overlapping objects. This default is that this optimization is disabled.

    ^M field_sensitive
    ^M Replaces the alias algorithm with an alternate implementation that tracks fields of individual pointers, i.e. it is field sensitive. This alternate implementation is also designed to be flow-insensitive, scalable and context-sensitive to heap allocations.

    no_f90_pointer_alias
    Assumes that any two different Fortran 90 pointers are pointing to distinct non-overlapping objects. The default is that this optimization is disabled.

• • -fno-math-errno
  • 

  • Do not set ERRNO after calling math functions that are executed with a single instruction, e.g. sqrt. A program that relies on IEEE exceptions for math error handling may want to use this flag for speed while maintaining IEEE arithmetic compatibility. Note specifying "-Ofast" implies "-fno-math-errno". The default is "-fmath-errno".

• • -ffast-math
  • 

  • "-fast-math" instructs the compiler to relax ANSI/ISO or IEEE rules/specifications for math functions in order to optimize floating-point computations to improve runtime. "-fno-fast-math" instructs the compiler to conform to ANSI and IEEE math rules. This option causes the preprocessor macro __FAST_MATH__ to be defined.
    Note:
    "-Ofast" implies "-ffast-math".
    "-ffast-math" sets options "-fno-math-errno" and "-OPT:IEEE_arithmetic=2".
    "-fno-fast-math" sets options "-fmath-errno" and "-OPT:IEEE arithmetic=1".

Commands and Options Used to Submit Benchmark Runs

Using numactl to bind processes and memory to cores

For multi-copy runs or single copy runs on systems with multiple sockets, it is advantageous to bind a process to a particular core. Otherwise, the OS may arbitrarily move your process from one core to another. This can effect performance. To help, SPEC allows the use of a "submit" command where users can specify a utility to use to bind processes. We have found the utility 'numactl' to be the best choice.

numactl runs processes with a specific NUMA scheduling or memory placement policy. The policy is set for a command and inherited by all of its children. The numactl flag "--physcpubind" specifies which core(s) to bind the process. "-l" instructs numactl to keep a process memory on the local node while "-m" specifies which node(s) to place a process memory. For full details on using numactl, please refer to your Linux documentation, 'man numactl'

Note that some versions of numactl, particularly the version found on SLES 10, we have found that the utility incorrectly interprets application arguments as it's own. For example, with the command "numactl --physcpubind=0 -l a.out -m a", numactl will interpret a.out's "-m" option as it's own "-m" option. To work around this problem, a user can put the command to be run in a shell script and then run the shell script using numactl. For example: "echo 'a.out -m a' > run.sh ; numactl --physcpubind=0 bash run.sh"

numactl is also used to invoke runspec so that mememory usage is spread evenly among NUMA nodes. This is accomplished as follows: runspec_command="numactl --interleave=all runspec"

Shell, Environment, and Other Software Settings

Linux Huge Page settings

In order to take full advantage of using x86 Open64's huge page runtime library, your system must be configured to use huge pages. It is safe to run binaries compiled with "-HP" on systems not configured to use huge pages, however, you will not benefit from the performance improvements huge pages offer. To configure your system for huge pages perform the following steps:

• Create a mount point for the huge pages: "mkdir /mnt/hugepages"
• The huge page file system needs to be mounted when the systems reboots. Add the following to a system boot configuration file before any services are started: "mount -t hugetlbfs nodev /mnt/hugepages"
• Set vm/nr_hugepages=N in your /etc/sysctl.conf file where N is the maximum number of pages the system may allocate.
• Reboot to have the changes take effect.

Note that further information about huge pages may be found in your Linux documentation file: /usr/src/linux/Documentation/vm/hugetlbpage.txt

HUGETLB_LIMIT

For the x86 Open64 compiler, the maximum number of huge pages an application is allowed to use can be set at run time via the environment variable HUGETLB_LIMIT. If not set, then the process may use all available huge pages when compiled with "-HP (or -HUGEPAGE)" or a maximum of n pages where the value of n is set via the compile time flag "-HP:limit=n".

Transparent Huge Pages (THP)

THP is an abstraction layer that automates most aspects of creating, managing, and using huge pages. THP is designed to hides much of the complexity in using huge pages from system administrators and developers, as normal huge pages must be assigned at boot time, can be difficult to manage manually, and often require significant changes to code in order to be used effectively.

Set transparent_hugepage boot parameter

In the file /boot/grub/menu.lst, add the boot parameter "transparent_hugepage=never" to the OS you plan to select during boot, to instruct it to disable Transparent Huge Pages (THP). A reboot is required for this setting to take effect.

Set Ubuntu power governor to performance

To produce the best performance on Ubuntu, the system power governor must be set to performance as follows:

• Install cpufrequtils: "sudo apt install cpufrequtils"
• Edit the following system file (create it if it doesn't exist): "sudo vi /etc/default/cpufrequtils"
• Add the following line: GOVERNOR="performance"
• Save and exit your editor.
• Disable the ondemand daemon so that your changes will not be overwritten: "sudo update-rc.d ondemand disable"

ulimit -s <n>

Sets the stack size to n kbytes, or unlimited to allow the stack size to grow without limit.

ulimit -l <n>

Sets the maximum size of memory that may be locked into physical memory.

dirty_ratio

Sets the percentage limit of system memory that can hold dirty cache data until it is written out via pdflush.

swappiness

The swappiness value can range from 1 to 100. A value of 100 will cause the kernel to swap out inactive processes frequently in favor of file system performance, resulting in large disk cache sizes. A value of 1 tells the kernel to only swap processes to disk if absolutely necessary.

zone_reclaim_mode

When zone_reclaim_mode is set to 0, the kernel will allocate memory from a remote node, rather than try to reclaim memory from the local node. A value of 1 will cause the page allocator to reclaim local page caches that are not currently used before allocating remote node memory.

sync, drop_caches

Used in conjunction, the two commands, sync and drop_caches, free disk cache memory for other uses. sync writes dirty pages to disk, while drop_caches reclaims clean disk cache pages.

OMP_NUM_THREADS

Sets the maximum number of OpenMP parallel threads auto-parallelized (-apo) applications may use.

O64_OMP_AFFINITY_MAP

Specifies the thread-CPU relationship when the operating system's affinity mechanism is used to assign OpenMP threads to CPUs.

O64_OMP_SPIN_USER_LOCK

Specifies whether or not to use the user-level spin mechanism for OpenMP locks. If the variable is set to TRUE then user-level spin mechanisms are used. If the variable is set to FALSE then pthread mutexes are used. The default if the variable is not set is the same as FALSE.

powersave -f (on SuSE)

Makes the powersave daemon set the CPUs to the highest supported frequency.

/etc/init.d/cpuspeed stop (on Red Hat)

Disables the cpu frequency scaling program in order to set the CPUs to the highest supported frequency.

LD_LIBRARY_PATH

An environment variable set to include the x86 Open64 and SmartHeap libraries used during compilation of the binaries. This environment variable setting is not needed when building the binaries on the system under test.

kernel/randomize_va_space

This option can be used to select the type of process address space randomization that is used in the system, for architectures that support this feature. 0 - Turn the process address space randomization off. This is the default for architectures that do not support this feature anyways, and kernels that are booted with the "norandmaps" parameter. 1 - Make the addresses of mmap base, stack and VDSO page randomized. This, among other things, implies that shared libraries will be loaded to random addresses. Also for PIE-linked binaries, the location of code start is randomized. This is the default if the CONFIG_COMPAT_BRK option is enabled. 2 - Additionally enable heap randomization. This is the default if CONFIG_COMPAT_BRK is disabled.

O64_OMP_SPIN_COUNT

Specify the number of times the spin loops will spin at user-level before falling back to operating system schedule/reschedule mechanisms. The default value is 20000.

Operating System Tuning Parameters

OS Tuning

ulimit:

Used to set user limits of system-wide resources. Provides control over resources available to the shell and processes started by it. Some common ulimit commands may include:

• ulimit -s [n | unlimited]: Set the stack size to n kbytes, or unlimited to allow the stack size to grow without limit.
• ulimit -l (number): Set the maximum size that can be locked into memory.

Zone Reclaim:

Zone reclaim allows the reclaiming of pages from a zone if the number of free pages falls below a watermark even if other zones still have enough pages available. Reclaiming a page can be more beneficial than taking the performance penalties that are associated with allocating a page on a remote zone, especially for NUMA machines.

Performance Governors (Linux):

In-kernel CPU frequency governors are pre-configured power schemes for the CPU. The CPUfreq governors use P-states to change frequencies and lower power consumption. The dynamic governors can switch between CPU frequencies, based on CPU utilization to allow for power savings while not sacrificing performance.

To set the governor, use the following commmand: "cpupower frequency-set -r -g {desired_governor}"

Disabling Linux services:

Certain Linux services may be disabled to minimize tasks that may consume CPU cycles.

irqbalance:

Disabled through "service irqbalance stop". Depending on the workload involved, the irqbalance service reassigns various IRQ's to system CPUs. Though this service might help in some situations, disabling it can also help environments which need to minimize or eliminate latency to more quickly respond to events.

numa_balancing:

Disabled through "echo 0 > /proc/sys/kernel/numa_balancing". This feature will automatically migrate data on demand so memory nodes are aligned to the local CPU that is accessing data. Depending on the workload involved, enabling this can boost the performance if the workload performs well on NUMA hardware. If the workload is statically set to balance between nodes, then this service may not provide a benefit.

Tuning Kernel parameters:

The following Linux Kernel parameters were tuned to better optimize performance of some areas of the system:

• vm.dirty_background_ratio: Set through "echo 40 > /proc/sys/vm/dirty_background_ratio". This setting can help Linux disk caching and performance by setting the percentage of system memory that can be filled with dirty pages.
• vm.dirty_ratio: Set through "echo 40 > /proc/sys/vm/dirty_ratio". This setting is the absolute maximum amount of system memory that can be filled with dirty pages before everything must get committed to disk.
• ksm/sleep_millisecs: Set through "echo 200 > /sys/kernel/mm/ksm/sleep_millisecs". This setting controls how many milliseconds the ksmd (KSM daeomn) should sleep before the next scan.
• khugepaged/scan_sleep_millisecs: Set through "echo 50000 > /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs". This setting controls how many milliseconds to wait in khugepaged is there is a hugepage allocation failure to throttle the next allocation attempt.

swappiness:

The swappiness value can range from 1 to 100. A value of 100 will cause the kernel to swap out inactive processes frequently in favor of file system performance, resulting in large disk cache sizes. A value of 1 tells the kernel to only swap processes to disk if absolutely necessary.

tuned-adm:

The tuned-adm tool is a commandline interface for switching between different tuning profiles available to the tuned tuning daeomn available in supported Linux distros. The default configuration file is located in /etc/tuned.conf and the supported profiles can be found in /etc/tune-profiles.

Some profiles that may be available by default include: default, desktop-powersave, server-powersave, laptop-ac-powersave, laptop-battery-powersave, spindown-disk, throughput-performance, latency-performance, enterprise-storage

To set a profile, one can issue the command "tuned-adm profile (profile_name)". Here are details about relevant profiles.

• throughput-performance: Server profile for typical throughput tuning. This profile disables tuned and ktune power saving features, enables sysctl settings that may improve disk and network IO throughphut performance, switches to the deadline scheduler, and sets the CPU governor to performance.
• latency-performance: Server profile for typical latency tuning. This profile disables tuned and ktune power saving features, enables the deadline IO scheduler, and sets the CPU governor to performance.
• enterprise-storage: Server profile to high disk throughput tuning. This profile disables tuned and ktune power saving features, enables the deadline IO scheduler, enables hugepages and disables disk barriers, increases disk readahead values, and sets the CPU governor to performance

Transparent Huge Pages (THP):

THP is an abstraction layer that automates most aspects of creating, managing, and using huge pages. THP is designed to hide much of the complexity in using huge pages from system administrators and developers, as normal huge pages must be assigned at boot time, can be difficult to manage manually, and often require significant changes to code in order to be used effectively. Transparent Hugepages increase the memory page size from 4 kilobytes to 2 megabytes. Transparent Hugepages provide significant performance advantages on systems with highly contended resources and large memory workloads. If memory utilization is too high or memory is badly fragmented which prevents hugepages being allocated, the kernel will assign smaller 4k pages instead. Most recent Linux OS releases have THP enabled by default.

Linux Huge Page settings:

If you need finer control and manually set the Huge Pages you can follow the below steps:

• Create a mount point for the huge pages: "mkdir /mnt/hugepages"
• The huge page file system needs to be mounted when the systems reboots. Add the following to a system boot configuration file before any services are started: "mount -t hugetlbfs nodev /mnt/hugepages"
• Set vm/nr_hugepages=N in your /etc/sysctl.conf file where N is the maximum number of pages the system may allocate.
• Reboot to have the changes take effect.

Note that further information about huge pages may be found in your Linux documentation file: /usr/src/linux/Documentation/vm/hugetlbpage.txt

Firmware / BIOS / Microcode Settings

Firmware Settings

One or more of the following settings may have been set. If so, the "Platform Notes" section of the report will say so; and you can read below to find out more about what these settings mean.

AMD SMT Option (Default = Enabled):

This feature allows enabling or disabling of logical processor cores on processors supporting AMD SMT. When enabled, each physical processor core operates as two logical processor cores. When disabled, each physical core operates as only one logical processor core. Enabling this option can improve overall performance for applications that benefit from a higher processor core count.

Thermal Configuration (Default = Optimal Cooling):

This feature allows the user to select the fan cooling solution for the system. Values for this BIOS option can be:

• Optimal Cooling: Provides the most efficient solution by configuring fan speeds to the minimum required to provide adequate cooling.
• Increased Cooling: Will run fans at higher speeds to provide additional cooling. Increased Cooling should be selected when non-HPE storage controllers are cabled to the embedded hard drive cage, or if the system is experiencing thermal issues that cannot be resolved in another manner.
• Maximum Cooling: Will provide the maximum cooling available by this platform.

Performance Determinism (Default = Performance Deterministic):

This option allows the processor to use a given performance level as the max cap, or to let the processor operate as close to the thermal design point (TDP) as possible. Values for this BIOS option can be:

• Power Deterministic: Processor operates as close to the TDP as possible.
• Performance Deterministic: Processor operates at a capped performance level as the max operating state.

Processor Power and Utilization Monitoring (Default = Enabled):

This BIOS option allows the enabling/disabling of iLo Processor State Mode Switching and Insight Power Management Processor Utilization Monitoring.

When set to disabled, the system will also set the Power Regulator mode to Static High Performance mode and the HP Power Profile mode to Custom. This option may be useful in some environments that require absolute minimum latency.

Workload Profile (Default = General Power Efficient Compute):

This option allows a user to choose a workload profile that best fits the user`s needs. The workload profiles control many power and performance settings that are relevant to general workload areas. Values for this BIOS option can be:

• General Power Efficient Compute, General Peak Frequency Compute, General Throughput Compute, Virtualization - Power Efficient, Virtualization - Max Performance, Low Latency, Mission Critical, Transaction Application Processing, High Performance Compute (HPC), Decision Support, Graphic Processing, I/O Throughput, and Custom.

Minimum Processor Idle Power Core C-State (Default = C6 State):

This option can only be configured if the Workload Profile is set to Custom, or this option is not a dependent value for the Workload Profile. This feature selects the processor's lowest idle power state (C-state) that the operating system uses. The higher the C-state, the lower the power usage of that idle state (C6 is the lowest power idle state supported by the processor). Values for this setting can be:

• C6 State: While in C6, the core PLLs are turned off, the core caches are flushed and the core state is saved to the Last Level Cache. Power Gates are used to reduce power consumption to close to zero. C6 is considered an inactive core.
• C1E State: C1E is defined as the enhanced halt state. While in C1E no instructions are being executed. C1E considered an active core.
• No C-states: No C-states is defined as C0, which is defined as the active state. While in C0, instructions are being executed by the core.

Memory Patrol Scrubbing (Default = Enabled):

This option allows for correction of soft memory errors. Over the length of system runtime, the risk of producing multi-bit and uncorrected errors is reduced with this option. Values for this BIOS setting can be:

• Enabled: Correction of soft memory errors can occur during runtime.
• Disabled: Soft memory error correction is turned off during runtime.

Last updated December 11, 2017.

Flag description origin markings:

	Indicates that the flag description came from the user flags file.
	Indicates that the flag description came from the suite-wide flags file.
	Indicates that the flag description came from a per-benchmark flags file.

For questions about the meanings of these flags, please contact the tester.
For other inquiries, please contact webmaster@spec.org
Copyright 2006-2018 Standard Performance Evaluation Corporation
Tested with SPEC CPU2006 v1.2.
Report generated on Tue Mar 6 11:48:23 2018 by SPEC CPU2006 flags formatter v6906.