CPU2017 Flag Description
Hewlett Packard Enterprise ProLiant DL365 Gen11 (3.00 GHz, AMD EPYC 9124)
Test sponsored by HPE
Compilers: AMD Optimizing C/C++ Compiler Suite
Base Compiler Invocation
-
- clang
![[user]](https://www.spec.org/auto/cpu2017/images/user.png)
- CC, LD
-
clang is a C compiler which encompasses preprocessing, parsing, optimization, code generation, assembly, and linking.
Depending on which high-level mode setting is passed, Clang will stop before doing a full link.
-
- clang++
- CXX, LD
-
clang++ C++ compiler which encompasses preprocessing, parsing, optimization, code generation, assembly, and linking.
Depending on which high-level mode setting is passed, Clang will stop before doing a full link.
-
- flang
- FC, LD
-
flang is a Fortran compiler which encompasses parsing, optimization, code generation, assembly, and linking. Depending on
which high-level mode setting is passed, Flang will stop before doing a full link.
Peak Compiler Invocation
-
- clang
- CC, LD
-
clang is a C compiler which encompasses preprocessing, parsing, optimization, code generation, assembly, and linking.
Depending on which high-level mode setting is passed, Clang will stop before doing a full link.
-
- clang++
- CXX, LD
-
clang++ C++ compiler which encompasses preprocessing, parsing, optimization, code generation, assembly, and linking.
Depending on which high-level mode setting is passed, Clang will stop before doing a full link.
-
- flang
- FC, LD
-
flang is a Fortran compiler which encompasses parsing, optimization, code generation, assembly, and linking. Depending on
which high-level mode setting is passed, Flang will stop before doing a full link.
Base Portability Flags
-
![[suite]](https://www.spec.org/auto/cpu2017/images/suite.png)
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
Peak Portability Flags
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
Base Optimization Flags
-
- -m64
- CC, LD
-
Generates code for a 64-bit environment. The 64-bit environment sets int to 32 bits and long and
pointer to 64 bits and generates code for AMD's x86-64 architecture. The compiler generates AMD64, INTEL64,
x86-64 64-bit ABI. The default on a 32-bit host is 32-bit ABI. The default on a 64-bit host is 64-bit ABI if the target
platform specified is 64-bit, otherwise the default is 32-bit.
-
-
-
-
- -O3
- COPTIMIZE
-
Like -O2, except that it enables optimizations that take longer to perform or that may generate larger code (in
an attempt to make the program run faster).
If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.
- Includes:
-
- -march=znver4
- COPTIMIZE
-
Specify that Clang should generate code for a specific processor family member and later. For example, if you specify
-march=znver1, the compiler is allowed to generate instructions that are valid on AMD Zen processors, but
which may not exist on earlier products. -march=znver4 enables AVX 512 ISA for Genoa (znver4) processors.
-
-
- -ffast-math
- COPTIMIZE
-
Enables a range of optimizations that provide faster, though sometimes less precise, mathematical operations that may
not conform to the IEEE-754 specifications. When this option is specified, the __STDC_IEC_559__ macro is
ignored even if set by the system headers.
-
-
-
- -fstruct-layout=7
- COPTIMIZE
-
Analyzes the whole program to determine if the structures in the code can be peeled, if dead or redundant fields can be deleted, and if
the pointer or integer fields in the structure can be compressed. If feasible, this optimization transforms the code to
enable these improvements. This transformation is likely to improve cache utilization and memory bandwidth. It is expected
to improve the scalability of programs executed on multiple cores.
This is effective only under flto as the whole program analysis is required to perform this optimization. You can choose
different levels of aggressiveness with which this optimization can be applied to your application; with 1 being the least
aggressive and 7 being the most aggressive level.
Possible values:
- fstruct-layout=0: disables structure peeling (default).
- fstruct-layout=1: enables structure peeling.
- fstruct-layout=2: enables structure peeling and selectively compresses self-referential pointers in these
structures to 32-bit pointers wherever safe.
- fstruct-layout=3: enables structure peeling and selectively compresses self-referential pointers in these
structures to 16-bit pointers wherever safe.
- fstruct-layout=4: enables structure peeling, pointer compression as in level 2 and further enables
compression of structure fields which are of 64-bit integer type to 32-bit integer type. This is performed under a
strict safety check.
- fstruct-layout=5: enables structure peeling, pointer compression as in level 3 and further enables compression
of structure fields which are of 64-bit integer type to 32-bit integer type. This is performed under a strict safety
check.
- fstruct-layout=6: enables structure peeling, pointer compression as in level 2 and further enables compression
of structure fields which are of type 64-bit integer type to 16-bit integer type. This is performed under a strict
safety check.
- fstruct-layout=7: enables structure peeling, pointer compression as in level 3 and further enables compression
of structure fields which are of type 64-bit integer type to 16-bit integer type. This is performed under a strict
safety check.
- fstruct-layout=8: enables structure peeling, pointer compression, 64 bit integer type compression
as in level 6 and creates optimal ordering of peeled structure fields which could improve runtime performance.
- fstruct-layout=9: enables structure peeling, pointer compression, 64 bit integer type compression
as in level 7 and creates optimal ordering of peeled structure fields which could improve runtime performance.
Note:
fstruct-layout=4 and fstruct-layout=5 are derived from fstruct-layout=2 and fstruct-layout=3 respectively with the added
feature of safe compression of 64-bit integer fields to 32-bit integer fields in structures. Going from
fstruct-layout=4 to fstruct-layout=5 may result in higher performance if the pointer values are such that the pointers
can be compressed to 16-bits.
fstruct-layout=6 and fstruct-layout=7 are derived from fstruct-layout=2 and fstructlayout=3 respectively, with the
added feature of safe compression of 64 bit integer fields to 16 bit integer in structures. Going from fstruct-layout=6
to fstruct-layout=7 may result in higher performance if the pointer values are such that the pointers can be
compressed to 16-bits.
-
-
-
-
-
-
-
- -zopt
- COPTIMIZE
-
This option enables a subset of scalar, vector and loop transformations including improved variants of loop invariant code motion, SLP and loop vectorizations, loop-fusion, loop-interchange, loop-unswitch, loop tiling and loop distribution.
-
-
-
-
-
-
- -m64
- CXX, LD
-
Generates code for a 64-bit environment. The 64-bit environment sets int to 32 bits and long and
pointer to 64 bits and generates code for AMD's x86-64 architecture. The compiler generates AMD64, INTEL64,
x86-64 64-bit ABI. The default on a 32-bit host is 32-bit ABI. The default on a 64-bit host is 64-bit ABI if the target
platform specified is 64-bit, otherwise the default is 32-bit.
-
-
-
- -O3
- CXXOPTIMIZE
-
Like -O2, except that it enables optimizations that take longer to perform or that may generate larger code (in
an attempt to make the program run faster).
If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.
- Includes:
-
- -march=znver4
- CXXOPTIMIZE
-
Specify that Clang should generate code for a specific processor family member and later. For example, if you specify
-march=znver1, the compiler is allowed to generate instructions that are valid on AMD Zen processors, but
which may not exist on earlier products. -march=znver4 enables AVX 512 ISA for Genoa (znver4) processors.
-
-
- -ffast-math
- CXXOPTIMIZE
-
Enables a range of optimizations that provide faster, though sometimes less precise, mathematical operations that may
not conform to the IEEE-754 specifications. When this option is specified, the __STDC_IEC_559__ macro is
ignored even if set by the system headers.
-
-
-
-
-
-
-
-
- -zopt
- CXXOPTIMIZE
-
This option enables a subset of scalar, vector and loop transformations including improved variants of loop invariant code motion, SLP and loop vectorizations, loop-fusion, loop-interchange, loop-unswitch, loop tiling and loop distribution.
-
-
-
-
-
-
-
-
- -m64
- FC, LD
-
Generates code for a 64-bit environment. The 64-bit environment sets int to 32 bits and long and
pointer to 64 bits and generates code for AMD's x86-64 architecture. The compiler generates AMD64, INTEL64,
x86-64 64-bit ABI. The default on a 32-bit host is 32-bit ABI. The default on a 64-bit host is 64-bit ABI if the target
platform specified is 64-bit, otherwise the default is 32-bit.
-
-
-
-
-
-
- -O3
- FOPTIMIZE
-
Like -O2, except that it enables optimizations that take longer to perform or that may generate larger code (in
an attempt to make the program run faster).
If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.
- Includes:
-
- -march=znver4
- FOPTIMIZE
-
Specify that Clang should generate code for a specific processor family member and later. For example, if you specify
-march=znver1, the compiler is allowed to generate instructions that are valid on AMD Zen processors, but
which may not exist on earlier products. -march=znver4 enables AVX 512 ISA for Genoa (znver4) processors.
-
-
- -ffast-math
- FOPTIMIZE
-
Enables a range of optimizations that provide faster, though sometimes less precise, mathematical operations that may
not conform to the IEEE-754 specifications. When this option is specified, the __STDC_IEC_559__ macro is
ignored even if set by the system headers.
-
-
-
-
-
-
-
-
-
-
Peak Optimization Flags
-
- -m64
- CC, LD
-
Generates code for a 64-bit environment. The 64-bit environment sets int to 32 bits and long and
pointer to 64 bits and generates code for AMD's x86-64 architecture. The compiler generates AMD64, INTEL64,
x86-64 64-bit ABI. The default on a 32-bit host is 32-bit ABI. The default on a 64-bit host is 64-bit ABI if the target
platform specified is 64-bit, otherwise the default is 32-bit.
-
-
-
-
-
- -march=znver4
- COPTIMIZE
-
Specify that Clang should generate code for a specific processor family member and later. For example, if you specify
-march=znver1, the compiler is allowed to generate instructions that are valid on AMD Zen processors, but
which may not exist on earlier products. -march=znver4 enables AVX 512 ISA for Genoa (znver4) processors.
-
-
- -ffast-math
- COPTIMIZE
-
Enables a range of optimizations that provide faster, though sometimes less precise, mathematical operations that may
not conform to the IEEE-754 specifications. When this option is specified, the __STDC_IEC_559__ macro is
ignored even if set by the system headers.
-
-
-
- -fstruct-layout=9
- COPTIMIZE
-
Analyzes the whole program to determine if the structures in the code can be peeled, if dead or redundant fields can be deleted, and if
the pointer or integer fields in the structure can be compressed. If feasible, this optimization transforms the code to
enable these improvements. This transformation is likely to improve cache utilization and memory bandwidth. It is expected
to improve the scalability of programs executed on multiple cores.
This is effective only under flto as the whole program analysis is required to perform this optimization. You can choose
different levels of aggressiveness with which this optimization can be applied to your application; with 1 being the least
aggressive and 7 being the most aggressive level.
Possible values:
- fstruct-layout=0: disables structure peeling (default).
- fstruct-layout=1: enables structure peeling.
- fstruct-layout=2: enables structure peeling and selectively compresses self-referential pointers in these
structures to 32-bit pointers wherever safe.
- fstruct-layout=3: enables structure peeling and selectively compresses self-referential pointers in these
structures to 16-bit pointers wherever safe.
- fstruct-layout=4: enables structure peeling, pointer compression as in level 2 and further enables
compression of structure fields which are of 64-bit integer type to 32-bit integer type. This is performed under a
strict safety check.
- fstruct-layout=5: enables structure peeling, pointer compression as in level 3 and further enables compression
of structure fields which are of 64-bit integer type to 32-bit integer type. This is performed under a strict safety
check.
- fstruct-layout=6: enables structure peeling, pointer compression as in level 2 and further enables compression
of structure fields which are of type 64-bit integer type to 16-bit integer type. This is performed under a strict
safety check.
- fstruct-layout=7: enables structure peeling, pointer compression as in level 3 and further enables compression
of structure fields which are of type 64-bit integer type to 16-bit integer type. This is performed under a strict
safety check.
- fstruct-layout=8: enables structure peeling, pointer compression, 64 bit integer type compression
as in level 6 and creates optimal ordering of peeled structure fields which could improve runtime performance.
- fstruct-layout=9: enables structure peeling, pointer compression, 64 bit integer type compression
as in level 7 and creates optimal ordering of peeled structure fields which could improve runtime performance.
Note:
fstruct-layout=4 and fstruct-layout=5 are derived from fstruct-layout=2 and fstruct-layout=3 respectively with the added
feature of safe compression of 64-bit integer fields to 32-bit integer fields in structures. Going from
fstruct-layout=4 to fstruct-layout=5 may result in higher performance if the pointer values are such that the pointers
can be compressed to 16-bits.
fstruct-layout=6 and fstruct-layout=7 are derived from fstruct-layout=2 and fstructlayout=3 respectively, with the
added feature of safe compression of 64 bit integer fields to 16 bit integer in structures. Going from fstruct-layout=6
to fstruct-layout=7 may result in higher performance if the pointer values are such that the pointers can be
compressed to 16-bits.
-
-
-
-
-
-
-
- -zopt
- COPTIMIZE
-
This option enables a subset of scalar, vector and loop transformations including improved variants of loop invariant code motion, SLP and loop vectorizations, loop-fusion, loop-interchange, loop-unswitch, loop tiling and loop distribution.
-
-
-
-
-
-
- -m64
- FC, LD
-
Generates code for a 64-bit environment. The 64-bit environment sets int to 32 bits and long and
pointer to 64 bits and generates code for AMD's x86-64 architecture. The compiler generates AMD64, INTEL64,
x86-64 64-bit ABI. The default on a 32-bit host is 32-bit ABI. The default on a 64-bit host is 64-bit ABI if the target
platform specified is 64-bit, otherwise the default is 32-bit.
-
-
-
-
-
-
- -O3
- FOPTIMIZE
-
Like -O2, except that it enables optimizations that take longer to perform or that may generate larger code (in
an attempt to make the program run faster).
If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.
- Includes:
-
- -march=znver4
- FOPTIMIZE
-
Specify that Clang should generate code for a specific processor family member and later. For example, if you specify
-march=znver1, the compiler is allowed to generate instructions that are valid on AMD Zen processors, but
which may not exist on earlier products. -march=znver4 enables AVX 512 ISA for Genoa (znver4) processors.
-
-
- -ffast-math
- FOPTIMIZE
-
Enables a range of optimizations that provide faster, though sometimes less precise, mathematical operations that may
not conform to the IEEE-754 specifications. When this option is specified, the __STDC_IEC_559__ macro is
ignored even if set by the system headers.
-
-
-
-
-
-
-
-
-
-
Implicitly Included Flags
This section contains descriptions of flags that were included implicitly
by other flags, but which do not have a permanent home at SPEC.
-
- -O2
-
Moderate level of optimization which enables most optimizations. This is the default when no "-O" option is
specified, or if no value is specified (i.e. "-O").
If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.
- Includes:
-
- -O1
-
Somewhere between -O0 and -O2.
If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.
-
- -O3
-
Like -O2, except that it enables optimizations that take longer to perform or that may generate larger code (in
an attempt to make the program run faster).
If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.
- Includes:
Using numactl to bind processes and memory to cores
For multi-copy runs or single copy runs on systems with multiple sockets, it is advantageous to bind a process to a
particular core. Otherwise, the OS may arbitrarily move your process from one core to another. This can affect
performance. To help, SPEC allows the use of a "submit" command where users can specify a utility to use to bind
processes. We have found the utility 'numactl' to be the best choice.
numactl runs processes with a specific NUMA scheduling or memory placement policy. The policy is set for a
command and inherited by all of its children. The numactl flag "--physcpubind" specifies
which core(s) to bind the process. "-l" instructs numactl to keep a process's memory on the
local node while "-m" specifies which node(s) to place a process's memory. For full details on using
numactl, please refer to your Linux documentation, 'man numactl'
Note that some older versions of numactl incorrectly interpret application arguments as its own. For
example, with the command "numactl --physcpubind=0 -l a.out -m a", numactl will interpret
a.out's "-m" option as its own "-m" option. To work around this problem, we put
the command to be run in a shell script and then run the shell script using numactl. For example:
"echo 'a.out -m a' > run.sh ; numactl --physcpubind=0 bash run.sh"
numactl --interleave=all runcpu
numactl --interleave=all runcpu executes the SPEC CPU command runcpu so that memory is consumed across NUMA nodes rather than consumed from a single node. This helps prevent local node out-of-memory conditions which can occur when runcpu is executed without interleaving.
For full details on using numactl, please refer to your Linux documentation, 'man numactl'
Transparent Huge Pages (THP)
THP is an abstraction layer that automates most aspects of creating, managing,
and using huge pages. It is designed to hide much of the complexity in using
huge pages from system administrators and developers. Huge pages
increase the memory page size from 4 kilobytes to 2 megabytes. This provides
significant performance advantages on systems with highly contended resources
and large memory workloads. If memory utilization is too high or memory is badly
fragmented which prevents huge pages being allocated, the kernel will assign
smaller 4k pages instead. Most recent Linux OS releases have THP enabled by default.
THP usage is controlled by the sysfs setting /sys/kernel/mm/transparent_hugepage/enabled.
Possible values:
- never: entirely disable THP usage.
- madvise: enable THP usage only inside regions marked MADV_HUGEPAGE using madvise(3).
- always: enable THP usage system-wide. This is the default.
The SPEC CPU benchmark codes themselves never explicitly request huge pages, as the mechanism to do that is OS-specific
and can change over time. Libraries such as amdalloc which are used by the benchmarks may explicitly request huge pages,
and use of such libraries can make the "madvise" setting relevant and useful.
When no huge pages are immediately available and one is requested, how the system handles the request for THP creation is
controlled by the sysfs setting /sys/kernel/mm/transparent_hugepage/defrag.
Possible values:
- never: if no THP are available to satisfy a request, do not attempt to make any.
- defer: an allocation requesting THP when none are available gets normal pages while requesting THP creation in the
background.
- defer+madvise: acts like "always", but only for allocations in regions marked MADV_HUGEPAGE using madvise(3); for all
other regions it's like "defer".
- madvise: acts like "always", but only for allocations in regions marked MADV_HUGEPAGE using madvise(3). This is the
default.
- always: an allocation requesting THP when none are available will stall until some are made.
An application that "always" requests THP often can benefit from waiting for an allocation until those huge pages can be assembled.
For more information see the Linux transparent hugepage documentation.
ulimit -s <n>
Sets the stack size to n kbytes, or unlimited to allow the stack size to grow without limit.
ulimit -l <n>
Sets the maximum size of memory that may be locked into physical memory.
powersave -f (on SuSE)
Makes the powersave daemon set the CPUs to the highest supported frequency.
/etc/init.d/cpuspeed stop (on Red Hat)
Disables the cpu frequency scaling program in order to set the CPUs to the highest supported frequency.
LD_LIBRARY_PATH
An environment variable that indicates the location in the filesystem of bundled libraries to use when running the
benchmark binaries.
LIBOMP_NUM_HIDDEN_HELPER_THREADS
target nowait is supported via hidden helper task, which is a task not bound to any parallel region. A hidden helper team with a number of threads is created when the first hidden helper task is encountered.
The number of threads can be configured via the environment variable LIBOMP_NUM_HIDDEN_HELPER_THREADS. The default is 8. If LIBOMP_NUM_HIDDEN_HELPER_THREADS is 0, the hidden helper task is disabled and support falls back to a regular OpenMP task. The hidden helper task can also be disabled by setting the environment variable LIBOMP_USE_HIDDEN_HELPER_TASK=OFF.
sysctl -w vm.dirty_ratio=8
Limits dirty cache to 8% of memory.
sysctl -w vm.swappiness=1
Limits swap usage to minimum necessary.
sysctl -w vm.zone_reclaim_mode=1
Frees local node memory first to avoid remote memory usage.
kernel/numa_balancing
This OS setting controls automatic NUMA balancing on memory mapping and process placement.
NUMA balancing incurs overhead for no benefit on workloads that are already bound to NUMA nodes.
Possible settings:
- 0: disables this feature
- 1: enables the feature (this is the default)
For more information see the numa_balancing entry in the
Linux sysctl documentation.
kernel/randomize_va_space (ASLR)
This setting can be used to select the type of process address space
randomization. Defaults differ based on whether the architecture supports
ASLR, whether the kernel was built with the CONFIG_COMPAT_BRK
option or not, or the kernel boot options used.
Possible settings:
- 0 - Turn the process address space randomization off. This is the default for architectures that do not support
this feature anyway, and kernels that are booted with the "
norandmaps" parameter.
- 1 - Randomize addresses of mmap base, stack, and VDSO pages.
This is the default if the
CONFIG_COMPAT_BRK option is enabled at kernel build time.
- 2 - Additionally enable heap randomization. This is the default if
CONFIG_COMPAT_BRK is
disabled.
Disabling ASLR can make process execution more deterministic and runtimes more consistent.
For more information see the randomize_va_space entry in the
Linux sysctl documentation.
vm/drop_caches
The two commands are equivalent:
echo 3> /proc/sys/vm/drop_caches
and
sysctl -w vm.drop_caches=3
Both must be run as root.
The commands are used to free up the filesystem page cache, dentries, and inodes.
Possible settings:
- 1 - Clear pagecache
- 2 - Clear dentries and inodes
- 3 - Clear pagecache, dentries, and inodes
MALLOC_CONF
The amdalloc library is a variant of jemalloc library. The amdalloc
library has tunable parameters, many of which may be changed at run-time via several mechanisms, one of which
is the MALLOC_CONF environment variable. Other methods, as well as the order in which they're referenced,
are detailed in the jemalloc documentation's TUNING section.
The options that can be tuned at run-time are everything in the jemalloc documentation's
MALLCTL NAMESPACE section that begins with
"opt.".
The options that may be encountered in SPEC CPU 2017 results are detailed here:
retain:true - Causes unused virtual memory to
be retained for later reuse rather than discarding it. This is the default for 64-bit Linux.thp:never - Attempts to never utilize huge pages
by using MADV_NOHUGEPAGE on all mappings. This option has no effect except when THP is set to
"madvise".
PGHPF_ZMEM
An environment variable used to initialize the allocated memory. Setting PGHPF_ZMEM to "Yes" has the effect of
initializing all allocated memory to zero.
GOMP_CPU_AFFINITY
This environment variable is used to set the thread affinity for threads spawned by OpenMP.
OMP_DYNAMIC
This environment variable is defined as part of the OpenMP standard.
Setting it to "false" prevents the OpenMP runtime from dynamically adjusting the number of threads to use for parallel
execution.
For more information, see chapter 4 ("Environment Variables") in the
OpenMP 4.5 Specification.
OMP_SCHEDULE
This environment variable is defined as part of the OpenMP standard.
Setting it to "static" causes loop iterations to be assigned to threads in round-robin fashion in the order of the thread
number.
For more information, see chapter 4 ("Environment Variables") in the
OpenMP 4.5 Specification.
OMP_STACKSIZE
This environment variable is defined as part of the OpenMP standard and controls the size of the stack for threads created
by OpenMP.
For more information, see chapter 4 ("Environment Variables") in the
OpenMP 4.5 Specification.
OMP_THREAD_LIMIT
This environment variable is defined as part of the OpenMP standard and limits the maximum number of OpenMP threads that
can be created.
For more information, see chapter 4 ("Environment Variables") in the
OpenMP 4.5 Specification.
Operating System (OS) Application/Service Tuning:
The following OS tunes could've been applied to better optimize performance of some areas of the system:
- ulimit: Used to set user limits of system-wide resources. Provides control over resources available to the shell and processes started by it. Some common ulimit commands may include:
- ulimit -s [n | unlimited]: Set the stack size to n kbytes, or unlimited to allow the stack size to grow without limit.
- ulimit -l (number): Set the maximum size that can be locked into memory.
- Performance Governors (Linux): In-kernel CPU frequency governors are pre-configured power schemes for the CPU. The CPUfreq governors use P-states to change frequencies and lower power consumption. The dynamic governors can switch between CPU frequencies, based on CPU utilization to allow for power savings while not sacrificing performance. To set the governor, use the following commmand: "cpupower frequency-set -r -g {desired_governor}"
- Disabling Linux services: Certain Linux services may be disabled to minimize tasks that may consume CPU cycles.
- irqbalance: Disabled through "service irqbalance stop". Depending on the workload involved, the irqbalance service reassigns various IRQ's to system CPUs. Though this service might help in some situations, disabling it can also help environments which need to minimize or eliminate latency to more quickly respond to events.
- tuned-adm: The tuned-adm tool is a commandline interface for switching between different tuning profiles available to the tuned tuning daemon available in supported Linux distros. The default configuration file is located in /etc/tuned.conf and the supported profiles can be found in /etc/tune-profiles. Some profiles that may be available by default include: default, desktop-powersave, server-powersave, laptop-ac-powersave, laptop-battery-powersave, spindown-disk, throughput-performance, latency-performance, enterprise-storage. To set a profile, one can issue the command "tuned-adm profile (profile_name)". Here are details about relevant profiles:
- throughput-performance: Server profile for typical throughput tuning. This profile disables tuned and ktune power saving features, enables sysctl settings that may improve disk and network IO throughput performance, switches to the deadline scheduler, and sets the CPU governor to performance.
- latency-performance: Server profile for typical latency tuning. This profile disables tuned and ktune power saving features, enables the deadline IO scheduler, and sets the CPU governor to performance.
- enterprise-storage: Server profile to high disk throughput tuning. This profile disables tuned and ktune power saving features, enables the deadline IO scheduler, enables hugepages and disables disk barriers, increases disk readahead values, and sets the CPU governor to performance
OS Kernel Parameter Tuning:
The following Linux Kernel parameters were tuned to better optimize performance of some areas of the system:
- dirty_background_ratio: Set through "echo 40 > /proc/sys/vm/dirty_background_ratio". This setting can help Linux disk caching and performance by setting the percentage of system memory that can be filled with dirty pages.
- dirty_ratio: Set through "echo 8 > /proc/sys/vm/dirty_ratio". This setting is the absolute maximum amount of system memory that can be filled with dirty pages before everything must get committed to disk.
- ksm/sleep_millisecs: Set through "echo 200 > /sys/kernel/mm/ksm/sleep_millisecs". This setting controls how many milliseconds the ksmd (KSM daemon) should sleep before the next scan.
- khugepaged/scan_sleep_millisecs: Set through "echo 50000 > /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs". This setting controls how many milliseconds to wait in khugepaged is there is a hugepage allocation failure to throttle the next allocation attempt.
- swappiness: The swappiness value can range from 1 to 100. A value of 100 will cause the kernel to swap out inactive processes frequently in favor of file system performance, resulting in large disk cache sizes. A value of 1 tells the kernel to only swap processes to disk if absolutely necessary. This can be set through a command like "echo 1 > /proc/sys/vm/swappiness"
- numa_balancing: Disabled through "echo 0 > /proc/sys/kernel/numa_balancing". This feature will automatically migrate data on demand so memory nodes are aligned to the local CPU that is accessing data. Depending on the workload involved, enabling this can boost the performance if the workload performs well on NUMA hardware. If the workload is statically set to balance between nodes, then this service may not provide a benefit.
- Zone Reclaim Mode: Zone reclaim allows the reclaiming of pages from a zone if the number of free pages falls below a watermark even if other zones still have enough pages available. Reclaiming a page can be more beneficial than taking the performance penalties that are associated with allocating a page on a remote zone, especially for NUMA machines. To tell the kernel to free local node memory rather than grabbing free memory from remote nodes, use a command like "echo 1 > /proc/sys/vm/zone_reclaim_mode"
- Free the file system page cache: The command "echo 1> /proc/sys/vm/drop_caches" is used to free up the filesystem page cache.
- kernel/randomize_va_space, also known as ASLR (Address Space Layout Randomization): This setting can be used to select the type of process address space randomization. Defaults differ based on whether the architecture supports ASLR, whether the kernel was built with the CONFIG_COMPAT_BRK option or not, or the kernel boot options used. Possible settings:
- 0: Turn process address space randomization off.
- 1: Randomize addresses of mmap base, stack, and VDSO pages.
- 2: Additionally randomize the heap. (This is probably the default.)
- Disabling ASLR can make process execution more deterministic and runtimes more consistent. For more information see the randomize_va_space entry in the Linux sysctl documentation.
- Transparent Hugepages (THP): THP is an abstraction layer that automates most aspects of creating, managing, and using huge pages. It is designed to hide much of the complexity in using huge pages from system administrators and developers. Huge pages increase the memory page size from 4 kilobytes to 2 megabytes. This provides significant performance advantages on systems with highly contended resources and large memory workloads. If memory utilization is too high or memory is badly fragmented which prevents hugepages being allocated, the kernel will assign smaller 4k pages instead. Most recent Linux OS releases have THP enabled by default. THP usage is controlled by the sysfs setting /sys/kernel/mm/transparent_hugepage/enabled. Possible values:
- never: entirely disable THP usage.
- madvise: enable THP usage only inside regions marked MADV_HUGEPAGE using madvise(3).
- always: enable THP usage system-wide. This is the default.
- THP creation is controlled by the sysfs setting /sys/kernel/mm/transparent_hugepage/defrag. Possible values:
- never: if no THP are available to satisfy a request, do not attempt to make any.
- defer: an allocation requesting THP when none are available get normal pages while requesting THP creation in the background.
- defer+madvise: acts like "always", but only for allocations in regions marked MADV_HUGEPAGE using madvise(3); for all other regions it's like "defer".
- madvise: acts like "always", but only for allocations in regions marked MADV_HUGEPAGE using madvise(3). This is the default.
- always: an allocation requesting THP when none are available will stall until some are made.
- An application that "always" requests THP often can benefit from waiting for an allocation until those huge pages can be assembled. For more information see the Linux transparent hugepage documentation.
Linux Huge Page settings:
If one prefers not to use Transparent Hugepages, one can always setup Huge Pages by following the below steps:
- Create a mount point for the huge pages: "mkdir /mnt/hugepages"
- The huge page file system needs to be mounted when the systems reboots. Add the following to a system boot configuration file before any services are started: "mount -t hugetlbfs nodev /mnt/hugepages"
- Set vm/nr_hugepages=N in your /etc/sysctl.conf file where N is the maximum number of pages the system may allocate.
- Reboot to have the changes take effect.
Note that further information about huge pages may be found in your Linux documentation file: /usr/src/linux/Documentation/vm/hugetlbpage.txt
Environment Variables:
The following Linux envrionment variables that could've possibly been tuned to better optimize performance of some areas of the system:
- GOMP_CPU_AFFINITY: Used to bind threads to specific CPUs. The variable should contain a space-separated or comma-separated list of CPUs. This list may contain different kinds of entries: either single CPU numbers in any order, a range of CPUs (M-N) or a range with some stride (M-N:S). CPU numbers are zero based. For example, GOMP_CPU_AFFINITY="0 3 1-2 4-15:2" will bind the initial thread to CPU 0, the second to CPU 3, the third to CPU 1, the fourth to CPU 2, the fifth to CPU 4, the sixth through tenth to CPUs 6, 8, 10, 12, and 14 respectively and then start assigning back from the beginning of the list. GOMP_CPU_AFFINITY=0 binds all threads to CPU 0. There is no libgomp library routine to determine whether a CPU affinity specification is in effect. As a workaround, language-specific library functions, e.g., getenv in C or GET_ENVIRONMENT_VARIABLE in Fortran, may be used to query the setting of the GOMP_CPU_AFFINITY environment variable. A defined CPU affinity on startup cannot be changed or disabled during the runtime of the application. If both GOMP_CPU_AFFINITY and OMP_PROC_BIND are set, OMP_PROC_BIND has a higher precedence. If neither has been set and OMP_PROC_BIND is unset, or when OMP_PROC_BIND is set to FALSE, the host system will handle the assignment of threads to CPUs.
- OMP_DYNAMIC: Dynamic adjustment of threads. Enable or disable the dynamic adjustment of the number of threads within a team. The value of this environment variable shall be TRUE or FALSE. If undefined, dynamic adjustment is disabled by default.
- OMP_SCHEDULE: How threads are scheduled. Allows to specify schedule type and chunk size. The value of the variable shall have the form: type[,chunk] where type is one of static, dynamic or guided. The optional chunk size shall be a positive integer. If undefined, dynamic scheduling and a chunk size of 1 is used.
- OMP_THREAD_LIMIT: Set the maximum number of threads. Specifies the number of threads to use for the whole program. The value of this variable shall be a positive integer. If undefined, the number of threads is not limited.
- MALLOC_CONF: This environment variable affects the execution of the allocation functions. If the environment variable MALLOC_CONF is set, the characters it contains will be interpreted as options.
Firmware Settings:
One or more of the following settings may have been set. If so, the "Platform Notes" section of the report will say so; and you can read below to find out more about what these settings mean.
- AMD SMT Option (Default = Enabled): This feature allows enabling or disabling of logical processor cores on processors supporting AMD SMT. When enabled, each physical processor core operates as two logical processor cores. When disabled, each physical core operates as only one logical processor core. Enabling this option can improve overall performance for applications that benefit from a higher processor core count.
- Thermal Configuration (Default = Optimal Cooling): This feature allows the user to select the fan cooling solution for the system. Values for this BIOS option can be:
- Optimal Cooling: Provides the most efficient solution by configuring fan speeds to the minimum required to provide adequate cooling.
- Increased Cooling: Will run fans at higher speeds to provide additional cooling. Increased Cooling should be selected when non-HPE storage controllers are cabled to the embedded hard drive cage, or if the system is experiencing thermal issues that cannot be resolved in another manner.
- Maximum Cooling: Will provide the maximum cooling available by this platform.
- Determinism Control (Default = Auto): This option allows the user to choose between an Auto and Manual mode for Determinism Control. Values for this BIOS option can be:
- Auto: The system will decide what Performance Determinism setting to use. Auto also uses the processor fused values for Determinism.
- Manual: Allows the user to select either Power Deterministic or Performance Deterministic as described below.
- Performance Determinism (Default = Performance Deterministic): This option allows the user to configure the AMD processor Determinism setting for AGESA ("AMD Generic Encapsulated Software Architecture", a bootstrap protocol by which system devices on AMD64-architecture mainboards are initialized) control or BIOS control. Values for this BIOS option can be:
- Performance Deterministic: This option allows for AGESA control and provides predictable (capped) performance across all processors of the same type in the system. This could cause the system to run at or report lower than expected power values.
- Power Deterministic: This option will not enable performance deterministic control and allows for BIOS control. This option will maximize performance with power limits defined by the system design and operate as close to the thermal design point (TDP) as possible.
- Package Power Limit Control Mode (Default = Auto): This is a per Processor Power Limit value applicable for all populated processors in the system. This can be set to limit the processor power to a certain value. Values for this BIOS option can be:
- Auto: Uses the default processor value.
- Manual: Lets the user set a power limit and exposes Package Power Limit value field that a user can enter a number into. If a number is entered higher than what the processor and/or system supports, the BIOS will limit the power to the max possible value.
- Memory Patrol Scrubbing (Default = Enabled): This option allows for correction of soft memory errors. Over the length of system runtime, the risk of producing multi-bit and uncorrected errors is reduced with this option. Values for this BIOS setting can be:
- Enabled: Correction of soft memory errors can occur during runtime.
- Disabled: Soft memory error correction is turned off during runtime.
- Memory Interleaving Mode (Default = Enabled): This option allows a user to enable or disable the Memory Interleaving Mode. ADD MORE DESCRIPTION BASED OFF HOW THE BIOS DESCRIBES THIS!!! Values for this BIOS setting can be:
- Enabled: What does this mean??
- Disabled: What does this mean??
- Workload Profile (Default = General Power Efficient Compute): This option allows a user to choose a workload profile that best fits the user`s needs. The workload profiles control many power and performance settings that are relevant to general workload areas. Values for this BIOS option can be:
- General Power Efficient Compute, General Peak Frequency Compute, General Throughput Compute, Virtualization - Power Efficient, Virtualization - Max Performance, Low Latency, Mission Critical, Transaction Application Processing, High Performance Compute (HPC), Decision Support, Graphic Processing, I/O Throughput, and Custom.
- Power Regulator (Default = Static High Performance Mode): This option can only be configured if the Workload Profile is set to Custom. This feature allows the user to select the following Power Regulator support:
- Dynamic Power Savings Mode: This mode allows the automatic variation of the processor speed and power usage based on processor utilization, resulting a reduction in overall power consumption with little or no impact on performance. It does not require OS support.
- Static Low Power Mode: This mode reduces the processor speed and power usage and guarantees a lower maximum power usage for the system.
- Static High Performance Mode: This mode allows the processors to run in their maximum power/performance state at all times, regardless of the OS power management policy.
- OS Control Mode: This mode allows the processors to run in their maximum power/performance state at all times unless the OS enables a power management policy.
- Minimum Processor Idle Power Core C-State (Default = C6 State): This option can only be configured if the Workload Profile is set to Custom. This feature selects the processor's lowest idle power state (C-state) that the operating system uses. The higher the C-state, the lower the power usage of that idle state (C6 is the lowest power idle state supported by the processor). Values for this setting can be:
- C6 State: While in C6, the core PLLs are turned off, the core caches are flushed and the core state is saved to the Last Level Cache. Power Gates are used to reduce power consumption to close to zero. C6 is considered an inactive core.
- C1E State: C1E is defined as the enhanced halt state. While in C1E no instructions are being executed. C1E considered an active core.
- No C-states: No C-states is defined as C0, which is defined as the active state. While in C0, instructions are being executed by the core.
- C-State Efficiency mode (Default = Enabled): This option allows to adjust the frequency along with change in C-State. Enabling will monitor the workload and modulate the frequency of the core to maintain a high C0 residency. It also has latency and power benefits when the CPU is not 100% utilized. Values for this BIOS setting can be:
- XGMI Force Link Width (Default = Auto): This option forces the value of XGMI link width to a value set by the user. Performance improvement should be observed while setting the correct value. Values for this BIOS setting can be:
- Memory Patrol Scrubbing (Default = Enabled): This option allows for correction of soft memory errors. Over the length of system runtime, the risk of producing multi-bit and uncorrected errors is reduced with this option. Values for this BIOS setting can be:
- Enabled: Correction of soft memory errors can occur during runtime.
- Disabled: Soft memory error correction is turned off during runtime.
- Memory Interleaving (Default = Enabled): This option allows the CPUs to increase the memory bandwidth for an application. Values for this BIOS setting can be:
- Enabled: Consecutive memory blocks, often cache lines, are read from the different memory banks which may result in increased throughput and increasing latency.
- Disabled: Consecutive memory blocks are in the same memory bank which may result in reduced throughput and increasing latency.
- NUMA memory domains per socket (Default = Auto): This option allows the user to divide the memory domains that each socket has into a certain number of NUMA memory domains for better memory bandwidth. Values for this BIOS setting can be:
- One memory domain per socket: Each processor socket will have one memory domain.
- Two memory domains per socket: Each processor socket will have two memory domains.
- Four memory domains per socket: Each processor socket will have four memory domains.
- Auto: The system will default to One memory domain per socket.
- Last-Level Cache (LLC) as NUMA Node (Default = Disabled): When enabled, this option allows the user to divide processor's cores into additional NUMA Nodes based on the L3 cache. Enabling this feature can increase performance for workloads that are NUMA aware and optimized. Values for this BIOS setting can be:
- Memory PStates (Default = Auto): This setting controls the power state support of the memory controllers. Values for this BIOS setting can be:
- Data Fabric C-State Enable (Default = Auto): Allows the Infinity Fabric to go into lower power states when idle, similar to CPU core C-States. There can be a delay changing back to full-power mode, causing latency jitter. In a low latency workload, or one core bursty I/O, one could disable this feature to achieve more performance with the tradeoff of higher power consumption. Values for this BIOS setting can be:
- Auto: Dynamically allow the Infinity Fabric to go to a lower-power state when the processor has entered C-States.
- Force Enabled: Force the Infinity Fabric to go to a lower-power state when the processor has entered C-States.
- Disabled: Do not allow the Infinity Fabric to go to a lower-power state when the processor has entered C-States.
- Infinity Fabric Power Management (Default = Enabled): When this feature is enabled, the EPYC processor will dynamically vary the clock frequency of the Infinity Fabric based on activity level. For NUMA optimized workloads, allowing the Infinity Fabric to run slower can lead to increased overall performance due to an increase in CPU boost. Disabling this feature may be necessary for latency sensitive workloads. Values for this BIOS setting can be:
- Disabled: Enable fixed Infinity Fabric P-State control
- Enabled: Dynamically switch Infinity Fabric P-State based on link usage.
- Infinity Fabric Performance State (Default = Auto): This feature allows for customizing the performance state (P-State) of the Infinity Fabric when Infinity Fabric Power Management is disabled. The default is Auto when Infinity Fabric Power Management is Enabled. P0 is ideal for latency sensitive or I/O centric workloads. P3 is ideal for achieving best core boost frequencies. Values for this BIOS setting can be:
- P0: Will force the Infinity Fabric and memory controllers into full-power mode, eliminating latency jitter. Highest performing Infinity Fabric P-State.
- P1: Next highest performing Infinity Fabric P-State, after P0.
- P2: Next highest performing Infinity Fabric P-State, after P1.
- P3: Minimum Infinity Fabric P-State.
- L1 HW Prefetcher (Default = Enabled): Use this option to disable L1 Stream HW prefetch feature. In some cases (e.g. workloads that are random in nature) setting this option to disabled can improve performance. However, most workloads will benefit from this setting being enabled as it gathers data and keeps the core pipeline busy. Values for this BIOS setting can be:
- L2 HW Prefetcher (Default = Enabled): Use this option to disable L2 Stream HW prefetch feature. In some cases (e.g. workloads that are random in nature) setting this option to disabled can improve performance. However, most workloads will benefit from this setting being enabled as it gathers data and keeps the core pipeline busy. Values for this BIOS setting can be:
- ACPI CST C2 Latency (Default = 800 microseconds): C2 Latency is a value that the BIOS communicates to the OS that helps it determine when it should enter lower power states. Some OS kernels and associated workloads will benefit from a larger latency while other combinations will benefit from lower values. Idle power will benefit from lower latency values. Values for this BIOS option can be:
- 800 microseconds : This is recommended for workloads where the processor cores switch less frequently between idle and busy states. The higher C-State residency allows the kernel to better balance power consumption and performance, based on workload behaviour. x86 systems that use the ACPI idle driver on older Linux kernels (pre-6.0) would be able to take advantage of this setting for such workloads.
- 18 microseconds : This is recommended for workloads that cause the processor to frequently switch between idle and busy states. More specifically on pre-6.0 Linux kernels using the ACPI idle driver, such workloads frequently transitions in and out of lower power state causing the kernel’s idle subsystem to force cores more often into low power state. Keeping the latency to this value helps avoid that performance penalty for such workloads.
Last updated November 11, 2022.
For questions about the meanings of these flags, please contact the tester.
For other inquiries, please contact info@spec.org
Copyright 2017-2022 Standard Performance Evaluation Corporation
Tested with SPEC CPU2017 v1.1.8.
Report generated on 2022-12-08 15:32:35 by SPEC CPU2017 flags formatter v5178.
![[benchmark]](https://www.spec.org/auto/cpu2017/images/benchmark.png)