aocc320-flags-A1 AMD Optimizing C/C++ Compiler Suite Version 3.2.0 Flag Descriptions Using numactl to bind processes and memory to cores

For multi-copy runs or single copy runs on systems with multiple sockets, it is advantageous to bind a process to a particular core. Otherwise, the OS may arbitrarily move your process from one core to another. This can affect performance. To help, SPEC allows the use of a "submit" command where users can specify a utility to use to bind processes. We have found the utility 'numactl' to be the best choice.

numactl runs processes with a specific NUMA scheduling or memory placement policy. The policy is set for a command and inherited by all of its children. The numactl flag "--physcpubind" specifies which core(s) to bind the process. "-l" instructs numactl to keep a process's memory on the local node while "-m" specifies which node(s) to place a process's memory. For full details on using numactl, please refer to your Linux documentation, 'man numactl'

Note that some older versions of numactl incorrectly interpret application arguments as its own. For example, with the command "numactl --physcpubind=0 -l a.out -m a", numactl will interpret a.out's "-m" option as its own "-m" option. To work around this problem, we put the command to be run in a shell script and then run the shell script using numactl. For example: "echo 'a.out -m a' > run.sh ; numactl --physcpubind=0 bash run.sh"

]]> Transparent Huge Pages (THP)

THP is an abstraction layer that automates most aspects of creating, managing, and using huge pages. It is designed to hide much of the complexity in using huge pages from system administrators and developers. Huge pages increase the memory page size from 4 kilobytes to 2 megabytes. This provides significant performance advantages on systems with highly contended resources and large memory workloads. If memory utilization is too high or memory is badly fragmented which prevents huge pages being allocated, the kernel will assign smaller 4k pages instead. Most recent Linux OS releases have THP enabled by default.

THP usage is controlled by the sysfs setting /sys/kernel/mm/transparent_hugepage/enabled. Possible values:

The SPEC CPU benchmark codes themselves never explicitly request huge pages, as the mechanism to do that is OS-specific and can change over time. Libraries such as jemalloc which are used by the benchmarks may explicitly request huge pages, and use of such libraries can make the "madvise" setting relevant and useful.

When no huge pages are immediately available and one is requested, how the system handles the request for THP creation is controlled by the sysfs setting /sys/kernel/mm/transparent_hugepage/defrag. Possible values:

An application that "always" requests THP often can benefit from waiting for an allocation until those huge pages can be assembled.
For more information see the Linux transparent hugepage documentation.

ulimit -s <n>

Sets the stack size to n kbytes, or unlimited to allow the stack size to grow without limit.

ulimit -l <n>

Sets the maximum size of memory that may be locked into physical memory.

powersave -f (on SuSE)

Makes the powersave daemon set the CPUs to the highest supported frequency.

/etc/init.d/cpuspeed stop (on Red Hat)

Disables the cpu frequency scaling program in order to set the CPUs to the highest supported frequency.

LD_LIBRARY_PATH

An environment variable that indicates the location in the filesystem of bundled libraries to use when running the benchmark binaries.

kernel/numa_balancing

This OS setting controls automatic NUMA balancing on memory mapping and process placement. NUMA balancing incurs overhead for no benefit on workloads that are already bound to NUMA nodes.

Possible settings:

For more information see the numa_balancing entry in the Linux sysctl documentation.

kernel/randomize_va_space (ASLR)

This setting can be used to select the type of process address space randomization. Defaults differ based on whether the architecture supports ASLR, whether the kernel was built with the CONFIG_COMPAT_BRK option or not, or the kernel boot options used.

Possible settings:

Disabling ASLR can make process execution more deterministic and runtimes more consistent. For more information see the randomize_va_space entry in the Linux sysctl documentation.

vm/drop_caches

The two commands are equivalent:

echo 3> /proc/sys/vm/drop_caches

and

sysctl -w vm.drop_caches=3

Both must be run as root.

The commands are used to free up the filesystem page cache, dentries, and inodes.

Possible settings:

MALLOC_CONF

The jemalloc library has tunable parameters, many of which may be changed at run-time via several mechanisms, one of which is the MALLOC_CONF environment variable. Other methods, as well as the order in which they're referenced, are detailed in the jemalloc documentation's TUNING section.

The options that can be tuned at run-time are everything in the jemalloc documentation's MALLCTL NAMESPACE section that begins with "opt.".

The options that may be encountered in SPEC CPU 2017 results are detailed here:

PGHPF_ZMEM

An environment variable used to initialize the allocated memory. Setting PGHPF_ZMEM to "Yes" has the effect of initializing all allocated memory to zero.

GOMP_CPU_AFFINITY

This environment variable is used to set the thread affinity for threads spawned by OpenMP.

OMP_DYNAMIC

This environment variable is defined as part of the OpenMP standard. Setting it to "false" prevents the OpenMP runtime from dynamically adjusting the number of threads to use for parallel execution.

For more information, see chapter 4 ("Environment Variables") in the OpenMP 4.5 Specification.

OMP_SCHEDULE

This environment variable is defined as part of the OpenMP standard. Setting it to "static" causes loop iterations to be assigned to threads in round-robin fashion in the order of the thread number.

For more information, see chapter 4 ("Environment Variables") in the OpenMP 4.5 Specification.

OMP_STACKSIZE

This environment variable is defined as part of the OpenMP standard and controls the size of the stack for threads created by OpenMP.

For more information, see chapter 4 ("Environment Variables") in the OpenMP 4.5 Specification.

OMP_THREAD_LIMIT

This environment variable is defined as part of the OpenMP standard and limits the maximum number of OpenMP threads that can be created.

For more information, see chapter 4 ("Environment Variables") in the OpenMP 4.5 Specification.

LIBOMP_NUM_HIDDEN_HELPER_THREADS

target nowait is supported via hidden helper task, which is a task not bound to any parallel region. A hidden helper team with a number of threads is created when the first hidden helper task is encountered.

The number of threads can be configured via the environment variable LIBOMP_NUM_HIDDEN_HELPER_THREADS. The default is 8. If LIBOMP_NUM_HIDDEN_HELPER_THREADS is 0, the hidden helper task is disabled and support falls back to a regular OpenMP task. The hidden helper task can also be disabled by setting the environment variable LIBOMP_USE_HIDDEN_HELPER_TASK=OFF.

]]>
Compilers: AMD Optimizing C/C++ Compiler Suite ]]>
Splitter rule for plugin arguments: -fplugin-arg-dragonegg-llvm-option="-flag[ -flag...]" Consumer rule for the tail of split up plugin arguments: -fplugin-arg-dragonegg-llvm-option="" -O Set the optimization level to -O2.

If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.

]]> -O0 Means "no optimization". This level compiles the fastest and generates the most debuggable code.

If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.

]]> -O1 Somewhere between -O0 and -O2.

If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.

]]> -O2 Moderate level of optimization which enables most optimizations. This is the default when no "-O" option is specified, or if no value is specified (i.e. "-O").

If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.

]]> -O3 Like -O2, except that it enables optimizations that take longer to perform or that may generate larger code (in an attempt to make the program run faster).

If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.

]]> -Os Like -O2 with extra optimizations to reduce code size.

If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.

]]> -Oz Like -Os (and thus -O2), but reduces code size further.

If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.

]]> -O4 Equivalent to -O3.

If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.

]]> -Ofast Enables all the optimizations from -O3 along with other aggressive optimizations that may violate strict compliance with language standards. Refer to the AOCC options document for the language you're using for more detailed documentation of optimizations enabled under -Ofast.

]]> -march=znver3 Specify that Clang should generate code for a specific processor family member and later. For example, if you specify -march=znver1, the compiler is allowed to generate instructions that are valid on AMD Zen processors, but which may not exist on earlier products.

]]> -fno-exceptions (For C++ only) Enable generation of unwind information. This allows exceptions to be thrown through Clang compiled stack frames. This is on by default in x86-64. -fno-exceptions disables C++ exception handling.

]]> -flto Generate output files in LLVM formats suitable for link time optimization. When used with -S this generates LLVM intermediate language assembly files, otherwise this generates LLVM bitcode format object files (which may be passed to the linker depending on the stage selection options).

]]> -m32 Generate code for a 32-bit environment. The 32-bit environment sets int, long and pointer to 32 bits and generates code that runs on any i386 system. The compiler generates x86 or IA32 32-bit ABI. The default on a 32-bit host is 32-bit ABI. The default on a 64-bit host is 64-bit ABI if the target platform specified is 64-bit, otherwise the default is 32-bit.

]]> -m64 Generate code for a 64-bit environment. The 64-bit environment sets int to 32 bits and long and pointer to 64 bits and generates code for AMD's x86-64 architecture. The compiler generates AMD64, INTEL64, x86-64 64-bit ABI. The default on a 32-bit host is 32-bit ABI. The default on a 64-bit host is 64-bit ABI if the target platform specified is 64-bit, otherwise the default is 32-bit.

]]> -fuse-ld=lld Instructs the compiler to use the LLVM linker and loader (lld) instead of the system linker when linking object files.

]]> -fuse-ld=gold Instructs the compiler to use the GNU gold linker (gold) instead of the system linker when linking object files.

]]> -fno-stack-arrays Instructs the compiler not to allocate arrays from the stack and instead use heap memory.

]]> -freciprocal-math Given the expression "a = b / c", instructs the compiler to calculate "a = b * (1/c)".

]]> -ffast-math Enables a range of optimizations that provide faster, though sometimes less precise, mathematical operations that may not conform to the IEEE-754 specifications. When this option is specified, the __STDC_IEC_559__ macro is ignored even if set by the system headers.

]]> -fopenmp Enable handling of OpenMP directives and generate parallel code. The openmp library to be linked can be specified through -fopenmp=library option.

]]> -fno-finite-math-only Do not allow optimizations for floating-point arithmetic that assume that arguments and results are not NaNs or +-Infs. The option instructs the compiler to follow exact implementation of IEEE or ISO rules/specifications for math functions.

]]> This option enables floating-point expression contraction such as forming of fused multiply-add operations if the target has native support for them.

]]> -Hz,1,0x1 flang option to preserve array access information for linearized arrays.

]]> -Kieee Instructs the compiler to conform to the IEEE-754 specifications. The compiler will Perform floating-point operations in strict conformance with the IEEE 754 standard. Some optimizations are disabled when this option is specified

]]> -lm Instructs the compiler to link with system math libraries.

]]> -lamdlibm Instructs the compiler to link with AMD-supported optimized math library.

]]> This option instructs the linker to link the executable with the pthread library.

]]> This option instructs the linker to link the executable with libdl, the interface to the dynamic loader.

]]> -z muldefs Instructs the linker to use the first definition encountered for a symbol, and ignore all others.

]]> -ljemalloc Use the jemalloc library, which is a general purpose malloc(3) implementation that emphasizes fragmentation avoidance and scalable concurrency support.

]]> -lomp Instructs the compiler to link with the OpenMP runtime libraries.

]]> -lflang Instructs the compiler to link with flang Fortran runtime libraries.

]]> -lflangrti Instructs the compiler to link with flang Fortran runtime libraries.

]]> The option combines compare/test instructions with branches wherever possible.

]]> -Wl,-mllvm -Wl,-disable-indvar-simplify The induction variable simplification optimization transforms induction variables to simpler forms. The option disables this optimization.

]]> Sets the default integer and logical types to an 8 byte type. It does not promote variables with explicit KIND declarations.

]]> -Wl,-mllvm -Wl,-enable-gvn-hoist This option enables the GVN hoist pass, which is used to hoist computations from branches.

]]> -Wl,-mllvm -Wl,-extra-vectorizer-passes Run cleanup optimization passes after vectorization.

]]> -Wl,-mllvm -Wl,-force-vector-width=4 Sets the SIMD width. Zero is autoselect.

]]> -Wl,-mllvm -Wl,-force-vector-interleave=1 Sets the vectorization interleave count. For example, force-vector-interleave=1 is used to force the unroll factor in loop unrolling to 1. Zero is autoselect.

]]> -Wl,-mllvm -Wl,-enable-vectorize-compares:false Certain loops with breaks may be vectorized by default at -O2 and above. In some extreme situations this may result in unsafe behavior. Use this option to disable vectorization of such loops.

]]> -Wl,-mllvm -Wl,-enable-iv-split Enables splitting of long live ranges of loop induction variables which span loop boundaries. This helps reduce register pressure and can help avoid needless spills to memory and reloads from memory.

]]> -Wl,-mllvm -Wl,-enable-partial-unswitch This optimization does partial unswitching of loops where some part of the unswitched control flow remains in the loop.

]]> -Wl,-mllvm -Wl,-aggressive-loop-unswitch This option enables aggressive loop unswitching heuristic (including -enable-partial-unswitch) based on the usage of the branch conditional values. Loop unswitching leads to code-bloat. Code-bloat can be minimized if the hoisted condition is executed more often. This heuristic prioritizes the conditions based on the number of times they are used within the loop. The heuristic can be controlled with the following options:

Note: These options may facilitate more unswitching in some of the workloads. Since loop-unswitching inherently leads to code bloat, facilitating more unswitching may significantly increase the code size and hence may also lead to longer compilation times.

]]> -Wl,-mllvm -Wl,-enable-x86-prefetching This optimization enables generation of prefetch instructions for tightly coupled loops

]]> This option enables an optimization that generates and calls specialized function versions when the loops inside function are vectorizable and the arguments are not aliased with each other. This optimization helps in function inlining and vectorization.

]]> This option enables an optimization that generates and calls specialized function versions when they are called with constant arguments. This optimization helps in function inlining.

]]> This option enables an optimization that does the slp vectorization across basic blocks. The SLP vectorizer vectorizes instructions within basic blocks. The global slp vectorizer analyzes instructions across basic blocks and vectorizes them.

]]> Converts the call to floating point exponent version of pow to its integer exponent version if the floating-point exponent can be converted to integer. This option is set to true by default.

]]> Enables estimation of the virtual register pressure before performing loop invariant code motion. This estimation is used to decide the invariants that will be hoisted during loop invariant code motion.

]]> Enables fusion of adjacent tiled loops as a part of loop tiling transformation. This option is set to false by default.

]]> This option enables an optimization that transforms the data layout of a single dimensional array to provide better cache locality by analysing the access patterns.

]]> Enables dead virtual function elimination optimization. Requires -flto=full.

]]> This flag enables vectorization of loops with complex control flow that can not be vectorized by loop and slp vectorizers.

]]> -mllvm -reduce-array-computations=3 This option eliminates the array computations based on their usage. The computations on unused array elements and computations on zero valued array elements are eliminated with this optimization. -flto as whole program analysis is required to perform this optimization.

Possible values:

]]> -mllvm -do-block-reorder=aggressive Block Reordering

Possible values:

This optimization also includes safety analysis which checks that it is safe to reorder basic blocks. However, when this optimization takes effect, the safety analysis ignores exceptions specifically within the call chain involved in the context of the blocks being reordered. This would be acceptable for most input programs, but if your program strictly needs to support throwing of exceptions at all points in the program, then it is advisable to avoid using this option.

]]> -fstruct-layout=5 Analyzes the whole program to determine if the structures in the code can be peeled and if pointer or integer fields in the structure can be compressed. If feasible, this optimization transforms the code to enable these improvements. This transformation is likely to improve cache utilization and memory bandwidth. This, in turn, is expected to improve the scalability of programs executed on multiple cores.

This is effective only under -flto as whole program analysis is required to perform this optimization. You can choose different levels of aggressiveness with which this optimization can be applied to your application with 1 being the least aggressive and 7 being the most aggressive level.

Possible values:

Note:

fstruct-layout=4 and fstruct-layout=5 are derived from fstruct-layout=2 and fstruct-layout=3 respectively with the added feature of safe compression of integer fields in structures. Going from fstruct-layout=4 to fstruct-layout=5 may result in higher performance if the pointer values are such that the pointers can be compressed to 16-bits.

fstruct-layout=6 and fstruct-layout=7 are derived from fstruct-layout=2 and fstruct-layout=3 respectively with the added feature of compression of integer fields in structures. These are similar to fstruct-layout=4 and fstruct-layout=5, but here, the integer fields of the structures are always compressed from 64-bits to 32-bits without any safety guarantee.

]]> Turns on LLVM's instrumenation based profiling.

]]> Uses the profiling files generated from a program compiled with -fprofile-instr-generate to guide optimization decisions.

]]> -fgnu89-inline In the 502/602.gcc benchmark description, "multiple definitions of symbols" is listed under the "Known Portability Issues" section, and this option is one of the suggested workarounds. This option causes Clang to revert to the same inlining behavior that GCC does when in pre-C99 mode.

]]> Sets the compiler's inlining heuristics to an aggressive level by increasing the inline thresholds.

]]> -Wl,-mllvm -Wl,-inline-threshold=99 Sets the compiler's inlining threshold level to the value passed as the argument. The inline threshold is used in the inliner heuristics to decide which functions should be inlined.

]]> -Wl,-mllvm -Wl,-inline-recursion=4 Enables inlining for recursive functions based on heuristics, with level 4 being most aggressive. Higher levels may lead to code bloat due to expansion of recursive functions at call sites.

Levels:

]]> This option enables aggressive loop unswitching heuristic based on usage of branch conditions.

]]> -Wl,-mllvm -Wl,-lsr-in-nested-loop Enables loop strength reduction for nested loop structures. By default, the compiler will do loop strength reduction only for the innermost loop.

]]> -mllvm -reroll-loops Run the loop rerolling pass.

]]> -Wl,-mllvm -Wl,-merge-constant This optimization merges duplicate constant uses into a register to reduce instruction width.

]]> -Mrecursive Allocate local variables on the stack, thus allowing recursion. SAVEd, data-initialized, or namelist members are always allocated statically, regardless of the setting of this switch.

]]> -suppress-fmas Disable generation of fma instructions when there is a chain of fma instructions and output of one fma instruction is used as input to other fma instruction.

]]> This option enables AVX2 (Advanced Vector Extensions, 2nd generation) support.

]]> This option enables generation of the adx instruction.

]]> This option disables generation of the adx instruction.

]]> This option enables the generation of SSE4a instructions.

]]> This option disables the generation of SSE4a instructions.

]]> -mllvm -unroll-threshold=100 Sets the limit at which loops will be unrolled. For example, if unroll threshold is set to 100 then only loops with 100 or fewer instructions will be unrolled.

]]> -mllvm -loop-unswitch-threshold=100 Sets the limit at which loops will be unswitched. For example, if unswitch threshold is set to 100 then only loops with 100 or fewer instructions will be unswtched.

]]> -mllvm -unroll-aggressive Enables aggressive heuristics to get loop unrolling.

]]> -mllvm -unroll-count=8 The unroll count can be specified explicitly with -unroll_count=<value> where <value> is a positive integer. If this value is greater than the trip count, the loop will be fully unrolled.

]]> -funroll-loops This option instructs the compiler to unroll loops wherever possible.

]]> -fveclib=AMDLIBM Use the given vector functions library.

]]> -mllvm -x86-use-vzeroupper=false This option controls whether AOCC emits (true) or does not emit (false) a vzeroupper instruction before a transfer of control flow. Not emitting the vzeroupper instruction can help minimize the AVX to SSE transition penalty.

]]> -Wl,-mllvm -Wl,-align-all-nofallthru-blocks=6 Force the alignment of all blocks that have no fall-through predecessors (i.e. don't add nops that are executed). In log2 format (e.g 4 means align on 16B boundaries).

]]> -std=gnu89 Selects the C language dialect.

]]> -std=c++98 Selects the C++ language dialect.

]]> -Wl,-allow-multiple-definition Do not generate an error when linking multiple symbols of the same name.

]]> -Wno-return-type Do not warn about functions defined with a return type that defaults to "int" or which return something other than what they were declared to.

]]> -Wno-unused-command-line-argument Do not warn about unused command line arguments.

]]> -fvisibility=hidden Set the default symbol visibility for all global declarations.

]]> -mllvm -enable-loop-fusion This option enables the classical loop fusion transformation where the bodies of multiple loop nests are fused into one loop nest. The transformation checks various legality criteria involving the bounds of the loop nests involved, the control flow nesting of the loop nests and so on.

Loop fusion enables reuse of memory access operations across the loop nests and is also beneficial for cache performance. As part of the profitability check for this transformation it uses code size thresholds which control the size of the fused loop body created.

This transformation is off by default and may be enabled by using this option.

]]> -mllvm -enable-loopinterchange This option enables the classical loop interchange or loop permutation transformation on a loop nest.

This transformation reorders the loops in a multi-dimensional loop nest, checking for various legality criteria in the process. The goal is to find a reordering of the loops such that the number of loop invariant expressions that may be hoisted out from an inner loop to a loop at a higher level is maximized.

This transformation is off by default and may be enabled by using this option.

]]> -mllvm -compute-interchange-order -fconvert=big-endian The binary datasets for some of the Fortran benchmarks in the SPEC CPU suites are stored in big-endian format. This option is necessary for those datasets to be read in correctly.

]]> -Mbyteswapio The binary datasets for some of the Fortran benchmarks in the SPEC CPU suites are stored in big-endian format. This option is necessary for those datasets to be read in correctly.

]]> Specifies size of off_t data type.

]]> -funsigned-char This option instructs the compiler to treat char type as unsigned.

]]> /path/to/{clang|clang++|flang} This flag is just to trim the path from the compiler line. clang clang is a C compiler which encompasses preprocessing, parsing, optimization, code generation, assembly, and linking. Depending on which high-level mode setting is passed, Clang will stop before doing a full link.

]]> clang++ clang++ C++ compiler which encompasses preprocessing, parsing, optimization, code generation, assembly, and linking. Depending on which high-level mode setting is passed, Clang will stop before doing a full link.

]]> flang flang is a Fortran compiler which encompasses parsing, optimization, code generation, assembly, and linking. Depending on which high-level mode setting is passed, Flang will stop before doing a full link.

]]> -L/path/to/libs Specifies a directory to search for libraries. Use -L to add directories to the search path for library files. Multiple -L options are valid. However, the position of multiple -L options is important relative to -l options supplied.

]]> -I /path/to/include Specifies a directory to search for include files. Use -I to add directories to the search path for include files. Multiple -I options are valid.

]]>