===================================================
  o-o           o           o--o   o-o   o-o  o   o 
 /              |           |   | |     |     |\  | 
O      oo  o-o -o- o  o o-o O--o   o-o   o-o  | \ | 
 \    | | |     |  |  |  \  |   |     |     | |  \| 
  o-o o-o- o-o  o  o--o o-o o--o  o--o  o--o  o   o 
===================================================

Jian Tao <jtao@cct.lsu.edu>,
Allen Gabrielle <gallen@cct.lsu.edu>,
Erik Schnetter <schnetter@cct.lsu.edu>

Feb 1st, 2009


1. Option file
--------------
For many architectures, Cactus can detect and set the compilation options
correctly without any input from users. However, in practice, it is 
advised to set those compilation options explicitly either for performance 
or debugging purpose. One can set and export environmental variables before 
the configuration takes place to let Cactus know what options one wants to 
use. The best practice is to use an file which includes all the preset
options. 

E.g. ($CACTIS_HOME/BENCHMARKS/sample_gnu_options)

CC                =   gcc
LD                =   g++
CXX               =   g++
F77               =   gfortran
F90               =   gfortran
OPTIMISE          =   yes
C_OPTIMISE_FLAGS  =  -O3
CXX_OPTIMISE_FLAGS=  -O3
F77_OPTIMISE_FLAGS=  -O3
F90_OPTIMISE_FLAGS=  -O3

Please follow the Cactus User's Guide for more compilation options. The Cactus
User's Guide can be generated with

#gmake UsersGuide

A file named UsersGuide.pdf will be created under $CACTUS_HOME/doc


2. ThornList
------------
The real work in Cactus is mainly done by the software 
modules, or thorns in Cactus terminology. We prepare several thorns which are 
necessary for this benchmark under $CACTUS_HOME/arrangements, and a list
of thorns which is required for compilation can be found in
$CACTIS_HOME/BENCHMARKS/spec.thornlist


3. Compiling
------------
Given the correct OPTIONS file for the working machine and THORNLIST, the compilation 
of Cactus is a joyful journey.

The command line to start compiling is:

#gmake spec-config options=BENCHMARKS/sample_gnu_options \ 
      THORNLIST=BENCHMARKS/spec.thornlist PROMPT=no 

One may want to speed up the compilation with "-j" option:

#gmake -j 2 spec-config options=BENCHMARKS/sample_gnu_options \
      THORNLIST=BENCHMARKS/spec.thornlist PROMPT=no 

Note that the code for this benchmark can be compiled against any standard 
MPI libraries and run in parallel out of the box. Please refer to 
the Cactus User's Guide for more info on compilation.


4. Parameter files
-----------------
Thorns in Cactus usually need user's input to set some parameters whose
default values might not suitable in some cases. The values of the 
parameters are set in a parameter file, in which users can also 
choose thorns to active during runtime. Those thorns which are compiled
but not activated in the parameter file will not be used during
runtime.

For instance, the following section in 
$CACTIS_HOME/BENCHMARKS/spec.par activate three thorns
GenericFD, ML_BSSN, and ML_BSSN_Helper, and set the values for 
several parameters of the three thorns.

...
ActiveThorns = "GenericFD ML_BSSN ML_BSSN_Helper"

ADMBase::evolution_method        = "ML_BSSN"
ADMBase::lapse_evolution_method  = "ML_BSSN"
ADMBase::shift_evolution_method  = "ML_BSSN"

ML_BSSN::my_boundary_condition   = "Minkowski"

ML_BSSN::harmonicN               = 1      # 1+log
ML_BSSN::harmonicF               = 2.0    # 1+log
ML_BSSN::ShiftGammaCoeff         = 0.75
ML_BSSN::BetaDriver              = 0.5
...

Two parameters at the very beginning of $CACTIS_HOME/BENCHMARKS/spec.par 
control the memory usage and run time.

The parameter Cactus::cctk_itlast controls the number of iterations that the code
will evolved forward in time, and driver::local_nsize controls the grid size.

E.g.,

Cactus::cctk_itlast              = 40
driver::local_nsize              = 60

means 60^3 local evolved grid points will be used, and the code
will iterated 40 time steps forward. 

The total run time depends on both parameters. The best practice is probably to 
set the driver::local_nsize to fit the physical memory of the running machine and 
then adjust Cactus::cctk_itlast to change the run time.

60^3 local evolved grid points require about 300MB physical memory.
With Cactus::cctk_itlast = 40, it takes a Lenovo T61P laptop with Intel Core2 Duo 
CPU T7700 @ 2.40GHz and 4GB memory about 4 minutes to finish.


5. Running
----------
An executable called cactus_spec will be created under
$CACTIS_HOME/exe after a successful compilation. Under $CACTIS_HOME, one
can start running the executable with

#./exe/cactus_spec BENCHMARKS/spec.par

To run in parallel,

#mpirun -np 2009 ./exe/cactus_spec BENCHMARKS/spec.par


6. Acknowledgments
------------------
Our thanks go to all authors of Cactus <http://www.cactuscode.org/> and 
Kranc <http://numrel.aei.mpg.de/Research/Kranc/>.

The McLachlan code <http://www.cct.lsu.edu/~eschnett/McLachlan/>, the 
numerical kernel of this benchmark suite, is generated with Kranc which 
requires Mathematica.  

We used the LoopControl thorn distributed with 
Carpet <http://www.carpetcode.org/>, an adaptive mesh refinement driver 
for Cactus.


7. Contact
----------
If you have any questions or suggestions, you can send email to 
Jian Tao <jtao@cct.lsu.edu>.

Thanks again for using the CactusBSSN benchmark suite !
