PFLOTRAN Tiger Team Instructions

PFLOTRAN uses PETSc and HDF5 libraries, please see below for more information about building PETSc and PFLOTRAN.

[edit] Building PETSc libraries

PFLOTRAN uses the developer version of PETSc libraries, clone the latest petsc-dev source from http://www.mcs.anl.gov/petsc/petsc-2/developers/index.html
- NOTE: To maintain consistency among different groups within PERI, please download the following version of PETSc dev:
- PETSc source changeset: fc78576e289c
- PETSc BuildSystem changeset: 19767ef5a5d8

Instructions excerpted from above website
- hg clone -r fc78576e289c http://petsc.cs.iit.edu/petsc/petsc-dev
- cd petsc-dev/config
- hg clone -r 19767ef5a5d8 http://petsc.cs.iit.edu/petsc/BuildSystem BuildSystem

Copy the relevant configuration script from /tmp/work/vamsi/tiger_team/petsc_config_scripts directory on JaguarPF (pgi_9.0.4_fast.py on Cray XT and bgp-ibm-opt.py on IBM BG/P) to ${PETSC_DIR}/config directory. You will have to set the environment variables PETSC_ARCH to the name of the configuration script and PETSC_DIR to point to petsc-dev directory before running the script.
cd $PETSC_DIR && config/$PETSC_ARCH
After the configuration is completed, manually define the variables PETSC_HAVE_F90_2PTR_ARG and PETSC_HANDSHAKE_IO to 1 in ${PETSC_DIR}/${PETSC_ARCH}/include/petscconf.h
Run make all.

On Cray XT (Jaguar), load the parallel HDF5 libaries, module load hdf5/1.6.8_par.
On IBM BG/P, define the environment variables HDF5_INCLUDE="/soft/apps/hdf5-1.6.6/include" and HDF5_LIB="/soft/apps/hdf5-1.6.6/lib"

Please use the PFLOTRAN source available at the following location on JaguarPF: /tmp/work/vamsi/pflotran_2b/pflotran_src.tgz (After extracting, "hg tip" on pflotran source directory should show you the changeset as 3799e94e5e6c. This version of code was last updated in Jan 2010.)
Make sure you have PETSC_DIR, PETSC_ARCH and required HDF5 flags defined properly before building PFLOTRAN.
- In the PFLOTRAN source directory (src/pflotran), to use the default I/O method, run
  - make pflotran hdf5=1 jaguar=1 (On Cray XT); make pflotran hdf5=1 bgp=1 (On IBM BG/P)
- To use the improved I/O method, use the following build command,
  - make pflotran hdf5-vamsi=1 jaguar=1; on Cray XT.

Please specify these arguments to the binary during job submission: -file_output no -flow_mat_type aij -log_summary
In order to use the improved BiCGStab solver within PETSc, use these arguments: -file_output no -flow_mat_type aij -flow_ksp_type ibcgs -flow_ksp_lag_norm -tran_ksp_type ibcgs -log_summary

Input files for this problem are available at the following location on JaguarPF: /tmp/work/vamsi/tiger_team/270m_dataset.tar.gz
To run the executable with the steady state version of 270 M cell problem, create a job directory that contains the files, input_1350_2500_80_v1_clipped.h5, pflotran.in and the relevant job submission script. (These files are available in 270m_dataset.tar.gz)
Important keywords in input file, pflotran.in
- MAX_STEPS sets the number of time steps of the simulation.
- FORMAT HDF5 to turn on/off HDF5 output; PERIODIC TIMESTEP to control the rate at which output is written.

Please note that in order to run this problem with PFLOTRAN, you will have to request at least 4k cores. Please see the Smaller datasets section if you wish to run at a smaller core count.
The input files for this problem set and PFLOTRAN source are available at the following locations:
- On Jaguar: /tmp/work/vamsi/pflotran_2b/pflotran_2b.tgz
- On Intrepid: /gpfs1/vamsi/pflotran_2b/pflotran_2b.tgz
To run the executable with the 2 billion cell problem, extract the tar file to a job submission directory and use the relevant job submission script.
Important keywords in input file, pflotran.in
- MAX_STEPS sets the number of time steps of the simulation.
- FORMAT HDF5 to turn on/off HDF5 output; PERIODIC TIMESTEP to control the rate at which output is written.

Performance data for the 2 billion problem is available at http://neptune.ce.ncsu.edu/~vamsi/2b_dof/

You can get the smaller data sets of PFLOTRAN from the following location on JaguarPF: /ccs/home/vamsi/pflotran_input_deck.tar.gz
There are 6 different problem sizes listed by directory. The directory name is the grid size (NXxNYxNZ). Multiply the grid size by 1 and 15 dofs(degrees of freedom) per cell for flow and transport to get the problem size.