Short Description

The Comprehensive Air quality Model with extensions (CAMx) is a publicly available open-source computer modeling system for the integrated assessment of gaseous and particulate air pollution.

CAMx template job

For the Grid enabled CAMx template job we will use a CAMx precompiled binary executable as well as the downloadable test case provided by Environ. We will also need a JDL file which in short describes the batch job (input files, output files, executables etc) and a wrapper C shell script.

Below we will go through the JDL and wrapper script files.

The JDL file

JobType = "Normal";
Executable = "camx.csh";
CpuNumber = 5;
Arguments = "noHDF MPICH2";
StdOutput = "std.out";
StdError = "std.err";
InputSandbox = "camx.csh";
OutputSandbox = {"std.err","std.out"};
Requirements = Member("MPICH2", other.GlueHostApplicationSoftwareRunTimeEnvironment)
        && (other.GlueHostArchitecturePlatformType == "x86_64");

In the first line we define that this is a normal batch job and on the next two lines we define the name of the executable wrapper C shell script and the number of CPUs we will be using (5 in this case).

Next we provide a series of Arguments to the wrapper C shell script. The number of Arguments for this template job varies from 2 to 4.

• The first one signifies if we want to use the HDF library. Possible values are noHDF and HDF5.
• The second one signifies if we want to use the MPI library. Possible values are noMPI and MPICH2.
• The third one, which is optional signifies if we want to use the OpenMP enabled version of CAMx. If we leave it blank we will not use OpenMP. If we want to use the OpenMP enabled version we need to provide the third argument as omp.
• The fourth argument is taken into account whenever both MPI and OpenMP are used. It is a number that signifies the number of OpenMP threads per MPI process.

Notice that if the pure MPI is used one additional daemon CAMx process should be counted for in the JDL file, thus if CpuNumber is equal to 5 this means that the job will be run on 1 master process and 4 slave (computing) processes (no computing is done by the master process).

For a pure OpenMP job you are advised to use only GR-01-AUTH computing resources and up to 8 cores per job. Here is the snip code you should use within the JDL file:

CpuNumber = 8;
Arguments = "noHDF noMPI omp";
Environment = {"OPENMP=true"};
Requirements = other.ClueCEInfoHostName == "ce01.grid.auth.gr";

Please DON'T USE the mixed MPI/!OpenMP mode unless you really know what you are doing. Here is an example for running CAMx on 16 cores with 2 OpenMP threads per MPI process:

CpuNumber = 16;
Arguments = "noHDF MPICH2 omp 2";
Environment = {"NODES_REQ=8:ppn=2"};
Requirements = other.ClueCEInfoHostName == "ce01.grid.auth.gr";

In the next few lines we define Input and Output files as usual.

Notice that we only stage the wrapper C shell script and the Standard output and error through the JDL file. All other CAMx related files (both input and output) will be staged through Storage Elements.

The wrapper C shell script

The wrapper script can be divided into 4 sections:

1. Definitions of environment variables
2. Downloading of input files and executable
3. Execution of CAMx model
4. Uploading of output files

For completeness we provide here the complete wrapper script and go through it step by step below.

#!/bin/csh

# Define necessary environment variables
set MODEL = "CAMx"
set VERSION = "5.30"
set HDF = "$1"
set MPI = "$2"
set OMP = "$3"
set COMP = "i_linux$OMP"
set DOMAIN = "v$VERSION"
set EXEC_STR = "$MODEL.$DOMAIN.$HDF.$MPI.$COMP"
set EXEC = "./$EXEC_STR"
setenv LFC_HOST lfc.grid.auth.gr

echo Starting with downloads: `date`

# Download input files and unpack
foreach i (CAMx5.3x.test_run.inputs_met.101223.tar.gz CAMx5.3x.test_run.inputs_other.101223.tar.gz v5.30.specific.inputs.101223.tar.gz)
  lcg-cp lfn:/grid/see/gridauth-users/camx/$VERSION/case-environ/inputs/$i file:$i
  tar zxf $i
end

# Download executable and chmod
lcg-cp lfn:/grid/see/gridauth-users/camx/$VERSION/bin/$EXEC_STR file:$EXEC_STR
chmod +x $EXEC

echo Finished with downloads: `date`

# Start CAMx execution
set RUN     = "v$VERSION.midwest.36.12.$HDF.$MPI.$COMP"
set INPUT   = "./inputs"
set MET     = "./inputs/met"
set EMIS    = "./emiss"
set PTSRCE  = "./ptsrce"
set OUTPUT  = "./outputs"
#
mkdir -p $OUTPUT
#
#  --- set the dates and times ----
#
set RESTART = "NO"
foreach today (03.154 04.155)
set YYYY = 2002
set MM = 06
set JUL = $today:e
set CAL = $today:r
set YESTERDAY = `echo ${CAL} | awk '{printf("%2.2d",$1-1)}'`
#
if( ${RESTART} == "NO" ) then
        set RESTART = "false"
else
        set RESTART = "true"
endif
#
#  --- Create the input file (always called CAMx.in)
#
cat << ieof > CAMx.in

 &CAMx_Control

 Run_Message      = 'CAMx 5.30 Test Problem -- Mech6 CF CB05 $RUN',

!--- Model clock control ---

 Time_Zone        = 0,                 ! (0=UTC,5=EST,6=CST,7=MST,8=PST)
 Restart          = .${RESTART}.,
 Start_Date_Hour  = 2002,06,${CAL},0000,   ! (YYYY,MM,DD,HHmm)
 End_Date_Hour    = 2002,06,${CAL},2400,   ! (YYYY,MM,DD,HHmm)

 Maximum_Timestep    = 15.,            ! minutes
 Met_Input_Frequency = 60.,            ! minutes
 Ems_Input_Frequency = 60.,            ! minutes
 Output_Frequency    = 60.,            ! minutes

!--- Map projection parameters ---

 Map_Projection           = 'LAMBERT',  ! (LAMBERT,POLAR,UTM,LATLON)
 UTM_Zone                 = 0,
 POLAR_Longitude_Pole     = 0.,        ! deg (west<0,south<0)
 POLAR_Latitude_Pole      = 0.,        ! deg (west<0,south<0)
 LAMBERT_Central_Meridian = -97.,      ! deg (west<0,south<0)
 LAMBERT_Center_Longitude = -97.,      ! deg (west<0,south<0)
 LAMBERT_Center_Latitude  =  40.,      ! deg (west<0,south<0)
 LAMBERT_True_Latitude1   =  45.,      ! deg (west<0,south<0)
 LAMBERT_True_Latitude2   =  33.,      ! deg (west<0,south<0)

!--- Parameters for the master (first) grid ---

 Number_of_Grids      = 2,
 Master_Origin_XCoord = -792.,         ! km or deg, SW corner of cell(1,1)
 Master_Origin_YCoord = -1656.,        ! km or deg, SW corner of cell (1,1)
 Master_Cell_XSize    = 36.,           ! km or deg
 Master_Cell_YSize    = 36.,           ! km or deg
 Master_Grid_Columns  = 68,
 Master_Grid_Rows     = 68,
 Number_of_Layers(1)  = 16,

!--- Parameters for the second grid ---

 Nest_Meshing_Factor(2) = 3,           ! Cell size relative to master grid
 Nest_Beg_I_Index(2)    = 22,          ! Relative to master grid
 Nest_End_I_Index(2)    = 51,          ! Relative to master grid
 Nest_Beg_J_Index(2)    = 22,          ! Relative to master grid
 Nest_End_J_Index(2)    = 58,          ! Relative to master grid
 Number_of_Layers(2)    = 16,

!--- Model options ---

 Diagnostic_Error_Check = .false.,      ! True = will stop after 1st timestep
 Advection_Solver       = 'PPM',        ! (PPM,BOTT)
 Chemistry_Solver       = 'EBI',        ! (EBI,IEH,LSODE)
 PiG_Submodel           = 'None',       ! (None,GREASD,IRON)
 Probing_Tool           = 'None',       ! (None,OSAT,GOAT,APCA,PSAT,DDM,PA,RTRAC)
 Chemistry              = .true.,
 Drydep_Model           = 'WESELY89',   ! (None,WESELY89,ZHANG03)
 Wet_Deposition         = .true.,
 TUV_Cloud_Adjust       = .false.,
 ACM2_Diffusion         = .false.,
 Staggered_Winds        = .true.,
 Super_Stepping         = .true.,
 Gridded_Emissions      = .true.,
 Point_Emissions        = .true.,
 Ignore_Emission_Dates  = .true.,

!--- Output specifications ---

 Root_Output_Name         = '$OUTPUT/CAMx.$RUN.200206${CAL}',
 Average_Output_3D        = .false.,
 HDF_Format_Output        = .false.,
 Number_of_Output_Species  = 21,
 Output_Species_Names(1)   = 'NO',
 Output_Species_Names(2)   = 'NO2',
 Output_Species_Names(3)   = 'O3',
 Output_Species_Names(4)   = 'SO2',
 Output_Species_Names(5)   = 'H2O2',
 Output_Species_Names(6)   = 'HNO3',
 Output_Species_Names(7)   = 'NH3',
 Output_Species_Names(8)   = 'PNO3',
 Output_Species_Names(9)   = 'PSO4',
 Output_Species_Names(10)  = 'PNH4',
 Output_Species_Names(11)  = 'POA',
 Output_Species_Names(12)  = 'PEC',
 Output_Species_Names(13)  = 'FPRM',
 Output_Species_Names(14)  = 'CPRM',
 Output_Species_Names(15)  = 'CCRS',
 Output_Species_Names(16)  = 'FCRS',
 Output_Species_Names(17)  = 'SOA1',
 Output_Species_Names(18)  = 'SOA2',
 Output_Species_Names(19)  = 'SOA3',
 Output_Species_Names(20)  = 'SOA4',
 Output_Species_Names(21)  = 'SOA5',

!--- Input files ---

 Chemistry_Parameters = '$INPUT/CAMx5.3.chemparam.6_CF',
 Photolyis_Rates      = '$INPUT/tuv.200206.STL.txt',
 Initial_Conditions   = '$INPUT/IC.vistas_2002gt2a_STL_36_68X68_16L.2002081',
 Boundary_Conditions  = '$INPUT/BC.vistas_2002gt2a_STL_36_68X68_16L.2002${JUL}',
 Albedo_Haze_Ozone    = '$INPUT/ahomap.200206.STL_36_68X68_12_92X113.txt',
 Point_Sources        = '$PTSRCE/ptsrce.stl.36km.2002${JUL}.a0.bin',
 Master_Grid_Restart  = '$OUTPUT/CAMx.$RUN.200206${YESTERDAY}.inst',
 Nested_Grid_Restart  = '$OUTPUT/CAMx.$RUN.200206${YESTERDAY}.finst',
 PiG_Restart          = ' ',

 Emiss_Grid(1)   = '$EMIS/emiss.stl.36km.200206${CAL}.a1.bin',
 Landuse_Grid(1) = '$INPUT/lu.STL_36_68X68.bin',
 ZP_Grid(1)      = '$INPUT/met/camx.zp.200206${CAL}.36k.bin',
 Wind_Grid(1)    = '$INPUT/met/camx.uv.200206${CAL}.36k.bin',
 Temp_Grid(1)    = '$INPUT/met/camx.tp.200206${CAL}.36k.bin',
 Vapor_Grid(1)   = '$INPUT/met/camx.qa.200206${CAL}.36k.bin',
 Cloud_Grid(1)   = '$INPUT/met/camx.cr.200206${CAL}.36k.bin',
 Kv_Grid(1)      = '$INPUT/met/camx.kv.200206${CAL}.36k.bin',
 Emiss_Grid(2)   = '$EMIS/emiss.stl.12kmsmall.200206${CAL}.a1.bin',
 Landuse_Grid(2) = '$INPUT/lu.STL_12_92X113.bin',
 ZP_Grid(2)      = '$INPUT/met/camx.zp.200206${CAL}.12ksmall.bin',
 Wind_Grid(2)    = '$INPUT/met/camx.uv.200206${CAL}.12ksmall.bin',
 Temp_Grid(2)    = '$INPUT/met/camx.tp.200206${CAL}.12ksmall.bin',
 Vapor_Grid(2)   = '$INPUT/met/camx.qa.200206${CAL}.12ksmall.bin',
 Cloud_Grid(2)   = '$INPUT/met/camx.cr.200206${CAL}.12ksmall.bin',
 Kv_Grid(2)      = '$INPUT/met/camx.kv.200206${CAL}.12ksmall.bin',

 /
!-------------------------------------------------------------------------------

ieof
#
#  --- Execute the model ---
#

set time0 = `date +%s`

if ( "x$MPI" == "xnoMPI" ) then
  if ( "x$OMP" == "x" ) then 
    echo "Using serial version of CAMx on: " $HOSTNAME
    $EXEC
  else if ( "x$OMP" == "xomp" ) then
    echo "Using OpenMP version of CAMx on:"
    cat $PBS_NODEFILE
    setenv OMP_NUM_THREADS `cat $PBS_NODEFILE | wc -l`
    $EXEC
  endif
else if ( "x$MPI" == "xMPICH2" ) then
  if ( "x$OMP" == "x" ) then
    echo "Using $MPI version of CAMx on:"
    cat $PBS_NODEFILE
    echo "$MPI_MPICH2_MPIEXEC -comm=pmi $EXEC"
    $MPI_MPICH2_MPIEXEC -comm=pmi $EXEC
  else if ( "x$OMP" == "xomp" ) then
    echo "Using mixed MPI/OMP version of CAMx"
    setenv NUM_CPUS `cat $PBS_NODEFILE | wc -l`
    setenv NUM_NODES `cat $PBS_NODEFILE | sort -u | wc -l`
    setenv OMP_NUM_THREADS $4
    echo OMP_NUM_THREADS = $OMP_NUM_THREADS
    setenv MPI_PROCESSES `echo $NUM_CPUS/$OMP_NUM_THREADS | bc`
    setenv MPI_PER_NODE `echo $MPI_PROCESSES/$NUM_NODES | bc`
    if ( $MPI_PROCESSES <= 0 ) echo "MPI_PROCESSES is zero or less..."
    if ( $MPI_PER_NODE <= 0 ) echo "MPI_PER_NODE is zero or less..."
    echo -n "$HOSTNAME " > tmp.config
    foreach i ( `seq 1 $MPI_PER_NODE` )
      foreach node ( `cat $PBS_NODEFILE | sort -u ` )
         echo -n "$node " >> tmp.config
      end
    end
    echo " : ./$EXEC_STR " >> tmp.config
    cat tmp.config
    echo "$MPI_MPICH2_MPIEXEC -comm=pmi -config tmp.config"
    $MPI_MPICH2_MPIEXEC -comm=pmi -config tmp.config
    rm -rf tmp.config
  endif
endif

set time1 = `date +%s`
echo dt = `echo $time1-$time0| bc`

echo Starting packing and uploading: `date`

# pack output folder and zip it
tar cvf outputs.tar outputs/*${YYYY}${MM}${CAL}*
gzip -9 outputs.tar

# delete old output tarball (if it exists)
lcg-del -a lfn:/grid/see/gridauth-users/camx/$VERSION/case-environ/outputs/${YYYY}.${MM}.${CAL}.outputs.tar.gz

# copy and register output files
lcg-cr -l lfn:/grid/see/gridauth-users/camx/$VERSION/case-environ/outputs/${YYYY}.${MM}.${CAL}.outputs.tar.gz file:outputs.tar.gz
rm -rf $PWD/outputs.tar.gz

echo Finished packing and uploading: `date`

end

Environment variables

In this short section we read the first three Arguments from the JDL file and appoint them to the appropriate variables. These variables define the executable we will download as well as the way this will be executed.

#!/bin/csh

# Define necessary environment variables
set MODEL = "CAMx"
set VERSION = "5.30"
set HDF = "$1"
set MPI = "$2"
set OMP = "$3"
set COMP = "i_linux$OMP"
set DOMAIN = "v$VERSION"
set EXEC_STR = "$MODEL.$DOMAIN.$HDF.$MPI.$COMP"
set EXEC = "./$EXEC_STR"
setenv LFC_HOST lfc.grid.auth.gr

Input files and executable

In the next section we download and unpack the environ test case input files.

You will need to make changes to lines 21-24 and provide your own input files in order to run your own case jobs.

We also download the CAMx executable.

Notice that we download input files and executables from lfc.grid.auth.gr. In the general case you may use whichever LFC service you want for input and output files. However, the CAMx executables are available only to licensed users through lfc.grid.auth.gr.

echo Starting with downloads: `date`

# Download input files and unpack
foreach i (CAMx5.3x.test_run.inputs_met.101223.tar.gz CAMx5.3x.test_run.inputs_other.101223.tar.gz v5.30.specific.inputs.101223.tar.gz)
  lcg-cp lfn:/grid/see/gridauth-users/camx/$VERSION/case-environ/inputs/$i file:$i
  tar zxf $i
end

# Download executable and chmod
lcg-cp lfn:/grid/see/gridauth-users/camx/$VERSION/bin/$EXEC_STR file:$EXEC_STR
chmod +x $EXEC

echo Finished with downloads: `date`

Execution of CAMx model

In the following code preview we have omitted the stuff related to the creation of the CAMx.in file. We directly display the execution of the model which is based on the Arguments we have provided within the JDL file.

As can be seen the way the execution of the model is fired up depends on the executable we have downloaded in the first place, which in turn depends on the variables we have used in the Arguments section in the JDL file. The if function is split on the first level between noMPI and MPICH2. Depending on whether OpenMP is used a second level of if functions is used to distinguish among possible execution scenarios.

The only non trivial execution scenario is the mixed mode where several additional environment variables are calculated and used in order to partition the job. Through these the usage of the Environment attribute in the JDL file is not required, but it is however suggested. If something goes wrong with the allocation the script will exit with a warning message.

set time0 = `date +%s`

if ( "x$MPI" == "xnoMPI" ) then
  if ( "x$OMP" == "x" ) then 
    echo "Using serial version of CAMx on: " $HOSTNAME
    $EXEC
  else if ( "x$OMP" == "xomp" ) then
    echo "Using OpenMP version of CAMx on:"
    cat $PBS_NODEFILE
    setenv OMP_NUM_THREADS `cat $PBS_NODEFILE | wc -l`
    $EXEC
  endif
else if ( "x$MPI" == "xMPICH2" ) then
  if ( "x$OMP" == "x" ) then
    echo "Using $MPI version of CAMx on:"
    cat $PBS_NODEFILE
    echo "$MPI_MPICH2_MPIEXEC -comm=pmi $EXEC"
    $MPI_MPICH2_MPIEXEC -comm=pmi $EXEC
  else if ( "x$OMP" == "xomp" ) then
    echo "Using mixed MPI/OMP version of CAMx"
    setenv NUM_CPUS `cat $PBS_NODEFILE | wc -l`
    setenv NUM_NODES `cat $PBS_NODEFILE | sort -u | wc -l`
    setenv OMP_NUM_THREADS $4
    echo OMP_NUM_THREADS = $OMP_NUM_THREADS
    setenv MPI_PROCESSES `echo $NUM_CPUS/$OMP_NUM_THREADS | bc`
    setenv MPI_PER_NODE `echo $MPI_PROCESSES/$NUM_NODES | bc`
    if ( $MPI_PROCESSES <= 0 ) echo "MPI_PROCESSES is zero or less..."
    if ( $MPI_PER_NODE <= 0 ) echo "MPI_PER_NODE is zero or less..."
    echo -n "$HOSTNAME " > tmp.config
    foreach i ( `seq 1 $MPI_PER_NODE` )
      foreach node ( `cat $PBS_NODEFILE | sort -u ` )
         echo -n "$node " >> tmp.config
      end
    end
    echo " : ./$EXEC_STR " >> tmp.config
    cat tmp.config
    echo "$MPI_MPICH2_MPIEXEC -comm=pmi -config tmp.config"
    $MPI_MPICH2_MPIEXEC -comm=pmi -config tmp.config
    rm -rf tmp.config
  endif
endif

set time1 = `date +%s`
echo dt = `echo $time1-$time0| bc`

Upload of output files

As a final step output files are packed and uploaded to Grid Storage Elements. We pack our results into a single tar archive in line 243 and compress this in line 244. Since we are using a predefined storage space for this test case we delete any previously created file on line 247. Then on lines 250, 251 we upload our results and delete their local copies.

Notice that for each day, month and year in the loop a different tar archive is created and uploaded.

echo Starting packing and uploading: `date`

# pack output folder and zip it
tar cvf outputs.tar outputs/*${YYYY}${MM}${CAL}*
gzip -9 outputs.tar

# delete old output tarball (if it exists)
lcg-del -a lfn:/grid/see/gridauth-users/camx/$VERSION/case-environ/outputs/${YYYY}.${MM}.${CAL}.outputs.tar.gz

# copy and register output files
lcg-cr -l lfn:/grid/see/gridauth-users/camx/$VERSION/case-environ/outputs/${YYYY}.${MM}.${CAL}.outputs.tar.gz file:outputs.tar.gz
rm -rf $PWD/outputs.tar.gz

echo Finished packing and uploading: `date`

Job submission and collection of results

The CAMx template job presented here should work as is. We encourage you to test this before proceeding with the modifications you need to make to adapt these two template files to your needs. To start off,

1. download the JDL and the wrapper script files and place them on a folder of your choice on the User Interface. Then
2. submit this template job.

On the User Interface use the following commands to complete these two steps

Notice that you need a valid voms proxy with see vo extensions for the third command to execute properly.

wget http://wiki.grid.auth.gr/wiki/pub/Groups/ALL/CAMx/camx.jdl
wget http://wiki.grid.auth.gr/wiki/pub/Groups/ALL/CAMx/camx.csh
glite-wms-job-submit -a -o camx.txt camx.jdl

For more information on job management please visit this guide.

Output files are retrievable via two channels, namely via the OutputSandbox and via the LFC.

OutputSandbox files: To retrieve the contents of the OutputSandbox (std.out and std.err in our case) you need to execute the following command once the job finishes execution.

glite-wms-job-output --dir ./camx-result -i camx.txt

Look for the camx-result folder. Within that there should be a subfolder containing the 2 files std.out and std.err.

LFC files: To retrieve the CAMx output files stored on the LFC once the job finishes execution you need to execute the following commands on the User Interface

You will need to define the LFC_HOST environmental variable using module load see/auth (in case you skipped this step earlier) for the following commands to work properly.

lcg-cp lfn:/grid/see/gridauth-users/camx/5.30/case-environ/outputs/2002.06.03.outputs.tar.gz file:2002.06.03.outputs.tar.gz
lcg-del -a lfn:/grid/see/gridauth-users/camx/5.30/case-environ/outputs/2002.06.03.outputs.tar.gz
tar zxf 2002.06.03.outputs.tar.gz
lcg-cp lfn:/grid/see/gridauth-users/camx/5.30/case-environ/outputs/2002.06.04.outputs.tar.gz file:2002.06.04.outputs.tar.gz
lcg-del -a lfn:/grid/see/gridauth-users/camx/5.30/case-environ/outputs/2002.06.04.outputs.tar.gz
tar zxf 2002.06.04.outputs.tar.gz

Notice that we erase the output tarball from the LFC (lcg-del) so that the next person to submit this template job does not get an error trying to overwrite an existing file from the Worker Node.

Adapt the template job to your own needs

Before making any changes to the template job we suggest that you use it once as is to make sure that everything works as expected. Afterwards you will need to make changes to the wrapper script and perhaps also to the JDL file. Regarding the JDL file the most crucial point to consider is the Arguments attribute. We suggest that you always use the pure MPI version whenever possible and switch to OpenMP or mixed MPI/!OpenMP whenever you want to use more than ~10 CPU cores. In this case don't hesitate to drop us an e-mail so that we may further consult you on the changes you need to do.

Regarding the wrapper script itself the major changes you need to consider are

• Downloading of input files (lines 21-24)
• CAMx.in file generator (lines 34-189)
• Uploading of output files (lines 246, 249)

The overall workflow one follows using this use case is presented in the following graph.

If something goes wrong after you make these changes please send us an e-mail describing the problem.

You may download the JDL and wrapper script from the Attachments table below.

References

• http://www.camx.com