Running Gray-Scott on Odo/Frontier

This instructions are for the Odo training system at OLCF. Which is almost identical to Frontier.

We assume the user has access to the trn036 training account

Access Odo/Frontier

 $ ssh USERNAME@login1.odo.olcf.ornl.gov

Obtain Gray-Scott, first access your scrath directory, clone the repository pointing at the GrayScott-JACC branch.
```
 cd /gpfs/wolf2/olcf/trn036/scratch/$USER
 git clone https://github.com/JuliaORNL/GrayScott.jl.git
```

Run the script prepared for this tutorial GrayScott.jl/scripts/config_odo.sh to set up modules, environment, and packages. This is a one-time step and might take a few minutes.

 source GrayScott.jl/scripts/config_odo.sh

 #!/bin/bash

 PROJ_DIR=/gpfs/wolf2/olcf/trn036/scratch/$USER
 export JULIA_DEPOT_PATH=$PROJ_DIR/.julia
 GS_DIR=$PROJ_DIR/GrayScott.jl

 # remove existing generated Manifest.toml
 rm -f $GS_DIR/Manifest.toml
 rm -f $GS_DIR/LocalPreferences.toml

 # good practice to avoid conflicts with existing default modules
 module purge

 # load required modules
 module load PrgEnv-cray/8.6.0
 module load cray-mpich
 module load rocm/6.3.1
 module load adios2/2.10.0-mpi
 module load Core/25.05
 module load julia/1.11.0

 # AMDGPU.jl requires rocm system libraries (default = /opt/rocm)
 export ROCM_PATH=/opt/rocm-6.3.1

 # Required to point at underlying modules above
 export JULIA_ADIOS2_PATH=$OLCF_ADIOS2_ROOT

 # Downloads Julia packages in Project.toml
 julia --project=$GS_DIR -e 'using Pkg; Pkg.instantiate()'

 # Set MPIPreferences to use Cray's MPICH in LocalPreferences.toml
 julia --project=$GS_DIR -e 'using MPIPreferences; MPIPreferences.use_system_binary(mpiexec="srun", vendor="cray")'

 # Set JACC AMDGPU back end in LocalPreferences.toml.
 # It will add AMDGPU to Project.toml
 julia --project=$GS_DIR -e 'using JACC; JACC.set_backend("AMDGPU")'

 # Verify the packages are installed correctly
 julia --project=$GS_DIR -e 'using Pkg; Pkg.build()'
 # Precompile dependencies
 julia --project=$GS_DIR -e 'using Pkg; Pkg.precompile()'

JULIA_DEPOT is where Julia packages and artifacts (e.g. extra data) will be installed for different local environments. Do not use $HOME for limited performance - not scalable - and storage reasons, but for local tasks (e.g. analysis).

Running Gray-Scott jobs on Odo/Frontier

Create an area for Gray-Scott runs outside the repository (e.g. run001, future runs will be in run002, run003, etc.)
```
 mkdir run001
```

Copy the Gray-Scott settings file and the job_odo.sh to the run directory

 cp GrayScott.jl/examples/settings-file-odo.json run001
 cp GrayScott.jl/scripts/job_odo.sh run001

Submit your first job to Odo. It should generate an adios bp file output, and total runtime should be around 17 seconds using a single MPI process and AMD GPU.
```
 cd run001
 sbatch job_odo.sh
```

Check the output files generated by the job with ADIOS2 bpls

 bpls -lav gs-1MPI-1GPU-64L-F32-JACC-AMDGPU.bp

 File info:
 of variables:  3
 of attributes: 13
 statistics:    Min / Max 

 double   Du                           attr   = 0.2
 double   Dv                           attr   = 0.1
 double   F                            attr   = 0.02
 string   Fides_Data_Model             attr   = "uniform"
 string   Fides_Dimension_Variable     attr   = "U"
 double   Fides_Origin                 attr   = {0, 0, 0}
 double   Fides_Spacing                attr   = {0.1, 0.1, 0.1}
 string   Fides_Variable_Associations  attr   = {"points", "points"}
 string   Fides_Variable_List          attr   = {"U", "V"}
 float    U                            100*{64, 64, 64} = -0.115931 / 1.46275
 float    V                            100*{64, 64, 64} = 0 / 1.04308
 double   dt                           attr   = 1
 double   k                            attr   = 0.048
 double   noise                        attr   = 0.1
 int32_t  step                         100*scalar = 10 / 1000
 string   vtk.xml                      attr   = 
 <VTKFile type="ImageData" version="0.1" byte_order="LittleEndian">
   <ImageData WholeExtent="0 64 0 64 0 64" Origin="0 0 0" Spacing="1 1 1">
     <Piece Extent="0 64 0 64 0 64">
       <CellData Scalars="U">
         <DataArray Name="U" />
         <DataArray Name="V" />
         <DataArray Name="TIME">
               step
             </DataArray>
       </CellData>
     </Piece>
   </ImageData>
 </VTKFile>

Future runs: repeats steps 1-3 above , and edit the settings file to set the desired parameters for the Gray-Scott simulation.
- L is the number of cells on each direction, Lx = Ly = Lz = L, so L^3 is the total number of cells.
- plotgap is the number of steps between each visualization output.
- steps is the total number of steps to run the simulation.
- output is the name of the adios2 output directory dataset (can be visualized with ParaView for small cases)docs.
- Other variables might not influence the simulation but can be changed for testing purposes.
```
 {
 "L": 64,
 "Du": 0.2,
 "Dv": 0.1,
 "F": 0.02,
 "k": 0.048,
 "dt": 1.0,
 "plotgap": 10,
 "steps": 1000,
 "noise": 0.1,
 "output": "gs-1MPI-1GPU-64L-F32-JACC-AMDGPU.bp",
 "checkpoint": false,
 "checkpoint_freq": 700,
 "checkpoint_output": "ckpt.bp",
 "restart": false,
 "restart_input": "ckpt.bp",
 "adios_config": "adios2.xml",
 "adios_span": false,
 "adios_memory_selection": false,
 "mesh_type": "image",
 "precision": "Float32"
 }
```

Edit the launch script to request more resources: nodes, gpus, time (follow instructors directions at this point)

 #!/bin/bash
 #SBATCH -A trn036
 #SBATCH -J gs-julia-1MPI-1GPU
 #SBATCH -o %x-%j.out
 #SBATCH -e %x-%j.err
 #SBATCH -t 0:02:00
 #SBATCH -N 1

 date
 #rocminfo

 GS_DIR=/gpfs/wolf2/olcf/trn036/scratch/$USER/GrayScott.jl
 GS_EXE=$GS_DIR/gray-scott.jl

 srun -n 1 --gpus=1 julia --project=$GS_DIR $GS_EXE settings-file-odo.json

 # launch this file with sbatch `$ sbatch job_odo.sh`