2 REDUS Framework (R Script) User Manual
This manual will guide you to prepare an assessment to be run in the REDUS framework by using only R scripting methods.
2.1 Pre-requisites
To be able to follow this guide, you must have these components ready on your system:
A working Docker installation (https://www.docker.com/get-started).
An R application.
R package
yaml
,git2r
,jsonlite
andbibtex
installed.## Installing packages into '/home/runner/work/_temp/Library' ## (as 'lib' is unspecified)
## also installing the dependencies 'sys', 'fansi', 'pkgconfig', 'purrr', 'cli', 'crayon', 'utf8', 'askpass', 'ellipsis', 'generics', 'lifecycle', 'R6', 'tibble', 'tidyselect', 'vctrs', 'pillar', 'Rcpp', 'curl', 'mime', 'openssl', 'dplyr', 'humaniformat', 'httr'
2.2 Getting the REDUS Framework container template
First, we need to download the REDUS Framework container template from Github. This template provides the skeleton that will automate the assessment processing. The template is available at https://github.com/REDUS-IMR/docker-redus-pipeline.
# (Optional) Set a working directory
setwd(tempdir())
# Clone the template
git2r::clone("https://github.com/REDUS-IMR/docker-redus-pipeline.git", "redus-framework")
## cloning into 'redus-framework'...
## Receiving objects: 2% (1/43), 8 kb
## Receiving objects: 11% (5/43), 8 kb
## Receiving objects: 23% (10/43), 8 kb
## Receiving objects: 32% (14/43), 8 kb
## Receiving objects: 41% (18/43), 8 kb
## Receiving objects: 51% (22/43), 8 kb
## Receiving objects: 62% (27/43), 8 kb
## Receiving objects: 72% (31/43), 8 kb
## Receiving objects: 81% (35/43), 8 kb
## Receiving objects: 93% (40/43), 8 kb
## Receiving objects: 100% (43/43), 8 kb, done.
## Local: master /tmp/RtmpojiQXs/redus-framework
## Remote: master @ origin (https://github.com/REDUS-IMR/docker-redus-pipeline.git)
## Head: [e99d35b] 2020-03-09: Put results in tmp to be picked by nginx path
2.3 Prepare an ID
Before preparing the assessment files,
you must write a unique ID for the framework run in a file named id.conf
in the redus-framework/projects
directory.
This ID is necessary for the several web services in the container to run.
2.4 Get the assessment bundle (code and data)
REDUS Framework works best with any ICES TAF -based assessment bundles. However, any assessment bundles from stockassessment.org will work too with some slight modifications.
REDUS Framework Github repository provides two example assessments for users to tryout:
- North East Artic Cod in 2018 (TAF based) [link]
- Norwegian Sea Herring in 2019 (Stockassessment.org based) [link]
Read more about converting your own assessment into REDUS Framework in this guide (TODO).
For this example let’s use the NEA Cod 2018 Assessment.
# Download from Github
download.file("https://github.com/REDUS-IMR/redus-framework/raw/master/extra/neacod-2018.tgz", "neacod-2018.tgz")
# See the content of the file
untar("neacod-2018.tgz", list = TRUE)
## [1] "neacod-2018/"
## [2] "neacod-2018/data.R"
## [3] "neacod-2018/output.R"
## [4] "neacod-2018/model.R"
## [5] "neacod-2018/README.md"
## [6] "neacod-2018/report.R"
## [7] "neacod-2018/bootstrap/"
## [8] "neacod-2018/bootstrap/initial/"
## [9] "neacod-2018/bootstrap/initial/data/"
## [10] "neacod-2018/bootstrap/initial/data/nm.dat"
## [11] "neacod-2018/bootstrap/initial/data/cw.dat"
## [12] "neacod-2018/bootstrap/initial/data/cn.dat"
## [13] "neacod-2018/bootstrap/initial/data/mo.dat"
## [14] "neacod-2018/bootstrap/initial/data/nmOldAdj.dat"
## [15] "neacod-2018/bootstrap/initial/data/pcc.zip"
## [16] "neacod-2018/bootstrap/initial/data/pm.dat"
## [17] "neacod-2018/bootstrap/initial/data/pf.dat"
## [18] "neacod-2018/bootstrap/initial/data/survey.dat"
## [19] "neacod-2018/bootstrap/initial/data/lf.dat"
## [20] "neacod-2018/bootstrap/initial/data/lw.dat"
## [21] "neacod-2018/bootstrap/initial/data/survey_var.dat"
## [22] "neacod-2018/bootstrap/initial/data/sw.dat"
## [23] "neacod-2018/bootstrap/initial/data/dw.dat"
## [24] "neacod-2018/bootstrap/initial/config/"
## [25] "neacod-2018/bootstrap/initial/config/model.cfg"
## [26] "neacod-2018/bootstrap/DATA.bib"
## [27] "neacod-2018/bootstrap/SOFTWARE.bib"
## [28] "neacod-2018/utilities.R"
# Unpack it
untar("neacod-2018.tgz")
# Remove the compressed file as we don't need it anymore
unlink("neacod-2018.tgz")
NOTE: It’s possible to have more than one assessment bundle. Just extract the another more assesssment bundles in the current
redus-framework/projects
directory.
2.5 Configure the assessment
2.5.1 Processing directives
All REDUS Framework configuration is stored in a configuration file redus.yaml
.
This file should be placed in the redus/
directory of the assessment bundle.
The configuration should start with these 3 lines:
default:
redustools.remote: "http://astarte.imr.no/ocpu/library/REDUStools/"
survey: "bootstrap/initial/data/survey.dat"
The redustools.remote
tells the framework the location of the REDUS Framework API
Server endpoint. And survey
tells the framework the survey data file that is used
in the assessment. Let’s build this using R:
# Create the default configuration
configuration <- list(default = list(
redustools.remote="http://astarte.imr.no/ocpu/library/REDUStools/",
survey = "bootstrap/initial/data/survey.dat"))
# Check the output
cat(yaml::as.yaml(configuration))
## default:
## redustools.remote: http://astarte.imr.no/ocpu/library/REDUStools/
## survey: bootstrap/initial/data/survey.dat
2.5.1.1 Survey data
To apply processing step to any survey indexes, we must examine the content
of the survey file (e.g., bootstrap/initial/data/survey.dat
and
note down the fleet names (e.g., NorBarTrSur
, NorBarLofAcSur
, etc.).
The processing mode can be either:
asis
: This means that you don’t want to modify the pre-defined data that comes with the assessment.remote
: Choose this to use the official result from the nightly StoX survey time series run. > NOTE: The official results are currently available here: http://astarte.imr.no/ocpu/library/REDUStools/www/imrsts.html . This was generated and served using the REDUStools R package available here: https://github.com/REDUS-IMR/REDUStools.build
: This option will force the survey time series re-processing using StoX/Rstox. For this you will need to fill-in the recipe for the StoX processing. The recipe is an XML document as. See here for more detailed information. Later you will need to save the recipe as an XML file (e.g.,NorBarTrSur.xml
). See below for an example.manual
: Here you can input manual values in the text format for the specified survey data.
Below are some examples to apply an extra processing to the NorBarTrSur
survey index data.
a. Re-Build the index using StoX (build mode)
We want to use StoX to re-process the “Barents Sea Northeast Arctic cod bottom trawl index in winter” survey time series from NMD, using the Recipe XML file available from here: https://git.imr.no/snippets/8.
# Create redus configuration directory
dir.create("neacod-2018/redus")
# Prepare XML recipe
download.file("https://git.imr.no/-/snippets/8/raw/master/snippetfile1.txt?inline=false",
"neacod-2018/redus/NorBarTrSur.xml")
# Prepare configuration (YAML)
## Important prefix
prefix <- "survey.update."
## The id of the survey configuration, (1 for the 1st, 2 for the 2nd, etc...)
iter <- 1
## The name of the target survey index
survey <- "NorBarTrSur"
## Process type (can be either 'asis', 'remote', 'build' or 'manual')
configuration[["default"]][[paste0(prefix, iter, '.mode')]] <- "build"
## REDUS Master recipe if mode is set to "build"
configuration[["default"]][[paste0(prefix, iter, '.surveyBuildConf')]] <- "./redus/NorBarTrSur.xml"
### Try to use Age information from the pre-defined data
configuration[["default"]][[paste0(prefix, iter, '.useSourceAge')]] <- "true"
### Try to use Year information from the pre-defined data
configuration[["default"]][[paste0(prefix, iter, '.useSourceYear')]] <- "true"
## Name of the header target in survey.dat file
configuration[["default"]][[paste0(prefix, iter, '.header')]] <- survey
## Leave these two below as empty, these are the manual data input
## in text mode (each for index and covar)
configuration[["default"]][[paste0(prefix, iter, '.data')]] <- ""
configuration[["default"]][[paste0(prefix, iter, '.data_var')]] <- ""
## We can leave below as empty as this is needed only for "remote" mode
configuration[["default"]][[paste0(prefix, iter, '.stssource')]] <- ""
configuration[["default"]][[paste0(prefix, iter, '.stsdate')]] <- ""
b. Replace an index using the official result (remote mode)
## The id of the survey configuration (2 = second)
iter <- 2
## The name of the target survey index
survey <- "NorBarLofAcSur"
## Process type (can be either 'asis', 'remote', 'build' or 'manual')
configuration[["default"]][[paste0(prefix, iter, '.mode')]] <- "remote"
## These are the necessary directive for the "remote" mode
### Survey time series name (from Dataset Explorer)
configuration[["default"]][[paste0(prefix, iter, '.stssource')]] <- "Barents Sea Northeast Arctic cod bottom trawl index in winter"
### Set a snapshot date if necessary
configuration[["default"]][[paste0(prefix, iter, '.stsdate')]] <- ""
### Try to use Age information from the pre-defined data
configuration[["default"]][[paste0(prefix, iter, '.useSourceAge')]] <- "true"
### Try to use Year information from the pre-defined data
configuration[["default"]][[paste0(prefix, iter, '.useSourceYear')]] <- "false"
## Name of the header target in survey.dat file
configuration[["default"]][[paste0(prefix, iter, '.header')]] <- survey
## Below is not used in the 'remote' mode
configuration[["default"]][[paste0(prefix, iter, '.surveyBuildConf')]] <- ""
## Leave these two below as empty, these are the manual data input
## in text mode (each for index and covar)
configuration[["default"]][[paste0(prefix, iter, '.data')]] <- ""
configuration[["default"]][[paste0(prefix, iter, '.data_var')]] <- ""
Write the configuration
You must write the configuration back to the assessment bundle.
2.5.1.2 Assessment program
To change the assessment program, you need to open the bootstrap/SOFTWARE.bib
file and change the source
value in the stockassessment
item:
# Read Software BIB
df <- bib2df::bib2df("neacod-2018/bootstrap/SOFTWARE.bib")
# Peek the pre-defined program
print(df[1, "SOURCE"])
## # A tibble: 1 × 1
## SOURCE
## <chr>
## 1 fishfollower/SAM/stockassessment@a295ef2
# Change assessment program to the latest SAM
df[1, "SOURCE"] <- "fishfollower/SAM/stockassessment@master"
# Write back the change
bib2df::df2bib(df, "neacod-2018/bootstrap/SOFTWARE.bib")
In addition to the above, if you are either using SAM or XSAM, it is necessary
to write use.SAM
or use.XSAM
in the REDUS configuration directory:
2.6 Build and run the REDUS Framework container
These are the final step needed to run the assessment. After all the files are ready we can proceed with building the Docker image and run the framework as a Docker container.
# Get back to 'redus-framework' directory
setwd("..")
# Select a name for the run
runname <- "testit"
# Select port for the web services
port <- 8000
# Build the image
Sys.setenv(RUN = runname, PORT = port)
system("docker build --pull --rm -t ${RUN} .")
# Run the image
system("docker run -p ${PORT}:8888 --name ${RUN} -dit ${RUN}")
2.7 Examine the run
As with the cloud version of the REDUS Framework, the run from the script provides the various Web-based tools, namely: 1. File explorer 2. Terminal 3. Logs 4. Results page
You can instruct R to open the above URLs using your web browser:
# Let's open the File Explorer
browseURL(paste0("http://localhost:", port, "/id/", ID, "/p/workspace/"))
# Let's open the Terminal
browseURL(paste0("http://localhost:", port, "/id/", ID, "/p/terminal"))
# Let's open the Logs
browseURL(paste0("http://localhost:", port, "/id/", ID, "/p/logs"))
# Let's open the Results page (only available when run is finish with success)
browseURL(paste0("http://localhost:", port, "/id/", ID, "/p/report"))
2.8 Stopping the run
Remember to stop the Docker container after you finished to ensure all the memory and disk space used are released:
2.9 (Advanced) Parsing the status of the run
For the brave soul who likes to tinker more with the program, you can utilize the status information to programmatically get the results and/or cleaning up the run.
You can use a more advanced scheduler such as taskscheduleR
package to automate
everything.
# Try to print the status
status <- jsonlite::fromJSON(paste0("http://localhost:", port, "/id/", ID, "/p/status"))
print(status)
# Show me the results ONLY when the run has successfully finished
status <- jsonlite::fromJSON(paste0("http://localhost:", port, "/id/", ID, "/p/status"))
# All statuses are 0 means that the pre-processing and the assessment runs
# have been finished without errors.
if(sum(status$redus) + sum(status$assessement) == 0) {
browseURL(paste0("http://localhost:", port, "/id/", ID, "/p/report"))
}