3 REDUS Master Recipe
REDUS framework uses a special kind of recipe to process the IMR Survey Time Series from NMD’s Dataset Explorer. This recipe is formulated in an XML format and contains many user configurable options.
3.1 Examples
Below are some recipe samples:
3.1.1 1. Single Survey Time series
<?xml version="1.0" encoding="UTF-8"?>
<redus_master xmlns="http://www.imr.no/formats/redus/master/v0.1" revision="1" version="0.1">
<configuration>
<saveOutputTable>true</saveOutputTable>
<saveRunStatus>false</saveRunStatus>
</configuration>
<globalParameter>
<!-- <fileFix from="biotic_cruiseNumber_1994001_Anny+Kræmer.xml" to="biotic_cruiseNumber_1994001_Anny+Kræmer.xml"/> -->
</globalParameter>
<parameters sts="Barents Sea Northeast Arctic cod bottom trawl index in winter" revision="1" version="0.1">
<configuration>
<stsName>Barents Sea Northeast Arctic cod bottom trawl index in winter</stsName>
<overwriteNMD>false</overwriteNMD>
<forceReProcess>true</forceReProcess>
<forceBioticV3>true</forceBioticV3>
<skipYear>1993</skipYear>
<startYear></startYear>
<endYear></endYear>
<levelRequested>bootstrapImpute</levelRequested>
<bootstrapSeed>77</bootstrapSeed>
<bootstrapImputeSeed>101</bootstrapImputeSeed>
<bootstrapIter>5</bootstrapIter>
<coresUse>1</coresUse>
<groupType>age</groupType>
<minAge>1</minAge>
<maxAge>12</maxAge>
<plusAge>7</plusAge>
<numberScale>1000000</numberScale>
<dataType>survey</dataType>
<applyOverrides>false</applyOverrides>
</configuration>
<override>
<!-- <parameter name="UseProcessData">true</parameter> -->
</override>
</parameters>
</redus_master>
3.1.2 2. Multiple survey time series
<?xml version="1.0" encoding="UTF-8"?>
<redus_master xmlns="http://www.imr.no/formats/redus/master/v0.1" revision="1" version="0.1">
<configuration>
<saveOutputTable>true</saveOutputTable>
<saveRunStatus>false</saveRunStatus>
</configuration>
<globalParameter>
<!-- <fileFix from="biotic_cruiseNumber_1994001_Anny+Kræmer.xml" to="biotic_cruiseNumber_1994001_Anny+Kræmer.xml"/> -->
</globalParameter>
<parameters sts="Barents Sea Northeast Arctic cod bottom trawl index in winter" revision="1" version="0.1">
<configuration>
<stsName>Barents Sea Northeast Arctic cod bottom trawl index in winter</stsName>
<overwriteNMD>false</overwriteNMD>
<forceReProcess>true</forceReProcess>
<forceBioticV3>true</forceBioticV3>
<skipYear>1993</skipYear>
<startYear></startYear>
<endYear></endYear>
<levelRequested>bootstrapImpute</levelRequested>
<bootstrapSeed>77</bootstrapSeed>
<bootstrapImputeSeed>101</bootstrapImputeSeed>
<bootstrapIter>5</bootstrapIter>
<coresUse>1</coresUse>
<groupType>age</groupType>
<minAge>1</minAge>
<maxAge>12</maxAge>
<plusAge>7</plusAge>
<numberScale>1000000</numberScale>
<dataType>survey</dataType>
<applyOverrides>false</applyOverrides>
</configuration>
<override>
<!-- <parameter name="UseProcessData">true</parameter> -->
</override>
</parameters>
<parameters sts="Barents Sea capelin acoustic abundance estimate in autumn" revision="1" version="0.1">
<configuration>
<stsName>Barents Sea capelin acoustic abundance estimate in autumn</stsName>
<overwriteNMD>false</overwriteNMD>
<forceReProcess>true</forceReProcess>
<forceBioticV3>true</forceBioticV3>
<skipYear>2016,2019</skipYear>
<startYear>2016</startYear>
<endYear>2020</endYear>
<levelRequested>bootstrapImpute</levelRequested>
<bootstrapSeed>77</bootstrapSeed>
<bootstrapImputeSeed>101</bootstrapImputeSeed>
<bootstrapIter>5</bootstrapIter>
<coresUse>1</coresUse>
<groupType>age</groupType>
<minAge>1</minAge>
<maxAge>12</maxAge>
<plusAge>7</plusAge>
<numberScale>1000000</numberScale>
<dataType>survey</dataType>
<applyOverrides>false</applyOverrides>
</configuration>
<override>
<parameter name="UseProcessData">true</parameter>
</override>
</parameters>
</redus_master>
3.2 Usage
The recipe can be used in a stand-alone processing or included in the REDUS framework workflow. In brief, the processing of the recipe can be done by using the REDUStools
R package (https://github.com/REDUS-IMR/REDUStools).
install.packages("remotes")
remotes::install_github("REDUS-IMR/REDUStools")
remotes::install_github("SEA2DATA/Rstox", ref="develop")
# Output below can be the processing status or result table
# subject to the 'saveOutputTable' parameter in the XML recipe
output <- REDUStools::processRstoxSTS("barents_sea_neacod_trawl_winter.xml")
Please see another guide that will explain more on the above stand-alone proces here.
3.3 Parameter details
saveOutputTable
: Setting this node text totrue
tells the script to save the resulting table from the StoX process in anrds
object on disk. Many of the REDUS framework requires this to be set astrue
. Setting it tofalse
tellsREDUStools::processRstoxSTS()
function in the example usage above to return the result table.saveRunStatus
: Setting this node text totrue
tells the script to save the processing status of the survey time series in anrds
object on disk.fileFix
: This is useful if user want to override data input filename(s) from all the survey time seriesproject.xml
files. It will search for a matching string as in thefrom
attribute and replace it to the filename specified in theto
attribute. This parameter have a global scope, means it will be affecting all the survey time series in the recipe.parameters
: This node contains the unique identifier of the survey time series, please set a unique name for thests
attribute (usually the same name as the survey time series itself). Don’t forget to set therevision
andversion
parameters as well if applicable.stsName
: This node text indicates the name of the survey time series that will be processed. The names have to be exactly the same as the name found in NMD Dataset Explorer.overwriteNMD
: Setting this node text totrue
indicates that you want to overwrite the data that you might have downloaded earlier in your local disk.forceReProcess
: The script will always re-process the survey time series if set totrue
. Otherwise it will use the result from the previous process.forceBioticV3
: Some recent biotic data from NMD have been saved with NMD Biotic v3.1 format. Since RstoX can’t yet process the data, this switch sometimes necessary to force the header of Biotic v3.1 into Biotic V3. Note that the script won’t change the content of the data.skipYear
: Write the years that you want to skip from the processing and separate them using comma sign if you have more that one. Setting this to empty means that you don’t want to skip any years.startYear
: The start of year that you want to process. Setting this empty will process the earliest year possible.endYear
: The last year that you want to process. Setting this empty will process the survey time series until the last year possible.levelRequested
: The StoX output data comes from three possible levels:SuperIndAbundance
,bootstrap
, andbootstrapImpute
. UsuallybootstrapImpute
is the preferred level of output.bootstrapSeed
: Specify a seed value to be used in the bootstrap process.bootstrapImputeSeed
: Specify a seed value to be used in the impute process.bootstrapIter
: Specify the number of iteration for the bootstrap and impute process.coresUse
: Specify the number of cores that will be utilize for the StoX process.groupType
: Specify the groupings for the indices. The possible values areage
andLenGrp
.minAge
,maxAge
,plusAge
: The script will prepare the output table using these age parameters. This won’t affect processing, only the output.numberScale
: This value will be used as a multiplier for the output table. This won’t affect processing, only the output.dataType
: This is internal identifier for the REDUSframework. Leave it tosurvey
for survey time series.applyOverrides
: If set totrue
, the script will try to override some parameters in theproject.xml
files for all the year in a survey time series. Populate the<override><parameter>...</parameter></override>
sub-nodes with all overrides that you wish to be applied. For example<parameter name="UseProcessData">true</parameter>
will searchUseProcessData
nodes in aproject.xml
file and set their values totrue
. You can have multiple overrides for a survey time series.