< Creating a start script | Overview | Spatial output analysis >

Level: ⚫⚫⚫⚪⚪ Advanced

Requirements

Content

  • Use model-internal R-scripts for output analysis.

  • Know where to find the automated validation PDF and how it is structured.

  • Use the evaluation tool appResultsLocal of the library shinyresults.

  • Use the magpie4 package for output analysis.

  • Analyse outputs with the gdx package.

Overview

Introduction

After having successfully started and accomplished a simulation run, the next step is to evaluate the simulation results. In case you have not yet conducted an own MAgPIE simulation or your simulation is still running, you can download model runs produced for the purpose of this tutorial (https://zenodo.org/records/10949419). These runs were created with version 4.7.2 of the MAgPIE model (https://github.com/magpiemodel/magpie/tree/master). You can download these runs as tgz folder from Zenodo (choose the folder magpie24.zip) and copy the folders containing the simulation results into the output folder of your local version of the MAgPIE model (Note: if you freshly cloned the model and did not set up a run yet, you will have to create an empty output folder first).

There are several ways to assess and evaluate MAgPIE results. This tutorial gives an overview on different tools and options that can be used to analyse model outputs.

For each simulation, results are written into a subfolder of the output folder. This subfolder is created automatically. Its name is a combination of the model title and date. This is defined in the default.cfg (cfg$results_folder <- "output/:title::date:").

Model-internal R-scripts for output analysis

Manual execution of model-internal R-scripts

There are several output scripts that can be executed after the model run has finished. These R scripts can be found in the folder scripts/output. They can be selected and executed via the command window. To do so, windows users can open a command line prompt in the MAgPIE model folder by using shift + rightClick and then selecting open command window here option.

In the command prompt, use the following command:

Rscript output.R

You are now asked to choose the simulation run for which you would like to execute an output script.

Selection of model runs

Next, you can choose which output scripts you want to execute:

Selection of model-internal output scripts

The last step is to select the run submission type, e.g. “Direct execution”:

Selection of run submission type

Now, the selected script will be executed. After completion, the results are written in the respective folder of the simulation run inside the output folder of the model.

Model-internal R-script selection in the config file

In the file config/default.cfg, it is possible to specify which of these output R scripts are executed after a model run is finished. In the default MAgPIE configuration, the scripts output_check rds_report (to be used in appResultsLocal; see below), and extra/disaggregation are selected via cfg$output:

cfg$output <- c("output_check", "extra/disaggregation", "rds_report")

More output scripts (e.g. “validation” or “validation_short”) can be added to this vector if you want to run them for each and every of your runs.

Automated model validation

The automated model validation is an example of output analysis based on model-internal scripts (see above). If the validation script is executed (either by selection via cfg$output as explained above or by execution via the command window as explained above, a standard evaluation PDF is created that validates numerous model outputs with a validation database containing historical data and projections for most outputs returned by the model, either visually or via statistical tests. A evaluation PDF consists of hundreds of evaluation outputs and usually has a length of around 1800 (short version) to 3600 (detailed version) pages. By evaluating the model outputs on such a broad level rather than focusing only on key outputs, it allows getting a more complete picture of the corresponding simulation. As an example of such validation files, you can download the evaluation documents produced for all runs shown in the MAgPIE 4 framework paper (https://doi.org/10.5281/zenodo.1485303) or run the “validation” or “validation_short” output scripts as explained above. The validation PDF is also included in the runs that you may have downloaded above.

The table of contents of the validation PDF gives a good overview over the model outputs that can be simulated with a MAgPIE standard simulation, even though the validation PDF only shows a subset of possible model outputs:

Table of contents of the validation pdf

Interactive scenario analysis

The automated model validation PDF is a good tool for visually evaluating a broad range of model outputs. However, comparison between model runs, i.e. between different scenarios, is rather difficult and inconvenient with the different model results being scattered across different large PDF files.

To overcome this issue, we developed the interactive scenario analysis and evaluation tools appResultsLocal (and appResults for the use within the PIK network) as part of the library shinyresults (https://github.com/pik-piam/shinyresults). It facilitates the evaluation of results with plots for multiple scenarios including historical data and other projections based on an interactive selection of regions and variables. You can use this tool by running the following R command in the main folder of your model, which will automatically collect all runs in the output folder and visualize them:

shinyresults::appResultsLocal()

This command opens an interactive window, where you can select the simulations that you want to evaluate.

Interactive MAgPIE app

You can use filters to select a subset of all runs stored in the output folder of the model, for example by searching for runs that have been finished at a certain day, have been created by a certain user or by searching for keywords in the title of the simulation runs:

How to use the title for filtering runs

After having selected the subset of runs that you want to analyse, click the button Load selection. Now, you can click on the tab LinePLot. You will then see on the right hand side line plots showing the development of population for historical and future time steps for all model regions and on the global scale:

Regional and global development of population

Now, choose a variable of your interest, either by scrolling through the drop-down menu or write a key word in the input field, e.g. “cropland”, to reduce the options in the menu.

Selection of variables for line plots

Make yourself familiar with the features of the app! You can, for example, select a subset of regions or a subset of time steps for which the results should be plotted. Moreover, you can free the y-axis, include or exclude validation data (if available) and download the plot.

Analysis of outputs with the magpie4 library

If you want to go beyond visual output analysis and predefined output evaluation facilitated by scripts, you can use the functionality of the R package magpie4 (https://github.com/pik-piam/magpie4). This library contains a list of functions for extracting outputs from the MAgPIE model, which are also the basis for the generation of the automated validation PDF. For a quick overview of the functions included in the library, you can have a look in the folder magpie4/R. The following figure shows a subset of R-files included in magpie4/R:

Subset of R functions of the magpie4 library

To make yourself familiar with this library, you can open an R/RStudio session and set the MAgPIE model folder as working directory. This can be done by using the following command:

setwd("/path/to/your/magpie/model/folder")

Then, load the library and call the help pages:

library(magpie4)
?magpie4

You can click on the index and search for interesting functions, e.g. carbonstock, and read the respective help page:

Help page of the carbonstock-function of the magpie4 library

Analyzing outputs with the gdx library

The gdx library (https://github.com/pik-piam/gdx) allows to directly access objects contained in the fulldata.gdx file via the function readGDX. A pragmatic way to learn how to use this function for the extraction of interesting information from the fulldata.gdx is to open R files of the magpie4 library within Rstudio. Most of the magpie4 functions make use of readGDX.

In the function magpie4::carbonstock(), we see several instances where gdx::readGDX is used, e.g.:

a <- readGDX(gdx, "ov_carbon_stock", select = list(type = "level"), react = "silent")
p32_carbon_density_ac <- readGDX(gdx, "p32_carbon_density_ac", react = "silent")
top <- readGDX(gdx, "f59_topsoilc_density")[, getYears(cshare), ]

It is possible to extract various GAMS objects like “sets”, “equations”, “parameters”, “variables” and “aliases” with readGDX.

With the argument select = list(type = “level”) you can select the level values of endogenous variables; With select = list(type = “marginal”) you can extract the marginal values of these variables.


Exercises (click on the arrows to uncover the solution)

Open a validation pdf (either in a folder containing your own simulation results or the downloaded MAgPIE simulation runs and (a) make yourself familiar with the structure of the document and the hierarchy of outputs as displayed by the table of contents and; (b) have a look at some figures displaying model outputs of your interest.
Execute the model-internal output script `rds report` via the command window. This script collects the results of several report-functions - that calculate many key output variables like Production, Land Use or Yields - and writes them into an rds file.
  1. open a command window in the main MAgPIE model folder
  2. select model run by typing in the respective number and press ENTER
  3. select rds report by pressing 2 and ENTER
  4. select Direct execution by pressing 3 and ENTER
Start the interactive scenario analysis tool using the shinyresults R library, select one particular model run using the title filter and look at a variable of your choice.
  1. Open an R session in the main folder structure of the MAgPIE model folder
  2. Execute the command shinyresults::appResultsLocal()
  3. After the interactive site has opened, select title as a filter in the Select Data tab and look for the simulation run of your choice
  4. Click on Load selection and open the LinePlot1 or AreaPlot1 tab
  5. Select a variable of your choice and optionally also a particular scenario or region.
Using the interactive scenario analysis tool using the shinyresults R library, look at a line plot and an area plot of land cover change.
  1. Click on the Line Plot tab 2. Select (one after another) the indicators Resources - Land Cover Change - Cropland, Resources - Land Cover Change - Pastures and Rangelands and compare the indicators for the given scenarios across the different regions 3. Click on the Area Plot tab 4. Select the indicator Resources - Land Cover Change and only one region to compare how different land cover types develop over time in the different scenarios.
Find out how to include a custom validation file in the interactive scenario analysis tool. (This can be useful when you have additional or newer validation data at hand)
  1. Open the documentation to the function shinyresults::appResultsLocal()
  2. Specify a particular validation file (.rds or .mif) when executing the command, e.g.: appResultsLocal(valfile = output/default_2024-04-03_14.47.55/validation.mif). Note that this file has to be located in the main folder or you need to provide the full path to its location.
Apply the function carbonstock() from the R library magpie4 using the file “fulldata.gdx” in the folder of a model simulation run as input for the function. (a) Use the default settings of the arguments of the function (b) Change the arguments of the function, e.g. change the regional resolution from cellular (cluster level) to regional.
  1. Open an R session or R Studio
  2. Set the working directory to the main MAgPIE model folder (e.g. by copying the path of the folder with RIGHT CLICK + COPY and executing the command setwd(readClipboard())
  3. Assign the gdx file: gdx <- ''fulldata.gdx''
  4. solution to (a) cellularCarbon <- carbonstock(gdx)
  5. solution to (b) regionalCarbon <- carbonstock(gdx = gdx, level = ''reg'').
Use the gdx R library to extract the level and marginal values of the MAgPIE variable ov43watavail.
  1. Open R and load the gdx library (library(gdx))
  2. Assign the gdx file gdx <- ''fulldata.gdx''
  3. Execute w <- readGDX(gdx, ''ov43_watavail'', select = list(type = c(''level'', ''marginal'')))

< Creating a start script | Overview | Spatial output analysis >