Input data repositories & Patches
< Git pull requests - feeding back | Overview | Data processing with magclass >
Level: ⚫⚫⚫⚫⚪ Advanced
Requirements
Local copy of the MAgPIE model (https://github.com/magpiemodel/magpie)
Have R installed (https://www.r-project.org/)
Have R package
gms
installedContent
Create a local input data repository.
Package a patch file.
Include a patch in the model configuration
Overview
Introduction
The input data for MAgPIE is prepared by a set of pre-processing routines that take the data from original sources (e.g. FAO, LPJmL…), execute additional calculations and convert it to the required MAgPIE parameter format (MADRaT). These pre-processing routines are not accessible as open source at the moment. A user is instead provided with a ready-made inputs that are necessary for the model execution.
The input files are setup in the config file config/default.cfg
,
usually at the beginning of the settings. Currently, the input data is
set as:
cfg$input <- c(regional = "rev4.101_h12_magpie.tgz",
cellular = "rev4.101_h12_fd712c0b_cellularmagpie_c200_MRI-ESM2-0-ssp370_lpjml-8e6c5eb1.tgz",
validation = "rev4.101_h12_validation.tgz",
additional = "additional_data_rev4.48.tgz",
calibration = "calibration_H12_per_ton_fao_may22_glo_13Mar24.tgz")
Once specified in the configuration as the input data, the data is automatically downloaded (if needed) when the model run is started.
The prepared input data are compressed tar archive files “.tgz
”, which
can be opened with software such as 7-Zip, or
in terminal by tar
and untar
commands. The data archive files
contain the following types of data:
- Cellular input data (e.g. land area, crop yields, water
requirements, carbon density):
cellular = "rev4.65_h12_1998ea10_cellularmagpie_c200_MRI-ESM2-0-ssp370_lpjml-8e6c5eb1.tgz"
- Regional input and validation data (e.g. food demand):
regional = "rev4.65_h12_magpie.tgz"
validation = "rev4.65_h12_validation.tgz"
- Calibration data:
calibration = "calibration_H12_sticky_feb18_free_30Nov21.tgz"
- Global and other input data (e.g. conversion factors, national
policies):
additional = "additional_data_rev4.07.tgz"
Patch input data
There is a specific procedure on how to handle the changing of the input data. It will be demonstrated by the example of changing the USA NDC policy on afforestation target at 2030.
Example: Change national land-based NDC policies
Once the input data is downloaded to the local MAgPIE repository in forms of different input files in designated input folders (in the core, scripts, modules and module realizations), a user can update or change these input data files. This can be done directly by manipulating the files, but this approach carries a risk that such changes are not documented and that in certain cases the made changes can be overwritten by the repeated download of the data from the declared repositories. In order to avoid this risk, it is recommended to create a local folder that serves as a repository for the patch files that will apply changes to the data by overwriting the original data.
Create a local data repository
The folder for local input data repository can be created anywhere and
it’s path must be provided to the settings in config/default.cfg
file.
Let us assume that the patch folder (patch_inputdata
) is created in
the main MAgPIE repository. One can do it in R:
dir.create("./patch_inputdata")
or in the command line:
mkdir patch_inputdata
Once the directory is created, provide its location to the configuration file. It’s important to keep the list structure of the repository information:
cfg$repositories <- append(list("https://rse.pik-potsdam.de/data/magpie/public"=NULL,
"./patch_inputdata"=NULL),
getOption("magpie_repos"))
Create a patch and package it
Create a sub-directory in the ./patch_inputdata
which is going to be
used for packaging of the patched files.
dir.create("./patch_inputdata/patch_ndc_usa")
Copy the original file the policy_definition.csv
in the patch folder.
In R:
file.copy(from="./scripts/npi_ndc/policies/policy_definitions.csv",
to="./patch_inputdata/patch_ndc_usa/.")
or in the command line:
cp scripts/npi_ndc/policies/policy_definitions.csv patch_inputdata/patch_ndc_usa/.
Edit the content, in this case update the USA afforestation NDC policy
(affore
) with a more ambitious target of 15 MHa of afforested area
starting in 2020 and reaching the target at 2030:
USA,affore,ndc,1,2020,2030,15
After saving the file, package it with the tardir function in R environment and delete the file patch folder:
gms::tardir(dir="patch_inputdata/patch_ndc_usa",
tarfile="patch_inputdata/patch_ndc_usa.tgz")
unlink("patch_inputdata/patch_ndc_usa", recursive=TRUE)
Add the patch file to the configuration
Finally, the configuration file should be informed about the change in
the input data and the existing patch file that replaces the existing
input data. For this, edit the config/default.cfg
file from:
cfg$input <- c(regional = "rev4.101_h12_magpie.tgz",
cellular = "rev4.101_h12_fd712c0b_cellularmagpie_c200_MRI-ESM2-0-ssp370_lpjml-8e6c5eb1.tgz",
validation = "rev4.101_h12_validation.tgz",
additional = "additional_data_rev4.48.tgz",
calibration = "calibration_H12_per_ton_fao_may22_glo_13Mar24.tgz")
to:
cfg$input <- c(regional = "rev4.101_h12_magpie.tgz",
cellular = "rev4.101_h12_fd712c0b_cellularmagpie_c200_MRI-ESM2-0-ssp370_lpjml-8e6c5eb1.tgz",
validation = "rev4.101_h12_validation.tgz",
additional = "additional_data_rev4.48.tgz",
calibration = "calibration_H12_per_ton_fao_may22_glo_13Mar24.tgz",
patch = "patch_ndc_usa.tgz")
It is very important to add the patch file at the end of the listings in
the cfg$input
listings, because every next .tgz
archive will
overwrite the files previously imported by the files that are contained
in it.
At the next start of the model by Rscript
, the new patch will place
the file with change inputs according to the changes in the settings.
Alternative way of adding a local repository
As an alternative to the steps described above, one can add a local
repository to getOption("magpie_repos")
. For this, one has to add an
entry in the .Rprofile
file. Typically .Rprofile
is located in the
users’ home directory (~/.Rprofile)
.
options(magpie_repos=list("~/input_data/"=NULL))
The local repository input_data
can be located anywhere on your
filesystem. You could then add patch_ndc_usa.tgz
in this
folder.
Hint: If you can’t locate your .Rprofile
, you can use
usethis::edit_r_profile()
.
cfg$repositories
in default.cfg
can be reverted back.
cfg$repositories <- append(list("https://rse.pik-potsdam.de/data/magpie/public"=NULL),
getOption("magpie_repos"))
With this setup, the download script (Rscript start.R -> 3 Download
data
) will first look into the public repo and secondly into your local
repo for downloading the files specified in cfg$input
.
Exercices
Write your own starting script that will test the scenario with changed
NDC policy for the USA described above. None of the changes should
occur in the default.cfg
, but instead the starting script
should introduce them to the loaded cfg object.
Add a MAgPIE start script here:
scripts/start/projects/name_of_your_script.R
Solution:
# | (C) 2008-2020 Potsdam Institute for Climate Impact Research (PIK)
# | authors, and contributors see CITATION.cff file. This file is part
# | of MAgPIE and licensed under AGPL-3.0-or-later. Under Section 7 of
# | AGPL-3.0, you are granted additional permissions described in the
# | MAgPIE License Exception, version 1.0 (see LICENSE file).
# | Contact: magpie@pik-potsdam.de
# ----------------------------------------------------------
# description: Test USA NDC
# ----------------------------------------------------------
######################################
#### Script to start a MAgPIE run ####
######################################
library(lucode2)
library(magclass)
library(gms)
# Load start_run(cfg) function which is needed to start MAgPIE runs
source("scripts/start_functions.R")
#start MAgPIE runs
source("config/default.cfg")
#cfg$force_download <- FALSE
cfg$results_folder <- "output/:title:"
cfg$output <- c("rds_report")
cfg$title <- "SSP2_NDC_default"
cfg <- gms::setScenario(cfg,c("SSP2","NDC"))
start_run(cfg,codeCheck=FALSE)
cfg$title <- "SSP2_NDC_USA"
cfg <- gms::setScenario(cfg,c("SSP2","NDC"))
cfg$input <- c(regional = "rev4.101_h12_magpie.tgz",
cellular = "rev4.101_h12_fd712c0b_cellularmagpie_c200_MRI-ESM2-0-ssp370_lpjml-8e6c5eb1.tgz",
validation = "rev4.101_h12_validation.tgz",
additional = "additional_data_rev4.48.tgz",
calibration = "calibration_H12_per_ton_fao_may22_glo_13Mar24.tgz",
patch = "patch_ndc_usa.tgz")
start_run(cfg,codeCheck=FALSE)
< Git pull requests - feeding back | Overview | Data processing with magclass >