Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 31 additions & 34 deletions _episodes/03-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ keypoints:

## The configuration file

For the purposes of this tutorial, we will create a directory in our home directory
First, for the purposes of this tutorial, we will create a directory in our home directory
called `esmvaltool_tutorial` and use that as our working directory. The following steps
should do that:

Expand All @@ -38,26 +38,28 @@ This is a [YAML file](https://yaml.org/spec/1.2/spec.html).
You can get the default configuration file by running:

~~~bash
esmvaltool config get_config_user --path=<target_dir>
esmvaltool config get_config_user
~~~
The default configuration file will be downloaded to the directory specified with
the `--path` variable. For instance, you can provide the path to your working directory
as the `target_dir`. If this option is not used, the file will be saved to the default
location: `~/.esmvaltool/config-user.yml`, where `~` is the
The default configuration file will be downloaded to the default location:
`~/.config/esmvaltool/config-user.yml`, where `~` is the
path to your home directory. Note that files and directories starting with a
period are "hidden", to see the `.esmvaltool` directory in the terminal use
`ls -la ~`. Note that if a configuration file by that name already exists in the default
period are "hidden", to see the `.config` directory in the terminal use
`ls -la ~`.
With the optional ``--path=<target_dir>`` you could specifiy the directory
in which the configuration file can be saved. For instance, you can provide
the path to your working directory as the `target_dir`.
Note, if a configuration file by that name already exists in the default
location, the `get_config_user` command will not update the file as ESMValTool will not
overwrite the file. You will have to move the file first if you want an updated copy of the
user configuration file.
default user configuration file.



We run a text editor called ``nano`` to have a look inside the configuration file
and then modify it if needed:

~~~bash
nano ~/.esmvaltool/config-user.yml
nano ~/.config/esmvaltool/config-user.yml
~~~

Any other editor can be used, e.g.vim.
Expand Down Expand Up @@ -120,7 +122,7 @@ using the format: YYYYMMDD_HHMMSS.

> ## Set the destination directory
>
> Let's name our destination directory ``esmvaltool_output`` in the working directory.
> Let's name our destination directory ``esmvaltool_output`` in the current directory.
> ESMValTool should write the output to this path, so make sure you have the disk space
> to write output to this directory.
> How do we set this in the `config-user.yml`?
Expand Down Expand Up @@ -172,7 +174,7 @@ rootpath:
default: ~/climate_data
```
These are typically available in the default configuration file you downloaded, so simply
removing the machine specific lines should be sufficient to access input data.
uncommenting the machine specific lines should be sufficient to access input data.

> ## Set the correct rootpath
>
Expand Down Expand Up @@ -236,15 +238,13 @@ quickstart/find_data.html).
## Directory structure for the data from different projects

Input data can be from various models, observations and reanalysis data that
adhere to the [CF/CMOR standard](https://cmor.llnl.gov/). The ``drs`` setting
describes the file structure.

adhere to the [CF/CMOR standard](https://cmor.llnl.gov/).
The ``drs`` setting describes the file structure for several projects (e.g.
CMIP6, CMIP5, obs4mips, OBS6, OBS) on several key machines
(e.g. BADC, CP4CDS, DKRZ, ETHZ, SMHI, BSC). For more
information about ``drs``, you can visit the ESMValTool documentation on
[Data Reference Syntax (DRS)](https://docs.esmvaltool.org/projects/esmvalcore/
en/latest/quickstart/find_data.html#cmor-drs).
[Data Reference Syntax (DRS)](https://docs.esmvaltool.org/projects/ESMValCore/
en/latest/quickstart/find_data.html#explaining-drs-cmip5-or-drs-cmip6).

> ## Set the correct drs
>
Expand Down Expand Up @@ -293,24 +293,15 @@ en/latest/quickstart/find_data.html#cmor-drs).

> ## Explain the default drs (if working on local machine)
>
> 1. In the previous exercise, we set the `drs` of CMIP5 data to `default`.
> In the previous exercise, we set the `drs` of CMIP5 data to `default`.
> Can you explain why?
> 2. Have a look at the directory structure of the `OBS` data.
> There is a folder called `Tier1`. What does it mean?
>
>> ## Solution
>>
>> 1. `drs: default` is one way to retrieve data from a ROOT directory that has
>> `drs: default` is one way to retrieve data from a ROOT directory that has
>> no DRS-like structure. ``default`` indicates that all the files are in a
>> folder without any structure.
>>
>> 2. Observational data are organized in Tiers depending on their level of
>> public availability. Therefore the default directory must be structured
>> accordingly with sub-directories `TierX` e.g. Tier1, Tier2 or Tier3, even
>> when `drs: default`. More details can be found in the
[documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/
quickstart/find_data.html#observational-data).
>>
> {: .solution}
{: .challenge}

Expand All @@ -329,8 +320,8 @@ if you want to feed some additional data (e.g. shape files) to your recipe.
> auxiliary_data_dir: ~/auxiliary_data
> ```
> See more information in ESMValTool
[document](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart
/configure.html?highlight=auxiliary_data#user-configuration-file).
[documentation](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/
quickstart/configure.html?highlight=auxiliary_data#top-level-configuration-options).
{: .callout}

> ## Number of parallel tasks
Expand All @@ -353,12 +344,18 @@ amount of memory available in your system.

> ## Make your own configuration file
>
> It is possible to have several configuration files with different purposes,
> for example: config-user_formalised_runs.yml, config-user_debugging.yml.
> In this case, you have to pass the path of your own configuration file
> as a command-line option when running the ESMValTool.
> Configuration files could live in the user configuration directory, which is
> by default ``~/.config/esmvaltool``. The directory could be also specified
> via the command line argument ``--config_dir``.
> We will learn how to do this in the
> [next lesson]({{ page.root }}{% link _episodes/04-recipe.md %}).
>
> It is possible to have several configuration files with different purposes,
> for example: config-user_formalised_runs.yml, config-user_debugging.yml.
> In this case, ESMValTool searches for all YAML files within each of the
> configuration directories and merges them together. How this is done is
> exlained [here](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/
> quickstart/configure.html#yaml-files).
{: .callout}

{% include links.md %}
4 changes: 2 additions & 2 deletions _episodes/04-recipe.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ esmvaltool run examples/recipe_python.yml
or if you have the user configuration file in your current directory then

```
esmvaltool run --config_file ./config-user.yml examples/recipe_python.yml
esmvaltool run --config_dir . examples/recipe_python.yml
```

If everything is okay, you should see that ESMValTool is printing a lot of
Expand Down Expand Up @@ -80,7 +80,7 @@ Let's dissect what's happening here.
> > ## Answers
> >
> > 1. The config file should be the one we edited in the previous episode,
> > something like `/home/<username>/.esmvaltool/config-user.yml` or
> > something like `/home/<username>/.config/esmvaltool/config-user.yml` or
`~/esmvaltool_tutorial/config-user.yml`.
> > 1. ESMValTool found the recipe in its installation directory,
>> something like
Expand Down