Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sites/docs/src/content/docs/contributing/documentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ Before you start writing, familiarize yourself with these essential resources:

### Style guide

The [style guide](../developers/documentation/style_guide) covers all the essential styling rules for nf-core documentation, including:
The [style guide](../developers/documentation/style_guide.md) covers all the essential styling rules for nf-core documentation, including:

- Voice and tone guidelines for conversational, concise writing
- Grammar and punctuation rules (British English, active voice, Oxford comma)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,17 +9,19 @@ This page shows you how to configure pipelines to match your system's capabiliti

## Workflow resources

The base configuration of nf-core pipelines defines default resource allocations for each workflow step (e.g., in the [`base.config`](https://github.com/nf-core/rnaseq/blob/master/conf/base.config) file).
The base configuration of nf-core pipelines defines default resource allocations for each workflow step (for example, in the [`base.config`](https://github.com/nf-core/tools/blob/main/nf_core/pipeline-template/conf/base.config) file).

These default values are generous to accommodate diverse workloads across different users.
Your jobs might receive more resources than needed, which can reduce system efficiency.
You might also want to increase resources for specific tasks to maximise speed.
However, your jobs might receive more resources than needed, which can reduce system efficiency.
In contrast, You might also want to increase resources for specific tasks to maximise speed.
Consider increasing resources if a pipeline step fails with a `Command exit status` of `137`.

Pipelines configure tools to use available resources when possible (e.g., with `-p ${task.cpus}`), where `${task.cpus}` is dynamically set from the pipeline configuration.
:::note
Pipelines are coded to configure tools to use available resources when possible (e.g., with `-p ${task.cpus}`), where `${task.cpus}` is dynamically set from the pipeline configuration.
Not all tools support dynamic resource configuration.
:::

Most process resources use process labels, as shown in this base configuration example:
Most nf-core pipelines use process labels to define resource requirements for each module, as shown in this base configuration example:

```groovy
process {
Expand Down Expand Up @@ -48,15 +50,15 @@ process {

The `resourceLimits` list sets the absolute maximum resources any pipeline job can request (typically matching your machine's maximum available resources).
The label blocks define the initial default resources each pipeline job requests.
When a job runs out of memory, most nf-core pipelines retry the job and increase the resource request up to the `resourceLimits` maximum.
When a job runs out of memory, most nf-core pipelines will attempt to retry the job and increase the resource request up to the `resourceLimits` maximum.

### Customize process resources

:::tip
Copy only the labels you want to change into your custom configuration file, not all labels.
:::

To set a fixed memory allocation for all large tasks across most nf-core pipelines (without increases during retries), add this to your custom configuration file:
To set a fixed memory allocation for all large tasks across most nf-core pipelines (without increases during retries), add this to a custom Nextflow configuration file:

```groovy
process {
Expand All @@ -66,6 +68,10 @@ process {
}
```

:::tip
To find the default labels and resources of the pipeline you want to optimise, go to its GitHub repository and look at the file `conf/base.config`.
:::

You can target a specific process (job) name instead of a label using `withName`.
Find process names in your console log when the pipeline runs.
For example:
Expand All @@ -88,13 +94,9 @@ process {
}
```

:::info
If you receive a warning about an unrecognised process selector, check that you specified the process name correctly.
:::

For more information, see the [Nextflow documentation](https://www.nextflow.io/docs/latest/config.html#process-selectors).

After writing your [configuration file](#custom-configuration-files), supply it to your pipeline command with `-c`.
After writing your [configuration file](#custom-configuration-files), supply it to your pipeline command with `-c <path>/<to>/<config>.conf`.

:::warning
Check your syntax carefully.
Expand All @@ -105,11 +107,16 @@ Use quotes with a space or no quotes with a dot: `"200 GB"` or `200.GB`.
See the Nextflow documentation for [memory](https://www.nextflow.io/docs/latest/process.html#memory), [cpus](https://www.nextflow.io/docs/latest/process.html#cpus), and [time](https://www.nextflow.io/docs/latest/process.html#time).
:::

:::info
If you receive a warning when about an unrecognised process selector when running a pipeline, check that you specified the process name correctly.
:::

If the pipeline defaults need adjustment, contact the pipeline developers on Slack in the pipeline channel or submit a GitHub issue on the pipeline repository.

## Change your executor

Nextflow pipelines run in local mode by default, executing jobs on the same system where Nextflow runs.
Nextflow pipelines run in 'local' mode by default, executing jobs on the same system where Nextflow runs and assuming all tools the pipeline need are already on the machine's environment `$PATH`.

Most users need to specify an executor to tell Nextflow how to submit jobs to a job scheduler (e.g., SGE, LSF, Slurm, PBS, or AWS Batch).

You can configure the executor in shared configuration profiles or in custom configuration files.
Expand All @@ -135,12 +142,12 @@ process {
```

When a job exceeds the default memory request, Nextflow retries the job with increased memory.
The memory increases with each retry until the job completes or reaches the `256.GB` limit.

These parameters cap resource requests to prevent Nextflow from submitting jobs that exceed your system's capabilities.
The memory increases with each retry until the job completes or reaches one of the limits, such as `256.GB` for memory.

:::warning
Specifying resource limits does not increase the resources available to pipeline tasks.
See [Tuning workflow resources](#tuning-workflow-resources) for more information.
:::

:::note{collapse title="Note on older nf-core pipelines"}

Expand All @@ -166,8 +173,10 @@ The `--max_<resource>` parameters represent the maximum for a single pipeline jo

## Customize Docker registries

Most pipelines use `quay.io` as the default Docker registry for Docker and Podman images.
When you specify a Docker container without a full URI, Nextflow pulls the image from `quay.io`.
Most nf-core pipelines use `quay.io` as the default Docker registry for Docker and Podman images.
In some cases, you may want to customise where a pipeline sources their images.

By default, when you specify a Docker container without a full URI, Nextflow pulls the image from `quay.io`.

For example, this container specification:

Expand All @@ -177,15 +186,11 @@ Pulls from `quay.io`, resulting in the full URI:

- `quay.io/biocontainers/fastqc:0.11.7--4`

If you specify a different `docker.registry` value, Nextflow uses that registry instead.
If you specify a different `docker.registry` value in a configuration file, Nextflow uses that registry instead.
For example, if you set `docker.registry = 'myregistry.com'`, the image pulls from:

- `myregistry.com/biocontainers/fastqc:0.11.7--4`

When you specify a full URI in the container specification, Nextflow ignores the `docker.registry` setting and pulls exactly as specified:

- `docker.io/biocontainers/fastqc:v0.11.9_cv8`

## Update tool versions

The [Nextflow DSL2](https://www.nextflow.io/docs/latest/dsl2.html) implementation of nf-core pipelines uses one container or Conda environment per process, which simplifies software dependency maintenance and updates.
Expand Down Expand Up @@ -232,13 +237,12 @@ You can override the default container by creating a custom configuration file a
```

:::warning
Pipeline developers provide no warranty when you update containers.
Major changes in the container tool may break the pipeline.
Note that when you specify a full URI in the container specification, Nextflow ignores the `docker.registry` setting and pulls exactly as specified.
:::

:::warning
Tool developers sometimes change version reporting between updates.
Container updates may break version reporting within the pipeline and create missing values in MultiQC version tables.
Pipeline developers provide no warranty when you update containers.
Major changes in the container tool may break the pipeline or see a degradation of output (e.g. missing versions in MultiQC reports when the tool changes how versions are reported).
:::

## Modifying tool arguments
Expand Down
26 changes: 18 additions & 8 deletions sites/docs/src/content/docs/running/configuration/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,14 @@ For pipeline-specific parameters, see the pipeline documentation.

## Configuration options

You can configure pipelines using three approaches:
You can configure pipelines to your infrastructure using three approaches:

1. [Default pipeline configuration profiles](#default-configuration-profiles)
2. [Shared nf-core/configs configuration profiles](#shared-nf-coreconfigs)
3. [Custom configuration files](#custom-configuration-files)
- [Configuration options](#configuration-options)
- [Choosing your configuration approach](#choosing-your-configuration-approach)
- [Default configuration profiles](#default-configuration-profiles)
- [Shared nf-core/configs](#shared-nf-coreconfigs)
- [Custom configuration files](#custom-configuration-files)
- [Additional resources](#additional-resources)

:::warning{title="Do not edit the pipeline code to configure nf-core pipelines"}
Editing pipeline defaults prevents you from updating to newer versions without overwriting your changes.
Expand All @@ -48,7 +51,7 @@ Use shared nf-core/configs when:

Use custom configuration files when:

- You need specific resource limits
- You need pipeline-specific resource limits
- Running on unique infrastructure
- You are the only user of the pipeline

Expand All @@ -65,6 +68,7 @@ Order matters.
Profiles load in sequence.
Later profiles overwrite earlier ones.
:::

nf-core provides these basic profiles for container engines:

- `docker`: Uses [Docker](http://docker.com/) and pulls software from quay.io
Expand All @@ -78,7 +82,7 @@ nf-core provides these basic profiles for container engines:
Use Conda only as a last resort (that is, when you cannot run the pipeline with Docker or Singularity).
:::

Without a specified profile, the pipeline runs locally and expects all software to be installed and available on the `PATH`.
Without a specified profile, the pipeline runs locally and expects all software to be installed and available on the `$PATH`.
This approach is not recommended.

Each pipeline includes `test` and `test_full` profiles.
Expand All @@ -97,9 +101,11 @@ If not, follow the repository instructions or the tutorial to add your cluster.

### Custom configuration files

If you run the pipeline alone, create a local configuration file.
If you run the pipeline alone on a local machine, create a local configuration file.
Nextflow searches for configuration files in three locations:

<!-- TODO this is not just local machine though, also works for HPC... ? -->

1. User's home directory: `~/.nextflow/config`
2. Analysis working directory: `nextflow.config`
3. Custom path on the command line: `-c path/to/config` (you can specify multiple files)
Expand All @@ -113,11 +119,15 @@ The loading order is:
4. Each `-c` file in the order you specify
5. Command line parameters (`--<parameter>`)

:::warning
<!-- TODO I think this _maybe_ should go before the explanation of how to customise with local configuration files, as this is the default behaviour BEFORe you start customising so the logic will maybe be more familiar. Relatd, maybe we should mention that in nf-core pipelines we have `base.config` as this is where we set our _Resource_ defaults -->

:::warnings
Parameters in `custom.config` files will not override defaults in `nextflow.config`.
Use `-params-file` with YAML or JSON format instead.
:::

<!-- TODO I'm pretty sure `custom.config` WILL overwrite `nextflow.config`, particularly given my comment above about `nextflow.config` being teh pipeline defaults -->

:::tip
Generate a parameters file using the **Launch** button on the [nf-co.re website](https://nf-co.re/launch).
:::
Expand Down
Loading
Loading