Skip to content

Commit b9b0df2

Browse files
boulundKristapsAroArzAron Arzoomand
authored
Merge 0.4.1 (#155)
* Update version to 0.4.1-dev * Update CircleCI badge in README to monitor develop branch * Add Singularity deffile for stag-mwc env * Add Singularity image for main env up to kraken2 * Add link to Singularity bind docs * Add stag-mwc-main singularity image to all rules that require it * Add Singularity images for biobakery and assembly * Set rule threads from cluster_config if defined * Update CHANGELOG * Docs: Update module intro paragraph, closes #151 * Add {cluster.extra} to slurm-submit.py script call * Update CHANGELOG * Fix typo in kraken2 rules * the host removal script was ignoring it's flag in config.yaml * Update CHANGELOG * Update CHANGELOG * Tweak docs, update changelog, prepare for release * Amrplusplus (#156) * amrplusplus * amrplusplus scripts * added scripts * changed input in config.yaml Co-authored-by: Fredrik Boulund <[email protected]> * default db empty Co-authored-by: Fredrik Boulund <[email protected]> * db -> megares Co-authored-by: Fredrik Boulund <[email protected]> * annotation --> megares_annotation Co-authored-by: Fredrik Boulund <[email protected]> * added amrplusplus to circleci * updated CHANGELOG.md * updated amrplusplus in config.yaml * added amrplusplus to modules.rst * updated amrplusplus.smk * added license and removed bin dir * updated modules.rst * added align_to_amr * fixed default database * amrplusplus db empty in conf * indentations * clarification regarding database * Update docs/source/modules.rst Co-authored-by: Fredrik Boulund <[email protected]> * Update rules/antibiotic_resistance/amrplusplus.smk Co-authored-by: Fredrik Boulund <[email protected]> * Update docs/source/modules.rst grammar Co-authored-by: Fredrik Boulund <[email protected]> * Update CHANGELOG.md Co-authored-by: Fredrik Boulund <[email protected]> * added bwa to stag-mwc.yaml * added conda environment * amrplusplus exectuable with conda * amrplusplus executable with conda * samtools version 1.10 * amrplusplus conda executable * added amrplusplus.yaml to not break stag-mwc.yaml dependencies * removed amrplusplus dependencies because they broke it * now using amrplusplus.yaml for conda instead of stag-mwc.yaml Co-authored-by: Fredrik Boulund <[email protected]> Co-authored-by: Aron Arzoomand <[email protected]> * Change report generation call (#161) * Change report generation call Ignore additional arguments to snakemake when creating report, Intended to avoid shell escaping issue with complex arguments to e.g. Singularity, issue #160 * removed snakemake_call * added report generation workaround * report generation Co-authored-by: Fredrik Boulund <[email protected]> * report generation Co-authored-by: Fredrik Boulund <[email protected]> * indentation Co-authored-by: Aron Arzoomand <[email protected]> * Slurm profile for CTMR UCP environment (#159) * First draft for CTMR UCP cluster profile * Remove references to rackham profile, remove account check * Updated resource requests for all rules * Final tweaks * Rename ctmr_ucp profile to ctmr_gandalf * Fix yml-yaml typo * Fix incorrectly named singularity images * Remove --use prefix for argument to metawrap * Add missing singularity container to create_kaiju_krona_plot * added dbdir and threads for align_to_amr * Disable area plot for Kaiju * Minor updates to docs (#162) * Update badge URL in README Co-authored-by: Kristaps <[email protected]> Co-authored-by: Aron Arzoomand <[email protected]> Co-authored-by: Aron Arzoomand <[email protected]> Co-authored-by: AroArz <[email protected]>
1 parent 0c46f88 commit b9b0df2

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+1123
-77
lines changed

‎.circleci/config.yml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,8 @@ jobs:
7171
sed -i 's/kraken2: False/kraken2: True/' config.yaml
7272
sed -i 's/metaphlan2: False/metaphlan2: True/' config.yaml
7373
sed -i 's/humann2: False/humann2: True/' config.yaml
74-
sed -i 's/antibiotic_resistance: False/antibiotic_resistance: True/' config.yaml
74+
sed -i 's/groot: False/groot: True/' config.yaml
75+
sed -i 's/amrplusplus: False/amrplusplus: True/' config.yaml
7576
sed -i 's/assembly: False/assembly: True/' config.yaml
7677
sed -i 's/binning: False/binning: True/' config.yaml
7778
sed -i 's|db_path: \"\"|db_path: \"db/hg19\"|' config.yaml

‎CHANGELOG.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,36 @@ files), and the patch version is typically incremented for any set of changes
1313
committed to the master branch that does not trigger any of the aforementioned
1414
situations.
1515

16+
## [0.4.1] Unreleased
17+
### Added
18+
- Created Singularity images for all conda environments. Run with
19+
`--use-singularity` (do not combine with `--use-conda`).
20+
- New cluster profile "pseudo-rules" for anonymous rules for mappers: `bbmap`
21+
and `bowtie2` can now accept threads from `n` in the cluster profile. They
22+
still use the time allocation for the `__default__` rule, however.
23+
- Added possibility to use `extra:` to define additional arguments passed on to
24+
Slurm submissions. Useful to request e.g. fat nodes with `extra: "-C fat"`
25+
- Added custom reimplementation of AMRPlusPlus v2.0 which can be executed with
26+
either `--use-singularity` or `--use-conda`.
27+
28+
### Fixed
29+
- The host removal module now correctly identifies setting `host_removal: False`
30+
in the config file. Thank you chrsb!
31+
32+
### Changed
33+
- Do not combine `--use-singularity` with `--use-conda` anymore. The new
34+
Singularity images already contain all dependencies.
35+
- All rules now define the number of threads from cluster_config if defined.
36+
Old defaults are still used for local execution.
37+
- The shebang of `area_plot.py` has been changed to work in more environments.
38+
- Implemented workaround for error caused by automatic report generation when
39+
using Singularity.
40+
- Disabled taxonomic area plot for Kaiju outputs due to issues processing the
41+
output files.
42+
43+
### Removed
44+
45+
1646
## [0.4.0] 2020-02-18
1747
### Added
1848
- Added resource limiter for HUMAnN2 due to its intense use of huge temporary

‎README.md‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,11 @@ Go to https://stag-mwc.readthedocs.org for the full documentation.
2020
[Snakemake](https://snakemake.readthedocs.io) are required to be able to use
2121
StaG-mwc. Most people would probably want to install
2222
[Miniconda](https://conda.io/miniconda.html) and install Snakemake into their
23-
base environment. Conda will automatically install the required versions of
24-
all tools required to run StaG-mwc.
23+
base environment. When running StaG with the `--use-conda` or
24+
`--use-singularity` flags, all dependencies are managed automatically. If
25+
using conda it will automatically install the required versions of all tools
26+
required to run StaG-mwc. There is no need to combine the flags: the
27+
Singularity images already contain all required dependencies.
2528

2629
### Step 1: Clone workflow
2730
To use StaG-mwc, you need a local copy of the workflow repository. Start by
@@ -58,11 +61,16 @@ Make sure you edit the Slurm project account in
5861
documentation](https://snakemake.readthedocs.io) for further details on how to
5962
run Snakemake workflows on other types of cluster resources.
6063

64+
Note that in all examples above, `--use-conda` and essentially be replaced
65+
with `--use-singularity` to run in Singularity containers instead of using a
66+
locally installed conda. Read more about it under the Running section in the
67+
docs.
6168

6269
## Testing
6370
A very basic continuous integration test is currently in place. It merely
6471
validates the syntax by trying to let Snakemake build the dependency graph if
65-
all outputs are activated.
72+
all outputs are activated. Suggestions for how to improve the automated
73+
testing of StaG-mwc are very welcome!
6674

6775

6876
## Contributing

‎Snakefile‎

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,7 @@ min_version("5.5.4")
1616

1717
from rules.publications import publications
1818

19-
stag_version = "0.4.0"
20-
singularity: "docker://continuumio/miniconda3:4.7.10"
19+
stag_version = "0.4.1"
2120

2221
onstart:
2322
print("\n".join([
@@ -76,6 +75,7 @@ include: "rules/functional_profiling/humann2.smk"
7675
# Antibiotic resistance
7776
#############################
7877
include: "rules/antibiotic_resistance/groot.smk"
78+
include: "rules/antibiotic_resistance/amrplusplus.smk"
7979

8080
#############################
8181
# Mappers
@@ -153,12 +153,10 @@ onsuccess:
153153
Path("citations.rst").unlink()
154154
Path("citations.rst").symlink_to(citation_filename)
155155

156-
snakemake_call = " ".join(argv)
157-
shell("{snakemake_call} --unlock".format(snakemake_call=snakemake_call))
156+
shell("{snakemake_call} --unlock".format(snakemake_call=argv[0]))
158157
shell("{snakemake_call} --report {report}-{datetime}.html".format(
159-
snakemake_call=snakemake_call,
158+
snakemake_call=argv[0],
160159
report=config["report"],
161160
datetime=report_datetime,
162161
)
163162
)
164-

‎cluster_configs/ctmr_gandalf/config.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
jobscript: "cluster_configs/ctmr_gandalf/slurm-jobscript.sh"
2+
cluster: "cluster_configs/ctmr_gandalf/slurm-submit.py --time {cluster.time} --error {cluster.stderr} --output {cluster.stdout} --job-name '{cluster.jobname}' {cluster.extra}"
3+
cluster-status: "cluster_configs/ctmr_gandalf/slurm-status.py"
4+
cluster-config: "cluster_configs/ctmr_gandalf/ctmr_gandalf.yaml"
5+
max-jobs-per-second: 10
6+
max-status-checks-per-second: 10
7+
local-cores: 1
8+
jobs: 999

‎cluster_configs/ctmr_gandalf/ctmr_gandalf.yaml‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# Cluster config file for StaG-mwc for use on CTMR UCP
2+
__default__:
3+
account: "bio"
4+
partition: "ctmr"
5+
extra: ""
6+
time: "03:00:00"
7+
n: 2
8+
stderr: "slurm_logs/slurm-{rule}-{wildcards}.stderr"
9+
stdout: "slurm_logs/slurm-{rule}-{wildcards}.stdout"
10+
jobname: "[{rule}]: {wildcards}"
11+
12+
13+
#############################
14+
# Pre-processing
15+
#############################
16+
fastp:
17+
n: 8
18+
time: "01:00:00"
19+
remove_host:
20+
n: 8
21+
time: "01:00:00"
22+
bbcountunique:
23+
n: 4
24+
time: "00:45:00"
25+
26+
#############################
27+
# Naive comparisons
28+
#############################
29+
sketch:
30+
n: 8
31+
time: "00:20:00"
32+
33+
#############################
34+
# Taxonomic profiling
35+
#############################
36+
kaiju:
37+
n: 10
38+
time: "02:00:00"
39+
kraken2:
40+
n: 10
41+
time: "02:00:00"
42+
metaphlan2:
43+
n: 8
44+
time: "01:30:00"
45+
bracken:
46+
n: 2
47+
time: "01:00:00"
48+
49+
#############################
50+
# Functional profiling
51+
#############################
52+
humann2:
53+
n: 12
54+
time: "12:00:00"
55+
56+
#############################
57+
# Antibiotic resistance
58+
#############################
59+
groot_align:
60+
n: 8
61+
time: "01:00:00"
62+
align_to_amr:
63+
n: 10
64+
time: "04:00:00"
65+
66+
#############################
67+
# Mappers
68+
#############################
69+
bbmap:
70+
n: 10
71+
time: "02:00:00"
72+
bowtie2:
73+
n: 10
74+
time: "02:00:00"
75+
76+
#############################
77+
# Assembly
78+
#############################
79+
assembly:
80+
n: 20
81+
time: "05:00:00"
82+
assembly:
83+
n: 20
84+
time: "05:00:00"
85+
consolidate_bins:
86+
n: 20
87+
time: "05:00:00"
88+
blobology:
89+
n: 20
90+
time: "05:00:00"

‎cluster_configs/ctmr_gandalf/slurm-jobscript.sh‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
#!/bin/bash
2+
# properties = {properties}
3+
{exec_job}

‎cluster_configs/ctmr_gandalf/slurm-status.py‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
#!/usr/bin/env python3
2+
import re
3+
import subprocess as sp
4+
import shlex
5+
import sys
6+
import time
7+
import logging
8+
logger = logging.getLogger("__name__")
9+
10+
STATUS_ATTEMPTS = 20
11+
12+
jobid = sys.argv[1]
13+
14+
for i in range(STATUS_ATTEMPTS):
15+
try:
16+
sacct_res = sp.check_output(shlex.split("sacct -P -b -j {} -n".format(jobid)))
17+
res = {x.split("|")[0]: x.split("|")[1] for x in sacct_res.decode().strip().split("\n")}
18+
break
19+
except sp.CalledProcessError as e:
20+
logger.error("sacct process error")
21+
logger.error(e)
22+
except IndexError as e:
23+
pass
24+
# Try getting job with scontrol instead in case sacct is misconfigured
25+
try:
26+
sctrl_res = sp.check_output(shlex.split("scontrol -o show job {}".format(jobid)))
27+
m = re.search("JobState=(\w+)", sctrl_res.decode())
28+
res = {jobid: m.group(1)}
29+
break
30+
except sp.CalledProcessError as e:
31+
logger.error("scontrol process error")
32+
logger.error(e)
33+
if i >= STATUS_ATTEMPTS - 1:
34+
print("failed")
35+
exit(0)
36+
else:
37+
time.sleep(1)
38+
39+
status = res[jobid]
40+
41+
if (status.startswith("BOOT_FAIL")):
42+
print("failed")
43+
elif (status.startswith("CANCELLED")):
44+
print("failed")
45+
elif (status.startswith("COMPLETED")):
46+
print("success")
47+
elif (status.startswith("DEADLINE")):
48+
print("failed")
49+
elif (status.startswith("FAILED")):
50+
print("failed")
51+
elif (status.startswith("NODE_FAIL")):
52+
print("failed")
53+
elif (status.startswith("PREEMPTED")):
54+
print("failed")
55+
elif (status.startswith("TIMEOUT")):
56+
print("failed")
57+
# Unclear whether SUSPENDED should be treated as running or failed
58+
elif (status.startswith("SUSPENDED")):
59+
print("failed")
60+
else:
61+
print("running")

0 commit comments

Comments
 (0)