Kubernetes plugin, validate_settings crashes pod when settings is <class 'NoneType'> #61
Description
This issue is related to the kubernetes plugin issue #33
I have dived into the container being ran as the pod on kubernetes and found the issue - in plugin_registry/plugin.py the function validate_settings crashes the entire pod when it receives the "settings" variable of type <class 'NoneType'>. The exact scenario where that happens is described in the above mentioned issue.
After some modifications to the function, I found out that it is in fact being called twice. Once with type "NoneType" (crashing the pod) and once with the correct class "snakemake_interface_common.plugin_registry.plugin.SettingsBase".
The fix appears to be inserting a check in the function, stopping it's further execution, for example:
validate_settings:
def validate_settings(self, settings):def get_description(thefield): envvar = ( f" (or environment variable {self.get_envvar(thefield.name)})" if thefield.metadata.get("env_var", None) else "" ) return f"{self.get_cli_arg(thefield.name)}{envvar}" if settings is None: return None # rewrite for settings missing = [ thefield for thefield in fields(settings) if thefield.metadata.get("required") and getattr(settings, thefield.name) is None ] if missing: cli_args = [get_description(thefield) for thefield in missing] raise WorkflowError( f"The following required arguments are missing for " f"plugin {self.name}: {', '.join(cli_args)}." )
full logs from repeated experimentation:
(base) root@3b905478bb08:/tmp/repo# /bin/sh -c "pip install --target '.snakemake/pip-deployments' snakemake-storage-plugin-fs && python -m snake make --deploy-sources snakemake-workflow-sources.bfdcb0b3ee03be3b2067e78819318ff12737df578e9d074be6bddc289acf3deb.tar.xz bfdcb0b3ee03be3b2067e78 819318ff12737df578e9d074be6bddc289acf3deb --default-storage-provider 'fs' --verbose && python -m snakemake --snakefile 'snakefile' --target-job s 'sleep:' --allowed-rules sleep --cores 8 --attempt 1 --force-use-threads --force --target-files-omit-workdir-adjustment --max-inventory-time 0 --nocolor --notemp --no-hooks --nolock --ignore-incomplete --verbose --rerun-triggers 'code' 'input' 'mtime' 'params' 'software-env' --deplo yment-method conda --conda-frontend 'conda' --shared-fs-usage 'none' --wrapper-prefix 'https://github.com/snakemake/snakemake-wrappers/raw/' --l atency-wait 5 --scheduler 'ilp' --local-storage-prefix .snakemake/storage --default-storage-provider 'fs' --default-resources base64//dG1wZGlyPX N5c3RlbV90bXBkaXI= --mode 'remote' --verbose" Collecting snakemake-storage-plugin-fs Using cached snakemake_storage_plugin_fs-1.1.1-py3-none-any.whl.metadata (1.2 kB) Collecting reretry<0.12.0,>=0.11.8 (from snakemake-storage-plugin-fs) Using cached reretry-0.11.8-py2.py3-none-any.whl.metadata (5.5 kB) Collecting snakemake-interface-common<2.0.0,>=1.17.0 (from snakemake-storage-plugin-fs) Using cached snakemake_interface_common-1.17.4-py3-none-any.whl.metadata (760 bytes) Collecting snakemake-interface-storage-plugins<5.0.0,>=4.1.0 (from snakemake-storage-plugin-fs) Using cached snakemake_interface_storage_plugins-4.2.1-py3-none-any.whl.metadata (11 kB) Collecting sysrsync<2.0.0,>=1.1.1 (from snakemake-storage-plugin-fs) Using cached sysrsync-1.1.1-py3-none-any.whl.metadata (7.2 kB) Collecting ConfigArgParse<2.0,>=1.7 (from snakemake-interface-common<2.0.0,>=1.17.0->snakemake-storage-plugin-fs) Using cached ConfigArgParse-1.7-py3-none-any.whl.metadata (23 kB) Collecting argparse-dataclass<3.0.0,>=2.0.0 (from snakemake-interface-common<2.0.0,>=1.17.0->snakemake-storage-plugin-fs) Using cached argparse_dataclass-2.0.0-py3-none-any.whl.metadata (7.2 kB) Collecting throttler>=1.2.2 (from snakemake-interface-storage-plugins<5.0.0,>=4.1.0->snakemake-storage-plugin-fs) Using cached throttler-1.2.2-py3-none-any.whl.metadata (7.4 kB) Collecting wrapt>=1.15.0 (from snakemake-interface-storage-plugins<5.0.0,>=4.1.0->snakemake-storage-plugin-fs) Using cached wrapt-1.17.2-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.4 kB) Collecting toml<0.11.0,>=0.10.0 (from sysrsync<2.0.0,>=1.1.1->snakemake-storage-plugin-fs) Using cached toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) Using cached snakemake_storage_plugin_fs-1.1.1-py3-none-any.whl (5.6 kB) Using cached reretry-0.11.8-py2.py3-none-any.whl (5.6 kB) Using cached snakemake_interface_common-1.17.4-py3-none-any.whl (14 kB) Using cached snakemake_interface_storage_plugins-4.2.1-py3-none-any.whl (16 kB) Using cached sysrsync-1.1.1-py3-none-any.whl (7.5 kB) Using cached argparse_dataclass-2.0.0-py3-none-any.whl (8.8 kB) Using cached ConfigArgParse-1.7-py3-none-any.whl (25 kB) Using cached throttler-1.2.2-py3-none-any.whl (7.6 kB) Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB) Using cached wrapt-1.17.2-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (89 kB) Installing collected packages: throttler, wrapt, toml, reretry, ConfigArgParse, argparse-dataclass, sysrsync, snakemake-interface-common, snakem ake-interface-storage-plugins, snakemake-storage-plugin-fs Successfully installed ConfigArgParse-1.7 argparse-dataclass-2.0.0 reretry-0.11.8 snakemake-interface-common-1.17.4 snakemake-interface-storage- plugins-4.2.1 snakemake-storage-plugin-fs-1.1.1 sysrsync-1.1.1 throttler-1.2.2 toml-0.10.2 wrapt-1.17.2 WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/snakemake_storage_plugin_fs-1.1.1.dist-info already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/reretry already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/snakemake_interface_storage_plugins-4.2.1.dist-info already exists. Specify --upg rade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/snakemake_interface_storage_plugins already exists. Specify --upgrade to force re placement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/throttler already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/argparse_dataclass.py already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/argparse_dataclass-2.0.0.dist-info already exists. Specify --upgrade to force rep lacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/sysrsync-1.1.1.dist-info already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/reretry-0.11.8.dist-info already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/configargparse.py already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/wrapt-1.17.2.dist-info already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/toml-0.10.2.dist-info already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/throttler-1.2.2.dist-info already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/wrapt already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/snakemake_interface_common already exists. Specify --upgrade to force replacement . WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/snakemake_interface_common-1.17.4.dist-info already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/__pycache__ already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/ConfigArgParse-1.7.dist-info already exists. Specify --upgrade to force replaceme nt. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/toml already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/snakemake_storage_plugin_fs already exists. Specify --upgrade to force replacemen t. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/sysrsync already exists. Specify --upgrade to force replacement. WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly ren dering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-acti on option if you know what you are doing and want to suppress this warning.------------------------------_
Validate settingsSettings value:None
Settings type:
Is dataclass: False
Datatype: <class 'NoneType'>------------------------------_
Validate settingsSettings value:SettingsBase()
Settings type:
Is dataclass: True
Datatype: <class 'snakemake_interface_common.plugin_registry.plugin.SettingsBase'>host: 3b905478bb08
Building DAG of jobs...
shared_storage_local_copies: False
remote_exec: True
Submitting maximum 100 job(s) over 1.0 second(s).
Using shell: /usr/bin/bash
Provided cores: 8
Rules claiming more threads will be scaled down.
Resources before job selection: {'_cores': 8, '_nodes': 9223372036854775807, '_job_count': 9223372036854775807}
Ready jobs: 1
Select jobs to execute...
Selecting jobs to run using greedy solver.
Selected jobs: 1
Resources after job selection: {'_cores': 7, '_nodes': 9223372036854775806, '_job_count': 100}
Execute 1 jobs...[Fri Apr 4 11:29:51 2025]
localrule sleep:
jobid: 0
reason: Rules with neither input nor output files are always executed.
resources: mem_mb=20, mem_mib=20, disk_mb=1000, disk_mib=954, tmpdir=/tmp, threads=1/opt/conda/envs/snakemake/lib/python3.12/site-packages/conda/base/context.py:198: FutureWarning: Adding 'defaults' to channel list implicitly is
deprecated and will be removed in 25.9.To remove this warning, please choose a default channel explicitly with conda's regular configuration system, e.g. by adding 'defaults' to the l
ist of channels:conda config --add channels defaults
For more information see https://docs.conda.io/projects/conda/en/stable/user-guide/configuration/use-condarc.html
deprecated.topic(
Waiting for more resources.
[Fri Apr 4 11:29:59 2025]
Finished jobid: 0 (Rule: sleep)
1 of 1 steps (100%) done
Storing output in storage.
I do not know whether this is the best solution, nor what effect it may have on other plugins. As such, I kindly ask to verify that this change is fit to be integrated into the repository, and if not, I'm more than eager to test more possible solutions for my problem.