Using wildcards in VCF file names

[bschilder]

In the 1000 Genomes dataset hosted here, all files follow the same pattern except (annoyingly) for the chrX VCF and its index file (which end with ".v2.vcf.gz" instead of just ".vcf.gz" like the other files).

https://ftp.1000genomes.ebi.ac.uk/vol1/ftp/data_collections/1000G_2504_high_coverage/working/20220422_3202_phased_SNV_INDEL_SV/

Atm, my solution is to download those files into the data/vcf/phased folder (and rename them to be consistent with the other VCFs) before running the ProHap snakemake pipeline.

wget https://ftp.1000genomes.ebi.ac.uk/vol1/ftp/data_collections/1000G_2504_high_coverage/working/20220422_3202_phased_SNV_INDEL_SV/1kGP_high_coverage_Illumina.chrX.filtered.SNV_INDEL_SV_phased_panel.v2.vcf.gz \
-O $HOME/projects/ProHap/data/vcf/phased/1kGP_high_coverage_Illumina.chrX.filtered.SNV_INDEL_SV_phased_panel.vcf.gz \
&& gunzip $HOME/projects/ProHap/data/vcf/phased/1kGP_high_coverage_Illumina.chrX.filtered.SNV_INDEL_SV_phased_panel.vcf.gz

But would it be possible to add a bit more flexibility to the pipeline by allowing for wildcards?

For example:

phased_FTP_URL: "https://ftp.1000genomes.ebi.ac.uk/vol1/ftp/data_collections/1000G_2504_high_coverage/working/20220422_3202_phased_SNV_INDEL_SV/"
phased_local_path: "" 
phased_vcf_file_name: "1kGP_high_coverage_Illumina.chr{chr}.filtered.SNV_INDEL_SV_phased_panel.vcf"

With the wildcard added:

phased_vcf_file_name: "1kGP_high_coverage_Illumina.chr{chr}.filtered.SNV_INDEL_SV_phased_panel*.vcf"

Not sure how hard this would be, but for cases like this it could be quite helpful

Many thanks again!,
Brian

[vasicek58]

I'll look into this shortly, thanks for the suggestion!

[bschilder]

Along these lines, I have another dataset where each chromosome-specific VCF is sharded into multiple VCFs (bc they'd be too massive to process otherwise).
Originally reported here #8

So like, instead of this:

chr22.vcf.gz

It's

chr22_0001.vcf.gz
chr22_0002.vcf.gz
chr22_0003.vcf.gz
....

Perhaps the wildcard solution described above could also be used to address situations like this. Though the pipeline would need to know how to merge the results of these VCFs appropriately.

[vasicek58]

Hi @bschilder,

Please excuse the delayed response! I have now allowed globbing for the ProHap input files, meaning that you can specify "1kGP_high_coverage_Illumina.chr{chr}.filtered.SNV_INDEL_SV_phased_panel*.vcf" and it will work in the first case you reported.

In the second case, you can also specify "chr{chr}_*.vcf.gz" and it will process all the files. However, these will be merged after formatting and filtering by allele frequency into a single (compressed) file per chromosome. What allele frequency threshold do you use when processing these large files (if any)?

Also note that when you have multiple VCFs per chromosome, the current version expects that the variants inside the VCFs will be sorted by their genomic location, and the files will come in alphabetical order. In your case, this should be safe, as sorting the files will give chr22_0001 first, chr22_0002 second, etc. Is that okay for you?

Let me know if you need me to look into processing these sub-sections of chromosomes separately due to the size.

[bschilder]

Hi @bschilder,

Please excuse the delayed response! I have now allowed globbing for the ProHap input files, meaning that you can specify "1kGP_high_coverage_Illumina.chr{chr}.filtered.SNV_INDEL_SV_phased_panel*.vcf" and it will work in the first case you reported.

Great, thank you!

In the second case, you can also specify "chr{chr}_*.vcf.gz" and it will process all the files. However, these will be merged after formatting and filtering by allele frequency into a single (compressed) file per chromosome. What allele frequency threshold do you use when processing these large files (if any)?

In my case I'm including all alleles, so no filtering on allele freq.

Also note that when you have multiple VCFs per chromosome, the current version expects that the variants inside the VCFs will be sorted by their genomic location, and the files will come in alphabetical order. In your case, this should be safe, as sorting the files will give chr22_0001 first, chr22_0002 second, etc. Is that okay for you?

Yes that should be fine, the shards all all split arranged into ascending genomic positions (within and across shards).

Let me know if you need me to look into processing these sub-sections of chromosomes separately due to the size.

This may actually be the bottleneck atm, since I think there's some steps in VCF parsing that become pretty memory-intensive even in a small genomic region due to there being >414,000 individuals.
Potentially related to #10

Perhaps using tools to simulate these kinds of many-sample VCFs could be helpful here:

• msprime: https://tskit.dev/msprime/docs/stable/mutations.html
• vcfsim https://www.biorxiv.org/content/10.1101/2025.01.29.635540v1.full-text