Daniel Fischer

Introduction

When running computationally intensive workflows with Snakemake, you might encounter issues where too many jobs are running in parallel, causing excessive I/O load, memory pressure, or high latency on your hard drive. This can lead to failed jobs or degraded performance.

Snakemake provides a way to limit parallel execution per rule using the resources directive, but this only works if you also specify a global resource limit when executing the workflow.

In this blog post, we will demonstrate how to properly limit the number of parallel jobs for a specific rule using Snakemake’s resource management system.

---

The Problem: Too Many Jobs Running at Once

Consider the following Snakemake rule:

rule process_data:
    input:
        "{sample}.raw"
    output:
        "{sample}.processed"
    resources:
        process_data_jobs=1  # Assign a resource unit for limiting the number of jobs
    shell:
        """
        some_tool --input {input} --output {output}
        """

Why Doesn’t resources Alone Limit Job Execution?

You might expect that setting resources: processing_jobs=1 would automatically limit Snakemake to running only 1 job at a time. However, Snakemake does not enforce resource-based scheduling unless you specify a global limit when launching the workflow.

Without a global limit, Snakemake may still launch too many jobs in parallel, overloading your system.

---

The Solution: Enforce Resource Limits

To actually restrict the number of parallel jobs, run Snakemake with:

snakemake --resources process_data_jobs=10

How Does This Work?

• Each job of process_data requests 1 unit of process_data_jobs.
• The global limit processing_jobs=10 ensures that at most 10 jobs (10 / 1 = 10) run in parallel. You can also set different units, if you like

Before setting this limit, too many jobs could be running at once! After applying it, only 10 jobs were allowed to run simultaneously.

---

Conclusion

If you are facing high disk latency, I/O pressure, or excessive job execution in Snakemake, the best way to control it is by:

1. Using resources to define per-job resource requirements.
2. Setting a global resource limit (--resources processing_jobs=10) when executing Snakemake.

This approach ensures your workflow runs efficiently and reliably without overloading your system!

Timothy (Phleum pratense L.) is a key forage grass for Finnish agriculture, and improving its yield, winter hardiness, and digestibility is crucial for sustainable production. Our recent study explored the potential of genomic prediction to accelerate breeding progress by leveraging genotyping-by-sequencing and advanced statistical models.

Key findings:
* Heritability estimates ranged from 0.13 (yield at first cut) to 0.86 (digestibility at second cut).
* Genetic correlations suggest trade-offs between yield and winter survival but positive links between digestibility traits.
* Genomic breeding values were estimated using advanced statistical approaches, including a novel scaling of the genomic relationship matrix.
* Predictive ability reached up to 0.62 for digestibility, and validation confirmed moderate accuracy with little dispersion.

Despite concerns that genotype quality might impact predictions, our results show that genomic prediction remains a powerful tool for Timothy breeding in Finland. This research highlights the potential for data-driven breeding strategies to enhance forage crop resilience and quality.

https://link.springer.com/article/10.1007/s00122-025-04860-9