Category: Know-how

Introduction

When running computationally intensive workflows with Snakemake, you might encounter issues where too many jobs are running in parallel, causing excessive I/O load, memory pressure, or high latency on your hard drive. This can lead to failed jobs or degraded performance.

Snakemake provides a way to limit parallel execution per rule using the resources directive, but this only works if you also specify a global resource limit when executing the workflow.

In this blog post, we will demonstrate how to properly limit the number of parallel jobs for a specific rule using Snakemake’s resource management system.

---

The Problem: Too Many Jobs Running at Once

Consider the following Snakemake rule:

rule process_data:
    input:
        "{sample}.raw"
    output:
        "{sample}.processed"
    resources:
        process_data_jobs=1  # Assign a resource unit for limiting the number of jobs
    shell:
        """
        some_tool --input {input} --output {output}
        """

Why Doesn’t resources Alone Limit Job Execution?

You might expect that setting resources: processing_jobs=1 would automatically limit Snakemake to running only 1 job at a time. However, Snakemake does not enforce resource-based scheduling unless you specify a global limit when launching the workflow.

Without a global limit, Snakemake may still launch too many jobs in parallel, overloading your system.

---

The Solution: Enforce Resource Limits

To actually restrict the number of parallel jobs, run Snakemake with:

snakemake --resources process_data_jobs=10

How Does This Work?

• Each job of process_data requests 1 unit of process_data_jobs.
• The global limit processing_jobs=10 ensures that at most 10 jobs (10 / 1 = 10) run in parallel. You can also set different units, if you like

Before setting this limit, too many jobs could be running at once! After applying it, only 10 jobs were allowed to run simultaneously.

---

Conclusion

If you are facing high disk latency, I/O pressure, or excessive job execution in Snakemake, the best way to control it is by:

1. Using resources to define per-job resource requirements.
2. Setting a global resource limit (--resources processing_jobs=10) when executing Snakemake.

This approach ensures your workflow runs efficiently and reliably without overloading your system!