Snakemake: set cluster cores per rule
1
1
Entering edit mode
6.5 years ago
samesense ▴ 40

I need to download hundreds of large files, and run each of them through my snakemake pipeline. The file download is fast compared to my downstream pipeline. I'd like to limit the number of parallel downloads to 5, but allow the downsteam processing to use 100 cores. In snakemake, is there a way to limit the number of cores used by a certain rule? I picture 5 cores constantly grabbing data, while my other cores are working on the data I've already downloaded. If I run snakemake as usual with 100 cores, it will try to download all files at once, and overload the server. 

snakemake • 2.7k views
ADD COMMENT
0
Entering edit mode

I tried setting a download resource, according to this answer

https://groups.google.com/forum/#!searchin/snakemake/parallell$20resources/snakemake/PAfWikWvmG4/WAArbq-_ea0J

This did not limit the downloads as expected.

 

ADD REPLY
0
Entering edit mode
6.5 years ago

Is this what you're looking for: https://bitbucket.org/johanneskoester/snakemake/wiki/Documentation#markdown-header-threads?

Otherwise, you can try posting in the official Snakemake Google group: https://groups.google.com/forum/#!forum/Snakemake The author is pretty responsive.

ADD COMMENT
0
Entering edit mode

I saw that too. I think it would multithread my download, but wait on the rule to finish before moving onto the downstream pipeline. I'll ask the user group.

ADD REPLY

Login before adding your answer.

Traffic: 2829 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6