Question: Calling Python Script From Bash Console While Using Gnu Parallel
3
gravatar for NPalopoli
8.0 years ago by
NPalopoli280
Argentina/Buenos Aires/Universidad Nacional de Quilmes & Fundación Instituto Leloir
NPalopoli280 wrote:

I am trying to run a python script which processes many FASTA files in parallel using GNU parallel 20110722. As it could be seen below, I am not able to run in any of the ways I tried. (^C marks the point where I interrupt the job with ctrl+C because there is no response from the system).

me@kubuntu:~/Programs/LeitMotifsParallel$ parallel python {1} :::: <(echo MainMult.18136.py)
^C
me@kubuntu:~/Programs/LeitMotifsParallel$ parallel python {1} ::: <(echo MainMult.18136.py)
File "/dev/fd/63", line 1
MainMult.18136.py
             ^
SyntaxError: invalid syntax
me@kubuntu:~/Programs/LeitMotifsParallel$ parallel MainMult.18136.py
parallel: Input is tty. Press CTRL-D to exit.
^C
me@kubuntu:~/Programs/LeitMotifsParallel$

However, the python script runs as expected when run directly from console.

me@kubuntu:~/Programs/LeitMotifsParallel$ MainMult.18136.py 
/home/me/Programs/LeitMotifsParallel/StAlg.py:6: DeprecationWarning: the sets module is deprecated
import sets #@UnusedImport
Start : 18:59:34 11Aug2011
['M']
['M']
(...)

Though I have watched the two tutorials on GNU Parallel in youtube and have gone through the many examples of the README file, I haven't been succesful in finding an answer for this situation, so I would really appreciate if you could help me solve this.

python parallel bash • 8.8k views
ADD COMMENTlink modified 8.0 years ago by tange190 • written 8.0 years ago by NPalopoli280
2

Note that you can also use threading really easily from within Python: http://docs.python.org/library/multiprocessing.html

ADD REPLYlink written 8.0 years ago by Michael Schubert6.9k
1

Not sure this is a bioinformatics question. What are the ":::" doing? did you try "parallel python -- your-prog.py"

ADD REPLYlink written 8.0 years ago by brentp23k

Is it possible/necessary to put the python call into a separate bash file (e.g. run.sh)? It might be a problem with the input/output redirection or piping...

ADD REPLYlink written 8.0 years ago by Cjt370

Thanks all for the suggestions. Though I don't think it could be strictly categorized as a bioinformatics question I assumed it may be relevant for the community.
brentp: The ":::" are used by GNU Parallel for specifying arguments from the command line. I tried with your option but the "Input from tty" line shows it is not useful.
cjt: I tried with calling python from a separate bash file and using parallel to call that file but it is not working either.
Michael Schubert: I would definitely use threading from within Python in other cases but for this job I need to use parallel.

ADD REPLYlink written 8.0 years ago by NPalopoli280
10
gravatar for tange
8.0 years ago by
tange190
tange190 wrote:

First it is good to see a fellow bioinformatician use GNU Parallel.

Secondly it is good to learn you have watched the 4 intro videos: http://www.youtube.com/watch?v=OpaiGYxkSuQ http://www.youtube.com/watch?v=P40akGWJ_gY http://www.youtube.com/watch?v=1ntxT-47VPA http://www.youtube.com/watch?v=fOX1EyHkQwc and that you have browsed through the examples: http://nd.gd/u3

GNU Parallel does not parallelize the internals of an existing program. What it can do, however, is call the same program with different arguments. So if you normally do:

MainMult.18136.py foo.fasta
MainMult.18136.py bar.fasta

you can parallelize this by running 2 MainMult.18136.py in parallel like this:

parallel MainMult.18136.py ::: foo.fasta bar.fasta

If your program has the filenames hardcoded in the program then GNU Parallel cannot parallelize the task for you. The filenames must be given on the command line.

Alternatively, if the program reads from standard input (stdin) and you normally do:

cat foo.fasta bar.fasta | MainMult.18136.py

then you can parallelize this by using GNU Parallel --pipe:

cat foo.fasta bar.fasta | parallel --pipe --block 10M --recstart '>'  MainMult.18136.py

This will chop the input into 10 MB sized chunks and pass them on to MainMult.18136.py on standard input (stdin). Each chunk will be chopped at a '>' which is where a FASTA record starts.

If what you normally do is:

MainMult.18136.py
MainMult.18137.py
MainMult.18138.py

that is, run a bunch of different programs that happen to have the arguments hardcoded in each of them, then you can do:

parallel ::: MainMult.*.py

or:

ls | grep MainMult | parallel

However, from your description of the Python script it seems you normally do:

MainMult.18136.py

(only one program with no arguments and no reading from standard input). In this situation GNU Parallel is unable to parallelize the task. My advice is to change the Python script so that it takes the filenames as its arguments from the command line.

ADD COMMENTlink modified 8.0 years ago • written 8.0 years ago by tange190

I have seen the files and read the examples but I couldn't make my script work.

The FASTA files are directly called in the MainMult.18136.py file in the following way: 1) A variable is defined pointing to the file: work_file = "/home/me/Prog/Seqs.fasta" 2) The filename is specified as one of many parameters in the proper function call: Run("Seqs.fasta", 7, 100, True, 0.63, True, False, 0.98, False, False)

The point here is that the MainMult.18136.py runs from command line, but not if called with parallel.

ADD REPLYlink written 8.0 years ago by NPalopoli280

@NPalopoli: Not sure I understand; are you iterating over lists of FASTA files and function arguments within the script itself?

ADD REPLYlink written 8.0 years ago by James Estevez90

The key is that (and I cite you), "GNU Parallel does not parallelize the internals of an existing program". I've made different python scripts for each Fasta file and managed to run the program by following your advice and calling: parallel ::: MainMult.*.py. Thanks a lot for your help!

ADD REPLYlink written 8.0 years ago by NPalopoli280
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1604 users visited in the last hour