Calling Qiime scripts sequencially in Python 2.7 program
1
1
Entering edit mode
8.0 years ago
uxue33 ▴ 20

Hello all,

I'm beginning to learn Python 2.7 and need to develop some applications for a lab project.

I want to run an external program and then use its output as input for another external call in the same script. The issue is that the script tries to call the second program even before the first one has finished working, so the output is not yet available to be used as its input. How can I solve this problem?

Here is my code:

picker = Popen ([pick_open_reference_otus.py, -i, allseq_file, -r, refseq_file, -o, noName, -s, 0.1, -m, usearch61, -p, param_file ], shell = True)
picker_out = os.path.join(path, "otu_output/otu_table_mc2_w_tax.biom")

Thank you all!

python • 2.9k views
ADD COMMENT
1
Entering edit mode

I'm not 100% sure because I have not done any work with the subprocesses module, however I think you can just use the wait() module. something like picker.wait(picker_out) or something to that effect.

ADD REPLY
1
Entering edit mode

or picker_out = picker.wait(os.path.join(path, "otu_output/otu_table_mc2_w_tax.biom") sorry. I'm not sure about the syntax

ADD REPLY
1
Entering edit mode

last one :)

picker = Popen ([pick_open_reference_otus.py, -i, allseq_file, -r, refseq_file, -o, noName, -s, 0.1, -m, usearch61, -p, param_file ], shell = True)
picker.wait()
picker_out = os.path.join(path, "otu_output/otu_table_mc2_w_tax.biom")
ADD REPLY
0
Entering edit mode

Thank you a lot for your help!

ADD REPLY
1
Entering edit mode

https://docs.python.org/2/library/subprocess.html

Subprocess has what you want.

Python can be rather clunky for this kind of thing. If you need data from some of these programs you're calling to end up being processed in python, it is better to pull the data into python from the "outside". You may want to explore other options for managing pipelines, there are plenty of posts and tutorials on here about it.

ADD REPLY
0
Entering edit mode

Thank you for the link!

ADD REPLY
1
Entering edit mode
8.0 years ago

I used the following code for similar jobs:

retmode = subprocess.call(shlex.split('your shell command with arguments'))
if retmode > 0:
    sys.exit("Prinseq returned error code {} for sample {}".format(retmode, target))

I find the shlex module useful to structure command prior to using subprocess.call, it allows you to just write the string of the command and it will take care of the formatting in a list.

ADD COMMENT
0
Entering edit mode

Than you a lot for your code!

ADD REPLY
0
Entering edit mode

Dear all,

I found this post very useful and tried to use the solution myself. The thing is that I presume I get an error in the variable interpolation process that I can't figure out how to fix (I haven't used Python very often for this kind of task, because I usually use Perl for external program related pipelines). Here's my code and the error it produces.

import sys, os
import subprocess
import shlex

allseq_file = "/home/jllavin/Metagenomics/Test.fasta"
refseq_file = "/home/jllavin/Metagenomics/Dbs/99_otus.fasta"

picker = subprocess.call(shlex.split('pick_open_reference_otus.py -i allseq_file -r refseq_file -o output2 -s "0.1" -m "usearch61" -p parameters.txt -f'))
    if picker > 0:
    sys.exit[(picker)]

The error:

Traceback (most recent call last):
  File "/usr/lib64/python2.7/lib-tk/Tkinter.py", line 1470, in __call__
    return self.func(*args)
  File "QIIME_shlex.py", line 45, in button_ex_callback
    sys.exit[(picker)]
TypeError: 'builtin_function_or_method' object has no attribute '__getitem__'
ADD REPLY
1
Entering edit mode

hi joluito,

for sure sys.exit[(picker)] is wrong. The square brackets don't serve a purpose here. Try sys.exit(picker). (This is just the return code of the script, in case something goes wrong)

for the rest, I don't think your -i allseq_file is going to work. Or is allseq_file an actual file in the current directory?

If it's a variable in your python code (what I suspect) you need to construct the string differently:

picker = subprocess.call(shlex.split('pick_open_reference_otus.py -i ' + allseq_file + ' -r ' + refseq_file + etc etc'))

As such the value of the variable is inserted in the string, fed to shlex.split() and used for the process.

Note the spaces!

It can be helpful to first construct your command:

cmd = 'pick_open_reference_otus.py -i ' + allseq_file + ' -r ' + refseq_file + etc etc'
print(cmd)
picker = subprocess.call(shlex.split(cmd))

and print that to see if everything is okay

ADD REPLY
0
Entering edit mode

Sorry, I was adding the rest of the code while you answered

;)

ADD REPLY
1
Entering edit mode

So as I suspected, you are passing the variable's name to pick_open_reference_otus.py, rather than the variable's value. Is that clear?

ADD REPLY
0
Entering edit mode

Interesting point you raised here.

So, I'm not interpolating the variable's "string value" by passing its "name" to the command...

Therefore the way to proceed is to generate the command in a "print style" and then using it via subprocess.call(shlex.split)

Thank you very much for your patience, I will try it and come back with the outcome.

Best

ADD REPLY
1
Entering edit mode

Yes, just concat the strings together with the variables and don't forget spaces.

ADD REPLY
0
Entering edit mode

It worked smoothly.

Thank you very much for your kind support.

Best

JL

ADD REPLY

Login before adding your answer.

Traffic: 2226 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6