Question: Emboss Needle Algorithm'S Memory Usage Increases When I Run Many Processes Of It One After Another
1
gravatar for TabeaH
6.7 years ago by
TabeaH30
TabeaH30 wrote:

Hey there,

maybe someone of you can help me or at least give me a hint on where else I could look for help.

My problem is as follows:

I have a Python script that invokes the EMBOSS needle program on command line (using os.system("needle ...")) with two sequences and performs an alignment. Then it does some other stuff. This I repeated many 1000 times.

At first, needle uses a reasonable amount of memory (anything between 1 to 400 MB). However, after a couple of thousand times, the amount of memory used by needle increases, until it reaches the maximum and my Mac is completely blocked. The python script however doesn't use more memory than in the beginning.

So it seems that needle reserves some part of the memory and never releases it.

I have also tried to use the subprocess module and start needle with process=subprocess.Popen("needle ..."). Then I performed process.kill(), process.terminate() and process.wait() (all separately in other versions of my script), as it was suggested in some forums, in order to avoid zombie processes. But it didn't change anything....

Does one of you have an idea how can I release this memory from the grabs of needle or how I can overcome this problem in another way?

Edit:

It is the needle process and not the python script that increases in memory usage. I added this information more specifically.

python alignment memory • 2.2k views
ADD COMMENTlink modified 6.7 years ago by Asaf8.2k • written 6.7 years ago by TabeaH30

First you really need to establish whether it is the python program, the needle program or the fact that you iterate with the former on the latter that uses up the memory.

  • Take the last inputs that fill up the memory,now invoke needle from the command line on these inputs. How much memory does that need?
  • Alternatively save each input into a file, then run a separate shell script and run needle on these inputs. Do you observe the same problem?
ADD REPLYlink modified 6.7 years ago • written 6.7 years ago by Istvan Albert ♦♦ 84k

Thank you for these ideas! I will check tomorrow how the memory usage will look like in these two cases.

ADD REPLYlink written 6.7 years ago by TabeaH30
3
gravatar for Asaf
6.7 years ago by
Asaf8.2k
Israel
Asaf8.2k wrote:

use stretcher of the EMBOSS package, it's a needleman-wunsch algorithm variation that takes O(N) memory instead of O(NM). Maybe you have a memory leak or maybe your sequences are getting longer and require a lot more memory, either way stretcher will solve it.

ADD COMMENTlink written 6.7 years ago by Asaf8.2k

In fact some sequences were longer than I expected them to be. stretcher solved that. Thank you!

ADD REPLYlink written 6.6 years ago by TabeaH30
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1440 users visited in the last hour