MISO test run fails error of bam file was not found
0
0
Entering edit mode
6.1 years ago
JoeDoasi ▴ 10

Hi all,

After I dowloaded the MISO software and run the test command as requested in the manual, I ended up having this message:

===========================
Error: BAM /Library/Python/2.7/site-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam cannot be found.
02/06/2016 05:39:49 PM - miso_main - ERROR - BAM file /Library/Python/2.7/site-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam not found.
----------------------------------------------------------------------
FAIL: test_a_sam_to_bam (misopy.test_miso.TestMISO)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/misopy/test_miso.py", line 51, in test_a_sam_to_bam
    "c2c12.Atp2b1.sorted.bam")))
AssertionError"
===========================

Please guide me on how to solve this issue. FYI, I have downloaded all prerequisite softwares that needed by MISO and Samtools can be run from terminal without any problems.

I appreciate your help

RNA-Seq Assembly software error next-gen genome • 3.2k views
ADD COMMENT
0
Entering edit mode

How did you run the test? What is the result if you run which python?

ADD REPLY
0
Entering edit mode

I ran this command: $ python -m unittest discover misopy

according to the manual : https://miso.readthedocs.io/en/fastmiso/#installing-the-fastmiso-version-of-miso

Python version is 2.7.6

The result is this error (part 1 - due to post restrictions to 5000 charcters):

Testing gene-level Psi... Executing: python /usr/local/lib/python2.7/dist-packages/misopy/index_gff.py --index /usr/local/lib/python2.7/dist-packages/misopy/gff-events/mm9/genes/Atp2b1.mm9.gff /usr/local/lib/python2.7/dist-packages/misopy/gff-events/mm9/indexed Indexing GFF... /usr/local/lib/python2.7/dist-packages/misopy/gff-events/mm9/genes/Atp2b1.mm9.gff appears to already be indexed. Aborting. Executing: python /usr/local/lib/python2.7/dist-packages/misopy/run_events_analysis.py --compute-genes-psi /usr/local/lib/python2.7/dist-packages/misopy/gff-events/mm9/indexed /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam --output-dir /usr/local/lib/python2.7/dist-packages/misopy/test-output/gene-psi-output --read-len 36 --paired-end 250 30 --use-cluster MISO (Mixture of Isoforms model) To run MISO, please use "miso" instead. .Testing single-end SE event interface... Executing: python /usr/local/lib/python2.7/dist-packages/misopy/run_events_analysis.py --compute-events-psi se-sample /usr/local/lib/python2.7/dist-packages/misopy/test-data/se-counts/se_test.counts --output-dir /usr/local/lib/python2.7/dist-packages/misopy/test-output/SE-output --read-len 35 --overhang-len 4 --event-type SE --use-cluster MISO (Mixture of Isoforms model) To run MISO, please use "miso" instead. .Testing fr-unstranded... Checking read f_read against + Checking read f_read against - Checking read r_read against + Checking read r_read against - Testing fr-firststrand... .Testing conversion of SAM to BAM... Executing: sam_to_bam --convert /usr/local/lib/python2.7/dist-packages/misopy/test-data/sam-data/c2c12.Atp2b1.sam /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output Converting SAM to BAM... - Executing: samtools view -Sbh /usr/local/lib/python2.7/dist-packages/misopy/test-data/sam-data/c2c12.Atp2b1.sam > /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.bam Sorting BAM file... - Executing: samtools sort /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.bam /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted [bam_sort] Use -T PREFIX / -o FILE to specify temporary and final output files Usage: samtools sort [options...] [in.bam] Options: -l INT Set compression level, from 0 (uncompressed) to 9 (best) -m INT Set maximum memory per thread; suffix K/M/G recognized [768M] -n Sort by read name -o FILE Write final output to FILE rather than standard output -T PREFIX Write temporary files to PREFIX.nnnn.bam -@, --threads INT Set number of sorting and compression threads [1] --input-fmt-option OPT[=VAL] Specify a single input file format option in the form of OPTION or OPTION=VALUE -O, --output-fmt FORMAT[,OPT[=VAL]]... Specify output format (SAM, BAM, CRAM) --output-fmt-option OPT[=VAL] Specify a single output file format option in the form of OPTION or OPTION=VALUE --reference FILE Reference sequence FASTA FILE [null] Indexing BAM... - Executing: samtools index /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam [E::hts_open_format] fail to open file '/usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam' samtools index: failed to open "/usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam": No such file or directory Conversion took 0.00 minutes. FTesting gene-level Psi... Testing GFF indexing of: /usr/local/lib/python2.7/dist-packages/misopy/gff-events/mm9/genes/Atp2b1.mm9.gff Executing: index_gff --index /usr/local/lib/python2.7/dist-packages/misopy/gff-events/mm9/genes/Atp2b1.mm9.gff /usr/local/lib/python2.7/dist-packages/misopy/gff-events/mm9/genes/Atp2b1/indexed Indexing GFF... /usr/local/lib/python2.7/dist-packages/misopy/gff-events/mm9/genes/Atp2b1.mm9.gff appears to already be indexed. Aborting. Executing: miso --run /usr/local/lib/python2.7/dist-packages/misopy/gff-events/mm9/genes/Atp2b1/indexed /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam --output-dir /usr/local/lib/python2.7/dist-packages/misopy/test-output/gene-psi-output --read-len 36 MISO (Mixture of Isoforms model) Probabilistic analysis of RNA-Seq data for detecting differential isoforms Use --help argument to view options.

ADD REPLY
0
Entering edit mode

part 2:

Using MISO settings file: /usr/local/lib/python2.7/dist-packages/misopy/settings/miso_settings.txt Computing Psi values... - GFF index: /usr/local/lib/python2.7/dist-packages/misopy/gff-events/mm9/genes/Atp2b1/indexed - BAM: /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam - Read length: 36 - Output directory: /usr/local/lib/python2.7/dist-packages/misopy/test-output/gene-psi-output Checking your GFF annotation and BAM for mismatches... Error: BAM /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam cannot be found. 08/20/2016 05:33:07 AM - miso_main - ERROR - BAM file /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam not found.

.

FAIL: test_a_sam_to_bam (misopy.test_miso.TestMISO)

Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/misopy/test_miso.py", line 51, in test_a_sam_to_bam "c2c12.Atp2b1.sorted.bam"))) AssertionError


Ran 5 tests in 1.192s

FAILED (failures=1)

ADD REPLY
0
Entering edit mode

I found this: https://github.com/yarden/MISO/blob/fastmiso/misopy/sam_to_bam.py The samtools sort function in is incorrect and that call fails. Perhaps in an older version it did work, I wouldn't know. Try to adjust the script on line 28 to have correct samtools synthax Samtools even prints it in your traceback:

Usage: samtools sort [options...] [in.bam] Options: -l INT Set compression level, from 0 (uncompressed) to 9 (best) -m INT Set maximum memory per thread; suffix K/M/G recognized [768M] -n
Sort by read name
-o FILE Write final output to FILE rather than standard output
-T PREFIX Write temporary files to PREFIX.nnnn.bam -@, --threads INT Set number of sorting and compression threads [1] --input-fmt-option OPT[=VAL] Specify a single input file format option in the form of OPTION or OPTION=VALUE -O, --output-fmt FORMAT[,OPT[=VAL]]... Specify output format (SAM, BAM, CRAM) --output-fmt-option OPT[=VAL] Specify a single output file format option in the form of OPTION or OPTION=VALUE --reference FILE Reference sequence FASTA FILE [null]

So instead of having cmd = "samtools sort %s %s" %(bam_filename, sorted_filename) you should have cmd = "samtools sort %s -o %s" %(bam_filename, sorted_filename)

ADD REPLY
0
Entering edit mode

Thanks a lot for your responses. I changed the command by adding -o and still the get the same result. I also tried older version of samtools 0.1.19 as @genomax2 suggested and still same problem..

I don't know if this page have something meaningful: https://github.com/nservant/HiC-Pro/issues/23

could you please look at it and see if it makes sense !

ADD REPLY
0
Entering edit mode

Is the error message exactly the same? And you did add the -o in front of the second %s? Remarkable.

ADD REPLY
0
Entering edit mode

Yes I added -o in front of the second %s . The same error message.

ADD REPLY
0
Entering edit mode

My bad, you also need to change the line above: sorted_filename = "%s.sorted" %(bam_filename.split(".bam")[0]) should become sorted_filename = "%s.sorted.bam" %(bam_filename.split(".bam")[0])

Nothing you have to worry about, but the script should use pysam instead of os.system() calls to samtools o.0 or at least subprocess.call() with proper error handling. It now executes shell commands without checking whether those were successful...

ADD REPLY
0
Entering edit mode

Thanks for your help..

It failed again even after editing the two lines!, so I took a screenshot for the screen for more convenience..please check the attached photo

I'm new to programming and I worked on perl not python so imagine how I'm lost !!!

ADD REPLY
0
Entering edit mode

Debugging from a distance isn't easy but I'm confident we'll get there.

Actually, the error is not completely the same, and that's a good thing. You'll notice the samtools error about usage is no longer there. It's just a wrong output file that is created, if I'm not mistaken :p Could you edit the code once more to print out print(sorted_filename) before the cmd = "samtools sort %s %s" %(bam_filename, sorted_filename)? Then we'll find out which file exactly is created.

Ehm edit: I notice multiple lines in which samtools not found is stated. You have samtools installed and in $PATH right?

ADD REPLY
0
Entering edit mode

Ah, got another error a bit lower. You can change line 34 final_filename = "%s.bam" %(sorted_filename) to final_filename = sorted_filename

ADD REPLY
0
Entering edit mode

I took a screenshot for the terminal when I run samtools and also when I ran the modules check command as requested in the installation manual..

what really kills is that modules check states that samtools is installed !!!!!!!

also please check the test script.

please see the outputs of both commands, $ samtools & $ module_availability

test_miso.py

ADD REPLY
0
Entering edit mode

This is getting rather odd. Where did you install samtools? Where do you test? Is samtools in your $PATH?

ADD REPLY
0
Entering edit mode

I really appreciate your help and time ...

samtools is downloaded on the desktop

Kindly look at the attached photo ($PATH and /.bashrc) screenshot for the $PATH

ADD REPLY
0
Entering edit mode

Good, could you tell me the output of which samtools? What I find most odd is that finding samtools wasn't the problem earlier, but now it is. Makes me wonder whether starting over from the beginning would be worth it.

ADD REPLY
0
Entering edit mode

ya73@Bioinftop2016-06 ~ $ which samtools /home/ya73/Desktop/tuxedo/samtools-1.3.1//samtools

I gave up !!!! more than a week trying to troubleshoot, yet the people made it do not respond !!!

I deleted it and I'm gonna reinstall it.. awaiting for your advice.

ADD REPLY
0
Entering edit mode

Alright, run the test again and let's try solve the error message again. I had the feeling we almost did fix it :p If this doesn't work out quickly I might rewrite the damn script myself.

ADD REPLY
0
Entering edit mode

Unfortunately didn't work..

I was doubting that sorting the bam file is the problem so I copied the original sam file (in the test directory: c2c12.Atp2b1.sam) to the desktop and converted it to bam (it worked using samtools conversion command). then I run the sorting command :

$ samtools sort c2c12.Atp2b1.bam c2c12.Atp2b1.sorted

guess what happened ?!

samtools 1.3.1 had an error samtools 0.1.19 worked and sorted the bam file !

I went again to run the test using samtools-0.1.19 and it failed !!!!!!!!!!!!!

please find here a snapshot for the terminal using version 0.1.19!

Thank you

ADD REPLY
2
Entering edit mode

Samtools sort command has changed between those two versions. For v. 1.3.1 you need to use samtools sort -o c2c12.Atp2b1.sorted.bam c2c12.Atp2b1.bam

ADD REPLY
0
Entering edit mode

thanks. it worked.

Could it be the pysam failed to call samtools ?

pysam is downloaded on : /usr/local/lib/python2.7/dist-packages

miso is downloaded on : /usr/local/bin

please look at this part of the process which is realated to samtools and pysam:

Testing conversion of SAM to BAM... Executing: sam_to_bam --convert /usr/local/lib/python2.7/dist-packages/misopy/test-data/sam-data/c2c12.Atp2b1.sam /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output Converting SAM to BAM... - Executing: samtools view -Sbh /usr/local/lib/python2.7/dist-packages/misopy/test-data/sam-data/c2c12.Atp2b1.sam > /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.bam sh: 1: samtools: not found Sorting BAM file... /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam - Executing: samtools sort /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.bam -o /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam sh: 1: samtools: not found Indexing BAM... - Executing: samtools index /usr/local/lib/python2.7/dist-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam sh: 1: samtools: not found

ADD REPLY
1
Entering edit mode

From what I saw MISO doesn't use pysam, it just does lazy calls to os.system(samtools)

Okay, could you try editing the script to replace samtools everywhere with the absolute path to the samtools executable?

ADD REPLY
0
Entering edit mode

Finally - Finalmente

I seems like it worked..

BIG THANKS to you @WouterDeCoster and also to @genomax2

take a look at snapshots 1 and 2

ADD REPLY
1
Entering edit mode

Believe me, I'm as happy as you that we finally got through. Now go do some awesome research!

ADD REPLY
0
Entering edit mode

one week to start my graduation project ... that was so stressful .. especially that I'm not a python dude !

Thanks again

ADD REPLY
0
Entering edit mode

You can either try @Wouter's change or install an old version of samtools.

ADD REPLY
0
Entering edit mode

Can you verify that this file is present on your system ls -lh /Library/Python/2.7/site-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam ?

ADD REPLY
0
Entering edit mode

I got this message :

ls: cannot access /Library/Python/2.7/site-packages/misopy/test-output/sam-output/c2c12.Atp2b1.sorted.bam: No such file or directory

I don't know why it doesn't create this file, c2c12.Atp2b1.sorted.bam when I run the test command !!!!

ADD REPLY

Login before adding your answer.

Traffic: 2106 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6