Question: Tophat Error : Oserror: [Errno 2] No Such File Or Directory
3
gravatar for Sukhdeep Singh
5.0 years ago by
Sukhdeep Singh8.7k
Germany
Sukhdeep Singh8.7k wrote:

Hi,

I am running tophat and getting a lot of different errors, I could understand some of them, but this one, I don't know the reason. So, the error is

OSError: [Errno 2] No such file or directory: 'test_yo/tmp/left_kept_reads.m2g_um.candidates_and_unspl.bam'

This appears, after mapping when the reporting of output tracks is done. Command used :

tophat --bowtie1 -G /projects/grub/archive/Mus_musculus/UCSC/mm9/Annotation/Genes/genes.gtf -o test_yo /biodata/biodb/ABG/genomes/bowtie/mm9 test/test_data/reads_1.fq

Please let me know, if you encountered this or know a solution. Same question appears here but no answer yet.

My tophat.log :

[2012-04-16 18:49:10] Beginning TopHat run (v2.0.0)
-----------------------------------------------
[2012-04-16 18:49:10] Checking for Bowtie
          Bowtie version:     0.12.7.0
[2012-04-16 18:49:10] Checking for Samtools
        Samtools version:     0.1.17.0
[2012-04-16 18:49:10] Checking for Bowtie index files
[2012-04-16 18:49:10] Checking for reference FASTA file
    Warning: Could not find FASTA file /biodata/biodb/ABG/genomes/bowtie/biodata/biodb/ABG/genomes/bowtie/mm9.fa
[2012-04-16 18:49:10] Reconstituting reference FASTA file from Bowtie index
  Executing: /state/partition1/apps/bin/bowtie-inspect /biodata/biodb/ABG/genomes/bowtie/mm9 > test_yo/tmp/mm9.fa

[2012-04-16 18:51:12] Generating SAM header for /biodata/biodb/ABG/genomes/bowtie/mm9
    format:         fastq
    quality scale:     phred33 (default)
[2012-04-16 18:51:32] Reading known junctions from GTF file
[2012-04-16 18:51:35] Preparing reads
     left reads: min. length=75, count=100
[2012-04-16 18:51:35] Creating transcriptome data files..
[2012-04-16 18:51:50] Building Bowtie index from genes.fa
[2012-04-16 18:56:57] Mapping left_kept_reads against transcriptome genes with Bowtie 
[2012-04-16 18:56:58] Converting left_kept_reads.m2g to genomic coordinates (map2gtf)
[2012-04-16 18:57:00] Resuming TopHat pipeline with unmapped reads
[2012-04-16 18:57:00] Mapping left_kept_reads.m2g_um against mm9 with Bowtie 
[2012-04-16 18:57:01] Mapping left_kept_reads.m2g_um_seg1 against mm9 with Bowtie (1/3)
[2012-04-16 18:57:02] Mapping left_kept_reads.m2g_um_seg2 against mm9 with Bowtie (2/3)
[2012-04-16 18:57:03] Mapping left_kept_reads.m2g_um_seg3 against mm9 with Bowtie (3/3)
[2012-04-16 18:57:04] Searching for junctions via segment mapping
[2012-04-16 18:58:38] Retrieving sequences for splices
[2012-04-16 19:00:14] Indexing splices
[2012-04-16 19:00:33] Mapping left_kept_reads.m2g_um_seg1 against segment_juncs with Bowtie (1/3)
[2012-04-16 19:00:41] Mapping left_kept_reads.m2g_um_seg2 against segment_juncs with Bowtie (2/3)
[2012-04-16 19:00:49] Mapping left_kept_reads.m2g_um_seg3 against segment_juncs with Bowtie (3/3)
[2012-04-16 19:00:57] Joining segment hits
[2012-04-16 19:02:31] Reporting output tracks
Traceback (most recent call last):
  File "/share/apps/bin/tophat", line 3778, in ?
    sys.exit(main())
  File "/share/apps/bin/tophat", line 3754, in main
    os.remove(m)
OSError: [Errno 2] No such file or directory: 'test_yo/tmp/left_kept_reads.m2g_um.candidates_and_unspl.bam'

If you want run.log, I can also provide that.

I can reproduce the error using bowtie2 (default in tophat2) as well and the test data used is from the tophat website.

Cheers

tophat error rna-seq • 11k views
ADD COMMENTlink modified 4.4 years ago • written 5.0 years ago by Sukhdeep Singh8.7k

My run with tophat (v2.0.0 with bowtie2) coincides exactly with what your output shows, except that after reporting output tracks, its says run complete ... seconds elapsed. So, I don't think there is any issue with tophat run until there. Which OS are you working in? Did you try copying your fasta to bowtie-index folder (with same name as bowtie-index)...?

ADD REPLYlink written 5.0 years ago by Arun2.2k
2

Also try running with the --keep-tmp option (just a thought as your error seems to be from the tmp directory created by tophat).

ADD REPLYlink written 5.0 years ago by Arun2.2k
1

Amazing Arun --keep-tmp solves it :)

ADD REPLYlink written 5.0 years ago by Sukhdeep Singh8.7k

I'm glad that fixed it! :)

ADD REPLYlink written 4.9 years ago by Arun2.2k

Had the same issue and tophat didn't generate the accepted_hits.bam.. So I am running now with the --keep-tmp option! fingers crossed!

ADD REPLYlink written 22 months ago by rodrigo.duarte8810

I am experiencing pretty much exactly this error. However, adding --keep-tmp did not solve the problem.

I am running tophat on a campus cluster machine with shared filesystem, and I wonder if the tmp files are getting removed for some reason related to that.

My command line:

tophat2 -p12 -o tophat-N3-ensembl -N3 --read-edit-dist=3 -G cf3ensembl.1.79.gtf --keep-tmp --transcriptome-index=cf3ensembl-transcriptome filtered.fastq 2> tophat-N3-err.txt

 

Output:

...

[2015-11-03 14:56:49] Joining segment hits
[2015-11-03 15:01:00] Reporting output tracks
    [FAILED]
Error: [Errno 2] No such file or directory: '/home/hekman2/scratch/transcriptome
/amygdala/aligned/tophat-N3-ensembl-530/tmp/accepted_hits0_sorted.bam'
Found 151163 junctions from happy spliced reads

 

Version: tophat2.1.0.

Interestingly, most of the files that I process complete just fine. About 1/6 of them have the above error. When I rerun them, they generally get the error again, although once or twice on the third try they succeeded. So maybe something to do with running out of memory, somehow?

 

Any suggestions would be appreciated!

 

Jessica

ADD REPLYlink written 17 months ago by jessica.hekman0

Yeah, its possible. Can you try to run it on a local machine.
I am unable to help in this regard as I am not sure what is the exact problem.

ADD REPLYlink written 17 months ago by Sukhdeep Singh8.7k

I can't run it on a local machine until probably next week, but I'll try it and report back.

I did run it on the cluster with 8 instead of 12 processors. Result: all the files that had caused errors succeeded (yay). I concluded that I had solved the problem and moved on to the next set of files, using 8 processors each. Result: new error (though clearly similar):

[2015-11-04 14:15:09] Indexing splices
Building a SMALL index
Traceback (most recent call last):
  File "/scratch/users/hekman2/bin/tophat", line 4095, in <module>
  File "/scratch/users/hekman2/bin/tophat", line 4061, in main
  File "/scratch/users/hekman2/bin/tophat", line 3706, in spliced_alignment
  File "/scratch/users/hekman2/bin/tophat", line 1161, in nonzeroFile
  File "/usr/lib64/python2.6/subprocess.py", line 642, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.6/subprocess.py", line 1238, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

 

So, again, an error about a missing file.

 

I'll report back when I have a chance to try these on a local machine.

ADD REPLYlink written 17 months ago by jessica.hekman0
2
gravatar for Sukhdeep Singh
5.0 years ago by
Sukhdeep Singh8.7k
Germany
Sukhdeep Singh8.7k wrote:

So, to conclude the tophat errors for future reference from my experience

Error1

Above mentioned error, can be solved using a parameter --keep-tmp. Thanks to Arun

Error2

Searching for junctions via segment mapping

[FAILED] Error: segment-based junction search failed with err =127 The exact error cause can be seen in the segment.juncs file present in the tmp dir under your output dir but for me the probable cause was multithread problem. Tophat/Cufflinks are not MPI (Message Parsing protocol) programs, so they can't be distributed to more than one node. So, submitting to a single node with whatever cpu's it has max limit is the max it has solves it. Another reason, is exporting the LD lib path using export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/share/apps/lib/ in your qsub or cluster execution script.

Also, there are logs stored in your tmp dir, so check for the last command in the run.log which is big and try running it on the console and you'll see where the error is. Same error happens with err=-9

Error3

Searching for junctions via segment mapping
[FAILED]
Error: segment-based junction search failed with err =-6

Most likely reason is you run out of memory

Error4

Bowtie problems, If you update Tophat to latest version i.e.2, be sure to update bowtie 1 to bowtie 2 which is called by default by tophat. If its not there, you'll get error and for using bowtie2, you should also fetch the bowtie2 indexes which are different than bowtie1.

Another error, occurs if your annotation file (genes.gtf) and the your bowtie index are from different sources (NCBI vs UCSC vs Ensembl), because they have a difference in how the chromosomes are named. Also, be sure your fastq is proper fastq else you get an execution error

Error5

As the user RT reported,

[2014-01-14 00:24:01] Searching for junctions via segment mapping

[FAILED]
Error: segment-based junction search failed with err =127

/shared/apps/tophat/2.0.10/bin/segment_juncs: error while loading shared libraries: libboost_thread.so.1.51.0: cannot open shared object file: No such file or directory

This is because of absence or old version of Boost C++ libraries.

Copying steps from the website
Building TopHat from source

In order to build TopHat2 you must have the following installed on your system:

  • the Boost C++ libraries (we recommend version 1.47 or higher so you can use it for building Cufflinks as well) 

Note: Starting with version 2.0.13 TopHat no longer requires the user to have the SAMtools library (libbam.a), headers or even the samtools program installed, as TopHat2 comes pre-packaged with a stable version of SAMtools which is known to work well with TopHat. Also, TopHat uses the SeqAn library that comes with the TopHat2 source code distribution. It is not necessary for the SeqAn library to be installed separately in order to run TopHat.

Building Boost

  1. Download a recent Boost source tarball, unpack it and cd to the newly unpacked Boost source directory.
  2. Prepare the build: 
    ./bootstrap.sh
  3. Build Boost. Note that you can specify where to install Boost with the --prefix option, which specifies the base path (prefix) where sub-directories ./include/ and ./lib/ will host the Boost headers and library files respectively. The default Boost installation directory prefix is /usr/local. Take note of this installation directory (if you specify your own) because you will need to provide it to the --with-boost option of TopHat's./configure script. Run the build and install command:

    ./bjam --prefix=<YOUR_BOOST_INSTALL_DIRECTORY> link=static \
     runtime-link=static stage install

I hope it helps someone.

Cheers

ADD COMMENTlink modified 22 months ago • written 5.0 years ago by Sukhdeep Singh8.7k

Hi Sukhdeep,

Have you ever encountered the following errors:

[2014-01-14 00:24:01] Searching for junctions via segment mapping

[FAILED]

Error: segment-based junction search failed with err =127

/cm/shared/apps/tophat/2.0.10/bin/segment_juncs: error while loading shared libraries: libboost_thread.so.1.51.0: cannot open shared object file: No such file or directory

ADD REPLYlink modified 3.2 years ago • written 3.2 years ago by RT280

Hi RT, 
I am quite a begginer to RNA-seq data and I am getting the same error [Error: segment-based junction search failed with err =127]. Did you find solution in the end? 

Cheers

ADD REPLYlink written 2.0 years ago by edin.hamzic0

Hi, coming back too late for this, but might help someone.

I will update my answer

ADD REPLYlink modified 22 months ago • written 22 months ago by Sukhdeep Singh8.7k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1028 users visited in the last hour