Question: Script makes different file then the manual command, but the command is the same
0
gravatar for stan.aanhane
5 weeks ago by
stan.aanhane30
stan.aanhane30 wrote:

Hi,

I am working on a pipeline in a script, and want to have a specific section of my blast file in a txt file. But for some reason Linux can not work with the txt file. This is the code i use for this part:

blastn -db /exports/minor_data/BNGSA/refgenome/lclav.db -outfmt 6 -max_target_seqs 5 -query 50biggest.fasta > blastresults.txt
awk -F ' ' '{if ($3 >= "50.000" && $11 <="0.0") print $0}' blastresults.txt > Results_50_0.0.txt

And for some reason i get the error:

awk: cannot open blastresults.txt (No such file or directory).

The file is there, with the same size as the file when i do this manually. The code works manually, but when i use it in a script, it can not find the blastresults.txt file.

After some time looking I found that the text files are called 'blastresults.txt'$'\r' instead of blastresults.txt. And I have no clue why Linux makes this file, and how i fix this problem.

Can someone help me out?

Thanks!

awk pipeline linux • 315 views
ADD COMMENTlink modified 5 weeks ago • written 5 weeks ago by stan.aanhane30

Hope you have printed output location and awk execution path.

ADD REPLYlink written 5 weeks ago by cpad011213k

Well when i use the blastn command normal i will get the .txt file and the awk command works. But when i use it in the script, it says it can not find the file produced in the blastn command. But it is there, but it says TXT-file instead of textile behind the file. They are both the same size, so I'm really confused.

ADD REPLYlink written 5 weeks ago by stan.aanhane30

While a redirect > may work, you should use -out blastresult.txt option since that is the proper way to specify an output file.

ADD REPLYlink written 5 weeks ago by genomax87k

can you try posting your script here?

ADD REPLYlink written 5 weeks ago by cpad011213k
1
    #! /bin/sh
echo "cp /exports/minor_home/reads/test/NINovo/contigs.fasta /exports/minor_home/reads/test"
cp /exports/minor_home/reads/test/NINovo/contigs.fasta /exports/minor_home/reads/test
wait
echo "awk '/^>/ {printf("\n%s\n",$0);next; } { printf("%s",$0);}  END {printf("\n");}' < contigs.fasta > contigs1line.fasta"
awk '/^>/ {printf("\n%s\n",$0);next; } { printf("%s",$0);}  END {printf("\n");}' < contigs.fasta > contigs1line.fasta
wait
echo "head -101 contigsoneline.fasta > 50biggest.fasta"
head -101 contigs1line.fasta > 50biggest.fasta
wait
echo "sed -i 's/NODE/lclav_config/g' 50biggest.fasta"
sed -i 's/NODE/lclav_config/g' 50biggest.fasta
wait
echo "sed -i 's/_/ /g' 50biggest.fasta"
sed -i 's/_/ /g' 50biggest.fasta
wait
echo "sed -i 's/length/lengte/g' 50biggest.fasta"
sed -i 's/length/lengte/g' 50biggest.fasta
wait

#Deelopdracht 8
echo "Deelopdracht 8"
echo "blastn -db /exports/minor_data/BNGSA/refgenome/lclav.db -outfmt 6 -max_target_seqs 5 -query 50biggest.fasta > BLASTResults.txt"
blastn -db /exports/minor_data/BNGSA/refgenome/lclav.db -outfmt 6 -max_target_seqs 5 -query 50biggest.fasta -out blastresults.txt
wait
echo "awk -F ' ' '{if ($3 >= "50.000" && $11 <="0.0") print $0}' blastresults.txt > Results_50_0.0.txt"
awk -F ' ' '{if ($3 >= "50.000" && $11 <="0.0") print $0}' exports/minor_home/reads/test/blastresults.txt > Results_50_0.0.txt
wait
cat Results_50_0.0.txt
echo "Pipeline Finished"
ADD REPLYlink written 5 weeks ago by stan.aanhane30
1

Looks like you are missing the first / in your path exports/minor_home/reads/test/blastresults.txt on awk command line.

ADD REPLYlink written 5 weeks ago by genomax87k

unfortunately, this doesnt solve the problem... Same error. I think for some reason, when i use the blast command with the script the .txt file have different properties then the one is done manually. As you can see in the picture the one above(done manually) looks like a text file, and works. But the other one looks a bit odd, and doesn't work...

Picture with the two files

ADD REPLYlink written 5 weeks ago by stan.aanhane30
  1. Can you try running ./script.sh or bash script.sh on small set and let us know if error still exists?
  2. Try also running dos2unix on the script file before execution.
  3. Can you edit this header line and see? from #! /bin/sh to #! /bin/bash (or wherever bash is installed.
ADD REPLYlink modified 5 weeks ago • written 5 weeks ago by cpad011213k
  1. I have tried this, with just 6 of the 50 configs, and the error still exists.
  2. At this point dos2unix is not installed, I have to wait until i am able to install this (I'm working on a VM) 3 This doesn't make a different either
ADD REPLYlink written 5 weeks ago by stan.aanhane30

OK,

let's take this from the top:

  • can you do (& post result) pwd in the directory where you see (or think) the blastresults.txt file is
  • can you do (and post the result) of doing ls /exports/minor_home/reads/test/blastresults.txt
  • can you do (& post result) pwd where you execute this script
ADD REPLYlink written 5 weeks ago by lieven.sterck8.3k
  • /exports/minor_home/reads/test
  • When ls command is used, this error appears: ls: cannot access '/exports/minor_home/reads/test/blastresults.txt': No such file or directory
  • /exports/minor_home/reads/test

The locations of the files look correct to me. The only thing is that i am not able to open the file in linux.

ADD REPLYlink written 5 weeks ago by stan.aanhane30

Just to ensure: there is actual output in the blastresults.txt?

ADD REPLYlink written 5 weeks ago by cschu1812.4k

Yes there is. When i open the file, with Notepad, I can see the blast results. But my PC also does not recognise this file, I have to select open with Notepad to see the results. When I use the blast command manually, the pc recognises the file.

ADD REPLYlink written 5 weeks ago by stan.aanhane30

Sounds like you are accessing this file on unix partition from windows side using Notepad. That may be the cause of issues you are observing.

This used to be an issue once upon a time. This was supposed to have been fixed recently. If you don't try to open the file using Notepad do you still get the error?

ADD REPLYlink modified 5 weeks ago • written 5 weeks ago by genomax87k

When I open the file in Windows, it will open. I have to select notepad, but that's it. So no error, but the file does not change in Linux, the file cant be found. When I use the command on its own, the file looks different. If I open it in windows, it automatically opens, and I don't have to select notepad. And the file can be used in the next step.

ADD REPLYlink written 5 weeks ago by stan.aanhane30

I had a look at the files. Copied them both to windows, and for some reason, the script makes a .txt_ file instead of a .txt file. So that's the issue, but i don't see the reason why its making a .txt_ file

ADD REPLYlink modified 5 weeks ago • written 5 weeks ago by stan.aanhane30

Not sure why that is happening. Closest issue that I could find where filenames got a trailing _ in the name is this thread but it is not directly applicable in your case.

You could try putting a mv blastresults.txt_ blastresults.txt before the awk command to see if it fixes the problem.

ADD REPLYlink modified 5 weeks ago • written 5 weeks ago by genomax87k

I tried, but the same result. Only at a different place. The error appears after the mv code now.

I was thinking, I can make the file with touch first, and add the data to this file. But even touch makes a .txt_ file. But this only shows when its copied to windows. When in Linux it still shows just .txt.

I am using Notepad++ for this script. Can the problem be within the program itself?

ADD REPLYlink written 5 weeks ago by stan.aanhane30
  1. Make sure you don't have the results file open in Windows (e.g. reboot)
  2. Delete the results file
  3. Confirm with ls that the results file no longer exists in that dir
  4. Run your script
ADD REPLYlink modified 5 weeks ago • written 5 weeks ago by 5heikki8.9k

This dindt fix the problem. But something else drew my attention. The weird text files are called: 'blastresults.txt'$'\r' when I type in ls. But I don't know why Linux makes these type of files?

ADD REPLYlink modified 5 weeks ago • written 5 weeks ago by stan.aanhane30

how are you running this script? From your windows machine/environment?

best would be to export that script to your linux environment and then do run dos2unix on it . (or on the output files generated by your script)

what you just descried is a typical thing when opening/running windows edited files in a linux enivronment (== differences in line endings between dos/windows and unix)

ADD REPLYlink modified 5 weeks ago • written 5 weeks ago by lieven.sterck8.3k

I am woking on my windows pc. Due to COVID-19 we are forced to work from home, so we made it possible, with the use of a VM(virtual machine) to work on the pc's at work, on our pc's at home. So I am working in Linux pc at a different location, from my own windows laptop.

I will have a look with dos2unix, but I think we need to install this, so I have to contact the right person.

If there is an other solution, that would be great!

ADD REPLYlink written 5 weeks ago by stan.aanhane30
1

you can custom implement what dos2unix does indeed, however, more efficient would be to just download the binary from the internet and install it in you home folder or such.

if you are working remotely you might need to look into an SSH terminal option, such as putty, mobaXterm, ... these will provide you access to your linux machines at the lab.

ADD REPLYlink written 5 weeks ago by lieven.sterck8.3k

Thank you, I will have a look at that.

We indeed work from a SSH terminal called Bitvise.

ADD REPLYlink written 5 weeks ago by stan.aanhane30

that should be OK as well ...

in any case: top tip: when switching data/scripts/... from windows to linux or viceversa do never consider it will work as such ;-)

if in any way possible it would be best to choose one environment (linux ;) ) and stick to it :) , thus try to avoid opening scripts in notepad and then run them in linux

ADD REPLYlink modified 5 weeks ago • written 5 weeks ago by lieven.sterck8.3k

so we made it possible, with the use of a VM(virtual machine) to work on the pc's at work, on our pc's at home. So I am working in Linux pc at a different location, from my own windows laptop.

In this instance, your windows laptop becomes a means to work on the remote server. I am not sure why you are using a VM at all. You can use a simple SSH program like MobaXTerm to access your linux machine at work.

Then you said this in a following comment.

We indeed work from a SSH terminal called Bitvise.

So perhaps you don't use a real VM and are confusing a local terminal with a virtual machine. It appears that BitVise program has an secure FTP/scp graphical interface. You may be accessing files that are on your server via what looks like windows explorer file interface.

As @lieven said, if you only use the server for all of your work this odd behavior should not happen.

ADD REPLYlink modified 5 weeks ago • written 5 weeks ago by genomax87k
1

Well, I fixed the problem. It's the output of the Notepad++ file. It was still in windows format and added ^M to every line. I converted the line to Unix format, at it works fine now. Feel stupid to overlook this... But thanks for the help!

ADD REPLYlink written 5 weeks ago by stan.aanhane30

when i use it in a script, it can not find the blastresults.txt file.

Add full explicit path to the result file. Especially if you are running the script in a separate shell.

ADD REPLYlink written 5 weeks ago by genomax87k

I tried this, but it gives the same error. I think the problem is in the file it makes. I ran it one time with the script(with -out) and one time manually. I added a picture so you can see how the files look. the top blastresults.txt file is the one made manually, the other is made with the script. The text files as shown in the Linux pc

ADD REPLYlink modified 5 weeks ago • written 5 weeks ago by stan.aanhane30

How is it possible to have two files with the same name in the same directory?

ADD REPLYlink written 5 weeks ago by 5heikki8.9k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1626 users visited in the last hour