Download of FASTQ files from the European Nucleotide Archive (ENA) using Aspera
1
1
Entering edit mode
2.9 years ago
r.castellan ▴ 10

Hi there, I am trying to retrieve sequencing data from ENA using the Aspera client and following the tutorial:

The tutorial has been really helpful and I followed each step. I get a download.txt file which is a list of items looking like:

ascp -QT -l 300m -P33001 -i $HOME/Applications/Aspera Connect.app/Contents/Resources/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/SRR898/002/SRR8989702/SRR8989702_1.fastq.gz .  I then followed the tutorial and ran: while read LIST; do$LIST; done < download.txt


This leads to either two errors

Startup failed, exit
ascp: no remote host specified


Or:

-bash: $LIST: ambiguous redirect  I also tried running: $HOME/Applications/Aspera Connect.app/Contents/Resources/asperaweb_id_dsa.openssh


And get a permission denied error.

I'm new to this and haven't found a solution yet.

RNA-Seq sequencing • 3.2k views
0
Entering edit mode

This leads to either two errors

What are you doing differently to get the two errors?

0
Entering edit mode

If I hit return after do I tend to get the second error and if I don't the first one. Thanks!

0
Entering edit mode

(base) [nikhil@hpc04 dmd_SRA]$ascp -QT -l 300m -P33001 -i ~/.aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:vol1/fastq/ERR353/008/ERR3537668/ERR3537668_1.fastq.gz ./ ascp: Target address not available Startup failed, exit  Please guide me in this regard ADD REPLY 0 Entering edit mode The command runs smoothly on my machine. Can you try: ascp -QT -l 300m -P33001 -i$HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:vol1/fastq/ERR353/008/ERR3537668/ERR3537668_1.fastq.gz .

0
Entering edit mode

Just wanted to mention that the problem persists even after specifying the home directory with $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh. ADD REPLY 0 Entering edit mode Hello, did you ever figure out what caused the issue? I encountered the exact same issue while running on server (let's call this server A): (fuc) [sbslee@bdcm03 ~]$ .aspera/connect/bin/ascp -QT -l 100m -P33001 -k 1 -i .aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:vol1/run/ERR323/ERR3239277/NA06986.final.cram .
Startup failed, exit


The funny thing is, the exact same command works just fine in my local Linux PC:

(fuc) [sbslee@localhost Desktop]$.aspera/connect/bin/ascp -QT -l 100m -P33001 -k 1 -i .aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:vol1/run/ERR323/ERR3239277/NA06986.final.cram . NA06986.final.cram 10% 1539MB 1.3Mb/s 38:04 ETA  So I was thinking it might be a server-related issue, but I can't think of any reason why. I also confirmed that in a different server (call it server B), the command works OK. So it got to be an issue in server A, but I didn't change any configuration, etc., so I'm not sure why I'm seeing this problem. Server A is connected to Internet for sure (I can download stuff such as $ git clone). Any help would be much appreciated!

0
Entering edit mode

I would talk to the server admins and discuss potential problems. It is obviously a specific problem with connection on that particilar server. Maybe some kind of firewall that is in place or ports that are not open, idk.

1
Entering edit mode
2.9 years ago
ATpoint 65k

Thanks for reporting the issue. I could reproduce the error and think it is due to the whitespace in the name of the Aspera folder. Could you try to rename Aspera Connect into something without whitespace e.g. AsperaConnect.app, then adjust the download command accordingly and try again. I guess that is Mac-specific because (wisely) Linux does not use a whitespace in the folder name. I Did it on my machine and the errors disappeared, still I got a time-out message. Not sure if this is due to an error at ENA which happens at times, so if you cannot download, maybe wait a day and try again. Connection errors are rather frequent in my experience. I will double check when I've access to my workstation tomorrow.

==> Edit: The whitespacing is not an issue if properly escaped. I updated the command in the tutorial so that the awk command does not mess it up. It should be fine towards that initial error you got.

0
Entering edit mode

Thanks a lot, will try that now and update.

0
Entering edit mode

Just checked on both Linux and Mac and the same problem, I think this is due to server problems at ENA. I will update the tutorial towards this whitespace problem, thanks again for reporting.

0
Entering edit mode

Oh right! Sorry only saw your reply now. No worries, thanks for helping

0
Entering edit mode

So I changed it and I get a new error:

ascp: Private key file not found at path /Users/Raphael/.ssh/$HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh  I'm novice, not sure it was worth pointing out, but does this path look strange? I also ran the following: $HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh


I get the permission denied error still

0
Entering edit mode

Is there a file at the location? What do you get if you do ls -l $HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh? If your SSH key is in some other location then you will need to use that path. Are you running macOS catalina BTW? ADD REPLY 0 Entering edit mode I get: -rw-rw-r--@ 1 Raphael staff 668 8 Nov 15:12 /Users/Raphael/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh I am running Mojave still ADD REPLY 0 Entering edit mode It is normal that you get the permission denied when calling this file, it is also not an executable so no point in trying to run it. Your problem is here: /Users/Raphael/.ssh/$HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh

Remove /Users/Raphael/.ssh/ from it as $HOME is already there. Try and it should work unless there is something odd going on with your environment: ascp -QT -l 300m -P33001 -i$HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/SRR898/002/SRR8989702/SRR8989702_1.fastq.gz .

0
Entering edit mode

So when I ran the code by itself it worked. However, it fails when I run:

while read LIST; do
$LIST; done < download.txt  when I head download.txt I get: ascp -QT -l 300m -P33001 -i$HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/SRR898/002/SRR8989702/SRR8989702_1.fastq.gz .

0
Entering edit mode

What is the error mesage?

0
Entering edit mode

sorry I had to wait 6 hours as I had put up 5 posts...

the error message is the same as before:

ascp: Private key file not found at path /Users/Raphael/.ssh/$HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh  ADD REPLY 1 Entering edit mode It is puzzling that your $HOME variable is expanding correctly to /Users/Raphael when you use it here ls -l $HOME/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh but when you try to run the loop it seems to change to something strange /Users/Raphael/.ssh/$HOME and does not properly expand.

What do you get when you echo $HOME? What happens if you simply try one file like this? Does this work? ascp -QT -l 300m -P33001 -i /Users/Raphael/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/SRR898/002/SRR8989702/SRR8989702_1.fastq.gz .  ADD REPLY 0 Entering edit mode So echo$HOME returns:

/Users/Raphael


The following worked, so I guess it's a problem in the loop which adds the extra bits?

ascp -QT -l 300m -P33001 -i /Users/Raphael/Applications/AsperaConnect.app/Contents/Resources/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/SRR898/002/SRR8989702/SRR8989702_1.fastq.gz .

1
Entering edit mode

It is indeed odd that $HOME seems to improperly expand. I used $HOME in the tutorial commands to provide a generic command, but glad it works now. You can simply edit the awk command that created the download.txt by replaceing the \$HOME with /Users/Raphael/. This should be independent of what the loop does as it only calls what is inside the download list.

0
Entering edit mode

Great! Thanks so much for your help!

0
Entering edit mode

It's running nicely, the session is timing out every now and then on certain files, is there a way to automatically determine which ones have failed and re-run them? Maybe matching the output files with the download.txt file?