Getting files from BaseMount (Using find command to loop through the symlinks)
3
0
Entering edit mode
3.7 years ago
SaltedPork ▴ 120

I'm using basemount on centos and want to download all the fastq.gz files. I want to use the find command to locate each zipped file and then download them in a bash loop. The problem is that the command (below) only returns File system loop detected errors, and Too many symlinks errors. Any ideas?

find -L . -type f -name 'fastq.gz'
basemount basespace • 1.5k views
ADD COMMENT
4
Entering edit mode
3.7 years ago
steve ★ 3.0k

I would suggest using rsync instead of a combination of find and cp. I have a workflow description here. The gist:

first use find to make a list of files to be copied

find "${basepace_projects}/${project_ID}" -name "*.fastq.gz" | sed -e "s|${basepace_projects}/||g" > "$project_files_list"

use that list as a filter pattern file for rsync (check the output with --dry-run first)

rsync --progress -vRt --files-from="$project_files_list" "$basepace_projects" "$output_directory"
ADD COMMENT
2
Entering edit mode
3.7 years ago
Paul ★ 1.4k

Hi, what about command: find path_bs -maxdepth 10 -name "*fastq.gz" -exec cp {} path/to/save \; Does it works?

ADD COMMENT
0
Entering edit mode

Hi, yes this is good, my mistake was using a . instead of the full path. I also used the exec command you used and it worked, so many thanks.

ADD REPLY
2
Entering edit mode

I moved this comment to an answer. You can accept (green check mark) it.

ADD REPLY
0
Entering edit mode

I was happy to help!

ADD REPLY
1
Entering edit mode
3.7 years ago
cp `find ./ -type f -name "*.fastq.gz"` ~/My_Fastq
ADD COMMENT
1
Entering edit mode

Does that command traverse/read symlinks?

ADD REPLY
1
Entering edit mode

When using basemount, it works no problem for me to retrieve fastq. AFAIK, basemount isn't (or shouldn't be symlinking), the files should appear normally, but are only fully retrieved from base space when a command is carried out on them. I wonder if invoking the -L flag in find is treating them all as symlinks, based on the way that basemount works?

ADD REPLY
0
Entering edit mode

I have also read that BaseMount isn't using symlinks, and yet when I use -L it worked, not sure what's going on there because they look like symlinks to me (in my terminal).

ADD REPLY

Login before adding your answer.

Traffic: 1800 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6