Getting files from BaseMount (Using find command to loop through the symlinks)
3
0
Entering edit mode
3.7 years ago
SaltedPork ▴ 120

I'm using basemount on centos and want to download all the fastq.gz files. I want to use the find command to locate each zipped file and then download them in a bash loop. The problem is that the command (below) only returns File system loop detected errors, and Too many symlinks errors. Any ideas?

find -L . -type f -name 'fastq.gz'

basemount basespace • 1.5k views
4
Entering edit mode
3.7 years ago
steve ★ 3.0k

I would suggest using rsync instead of a combination of find and cp. I have a workflow description here. The gist:

first use find to make a list of files to be copied

find "${basepace_projects}/${project_ID}" -name "*.fastq.gz" | sed -e "s|${basepace_projects}/||g" > "$project_files_list"


use that list as a filter pattern file for rsync (check the output with --dry-run first)

rsync --progress -vRt --files-from="$project_files_list" "$basepace_projects" "\$output_directory"

2
Entering edit mode
3.7 years ago
Paul ★ 1.4k

Hi, what about command: find path_bs -maxdepth 10 -name "*fastq.gz" -exec cp {} path/to/save \; Does it works?

0
Entering edit mode

Hi, yes this is good, my mistake was using a . instead of the full path. I also used the exec command you used and it worked, so many thanks.

2
Entering edit mode

I moved this comment to an answer. You can accept (green check mark) it.

0
Entering edit mode

I was happy to help!

1
Entering edit mode
3.7 years ago
cp find ./ -type f -name "*.fastq.gz" ~/My_Fastq

1
Entering edit mode

1
Entering edit mode

When using basemount, it works no problem for me to retrieve fastq. AFAIK, basemount isn't (or shouldn't be symlinking), the files should appear normally, but are only fully retrieved from base space when a command is carried out on them. I wonder if invoking the -L flag in find is treating them all as symlinks, based on the way that basemount works?

0
Entering edit mode

I have also read that BaseMount isn't using symlinks, and yet when I use -L it worked, not sure what's going on there because they look like symlinks to me (in my terminal).