Create files in multiple directories via for loop
0
0
Entering edit mode
18 months ago
vinayjrao ▴ 210

Hi, I have around 500 directories, and I wish to first create an empty text file within each of these and then process my data files contained within sub-directories and redirect the output into the respective text files.

An example of my existing folders - exisiting/main/sub1/sub2/sub3/sub4/sub5/sub6/data.tsv.bz2

What I wish to do - create main/sub1.txt & extract data.tsv.bz2 > main/sub1.txt

I have figured how to redirect my output into the desired files, but I am facing issues with creating the .txt files.

Tha command tried for the same is for i in */; do touch $i.txt; done. This creates a file named .txt within the main folder. So I also tried for i in */ ; do cd "$i"; for j in */; do ( touch $j.vcf ) done; done, which gave me an error saying

bash: cd: main/: No such file or directory

Since I have typed the command from my existing directory, it first identifies the main directory. I want to be able to provide the name of the sub1 directory while creating sub1.txt

Is there any way to get around this problem?

Thanks in advance

shell linux • 1.3k views
ADD COMMENT
1
Entering edit mode

You didn't mention where the sub directories are. Here is the solution to OP problem:

$ find . -type f -name "*.bz2"                                  
./exisiting/main/sub1/sub2/sub3/sub4/sub5/sub6/data.tsv.bz2

$ parallel --plus --dry-run 'bzip2 -dc {} > main/{/.}' ::: $(find . -type f -name "*.bz2") 

bzip2 -dc ./exisiting/main/sub1/sub2/sub3/sub4/sub5/sub6/data.tsv.bz2 > main/data.tsv
ADD REPLY
0
Entering edit mode

As mentioned by @cpad0112 The problem statement is not very clear.

#Create random subdirectories for test
$mkdir sub{1..6}
#list the contents of the current directory
$ls -l
data.tsv.bz2
sub1/
sub2/
sub3/
sub4/
sub5/
sub6/

Probable solution

#extract the data file in each of the subdirectories with the directory name as file name
$for dir in `ls -d */`;do bzcat data.tsv.bz2 > ${dir}/${dir%/}.txt;done
$tree
├── data.tsv.bz2
├── sub1
│   └── sub1.txt
├── sub2
│   └── sub2.txt
├── sub3
│   └── sub3.txt
├── sub4
│   └── sub4.txt
├── sub5
│   └── sub5.txt
└── sub6
    └── sub6.txt
ADD REPLY
0
Entering edit mode

Thank you both for your response. I am sorry that my problem was not very clear. I am working from the directory named existing/, which contains subdirectories such as main/sub1/sub2/sub3/sub4/sub5/sub6/. sub6/ contains the file data.tsv.bz2. I wish to create a file called sub1.txt in the main/ directory. Technically, I would have said touch main/sub1.txt, but since I have 500 directories, say for example main1/, main2/, main13/, main27/ etc, I wish to do this using a for loop such as for i in */ ; do cd "$i"; for j in */; do ( touch $j.vcf ) done; done. Here, I tried to identify main/ with $i and sub1/ with $j

ADD REPLY
1
Entering edit mode

is sub1/sub2/sub3/sub4/sub5/sub6/ a fixed string? As I understand, main is not (main 1..main 500). It would help if you could post directory tree.

ADD REPLY
0
Entering edit mode

Starting from the directory existing/

├── main/

│          └── sub1/

│                     └── sub2/

│                                 └── sub3/

│                                            └── sub4/

│                                                       └── sub5/

|                                                                  └── sub6/

|                                                                              └── data.tsv.bz2

ADD REPLY
1
Entering edit mode

Since OP information is incomplete, the best you can do is:

 1. cd existing
 2. find . -type f -name "*.bz2" | while read line; do echo bzip2 -dc $line ">"  ${line%%/sub2/*}.txt;done

output should look like this. Bash command above dry-runs the code:

bzip2 -dc ./main1/sub1/sub2/sub3/sub4/sub5/sub6/data.tsv.bz2 > ./main1/sub1.txt
bzip2 -dc ./main2/sub1/sub2/sub3/sub4/sub5/sub6/data.tsv.bz2 > ./main2/sub1.txt

Dry run with parallel:

$ cd existing
$ parallel --dry-run 'bzip2 -dc {} > {=s/\/sub2.*//=}.txt' ::: $(find . -type f -name "*.bz2")

bzip2 -dc ./main1/sub1/sub2/sub3/sub4/sub5/sub6/data.tsv.bz2 > ./main1/sub1.txt
bzip2 -dc ./main2/sub1/sub2/sub3/sub4/sub5/sub6/data.tsv.bz2 > ./main2/sub1.txt
ADD REPLY
0
Entering edit mode

This works fine. Thanks a lot. You may move it to an answer so that I can accept it. Thank you once again :)

ADD REPLY
0
Entering edit mode

So, will this extract data.tsv.bz2 into sub1.txt?

I would also like to know if I can create files with touch in the way I mentioned in the post?

Thanks once again.

ADD REPLY
0
Entering edit mode

Take a back up of one file and run one command from the dry-run on the backed-up directory/file. you would know if it works or not.

ADD REPLY

Login before adding your answer.

Traffic: 840 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6