Has Anyone Got Plink Working With Gnu Parallel?

Question

[jvijai]

I usually use a script run.sh) to run my PLINK analysis (sort of like a makefile).
Has anyone managed to run PLINK with GNU Parallel? If so, how to use GNU Parallel with a script?
I tried the following but I cannot see 20 cores being run (by top).

>parallel -j 20 --progress  | ./run.sh

Any help appreciated. Thanks

Edits: Yes, I have 40 cores. Normally I would do like this:

Using the file MyCovarfile.raw (structure shown at the bottom) for Analysis of Pheno1 and covars Age, PC1, PC2, PC3.

plink --bfile myfile \  
--pheno MyCovarfile.raw \  
 --pheno-name Pheno1 \  
 --covar MyCovarfile.raw \  
 --covar-name Age-PC3 \  
 --logistic \  
 --adjust \  
 --qq-plot \  
 --out Pheno1_Age_PC3

Now, I can do the same with many different covariate models (and my run.sh is a list of such commands with different combinations of the covariates and phenotypes), but right now, they get excecuted on a single core one after the other serially.

Using the file MyCovarfile.raw (structure shown at the bottom) for Analysis of Pheno2 and covars Age, PC1, PC2, PC3.

plink --bfile myfile \  
 --pheno MyCovarfile.raw \  
 --pheno-name Pheno2 \  
 --covar MyCovarfile.raw \  
 --covar-name Age-PC3 \  
 --logistic \  
 --adjust \  
 --qq-plot \  
 --out Pheno2_Age_PC3

Using the file MyCovarfile.raw (structure shown at the bottom) for Analysis of Pheno2 and covars Age, PC1, PC2, PC3, PC4.

plink --bfile myfile \  
 --pheno MyCovarfile.raw \  
 --pheno-name Pheno1 \  
 --covar MyCovarfile.raw \  
 --covar-name Age-PC4 \  
 --logistic \  
 --adjust \  
 --qq-plot \  
 --out Pheno1_Age_PC4

Using the file MyCovarfile.raw (structure shown at the bottom) for Analysis of Pheno2 and covars Age, PC1, PC2, PC3, PC4.

plink --bfile myfile \  
 --pheno MyCovarfile.raw \  
 --pheno-name Pheno2 \  
 --covar MyCovarfile.raw \  
 --covar-name Age-PC4 \  
 --logistic \  
 --adjust \  
 --qq-plot \  
 --out Pheno2_Age_PC4

Hope this helps. http://pngu.mgh.harvard.edu/~purcell/plink/data.shtml#covar

Structure of MyCovarfile.raw

FID    IID    AFF    Pheno1    Pheno2    Pheno3    Pheno4    Pheno5    Pheno6    Pheno7    Bin    AGE    PC1    PC2    PC3    PC4  
0001    9542    1    1    1    1    1    1    1    1    1    8    -0.0053    -0.0046    0.0036    -0.0052  
0002    9606    1    1    1    1    1    1    1    1    1    3    -0.0052    -0.0045    0.0035    -0.0021  
0003    9702    2    1    1    1    1    1    1    1    1    3    -0.0045    -0.0041    0.0032    0.0016  
0004    9544    2    1    1    1    1    1    1    1    1    5    -0.0037    -0.0028    0.0032    0.0003

where FID, IID and AFF means familyID, Individual-ID and Affection status of the individual.

[Chris Miller]

let's start with the obvious - do you have 20 cores in your machine?

[tange]

I don't know how to normally run PLINK on 2 sets of data. Please show how you would run PLINK on to sets of data in serial. Also please state whether you have watched the intro videos:

• http://www.youtube.com/watch?v=OpaiGYxkSuQ
• http://www.youtube.com/watch?v=1ntxT-47VPA
• http://www.youtube.com/watch?v=fOX1EyHkQwc

[tange]

I don't know how to normally run PLINK on 2 sets of data. Please show how you would run PLINK on two sets of data in serial (if you cannot do that you most likely cannot use GNU Parallel). Also please state whether you have watched the intro videos:

[jvijai]

Hi Chris, Yes, I have 40 cores.

Ole Tange, Yes watched the intro videos and tried reading the manual too.

Normally I would do like this:

plink --bfile myfile  
      --pheno MyCovarfile.raw  
      --pheno-name Pheno1    
      --covar MyCovarfile.raw   
      --covar-name Age-PC3  
      --logistic   
      --adjust    
      --qq-plot    
      --out Pheno1_Age_PC3

Now, I can do the same with many different covariate models, but right now, they get excecuted on a single core one after the other serially.

Hope this helps.

[jvijai]

Yes, I have 40 cores, Yes, watched the video intro and read a buit through the manual without really understanding it.

[jvijai]

There is a really nice intro to installation of GNU Parallel here.

Answer 1

[Giovanni M Dall'Olio]

What is the content of your run.sh script?

To use GNU/Parallel effectively, you need to write a script that launches a single job (e.g. a single plink work) and then call it through the parallel syntax.

Example:

$: seq 1 20 | parallel -j 20 --progress run_single_plink_job.sh

# you can use {} to differentiate each call
$: seq 1 20 | parallel -j 20 --progress run_single_plink_job.sh --job_id {}
$: seq 1 22 | parallel -j 20 --progress run_single_plink_job.sh --chromosome {}

seq 1 20 is a common shortcut to tell parallel that run_single_plink_job.sh must be executed twenty times. For each of these calls, the variable {} will take a different value from 1 to 20. Example:

$: seq 1 20 | parallel "echo This is job {}"
This is job 1
This is job 2
This is job 3
This is job 4
...
This is job 20

Alternatively, you can provide arguments to parallel by putting them at the end of the command, separated by a ":::"

$: parallel "echo This is job {}" ::: {1..20}
This is job 1
This is job 2
This is job 3
This is job 4
....
This is job 20

or

$: parallel "echo Hello {}" ::: Marc Caius Julius Caesar
Hello Marc
Hello Caius
Hello Julius
Hello Caesar

If your run.sh script contains a series of commands that can be executed in parallel, each line being a single job, you can just cat it and pipe it to parallel:

$: cat run.sh | parallel -j 20 --progress
$: paste run.sh | parallel -j 20 --progress

Example:

$: cat >parallel_job.sh
sleep 2; echo "job1"
sleep 1; echo "job2"
echo "job3"

$: cat parallel_job.sh | parallel -u -j 3
job3
job2
job1

# alternatively, use paste
$: paste parallel_job.sh | parallel -u -j 3
job3
job2
job1

I suggest you to use htop to monitor the execution of the jobs. Moreover, remember that if you don't use the -u option, you won't see any output untill the job is finished.

Note: be sure that you are using GNU/Parallel and not parallel from moreutils, or the syntax will be different.

[brentp]

excellent intro on using parallel, thanks!

[Giovanni M Dall'Olio]

seq 1 20 simply prints numbers from 1 to 20. It is a common shortcut to tell parallel that he must execute the command 20 times. I have updated the answer.

[jvijai]

Thanks Giovanni, I will try this. Can you explain what the "seq 1 20" does in this instance?

Answer 2

[tange]

Let us assume you want to run this:

plink --bfile myfile \
--pheno MyCovarfile.raw \
--pheno-name Pheno1 \
--covar MyCovarfile.raw \
--covar-name Age-PC3 \
--logistic \
--adjust \
--qq-plot \
--out Pheno1_Age_PC3

But instead of Pheno1 you want Pheno1, Pheno2, Pheno3, Pheno4, Pheno5, Pheno6, Pheno7 and instead of PC3 you want PC1, PC2, PC3, PC4. And you want all combinations of those:

parallel plink --bfile myfile \
--pheno MyCovarfile.raw \
--pheno-name {1} \
--covar MyCovarfile.raw \
--covar-name Age-{2} \
--logistic \
--adjust \
--qq-plot \
--out {1}_Age_{2} \
::: Pheno1 Pheno2 Pheno3 Pheno4 Pheno5 Pheno6 Pheno7 \
::: PC1 PC2 PC3 PC4

Try prepeding plink with echo to see if this will execute what you want.

[jvijai]

Hi Tange, for most part that is correct, however as shown in the link to the PLINK site, the pheno-name and covar-name are columns within the MyCovarfile.raw. The way the analysis works is it picks a column specified as the pheno values for an analysis, and so you can have columns such as Pheno1, Pheno2, Pheno3 and so on in the same file (not separate file). Yes, I think I can split those columns to individual files and run the analysis you suggested. But the general feeling I get is that this is not a parallelizable job in the way PLINK treats it.. .

[tange]

Please give us 3 examples instead of describing those. You are making it much harder to give you a useful answer.

[jvijai]

Added more examples. The Covarfile structure is important to understand how PLINK calls the column for analysis. If GNU parallel can help with that part, it would be quite helpful. If this works.. I have another question with IMPUTE2 that would be a help to many many researchers I know..!

[jvijai]

I successfully used gtools to create gens file from ped file using GNU Parallel.. ! :)

[jvijai]

That worked perfectly! Thank you. Nice to know how parallel can take columns in a file as parameters. I had a related question. If your arguments such as Pheno1 Pheno2... Pheno7 are new lines in a file, how would the parallel command look like?