Question: how to run Jupyter script on Slurm HPC
0
gravatar for Bioinfonext
6 weeks ago by
Bioinfonext250
Korea
Bioinfonext250 wrote:

Hello,

I need to run a Jupyter script, could you please suggest how can run it on HPC server? Do I need to load any module? Many thanks

jupyter hpc • 179 views
ADD COMMENTlink modified 6 weeks ago by i.sudbery9.1k • written 6 weeks ago by Bioinfonext250
2
gravatar for i.sudbery
6 weeks ago by
i.sudbery9.1k
Sheffield, UK
i.sudbery9.1k wrote:

This very much depends on how your HPC is set up and what format the script is in.

Mostly likely you will need a python installation that has the jupyter package installed in it.

If the script is an ipynb file, once you have jupyter installed, running the script is easy. The command to run the script is is, strangely, nbconvert.

jupyter nbconvert --to notebook --execute mynotebook.ipynb

You would wrap that in what ever submission script you would normally use with your HPC system. For example:

[USER@login1]$ cat submission.sh
#!/bin/bash
#SBATCH --nodes=1
srun jupyter nbconvert --to notebook --execute mynotebook.ipynb  

[USER@login1]$ sbatch submission.sh

(disclaimer: I am not a regular SLURM user)

So the hard bit is getting jupyter installed on your HPC system. Almost all HPCs have systems for installing python packages without the need for root permissions, but the route varies from system to system (this is independent of the scheduller - SLURM in this case).

Our system uses conda to manage package installation. You would first need to create an environment with jupyter installed so you would want to do something like:

module load conda
conda create -n jupyter-env python=3 jupyter

You would also need to install any packages that the notebook required as well. Then you would need to load the env in batch submission script, so your script might look like:

[USER@login1]$ cat submission.sh
#!/bin/bash
#SBATCH --nodes=1
module load conda
source activate jupyter-env
srun jupyter nbconvert --to notebook --execute mynotebook.ipynb

Other systems might rely on pip and/or virtualenv. In pip you might try

[USER@login1]$ pip install jupyter --user
ADD COMMENTlink written 6 weeks ago by i.sudbery9.1k

Hi @i.sudbery

Thanks for your detail response. I was able to install Jupyter on the server but while running script it shows the error? I have attached the script, could it be possible to convert it as python script? Thank you so much for your time and help.

#Run it in jupyter lab or notebook


import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd

%matplotlib inline 


qza = 'alpha-rarefaction_20000.qzv' # path to rarefaction file from qiime2
a = !unzip $qza
digest = a[1].split('/')[0].replace('  inflating: ','')
inf = digest + '/data/observed_otus.csv'
otus = pd.read_csv(inf,index_col=0, sep=',')
!rm -r $digest 

cols = [col for col in otus.columns if 'iter' not in col]
mean,data = otus[cols],pd.DataFrame(columns=cols)
depths = [col.split('_')[0] for col in otus.columns if 'depth' in col]
otus = otus.drop(cols,axis=1)
otus.columns = depths
for depth in depths:
    mean['ASV'] = otus[depth].mean(axis=1)
    mean['depth']= depth.split('-')[-1]
    data = pd.concat([data,mean])

# here provide colors for each item that will be plotted
pal={'Leaf':'brown','Root':'green','Soil':'blue'} 
fig,ax = plt.subplots(figsize=(2.5,2),dpi=600,tight_layout=True)
sns.set(style='ticks',rc={"lines.linewidth":0.7,"axes.linewidth":0.5})
# use your column name to plot here instead of 'BodySite'

sns.lineplot(x='depth',y='ASV',data=data,palette=pal,hue='Compartment',sort=False,err_style='bars',\
             dashes=True,style='BodySite',ci=67)
ax.set_xlabel('Sequencing depth',fontsize=8)
ax.set_ylabel('Observed ASVs',fontsize=8)
ax.tick_params(axis='x', labelrotation=90)
ax.tick_params(axis='both',which='major',length=2,pad=0.5,labelsize=6)
handles, labels = ax.get_legend_handles_labels()
ax.legend(handles=handles[1:],labels=labels[1:],fontsize=5,frameon=False,numpoints=4,borderaxespad=0,handletextpad=0.2,loc=2,)
plt.savefig('rarefaction_ASVs.png', bbox_inches='tight')

Many thanks

ADD REPLYlink written 6 weeks ago by Bioinfonext250

This should work fine as a script with the excpetion of the lines

%matplotlib inline

which can be removed

and

!rm -r $digest

This shell command should be replaced with the equivalent python command. Try a google search for "python delete directory".

ADD REPLYlink written 6 weeks ago by i.sudbery9.1k

Hi @i.sudbery,

thanks, but this line also shows the error and not sure what command to use to unzip the file in python?

a = !unzip $qza

Many thanks,

ADD REPLYlink written 6 weeks ago by Bioinfonext250

There is not python equivalent for unzipping a file as far as I'm aware (I might be wrong). You will need to run a shell command from python.

ADD REPLYlink written 6 weeks ago by i.sudbery9.1k

Hi,

Now jupyter installed on the server and I am using below jupyter code to plot the rarefaction plot but I am still getting some error. could you please suggest how I can get rid of it?

#!/bin/bash
#SBATCH --job-name=plot    # Job name
#SBATCH --nodes=3                  # Number of nodes           
#SBATCH --time=2000:00:00              # Time limit hrs:min:sec
#SBATCH --partition=k2-lowpri
#SBATCH --mem=150G


module load apps/jupyter/4.6.3
module load libs/pandas/0.24.2/gcc-4.8.5+python-2.7.8+setuptools-39.2.0+numpy-1.16.2
module load apps/python3/3.7.4/gcc-4.8.5
module load apps/qiime2/2019.4.0/bin
module load libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3



#Run it in jupyter lab or notebook


import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd

%matplotlib inline 

qza = 'alpha-rarefaction-20000.qzv' # path to rarefaction file from qiime2
a = !unzip $qza
digest = a[1].split('/')[0].replace('  inflating: ','')
inf = digest + '/data/observed_otus.csv'
otus = pd.read_csv(inf,index_col=0, sep=',')
!rm -r $digest 

cols = [col for col in otus.columns if 'iter' not in col]
mean,data = otus[cols],pd.DataFrame(columns=cols)
depths = [col.split('_')[0] for col in otus.columns if 'depth' in col]
otus = otus.drop(cols,axis=1)
otus.columns = depths
for depth in depths:
    mean['ASV'] = otus[depth].mean(axis=1)
    mean['depth']= depth.split('-')[-1]
    data = pd.concat([data,mean])
# here provide colors for each item that will be plotted
pal={'Irrigated':'brown','Rainfed':'red'} 
fig,ax = plt.subplots(figsize=(2.5,2),dpi=600,tight_layout=True)
sns.set(style='ticks',rc={"lines.linewidth":0.7,"axes.linewidth":0.5})
# use your column name to plot here instead of 'BodySite'
sns.lineplot(x='depth',y='ASV',data=data,palette=pal,hue='Season',sort=False,err_style='bars',\
             dashes=True,style='Season',ci=67)
ax.set_xlabel('Sequencing depth',fontsize=8)
ax.set_ylabel('Observed ASVs',fontsize=8)
ax.tick_params(axis='x', labelrotation=90)
ax.tick_params(axis='both',which='major',length=2,pad=0.5,labelsize=6)
handles, labels = ax.get_legend_handles_labels()
ax.legend(handles=handles[1:],labels=labels[1:],fontsize=5,frameon=False,numpoints=4,borderaxespad=0,handletextpad=0.2,loc=2,)
plt.savefig('Infected_observed_ASVs.png', bbox_inches='tight')

Error;
import: unable to open X server `' @ error/import.c/ImportImageCommand/344.
import: unable to open X server `' @ error/import.c/ImportImageCommand/344.
import: unable to open X server `' @ error/import.c/ImportImageCommand/344.
/var/spool/slurmd.spool/job1452532/slurm_script: line 21: fg: no job control
/var/spool/slurmd.spool/job1452532/slurm_script: line 23: qza: command not found
/var/spool/slurmd.spool/job1452532/slurm_script: line 24: a: command not found
/var/spool/slurmd.spool/job1452532/slurm_script: line 25: syntax error near unexpected token `('
/var/spool/slurmd.spool/job1452532/slurm_script: line 25: `digest = a[1].split('/')[0].replace('  inflating: ','')'

metadata file ;
#SampleID       Villages        Region  Treatment       Batch   Season
PN0086D.1.S1    Kholabari       Mymensingh      T1      batch1  Irrigated
PN0086D.2.S2    Bogajan Mymensingh      T1      batch1  Irrigated
PN0086D.3.S3    Bonkua  Mymensingh      T1      batch1  Irrigated

many thanks

ADD REPLYlink modified 2 days ago • written 6 weeks ago by Bioinfonext250
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1194 users visited in the last hour