Entering edit mode
5 months ago
Maverick
▴
10
Hello, I am trying to run GATK's CNV pipeline.
Has anyone faced a similar error like this? I am able to run the entire pipeline at my login node from the same python environment. But when I submit my job script to the compute node of our HPC, the import gcnvkernel throws this error.
22:50:32.029 INFO DetermineGermlineContigPloidy - Initializing engine
22:50:32.033 DEBUG ScriptExecutor - Executing:
22:50:32.033 DEBUG ScriptExecutor - python
22:50:32.033 DEBUG ScriptExecutor - -c
22:50:32.033 DEBUG ScriptExecutor - import gcnvkernel
ERROR (theano.gof.opt): Optimization failure due to: local_subtensor_merge
ERROR (theano.gof.opt): node: Subtensor{:int64:}(log_emission_tc[0:], ScalarFromTensor.0)
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/opt.py", line 2034, in process_node
replacements = lopt.transform(node)
File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/tensor/opt.py", line 3091, in local_subtensor_merge
ushape[pos_2]))
File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/tensor/opt.py", line 3039, in merge_two_slices
start = pre_greedy_local_optimizer(list_opt, start)
File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/opt.py", line 2923, in pre_greedy_local_optimizer
list_optimizations, out, {}, 0)
File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/opt.py", line 2895, in local_recursive_function
depth + 1)
File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/opt.py", line 2895, in local_recursive_function
depth + 1)
File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/opt.py", line 2907, in local_recursive_function
ret = opt.transform(node)
File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/tensor/opt.py", line 6516, in constant_folding
no_recycling=[], impl=impl)
File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/op.py", line 955, in make_thunk
no_recycling)
File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/op.py", line 858, in make_c_thunk
output_storage=node_output_storage)
File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/cc.py", line 1217, in make_thunk
keep_lock=keep_lock)
File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/cc.py", line 1157, in __compile__
keep_lock=keep_lock)
File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/cc.py", line 1624, in cthunk_factory
key=key, lnk=self, keep_lock=keep_lock)
File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/cmodule.py", line 1189, in module_from_key
module = lnk.compile_cmodule(location)
File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/cc.py", line 1527, in compile_cmodule
preargs=preargs)
File "~/miniconda3/envs/gatk/lib/python3.6/site-packages/theano/gof/cmodule.py", line 2411, in compile_str
(status, compile_stderr.replace('\n', '. ')))
Exception: ('Compilation failed (return status=1): In file included from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/syslimits.h:7:0,. from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/limits.h:34,. from ~/miniconda3/envs/gatk/include/python3.6m/Python.h:11,. from /projectnb/pegasus/~/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-centos-7.6.1810-Core-x86_64-3.6.10-64/tmpi51kx06_/mod.cpp:1:. /usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/limits.h:168:61: error: no include path in which to search for limits.h. #include_next <limits.h> /* recurse down to the real one */. ^. In file included from /projectnb/pegasus/~/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-centos-7.6.1810-Core-x86_64-3.6.10-64/tmpi51kx06_/mod.cpp:1:0:. ~/miniconda3/envs/gatk/include/python3.6m/Python.h:25:19: fatal error: stdio.h: No such file or directory. #include <stdio.h>. ^. compilation terminated.. ', '[InplaceDimShuffle{}(TensorConstant{0})]')
I activate the environment and load all required modules as mentioned below in my job script:
source ~/miniconda3/etc/profile.d/conda.sh
module load java/17
conda activate gatk
Are there any changes that need to be made when submitting a job to HPC? I can see that the error says certain files like stdio.h and limits.h are not there but they are present as seen below -
find $CONDA_PREFIX -name limits.h
~/miniconda3/envs/gatk/x86_64-conda-linux-gnu/include/c++/7.5.0/tr1/limits.h
~/miniconda3/envs/gatk/x86_64-conda-linux-gnu/sysroot/usr/include/linux/limits.h
~/miniconda3/envs/gatk/x86_64-conda-linux-gnu/sysroot/usr/include/limits.h
~/miniconda3/envs/gatk/lib/gcc/x86_64-conda-linux-gnu/7.5.0/include-fixed/limits.h
~/miniconda3/envs/gatk/lib/gcc/x86_64-conda-linux-gnu/7.5.0/install-tools/include/limits.h
find $CONDA_PREFIX -name stdio.h
~/miniconda3/envs/gatk/x86_64-conda-linux-gnu/include/c++/7.5.0/tr1/stdio.h
~/miniconda3/envs/gatk/x86_64-conda-linux-gnu/sysroot/usr/include/bits/stdio.h
~/miniconda3/envs/gatk/x86_64-conda-linux-gnu/sysroot/usr/include/stdio.h
Thank you for your time.
Incase anyone has a similar issue - the HPC's gcc was overriding the gcc provided by the gatk package in the conda environment.
I followed conda-gcc-does-not-replace-system-gcc to create a symbolic link and it was sorted.
It was however followed by another error "missing crypt.h". Depending on the conda version the file can be placed in a location other than the anticipated one. I followed missing crypt.h to copy and paste the file and the script now runs without issues.