Question

PHG: liquibase faillures when trying to run the imputation pipeline

0

Entering edit mode

2.1 years ago

charles.beauchamps • 0

Hello there :)

I am trying to run the PHG pipeline end to end.

I had no insurmontable issue up until the imputation step.

My set up is a PHG haplotype DB build with the v0.0.37 of phg, using the singularity container platform, I used only assembly to create my DB and want to imput fastq files.

However when I tried to run the command:

/tassel-5-standalone/run_pipeline.pl -Xmx75G -debug -configParameters /phg_run1/config.txt -ImputePipelinePlugin -imputeTarget path -endPlugin

Note: my working directory within the container is not /phg similarly to how the wiki is advising but /phg_run1

I hit a liquibase error. In the pipeline logs here are the relevant extract:

[pool-1-thread-1] INFO net.maizegenetics.plugindef.AbstractPlugin - Finished net.maizegenetics.pangenome.liquibase.CheckDBVersionPlugin: time: Apr 14, 2022 9:44:51
[pool-1-thread-1] INFO net.maizegenetics.plugindef.AbstractPlugin - net.maizegenetics.pangenome.liquibase.CheckDBVersionPlugin  Citation: Bradbury PJ, Zhang Z, Kroon DE, Casstevens TM, Ramdoss Y, Buckler ES. (2007) TASSEL: Software for association mapping of complex traits in diverse samples. Bioinformatics 23:2633-2635.
[pool-1-thread-1] INFO net.maizegenetics.plugindef.AbstractPlugin - Starting net.maizegenetics.pangenome.liquibase.LiquibaseUpdatePlugin: time: Apr 14, 2022 9:44:51
[pool-1-thread-1] INFO net.maizegenetics.plugindef.AbstractPlugin -
LiquibaseUpdatePlugin Parameters
outputDir: /phg_run1/outputDir
command: status

[pool-1-thread-1] INFO net.maizegenetics.pangenome.liquibase.LiquibaseUpdatePlugin - Please wait, begin Command:liquibase --driver=org.sqlite.JDBC --url=jdbc:sqlite:/phg_run1/soy_1_phg.db --username=sqlite --password=sqlite --changeLogFile=/liquibase/changelogs/db.changelog-master.xml status --verbose
[pool-1-thread-1] ERROR net.maizegenetics.pangenome.liquibase.LiquibaseUpdatePlugin - liquibase run via ProcessBuilder returned error code 255
[pool-1-thread-1] INFO net.maizegenetics.plugindef.AbstractPlugin - Finished net.maizegenetics.pangenome.liquibase.LiquibaseUpdatePlugin: time: Apr 14, 2022 9:44:51
[pool-1-thread-1] DEBUG net.maizegenetics.plugindef.AbstractPlugin - ERROR!! - liquibase status command encountered an error.  See file /phg_run1/outputDir/liquibase_status_error.log for details.

in the liquibase_status_error.log I had an empty file, however in the liquibase_status_output.log file I had the following content:

Liquibase Community 4.7.0 by Liquibase
####################################################
##   _     _             _ _                      ##
##  | |   (_)           (_) |                     ##
##  | |    _  __ _ _   _ _| |__   __ _ ___  ___   ##
##  | |   | |/ _` | | | | | '_ \ / _` / __|/ _ \  ##
##  | |___| | (_| | |_| | | |_) | (_| \__ \  __/  ##
##  \_____/_|\__, |\__,_|_|_.__/ \__,_|___/\___|  ##
##              | |                               ##
##              |_|                               ##
##                                                ##
##  Get documentation at docs.liquibase.com       ##
##  Get certified courses at learn.liquibase.com  ##
##  Free schema change activity reports at        ##
##      https://hub.liquibase.com                 ##
##                                                ##
####################################################
Starting Liquibase at 09:54:16 (version 4.7.0 #1140 built at 2022-01-07 19:26+0000)
Unexpected error running Liquibase: The file /liquibase/changelogs/db.changelog-master.xml was not found in
    - /liquibase/lib
    - /liquibase/lib/h2-2.0.206.jar
    - /liquibase/lib/hsqldb-2.5.2.jar
    - /liquibase/lib/jaxb-api-2.3.0.jar
    - /liquibase/lib/jaxb-core-2.3.0.jar
    - /liquibase/lib/jaxb-impl-2.3.0.jar
    - /liquibase/lib/jaybird-4.0.3.java8.jar
    - /liquibase/lib/jcc-11.5.6.0.jar
    - /liquibase/lib/mariadb-java-client-2.7.0.jar
    - /liquibase/lib/mssql-jdbc-8.4.0.jre8.jar
    - /liquibase/lib/ojdbc8-18.3.0.0.jar
    - /liquibase/lib/picocli-4.6.1.jar
    - /liquibase/lib/postgresql-42.2.15.jar
    - /liquibase/lib/snakeyaml-1.27.jar
    - /liquibase/lib/sqlite-jdbc-3.34.0.jar
    - /liquibase/liquibase.jar
    - /synhpc/home/u772624
    - /synhpc/home/u772624/.
    - /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/cldrdata.jar
    - /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/dnsns.jar
    - /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/icedtea-sound.jar
    - /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/jaccess.jar
    - /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/java-atk-wrapper.jar
    - /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/libatk-wrapper.so
    - /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/localedata.jar
    - /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/nashorn.jar
    - /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/sunec.jar
    - /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/sunjce_provider.jar
    - /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/sunpkcs11.jar
    - /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/zipfs.jar
Specifying files by absolute path was removed in Liquibase 4.0. Please use a relative path or add '/' to the classpath parameter.
For more information, please use the --logLevel flag

Hence to go over that issue I edited the /liquibase/liquibase script last line from:

java -cp "$CP" $JAVA_OPTS liquibase.integration.commandline.Main ${1+"$@"}

into:

java -cp "$CP:/" $JAVA_OPTS liquibase.integration.commandline.Main ${1+"$@"}

in order to add the root '/' to the classpath as advised by liquibase error message.

Note: to do that with singularity I had to bind a copy of the script from outside the container and edit it

At this stage to check if everything was alright I logged within the container (singularity shell command) and manually ran the following liquibase status command:

liquibase --driver=org.sqlite.JDBC --url=jdbc:sqlite:/phg_run1/soy_1_phg.db --username=sqlite --password=sqlite --changeLogFile=/liquibase/changelogs/db.changelog-master.xml status --verbose

and I got :

Starting Liquibase at 09:57:56 (version 4.7.0 #1140 built at 2022-01-07 19:26+0000)
null@jdbc:sqlite:/phg_run1/soy_1_phg.db is up to date
Liquibase command 'status' was executed successfully.

So I tried once again to run the imputation command (cf on the top of this post)

but I still face a similar error within the pipeline logs:

[pool-1-thread-1] INFO net.maizegenetics.plugindef.AbstractPlugin - Finished net.maizegenetics.pangenome.liquibase.CheckDBVersionPlugin: time: Apr 14, 2022 10:00:6
[pool-1-thread-1] INFO net.maizegenetics.plugindef.AbstractPlugin - net.maizegenetics.pangenome.liquibase.CheckDBVersionPlugin  Citation: Bradbury PJ, Zhang Z, Kroon DE, Casstevens TM, Ramdoss Y, B
[pool-1-thread-1] INFO net.maizegenetics.plugindef.AbstractPlugin - Starting net.maizegenetics.pangenome.liquibase.LiquibaseUpdatePlugin: time: Apr 14, 2022 10:00:6
[pool-1-thread-1] INFO net.maizegenetics.plugindef.AbstractPlugin -
LiquibaseUpdatePlugin Parameters
outputDir: /phg_run1/outputDir
command: status

[pool-1-thread-1] INFO net.maizegenetics.pangenome.liquibase.LiquibaseUpdatePlugin - Please wait, begin Command:liquibase --driver=org.sqlite.JDBC --url=jdbc:sqlite:/phg_run1/soy_1_phg.db --usernam
[pool-1-thread-1] INFO net.maizegenetics.plugindef.AbstractPlugin - Finished net.maizegenetics.pangenome.liquibase.LiquibaseUpdatePlugin: time: Apr 14, 2022 10:00:7
[pool-1-thread-1] DEBUG net.maizegenetics.plugindef.AbstractPlugin - ERROR!! - liquibase status command encountered an error.  See file /phg_run1/outputDir/liquibase_status_error.log for details.

as previously the liquibase_status_error.log file was empty but liquibase_status_output.log was containing the same successful message that I had previously when running the command within the container.

So I do not have any idea why the pipeline “think” that liquibase hit an error when it doesn't appear to do so.

And I'm looking for any guidance to overcome my current issue.

Best regards.

phg liquibase singularity • 1.2k views

ADD COMMENT • link updated 2.1 years ago by lcj34 ▴ 420 • written 2.1 years ago by charles.beauchamps • 0

score 0 · Answer 1 · 2022-04-14

0

Entering edit mode

2.1 years ago

lcj34 ▴ 420

Could you clarify for me if you are running the liquibase commands from within, or outside of the container? I have seen the error message you post when running liquibase from inside a docker container (I realize you are using Singularity). But it has always worked when running the liquibase commands from outside the container. I'd like to verify if that has changed.

Are you running your imputation commands from within, or outside the container?

ADD COMMENT • link 2.1 years ago by lcj34 ▴ 420

0

Entering edit mode

All is running within the container, as I have no liquibase installation on the Host

ADD REPLY • link 2.1 years ago by charles.beauchamps • 0

score 0 · Answer 2 · 2022-04-14

I wasn't clear with my question. Are you entering the container to manually run the scripts, or are you sending the command from outside the container?

IE: are you entering the shell, e.g. "singularity shell myImage.sif", and then typing the commands. Or are you running the full request from the command line, e.g. "singularity run -B <working path>:/phg/ myImage.sif /tassel-5-standalone/run_pipeline.pl -debug -configParameters /phg_run1/config.txt -ImputePipelinePlugin ..."

Will you post the commands you are running? How did you populate the db? Was it with the same versions of the PHG? Which commands did you run and again, were those commands run from a Singularity shell, or from outside the container using "singularity run ... ?"