Question: Getting a bash pipeline to output non-zero exit code when a step fails
1
gravatar for SaltedPork
12 months ago by
SaltedPork80
SaltedPork80 wrote:

I have a pipeline which looks something like below, its written in bash

Trimmomatic --> Samtools --> Smalt --> Lastz --> custom perl programs.

If anyone of those processes fails they usually return a non-0 exit code, usually a 1. However, my pipeline program will continue to run, report the error and return an exit code of 0, indicating success.

How can I get the pipeline to quit/stop with a non-0 exit code when a part of the pipeline fails?

command-line sh bash • 2.8k views
ADD COMMENTlink modified 12 months ago by bernatgel1.8k • written 12 months ago by SaltedPork80
2

Be aware that some software can "fail" when they don't fail.

ADD REPLYlink modified 12 months ago • written 12 months ago by h.mon24k
14
gravatar for RamRS
12 months ago by
RamRS20k
Houston, TX
RamRS20k wrote:

You're looking for set -eo pipefail. Add that after the first (#!/bin/bash) line and your script will fail if any part of your script fails.

See for more information: https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_pipefail/

ADD COMMENTlink modified 12 months ago • written 12 months ago by RamRS20k
1

Much better solution than mine!

ADD REPLYlink written 12 months ago by jrj.healey11k

Thanks, this is a really good solution. Do you have any advice for if I wanted to just have the pipeline fail at certain points in the pipeline. Perhaps put a command in a loop and check that echo $? returns non-zero or not?

ADD REPLYlink written 12 months ago by SaltedPork80
2

If you follow Ram's link, you can combine it with the logic of mine below but invert it. i,e. if you have pipefail set, you can just invert the non-zero exit codes by doing something like:

mycommand || true

This means even if the command fails, the parent shell process will interpret is as having worked correctly (0), so you can use this to selectively override certain sections.

ADD REPLYlink written 12 months ago by jrj.healey11k
2

Exactly. This works if the minority of statements are non-sensitive where OP wants to ignore the exit code. If the majority are non-sensitive, we're better off isolating the sensitive to separate scripts.

ADD REPLYlink written 12 months ago by RamRS20k
1

Yep. You'll need to check that. It will take a few trials though, as this is almost an implementation of try-catch blocks.

You could also place the sensitive parts in a separate script that has set -eo pipefail enabled and call it from your non-sensitive script.

ADD REPLYlink written 12 months ago by RamRS20k
4
gravatar for kloetzl
12 months ago by
kloetzl1.0k
European Union
kloetzl1.0k wrote:

To make the whole pipeline fail when any of the parts fail set the pipefail option. In zsh that can be done via setopt pipefail (don't know the bash equivalent by heart)/

ADD COMMENTlink written 12 months ago by kloetzl1.0k
1

bash equivalent would be set -o pipefail

ADD REPLYlink written 12 months ago by RamRS20k
3
gravatar for bernatgel
12 months ago by
bernatgel1.8k
Barcelona, Spain
bernatgel1.8k wrote:

I like the explicit error management described here, with a try, yell and die functions. As a bonus, you can modify die to do some additional logging of errors before actually dying and returning. It might be useful in addition to pipefail

yell() { echo "$0: $*" >&2; }
die() { yell "$*"; exit 111; }
try() { "$@" || die "cannot $*"; }
ADD COMMENTlink written 12 months ago by bernatgel1.8k
2
gravatar for jrj.healey
12 months ago by
jrj.healey11k
United Kingdom
jrj.healey11k wrote:

mycommand || exit 1

Should work I think. Basically saying, either mycommand runs successfully to completion or exit the script with a non-zero status.

ADD COMMENTlink modified 12 months ago • written 12 months ago by jrj.healey11k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1272 users visited in the last hour