Minimal CWL (Common Workflow Language) workflow from Makefile
5
13
Entering edit mode
8.2 years ago

Common Workflow Language (CWL) https://github.com/common-workflow-language/common-workflow-language / http://common-workflow-language.github.io/draft-3/ has been trending on my twitter timeline during the last weeks.

However the spec is quite large and I find it hard to get some simple examples.

Furthermore, I have the feeling that all engines require a lot of dependencies or docker. I'd like to test my makefile-based workflows using CWL, how should I write and test the following simple Makefile using CWL:

SHELL=/bin/bash
.PHONY:   all
all :  database.dna

database.dna :  seq1.dna seq2.dna seq3.dna
    cat seq1.dna seq2.dna seq3.dna > database.dna

seq3.dna :  seq3.rna
    tr "U" "T" < seq3.rna > seq3.dna

seq3.rna :
    echo "AUGCGAUCGAUCG" > seq3.rna

seq2.dna :  seq2.rna
    tr "U" "T" < seq2.rna > seq2.dna

seq2.rna :
    echo "AUGAAGACUGCGAUCGAUCG" > seq2.rna

seq1.dna :  seq1.rna
    tr "U" "T" < seq1.rna > seq1.dna

seq1.rna :
    echo "AUGAAGACUGACUCGUCG" > seq1.rna

EDIT: feel free to add the file for your favorite workflow-engine as an answer.

makefile cwl • 5.7k views
ADD COMMENT
0
Entering edit mode

Responded with an example on this github issue: https://github.com/common-workflow-language/workflows/issues/1

ADD REPLY
4
Entering edit mode
8.2 years ago

To describe this workflow using CWL you need to:
- describe the 3 different tools (tr, echo, cat)

- describe the workflow structure

- describe the input data for the workflow job, i.e. the 3 input strings (AUGAAGACUGACUCGAUCGAUCG. etc.)

As an example to see the implementation of "tr", you can see this gist: . The github repository for common-workflow-language also has a number of tool and workflow descriptions that might help (https://github.com/common-workflow-language/common-workflow-language/tree/master/conformance/draft-2).

The reference implementation should be fairly easy to set up (pip install cwl-runner) but it is minimal (no cluster integration, etc.). Docker is not and should never become a requirement of CWL, some tools have been described with docker "requirements" but this is absolutely not mandatory. You can also ask your questions on all the channels mentionned here: https://github.com/common-workflow-language/common-workflow-language#community-and-contributing

Hope this helps,

Hervé

ADD COMMENT
0
Entering edit mode

many thanks Hervé !

ADD REPLY
3
Entering edit mode
8.2 years ago
peter.amstutz ▴ 300

Responded with an example on this github issue: https://github.com/common-workflow-language/workflows/issues/1

ADD COMMENT
2
Entering edit mode
8.0 years ago
sahiilseth ▴ 30

I do plan to create a CWL format importer for flowr in the future. Though, here is a example, using two simple tab-delim files. Feedback, really welcome!

Install flowr following these steps.

You may edit this file, to run the same flow in various clusters, or local envir.

vi simple_make.def

Additionally, for easy visualization, you may use this command:

flowr plot_flow x=simple_make.def pdffile=simple_make.pdf

When done, submit:

flowr to_flow x=simple_make.tsv def=simple_make.def execute=TRUE

Output: https://github.com/sahilseth/flowr/files/4729/simple_make.pdf

ADD COMMENT
1
Entering edit mode
8.2 years ago

I'm copying Paolo Di Tommaso's gist for http://www.nextflow.io/ below

ADD COMMENT
1
Entering edit mode

correct me if I'm wrong but this isn't CWL-compliant

ADD REPLY
1
Entering edit mode

"EDIT: feel free to add the file for your favorite workflow-engine as an answer."

That's right, that's nextflow.io.

ADD REPLY
0
Entering edit mode

is CWL a workflow engine itself or a standard for describing the interface between components?

ADD REPLY
1
Entering edit mode

The second one. It's a specification for a declarative document format for describing portable workflows. The goal is to have lots of interoperable implementations.

ADD REPLY
0
Entering edit mode
8.2 years ago
PoGibas 5.1k

Will try to contribute. This is not an actual answer, but how I try to keep my projects tidy using Makefile (aka "reproducible research").

Dependency for every script is a dummy file (shed for sh or Red for R)

analysis=\
    download.shed \
    clean.shed \
    statistics.Red \
    figures.Red

###########################################
# Analysis Workflow
###########################################

all: $(analysis)

# Download data
download.shed: download.sh

# Clean data
clean.shed: clean.sh download.shed

# Summary statistcs
statistics.Red: statistics.R clean.shed

# Plot data
figures.Red: figures.R statistics.Red

###########################################
# Execute
###########################################

%.shed: %.sh
    ./$< 2> $(basename $<).out && touch $@

%.Red: %.R
    R CMD BATCH --no-save --no-restore $< && touch $@

clean:
    rm -f *out *.shed *.Red
ADD COMMENT
0
Entering edit mode

thanks, I know how to write a makefile ( e.g. https://github.com/lindenb/ngsxml ). However, what I need is a solution for the alternative solutions, especially CWL.

ADD REPLY

Login before adding your answer.

Traffic: 1651 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6