Question: Minimal CWL (Common Workflow Language) workflow from Makefile
13
gravatar for Pierre Lindenbaum
3.9 years ago by
France/Nantes/Institut du Thorax - INSERM UMR1087
Pierre Lindenbaum120k wrote:

Common Workflow Language (CWL) https://github.com/common-workflow-language/common-workflow-language / http://common-workflow-language.github.io/draft-3/ has been trending on my twitter timeline during the last weeks.

However the spec is quite large and I find it hard to get some simple examples.

Furthermore, I have the feeling that all engines require a lot of dependencies or docker. I'd like to test my makefile-based workflows using CWL, how should  I write and test the following simple Makefile using CWL:

 

SHELL=/bin/bash
.PHONY:   all
all :  database.dna

database.dna :  seq1.dna seq2.dna seq3.dna
    cat seq1.dna seq2.dna seq3.dna > database.dna

seq3.dna :  seq3.rna
    tr "U" "T" < seq3.rna > seq3.dna

seq3.rna :
    echo "AUGCGAUCGAUCG" > seq3.rna

seq2.dna :  seq2.rna
    tr "U" "T" < seq2.rna > seq2.dna

seq2.rna :
    echo "AUGAAGACUGCGAUCGAUCG" > seq2.rna

seq1.dna :  seq1.rna
    tr "U" "T" < seq1.rna > seq1.dna

seq1.rna :
    echo "AUGAAGACUGACUCGUCG" > seq1.rna

 

EDIT: feel free to add the file for your favorite workflow-engine as an answer.

 

 

 

 

makefile cwl • 4.3k views
ADD COMMENTlink modified 3.7 years ago by sahiilseth30 • written 3.9 years ago by Pierre Lindenbaum120k

Responded with an example on this github issue:  https://github.com/common-workflow-language/workflows/issues/1

ADD REPLYlink written 3.9 years ago by peter.amstutz300
4
gravatar for herve.menager
3.9 years ago by
France
herve.menager40 wrote:

To describe this workflow using CWL you need to:
- describe the 3 different tools (tr, echo, cat)

- describe the workflow structure

- describe the input data for the workflow job, i.e. the 3 input strings (AUGAAGACUGACUCGAUCGAUCG. etc.)

As an example to see the implementation of "tr", you can see this gist: . The github repository for common-workflow-language also has a number of tool and workflow descriptions that might help (https://github.com/common-workflow-language/common-workflow-language/tree/master/conformance/draft-2).

The reference implementation should be fairly easy to set up (pip install cwl-runner) but it is minimal (no cluster integration, etc.). Docker is not and should never become a requirement of CWL, some tools have been described with docker "requirements" but this is absolutely not mandatory. You can also ask your questions on all the channels mentionned here: https://github.com/common-workflow-language/common-workflow-language#community-and-contributing

Hope this helps,

Hervé

ADD COMMENTlink written 3.9 years ago by herve.menager40

many thanks Hervé !

ADD REPLYlink written 3.9 years ago by Pierre Lindenbaum120k
3
gravatar for peter.amstutz
3.9 years ago by
peter.amstutz300
peter.amstutz300 wrote:

Responded with an example on this github issue:  https://github.com/common-workflow-language/workflows/issues/1

ADD COMMENTlink written 3.9 years ago by peter.amstutz300
2
gravatar for sahiilseth
3.7 years ago by
sahiilseth30
United States
sahiilseth30 wrote:

I do plan to create a CWL format importer for flowr in the future. Though, here is a example, using two simple tab-delim files. Feedback, really welcome !

Install flowr following these steps.

You may edit this file, to run the same flow in various clusters, or local envir.

vi simple_make.def

Additionally, for easy visualization, you may use this command:

flowr plot_flow x=simple_make.def pdffile=simple_make.pdf

When done, submit:

flowr to_flow x=simple_make.tsv def=simple_make.def execute=TRUE

 

ADD COMMENTlink modified 3.7 years ago • written 3.7 years ago by sahiilseth30
1
gravatar for Pierre Lindenbaum
3.9 years ago by
France/Nantes/Institut du Thorax - INSERM UMR1087
Pierre Lindenbaum120k wrote:

I'm copying Paolo Di Tommaso 's gist for http://www.nextflow.io/ below

ADD COMMENTlink modified 3.9 years ago • written 3.9 years ago by Pierre Lindenbaum120k
1

correct me if I'm wrong but this isn't CWL-compliant

ADD REPLYlink modified 3.9 years ago • written 3.9 years ago by Jeremy Leipzig18k
1

"EDIT: feel free to add the file for your favorite workflow-engine as an answer."

That's right, that's nextflow.io.

ADD REPLYlink written 3.9 years ago by peter.amstutz300

is CWL a workflow engine itself or a standard for describing the interface between components?

ADD REPLYlink written 3.9 years ago by Jeremy Leipzig18k
1

The second one.  It's a specification for a declarative document format for describing portable workflows.  The goal is to have lots of interoperable implementations.

ADD REPLYlink written 3.9 years ago by peter.amstutz300
0
gravatar for PoGibas
3.9 years ago by
PoGibas4.8k
Vilnius
PoGibas4.8k wrote:

Will try to contribute. This is not an actual answer, but how I try to keep my projects tidy using Makefile (aka "reproducible research").

Dependency for every script is a dummy file (shed for sh or Red for R)

analysis=\
    download.shed \
    clean.shed \
    statistics.Red \
    figures.Red

###########################################
# Analysis Workflow
###########################################

all: $(analysis)

# Download data
download.shed: download.sh

# Clean data
clean.shed: clean.sh download.shed

# Summary statistcs
statistics.Red: statistics.R clean.shed

# Plot data
figures.Red: figures.R statistics.Red

###########################################
# Execute
###########################################

%.shed: %.sh
    ./$< 2> $(basename $<).out && touch $@

%.Red: %.R
    R CMD BATCH --no-save --no-restore $< && touch $@

clean:
    rm -f *out *.shed *.Red
ADD COMMENTlink modified 3.9 years ago • written 3.9 years ago by PoGibas4.8k

thanks, I know how to write a makefile ( e.g. https://github.com/lindenb/ngsxml ). However, what I need is a solution for the alternative solutions, especially CWL.

ADD REPLYlink written 3.9 years ago by Pierre Lindenbaum120k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1455 users visited in the last hour