When Reviewing A Software Paper, Do You Talk About The Code Quality?
7
12
Entering edit mode
13.1 years ago
brentp 24k

This is for either

A) a more biological paper, where software is a "means to an end"

B) a software paper where the software itself is the focus of the paper.

EDIT:

This is assuming that the code is made available to the reviewer.

publication software software • 4.8k views
ADD COMMENT
3
Entering edit mode

Most editors would never reject a paper based on the accusation of bad code - I mean, sure, as a reviewer you can talk about it - but that's probably the only influence you can. You hope that the authors will listen, go back and improve. They probably never will.

ADD REPLY
2
Entering edit mode

Are your experiments and experimental conditions ever a "means to an end"?

ADD REPLY
0
Entering edit mode

Your question seems to assume that code quality for means is not important... but that's just accidental, right?

ADD REPLY
0
Entering edit mode

I don't intend to imply that. I just guess that there my be different responses depending on those 2 cases.

ADD REPLY
0
Entering edit mode

@mndoci, I trust you comment is taken to be rhetorical and play-up the value of software in science, but clearly the answer to your question is "yes" - experiments are done with a goal in mind, they are not an end in themselves. See the definition of an experiment in any dictionary:

ADD REPLY
9
Entering edit mode
13.1 years ago
lh3 33k

A brief google search reveals that code quality seems to mostly mean a combination of readability, robustness, extensibility and maintainability. It has less to do with efficiency. In that case (and if you define code quality this way, too), code quality may be frequently subjective. I would be cautious of talking about it. Even if the code is obviously in bad shape (e.g. full of global variables), I may not measure a the quality of manuscript with code quality. To a scientific program, the underlying algorithm is far more important. Code quality is more in the engineering aspect and is only a "good-to-have" thing. We can write bad-looking but efficient programs. And I have indeed seen popular programs with bad code quality (in my standard). We may argue they are not easy to read/use, but we cannot find good alternatives. In addition, many tools are published not for others to use, either intentionally or effectively. In this case, code quality is not important, either.

ADD COMMENT
2
Entering edit mode

I think it is fine to suggest improving code quality as minor comments.

ADD REPLY
1
Entering edit mode

Code quality is like tiding up your home. Some people can live in total chaos, some people cannot tolerate a spec of dust. Whenever I look at code I wrote 3 months ago I say "I cannot believe I wrote such a crappy code!", but I understand that most people don't even care. Specially in science where most code is written for and by a single user. The real solution would be to have a simple "code quality metric".

ADD REPLY
0
Entering edit mode

You say: "I would be cautious of talking about it.". Why is that? Why not mention it? Even if it does not affect your recommendation?

ADD REPLY
7
Entering edit mode
13.1 years ago

To be honest, no. I'm not a computer scientist, so "code quality" to me is all about usability. I will comment on:

  • availability of source code (not on a university website)
  • ease of installation
  • documentation
  • tests/use-cases
  • run-time performance
  • input/output formats
  • help menus
  • error messages
  • accuracy of results

If the substance of what code produces is scientifically rigorous and has utility, then its style, readability, etc. is not that important to me. Furthermore, as long as source code is available, post-publication use, critique and extension will allow others to judge if the code is good or not.

ADD COMMENT
5
Entering edit mode
13.1 years ago

for (1) I would ask the authors to make the source available as a supplementary file, or better, to publish it on github/sourceforge/etc... A workflow could be posted on myexperiment.org

for (2) I would suggest to have a look at BMC - "Open Research Computation" : Pre-submission guide for software article authors http://www.openresearchcomputation.com/authors/presubmissionguide

Is the software source code available on a public repository? * Please provide the URL for the public repository

Is the source code made available under an Open Source Initiative compliant license? * A list of OSI compliant licences is available at: http://www.opensource.org/licenses/category

Are project authors and contributors clearly defined, ideally through a Description of a Project [http://en.wikipedia.org/wiki/DOAP, http://trac.usefulinc.com/doap] document? * We recommend the use of the automatic DOAP generator such as those linked here:http://trac.usefulinc.com/doap/wiki/Generators

Documentation Source code documentation as well as instructions for use are expected to a high standard.

etc...

ADD COMMENT
2
Entering edit mode

It is subjective, but there are things that are pretty widely recognized as poor practice. If I see a 30-line block of code with "A" hard-coded, then the same 30-line block for "G", then the same 30-line block for "T" , "G" (and dont forget lower-case) ... Well, the code may work fine, is that good code quality?

ADD REPLY
1
Entering edit mode

My thoughts exactly. Code quality is very subjective but asking for proper development practices is a fair request.

ADD REPLY
0
Entering edit mode

(Open Research Computing)++

ADD REPLY
4
Entering edit mode
13.1 years ago
David ▴ 50

Reproducibility is one of the most important and fundamental components of science. The 'quality' of experimental tools are not. It is much more important to encourage that code necessary to reproduce an analysis be submitted than it is to require that code meet a certain standard of quality.

As a reviewer, do you consider the quality of the materials used in an experiment, assuming that different materials have been demonstrated to produce equivalent results? If so, this would provide an unnecessary barrier to science. (Does it matter if a spectrometer cuvette is made of plastic vs. glass? That a microscope was made by Leica vs brand X? That Galileo used a primitive telescope?)

The philosophy that "if your code is good enough to do the job, it is good enough to publish" is outlined in a 2010 Nature column by Nick Barnes: 'Publish your code, it's good enough!'.

For publishing code intended for use as software, it is appropriate to comment on the functionality of the code, but not its 'smell'. If it is published as open source, it is available for others to improve upon. Important advances can be made in a fraction of the time that it would take to produce high quality code, and many researchers do not have the time that would be required to cleaning up 'good enough' code.

ADD COMMENT
0
Entering edit mode

An interesting (and well supported) viewpoint, thanks.

ADD REPLY
0
Entering edit mode

Good points. I agree that things like brand of scope or cuvette material are not important. But to continue the analogy, if the authors invented their own spectrophotometer or built their own microscope to generate their results, I'd want to evaluate the quality of what they created. For code, this could just be unit tests -- doesn't have to look pretty, but needs to work. In other words, functional quality over aesthetic quality (or other kinds of quality as described ind @lh3's answer)

ADD REPLY
2
Entering edit mode
13.1 years ago

Interesting question. I usually take quality to mean efficiency and success in doing what the software was designed to do.

I never write about details regarding code quality as a reviewer, just as I don't question whether an animal study using 20 cages (5 control + 5 treatment 1; 5 control + 5 treatment 2) where run concurrently or successively. This is for paper type A, biological and using software as a tool. The exception to this would be a general statement that the methods used and experiments conducted are well suited to the questions of X that the researchers proposed to address, etc.

A known tool (BLAST, BOWTIE, GenePatterns, e.g.) generally need not be explained in terms of efficiency and success - unless they were applied for the wrong purpose. A new tool may be difficult to assess, say if the code is not submitted or made available to the reviewers.

ADD COMMENT
2
Entering edit mode
13.1 years ago
Marvin ▴ 900

As a reviewer, do you talk about the quality of writing in the Methods section of a paper? I think you should. Supporting code is no different. If it's unreadable, the method isn't documented.

ADD COMMENT
0
Entering edit mode

If the method is clearly written in the paper, why the method is not documented? IMHO, it is usually a bad practice to ask users to read the source code in order to understand how the method works.

ADD REPLY
0
Entering edit mode

...and for that to work, the reviewer must be able to see that the method described in the paper and the method implemented in the code are actually the same.

ADD REPLY
1
Entering edit mode
13.1 years ago
Georg Summer ▴ 140

Depends on the software. If the software is a use-me-as-i-am application code quality does not matter that much. If the focus of the software is that you can extend it or use parts of it elsewhere code quality gets a lot more important. Imagine environments like Cytoscape or Galaxy that are just horrible to extend with your individual solutions and problems because the code is a messy blob of characters. What would you choose? Clearly defined and well designed interfaces vs messy blob? In the review of such a paper code quality should be considered.

ADD COMMENT

Login before adding your answer.

Traffic: 705 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6