Is it valid to stack brightfield and fluorescence channels in a CNN input?

Question

[Antonio]

I’m working on a deep learning task to classify whether a single cell has been exposed to carbon dots or not. Each sample consists of three spatially aligned grayscale microscopy images of the same cell, acquired using different modalities: one brightfield channel and two fluorescence channels highlighting the nucleus and the cell membrane, respectively. Since I’m not an expert in microscopy or biological imaging, I’m unsure whether it is correct to stack all three modalities into a single 3-channel image (as often done with RGB in CNNs). My concern is whether combining brightfield (which is transmitted light) with fluorescence modalities (which are emitted light) into the same tensor might introduce noise, confusion, or inconsistencies for the model. Would an expert in microscopy imaging consider this a flawed approach biologically or visually? Alternatively, would it make more sense to stack only the two fluorescence images (nuclear and membrane), assuming they are more coherent in signal type and structure, and possibly use brightfield separately? It is worth considering whether fluorescence channels, which highlight specific cellular structures, may generally provide more informative features than the brightfield channel for the task of detecting the presence of carbon dots? I’d appreciate any advice from professionals in microscopy, biomedical imaging, or multimodal data analysis on whether this kind of stacking is biologically meaningful and appropriate for classification tasks.

[LChart]

Not a real answer, but it should be pointed out that the only way to learn contrastive filters across modalities (i.e., fluorescence-only edge detectors vs fluorescence-plus-brightfield edge detectors) is to have these modalities stacked.

Another thing to point out is you don't need so much to worry about "adding noise" - the network can learn to effectively ignore an input channel - but instead augmenting the potential to overfit. Small difference, but important.

Answer 1

[Mensur Dlakic]

I think this is a borderline bioinformatics question, but I'll still attempt to answer.

You won't break the computer by merging all three channels and using that for training. CNNs don't know, nor do they care, about the nature of signal in your individual images. Even though I think the two fluorescent channels are what you need, combining them alone may not give the best training dataset. We have no idea what you are trying to classify from these images, and adding a brightfield channel could contain some information not found in the other two channels. You may need to invert the brightfield images, or play with different contrast values, before you add them to the other two. No matter what anybody tells you, the best way to answer your question is by experimenting with different image combinations.