The new model puts a whole new spin on deconvolution (deconvolution) image processing technology. This computationally intensive process improves the contrast and resolution of digital images from optical microscopes such as wide-field, confocal or transmitted light microscopes. Deconvolution reduces the blurring caused by the microscopy system used. There are two main methods for this: explicit deconvolution (direct deconvolution) and deconvolution based on deep learning.
The direct deconvolution approaches are based on the concept of the point spread function (PSF). A PSF describes how an infinitely small light source emanating from the sample is expanded by the optical system and distributed into a three-dimensional diffraction pattern. This means that in a captured (two-dimensional) image there is always some light from structures that are not in the focal point, which creates the blur. If you know the PSF of the microscopic system, you can remove this blurring and obtain an image that is much closer to reality than the unprocessed image.
“The big problem with PSF-based deconvolution techniques is that the PSF of a particular microscopic system is often unavailable or inaccurate,” says Dr. Artur Yakimovich, head of an HZDR junior research group and corresponding author of the ECCV publication. “For decades, work has been done on so-called blind deconvolution, where the PSF is estimated from the image or a set of images. However, blind deconvolution is still a rather tricky problem and the progress achieved is rather modest.”
The Yakimovich team has already proven in the past that the toolkit can be usefully applied to solve inverse problems in the field of microscopy. Inverse problems are about determining the original factors that lead to certain observations made. As a rule, large amounts of data and deep learning algorithms are required to successfully solve this type of problem. As with direct deconvolution methods, deep learning-based deconvolution results in images with higher resolution or better quality. For the approach presented at ECCV, the scientists used a physics-based neural network called Multi-Stage Residual-BCR Net (m-rBCR).
Deep learning used differently
In general, there are two basic variants for image processing. It either starts with the classic spatial representation of an image or (after a transformation of this representation) with its frequency version. In the latter, each image is represented as a collection of waves. Both forms of representation offer specific advantages and, depending on the type of image processing, one or the other form is more likely to be used. The vast majority of deep learning architectures in use work with the spatial representation. It is well suited for photos. However, microscopy images are different. They are usually monochromatic. In techniques such as fluorescence microscopy, for example, specific light sources are used on a black background. m-rBCR therefore uses the frequency representation as a starting point.
“The use of the frequency domain can help to create visually meaningful data representations, especially in such cases – a concept that enables m-rBCR to master the deconvolution task with surprisingly few parameters compared to other modern deep learning architectures,” explains Rui Li, first author and speaker at the ECCV. Li proposed to further develop the neural network architecture of the BCR-Net model. This model was in turn inspired by a frequency representation-based signal compression scheme introduced in the 1990s by Gregory Beylkin, Ronald Coifman and Vladimir Rokhlin (hence the name BCR-Transform).
The team validated the m-rBCR model using four different datasets, two simulated microscopy image datasets and two real microscopy datasets. Compared to the latest deep learning-based models, it shows high performance with significantly fewer training parameters and a shorter runtime. In addition, it also significantly outperforms explicit deconvolution methods.
A model tailored to microscopic images
“This new architecture uses an approach that has received little attention so far to enable representations beyond classical neural convolutional networks,” summarizes co-author Prof. Misha Kudryashev, head of the “In situ Structural Biology” research group at the Max Delbrück Center for Molecular Medicine in Berlin. “Our model significantly reduces potentially redundant parameters. However, as the results show, this is not accompanied by a loss of performance. The model is explicitly suitable for microscopic images and, due to its lean architecture, challenges the trend towards ever larger models that require correspondingly higher computing power.”
The Yakimovich group recently published a model for improving image quality based on generative AI. Their Conditional Variational Diffusion Model (CVDM) achieves results that are state of the art and also outperform the m-rBCR model presented here. “However, this requires training data and corresponding computing resources, including powerful graphics processors, which are in high demand nowadays,” Yakimovich points out. “The lightweight m-rBCR model does not have these limitations and still delivers very good results. I am therefore confident that we will receive a lot of encouragement from the research community. This is another reason why we have started to optimize the user-friendliness.”
The Yakimovich group “Machine Learning in Infection and Disease” aims to understand the complex network of molecular interactions that is active in the body after infection with pathogens. The use of the new possibilities of machine learning is central to this. Areas of interest include improving image resolution, 3D image reconstruction, automated disease diagnosis and evaluating the reconstruction quality of images.
Publication:
R. Li, M. Kudryashev, A. Yakimovich, Solving the inverse problem of microscopy deconvolution with a residual Beylkin-Coifman-Rokhlin neural network, In: A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, G. Varol (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15133. Springer, Cham. (doi: 10.1007/978-3-031-73226-3_22)
Weitere Informationen:
Dr. Artur Yakimovich | Junior Research Group Leader
Center for Advanced Systems Understanding (CASUS) at HZDR
Email: a.yakimovich@hzdr.de
– – – – –
Further links
👉 www.hzdr.de
Image: HZDR / A. Yakimovich
