IMAGES team research axes

Acquisition aware image restoration and enhancement

The following methods developed for multi-resolution and noise filtering rely on a rigorous modeling of the image acquisition principles.

A new framework for the decomposition of complex multi-channel data from coherent imagery has been proposed, allowing for the use of Gaussian denoisers to Wishart distributed data. An automatic selection of the best set of parameters (window size, patch size, pre-filtering strength), by aggregation strategy driven by variance / bias measurement thanks to noise modeling, has provided state of the art results for coherent image denoising. HDR performances have been improved using patch-based single shot HDR and patch-based exposure fusion. An innovative method for automatic white balance through projections on the Planckian locus has been presented, and the robustness of sparse reconstruction towards outliers in the context of multi-image super-resolution has been improved.

More precisely, the fusion of multiple images to obtain a higher quality image is an important approach in image restoration, especially when the physical support of images becomes so potent, cheap and ubiquitous. In such a setting, the common assumption is that the same scene is acquired many times. But actually it is seldom true that the scene does not change between acquisitions. The theory of compressed sensing allows us, under sparsity hypotheses, to reconstruct exactly a signal by means of convex optimization. We investigated the scenario of realistic multi-image restoration by handling the minor changes in the scene as outliers, in order to determine whether the sparsity hypothesis leads to an exact reconstruction also in the presence of outliers. We showed that the tructure of the outliers and signal jumps (using the popular total variation as the « L1-norm ») must conform to certain conditions in order to achieve robustness to outliers. We also studied the choice of the regularization parameter in this setting.

We provided a fine modeling and analysis of the so-called flutter-shutter cameras, which reduce motion blur. This study yielded the development of the first methodological, mathematical and numerical framework in order to optimize flutter-shutter cameras. This innovative work was highlighted in SIAMS News, a wide audience publication by SIAM.

Stochastic image modeling

A recurrent activity within our group is the mathematical modeling of natural images and texture. This axis has been investigated along different modalities. First, we have demonstrated the usefulness of different statistical constraints (sparsity, local dependency, Fourier spectrum, CNN features) for the purpose of both texture synthesis and texture-aware image restoration. In the same direction, a simple explanation for the second order statistical structure of color images has been provided. Going beyond the second order property of natural images, our team has also proposed two contributions for the stochastic modeling of image patches. We have introduced geometric and radiometric invariance properties for the Gaussian mixture modeling of patches. Still in the framework of a Gaussian modeling of patches, we have introduced a hyperprior-based approach for the restoration of images, with specific applications to denoising, interpolation and single-image HDR imaging. Some of the mathematical models we have developed have also been explored for the synthesis of abstract images.

Another activity is the development of statistical methods for low level detection tasks. The generic a contrario framework has been extended to enable the grouping of non-independent events. This generalization has proven useful for part-based object detection and line segment detection in SAR images. Another methodological contribution to a contrario methods was the introduction of a locally adaptive detection method for the purpose of default detection in old movies. Still in the same framework, methods for the matching of images through local features have been developed for SAR images and color natural images. Another contribution models convergent structures in an a contrario method with applications in mammography. The non-local patch paradigm was also exploited for image and video inpainting, combining geometry and texture. This work was awarded the Google best student paper, CVMP 2013, for A. Newson, and led to an international patent.

Machine Learning and deep neural networks

The contributions of the IMAGES team in machine learning are varied, and a recent turn has been taken to investigate deep learning methods. In particular, we developed original methods to explore the huge space of parameters, based on random search and Gaussian processes combined with Hyperband, in collaboration with Philips. To face the problems of reduced training sets, as is often the case in medical imaging, our recent focus is on transfer learning (collaborations with Philips, Epita and Huazhong University of Science and Technology). Relying on pre-trained networks (e.g. VGG) on sometimes completely different images, we discard the fully connected layers, and add specialized convolutional layers at the end of each of the five convolutional stages in VGG network. A linear combination of these specialized layers (i.e. fine to coarse feature maps) results in the final segmentation. Additionally, 3D information is taken into account.

In computer graphics, methodologies based on machine learning have also been developed. In particular SimSelect, a smart 3D interaction system, has been designed to automatically recognize regions of a 3D surface which are similar to the one currently selected, in real time. In rendering, a new local learning mechanism based on a specialized Bayesian model has been developed to denoise Monte Carlo rendering solutions. In virtual reality, the LazyNav system was proposed to easily navigate in a virtual environment, using body shape recognition to control the camera.

Noting the growing use of machine learning and deep neural networks for mimicking the human perceptual faculties, we examined one of the possible ultimate frontiers of this domain: the automatic assessment of beauty in images. More specifically, we draw a comparison between the tracks used today by computer scientists with those which have been cleared in the past literature: philosophical (aesthetics, experimental psychology, psycho-sociology), biologic (the so-called neuro-aesthetics), artistic and photographic, through handbooks and textbooks. We propose some paths to make digital methods more efficient.

The domain of deep learning is evolving very rapidly, and the team has decided to increase its research efforts in this direction.

Discrete mathematics, algebraic and structural models, and artificial intelligence

Our contributions in this domain are at the cross-road of lattice based formalisms for structured knowledge representation and information processing, with an original and strong anchoring in both mathematical morphology and symbolic artificial intelligence. Based on common algebraic features of mathematical morphology and of logics, we proposed and developed the so-called morpho-logics, in different settings. Our initial work in propositional logics was extended in two directions: exploiting morphological operations on propositional formulas to design concrete operators answering classical questions in artificial intelligence, such as revision, fusion, abduction, and extending these ideas to different logics, such as description logics, or more generally satisfaction systems and stratified institutions, encompassing large families of logics. In this general setting, revision was defined based on the notions of relaxation and dilation, and abduction on the notions of cutting, retraction and erosion. This work was exploited for spatial reasoning based on a knowledge base. For instance, interpreting an image can be formalized as an abduction process, where the « best » interpretation has to be inferred from the observations (the image) and a knowledge base (expressed in description logics and using formal concept analysis). To account for the structural information, we proposed to define mathematical morphology on hypergraphs, leading in particular to similarity measures between hypergraphs that are robust to some small transformations. Graph-based representations of image information were also used for multi-object tracking in videos, where the graph structure leads to more robustness with respect to occlusions and ambiguities for instance. The knowledge representation, both in logics and using graphs, can be further enhanced by spatial relations, which are best modeled using fuzzy sets. Our recent work in this domain includes the extension of constraint satisfaction problems to deal with complex fuzzy relations, for image understanding applications, and the comparison between distributions (for instance representing spatial relations) using mathematical morphology and optimal transport. Based on the links we established between fuzzy sets, mathematical morphology, rough sets, F-transforms, hypergraphs and formal concept analysis, we are currently investigating applications to mathematical musical representations. Applications in medical imaging are detailed.

Shape Modeling

A large part of the computer graphics activities are centered on discrete 3D surface representations such as point clouds and meshes. Several methods have been developed to approximate efficiently such typically dense data sets, with in particular (i) the sphere-mesh representation, which helps approximating complex 3D shapes with a handful of numbers and has been further developed to tackle animated 3D data; (ii) morphological frameworks to process and analyze detailed 3D shapes at large scale, including the point morphology framework for point-based mathematical morphology and bounding proxies for mesh-based conservative approximations; (iii) statistical modeling approaches to both enrich interactive modeling tools with live shape recognition primitives and improve automatic processing chains with self-similarity aware reconstruction and filtering; (iv) real time geometry processing methods, based on new parallel scalable operators designed to run efficiently on GPU and reducing by several orders of magnitude the time needed to process large 3D data sets (patent pending); (v) volume meshers, with a new fast, scalable and feature-preserving volume remesher which was published in the Computer and Graphics journal and presented at the Shape Modeling International 2016 conference, where it received the best paper award.

Geometrical and topological analysis

With the aim of processing and visualizing complex data, the team contributed to works on least squares affine transitions for global parameterization, conformal factor persistence for fast hierarchical cone extraction, weighted triangulations for geometry, topological analysis and visualization based on the analysis of critical points on 3D uncertain scalar fields (implemented in the TTK software.

Rendering

The computer graphics activities also target image synthesis applications, with several contributions related to the efficient simulation of global illumination, including a collection of works for the many-lights rendering approach, in particular regarding point-based global illumination framework which makes possible scalable rendering and non-diffuse lighting effects for this framework, as well as light cuts for the real time rendering of global illumination on large dynamic CAD models (patent pending). Physically-based rendering has also been addressed in the context of Monte Carlo simulations, with in particular a new efficient guiding strategy based on a product-space hierarchy for importance sampling, as well as a new generic Monte Carlo rendering denoiser, now used in production at Ubisoft and integrated to several major renders thanks to its open source distribution.

Vision, virtual reality and 3D printing results

The team has developed a collaboration with Osaka University, Japan to study how interactive 3D navigation could be achieved in mid-air for virtual reality applications. One of the key challenges was to let users navigate a virtual world, while still being able to use freely their hands, head and eyes. This collaboration led to the LazyNav system, which tracks subtle body motions to translate them into camera paths and was evaluated in a complete user study. This work was published at the IEEE 3DUI 2015 and received the best paper award. An extension of this work was then published in the IEEE TVCG journal, with in particular an adaptation to head-mounted displays (HMD) such as the Oculus Rift or the HTC Vive.

Regarding 3D printing, the team has developed a collaboration with TU Berlin, Germany, to create new geometry processing operators designed to enhance the visual perception of 3D shapes when printed at different scales. The main contribution of this work boils down to a new form of unsharp masking filter for 3D geometry, which scales to large data sets as it does not involve any global optimization. This work was published in the Computer and Graphics journal and presented at the Shape Modeling International 2017 conference, where it received the best paper award. Regarding 3D computer vision, the team has developed a collaboration with CNR Pisa, Italy, during which the team hosted Italian students for several months, to develop new point-based methods for captured 3D content. In particular, a new change detection algorithm has been proposed to address the problem of large « re-scans » performed over previously digitalized areas.

Remote sensing and SAR imagery

The team is leading research activities at different levels for the improvement and exploitation of remote sensing data, specially SAR images.

Concerning the acquisition system and the modeling of SAR image synthesis, different works have been led. A new model has been developed exploiting the complex spectrum and target extraction to reduce the side-lobe effects and the between pixel correlation, thus improving further processing. A method to combine a high resolution single polarization image with a polarimetric data of lower resolution through complex spectrum exploitation has been proposed. It received the best student paper award of EUSAR conference. Concerning moving targets, an approach for ground moving trajectory reconstruction with single-channel circular SAR data has been presented.

At the level of data modeling and speckle reduction, we have carried on our modeling work on Mellin framework by developing the following topics: the use of Meijer distribution as a generic law for SAR data; the study of the mimicking potential of Fisher and Generalized Gamma distributions; the generalization of the wavelet operators to deal with multiplicative noise. To reduce speckle, efficient patch-based approaches have been developed along the past years. A general framework is presented, able to process complex vectorial SAR data from a wide range of SAR images (interferometric, polarimetric, and combination) and to select automatically the best local parameters. This paper has been awarded the best paper award of IEEE TGRS. We proposed to combine patch-based methods with TV (Total Variation) regularization to take benefit from both models and applied it to elevation reconstruction with multi-channel interferometric data. Recently, a decomposition framework of complex SAR data called MuLOG has been proposed to take benefit from the advances in Gaussian noise reduction and opening the way to new speckle reduction methods.

At a higher level of information extraction, Markovian models (such as triplet Markov fields) have been defined for segmentation and elevation retrieval, with graph-cuts as optimization tools. Markov random fields have also been exploited for water surface extraction in the context of the future SWOT mission, or for road network detection combining optic and SAR sensors. By introducing contextual information in the context of SAR tomography, elevation information inside the resolution cells has been recovered. The popular SIFT control points, widespread in computer vision, have been adapted to the statistical specificities of SAR data with « SAR-SIFT » key-points and descriptors.

The main applications in remote sensing benefiting from the previously described methods are the followings: multi-temporal data processing and analysis, urban area monitoring, and the SWOT mission. First, we have worked in the past years on SAR time series. Methods for time series decomposition as a mixture of background and bright targets (possibly changing in time) have been proposed. An adaptation of patch-based approaches for multi-temporal despeckling has been presented, followed by change detection and analysis. A visualization tool for change detection in SAR time series has been recently proposed (best paper award of the CFPT conference). We also worked with optical range images on change detection and the use of expert systems to analyze time series. Secondly, we are focusing on 3D building monitoring in urban areas using interferometry, polarimetry and tomography. Besides, the team is also involved since 2016 as an ADT (Algorithm Definition Team) member of the coming SWOT (Surface Water and Ocean Topography) mission for water surface and hydrological networks survey. Other applications deal with Ground Penetrating SAR in the framework of the FUI project G4M, object classification (e.g. ships from remote sensing data) using machine learning approaches and 3D reconstruction focused on shallow water seabed.

A book on remote sensing imagery has been co-editored.

Collaborations: Telecom Saint-Etienne (laboratoire Hubert-Curien), IMB, Universities: Rennes I (IETR), Savoie Mont-Blanc (LISTIC), SupCom Tunis (Tunisia), Parthenope University (Italy), ONERA, CNES, CEA.

Medical imaging

The paradigm underlying all our contributions in medical imaging is the modeling of available knowledge and information, which is then used for image understanding. The knowledge to be modeled includes acquisition characteristics (geometry, statistical signal or noise properties), anatomy, shape and appearance, spatial relations, pathology characteristics. This leads to mathematical models, that are then exploited in segmentation, recognition and higher level understanding. For instance, X-ray acquisitions were modeled in the case of low-dose imaging and for dose estimation, for dual acquisitions in brain C-arm CBCT (Cone beam computed tomography), or for new tomosynthesis methods. Shape and appearance models were developed, together with spatial relations, for vessels in high resolution CT, abdominal multi-organ localization and segmentation, teeth segmentation and recognition in CBCT, and retina vessels in optical coherence tomography and adaptive optics with clinical applications. The developed methods strongly rely on the theoretical developments described in Section, and were applied on different imaging modalities and for different types of anatomical structures and pathologies, with two main directions: brain and pediatric imaging. Our most recent works in these directions include the localization and segmentation of brain lesions in MRI (Magnetic Resonance Imaging), guided by PET (Positon Emission Tomography) images (from combined PET-MRI acquisitions) using max-trees and deformable models, the segmentation of neonatal brain images (normal structures and white matter hyperintensities) using mathematical morphology, spatial relations and deep-learning, denoising of biological images, and new applications in image-guided surgery of the pelvis for pediatric patients, for which we developed segmentation and 3D modeling methods, again relying on computational models of the anatomy. A new topic recently emerged in the team on brain tractography with contributions in: 1) the analysis of structural connectivity using diffeomorphisms, 2) a reproducible and anatomically relevant segmentation of tractograms based on fuzzy spatial relations modeling their qualitative anatomical definitions, 3) the definition and comparison of appropriate distances suited for the analysis of tractograms.

Conversely, images can also be used to build realistic anatomical models. This work originates from the need to base simulations of wave propagation (typically to study the influence of electromagnetic waves on biological tissues) on realistic models. Within several ANR and ANSES national and international projects, we developed original voxelized and meshed 3D models of adult and children brains, fetus and pregnant women, whole body children, from MRI and ultrasound 3D images, and associated deformation models to simulate different positions and sizes~\cite{SD:MEDIA-15,SD:PMB-16}. Models are publicly available for research purposes at http://femonum.telecom-paristech.fr/.

Collaborations: RHU (Lariboisière and XV-XX hospital, ISEP), IMABRAIN (Sainte-Anne), IMAG2 (Necker), academic (at international level: Trento, Sao Paulo, HUST…; at national level: LRDE-Epita, Aramis team, ISEP, LIX…), medical (several CHU in Paris, joint teams, C. Adamsbaum (Bicêtre hospital) associated researcher, PhD and master theses of medical doctors) and industrial collaborations (Philips, General Electric, KeenEyes…).

Computational Photography

Several applications described in previous sections are related to computational photography and have been developed with partners: Technicolor, DxO, Adobe in order to improve the efficiency and capability of modern cameras. Moreover, an in-depth study of the modern photographic camera under various aspects: optical, electronic, digital \ldots led to a comprehensive handbook, the first one published on this topic.

Creative Industries

The computer graphics activities of IMAGES above presented have strong ties to video game, computer-aided design (CAD) and VFX/Animated Picture industries. First, an Industrial Research Chair has been co-conducted during five years, funded by Dassault Systèmes, PSA, Ubisoft and Orange, with the aim to determine how imaginary worlds can be modeled, fed and used to develop innovative concepts in various industries. One key challenge occurring in this project relates to the ability to convey the mental model of a 3D scene to a computer, interactively and intuitively. Secondly, a long term collaboration with Ubisoft Motion Picture has exploited the research results coming from the group for actual movie and TV series productions. Last, a joint research laboratory with Allegorithmic has been created to invent the future technologies of 3D digital content creation (3D DCC) for games, design and VFX.