Search

Audio-visual correlation learning aims to capture and understand natural phenomena between audio and visual data. The rapid growth of Deep Learning propelled the development of proposals that process audio-visual data and can be observed in the number of proposals in the past years. Thus encouraging the development of a comprehensive survey. Besides analyzing the models used in this context, we also discuss some tasks of definition and paradigm applied in AI multimedia. In addition, we investigate objective functions frequently used and discuss how audio-visual data is exploited in the optimization process, i.e., the different methodologies for representing knowledge in the audio-visual domain. In fact, we focus on how human-understandable mechanisms, i.e., structured knowledge that reflects comprehensible knowledge, can guide the learning process. Most importantly, we provide a summarization of the recent progress of Audio-Visual Correlation Learning (AVCL) and discuss the future research directions., Comment: arXiv admin note: text overlap with arXiv:2202.13673

Surgical instrument segmentation in laparoscopy is essential for computer-assisted surgical systems. Despite the Deep Learning progress in recent years, the dynamic setting of laparoscopic surgery still presents challenges for precise segmentation. The nnU-Net framework excelled in semantic segmentation analyzing single frames without temporal information. The framework's ease of use, including its ability to be automatically configured, and its low expertise requirements, have made it a popular base framework for comparisons. Optical flow (OF) is a tool commonly used in video tasks to estimate motion and represent it in a single frame, containing temporal information. This work seeks to employ OF maps as an additional input to the nnU-Net architecture to improve its performance in the surgical instrument segmentation task, taking advantage of the fact that instruments are the main moving objects in the surgical field. With this new input, the temporal component would be indirectly added without modifying the architecture. Using CholecSeg8k dataset, three different representations of movement were estimated and used as new inputs, comparing them with a baseline model. Results showed that the use of OF maps improves the detection of classes with high movement, even when these are scarce in the dataset. To further improve performance, future work may focus on implementing other OF-preserving augmentations.

International benchmarking competitions have become fundamental for the comparative performance assessment of image analysis methods. However, little attention has been given to investigating what can be learnt from these competitions. Do they really generate scientific progress? What are common and successful participation strategies? What makes a solution superior to a competing method? To address this gap in the literature, we performed a multi-center study with all 80 competitions that were conducted in the scope of IEEE ISBI 2021 and MICCAI 2021. Statistical analyses performed based on comprehensive descriptions of the submitted algorithms linked to their rank as well as the underlying participation strategies revealed common characteristics of winning solutions. These typically include the use of multi-task learning (63%) and/or multi-stage pipelines (61%), and a focus on augmentation (100%), image preprocessing (97%), data curation (79%), and postprocessing (66%). The "typical" lead of a winning team is a computer scientist with a doctoral degree, five years of experience in biomedical image analysis, and four years of experience in deep learning. Two core general development strategies stood out for highly-ranked teams: the reflection of the metrics in the method design and the focus on analyzing and handling failure cases. According to the organizers, 43% of the winning algorithms exceeded the state of the art but only 11% completely solved the respective domain problem. The insights of our study could help researchers (1) improve algorithm development strategies when approaching new problems, and (2) focus on open research questions revealed by this work., Comment: accepted to CVPR 2023

Formalizing surgical activities as triplets of the used instruments, actions performed, and target anatomies is becoming a gold standard approach for surgical activity modeling. The benefit is that this formalization helps to obtain a more detailed understanding of tool-tissue interaction which can be used to develop better Artificial Intelligence assistance for image-guided surgery. Earlier efforts and the CholecTriplet challenge introduced in 2021 have put together techniques aimed at recognizing these triplets from surgical footage. Estimating also the spatial locations of the triplets would offer a more precise intraoperative context-aware decision support for computer-assisted intervention. This paper presents the CholecTriplet2022 challenge, which extends surgical action triplet modeling from recognition to detection. It includes weakly-supervised bounding box localization of every visible surgical instrument (or tool), as the key actors, and the modeling of each tool-activity in the form of triplet. The paper describes a baseline method and 10 new deep learning algorithms presented at the challenge to solve the task. It also provides thorough methodological comparisons of the methods, an in-depth analysis of the obtained results across multiple metrics, visual and procedural challenges; their significance, and useful insights for future research directions and applications in surgery., Comment: MICCAI EndoVis CholecTriplet2022 challenge report. Published at Elsevier journal of Medical Image Analysis. 25 pages, 15 figures, 8 tables

This paper introduces the ``SurgT: Surgical Tracking" challenge which was organised in conjunction with MICCAI 2022. There were two purposes for the creation of this challenge: (1) the establishment of the first standardised benchmark for the research community to assess soft-tissue trackers; and (2) to encourage the development of unsupervised deep learning methods, given the lack of annotated data in surgery. A dataset of 157 stereo endoscopic videos from 20 clinical cases, along with stereo camera calibration parameters, have been provided. Participants were assigned the task of developing algorithms to track the movement of soft tissues, represented by bounding boxes, in stereo endoscopic videos. At the end of the challenge, the developed methods were assessed on a previously hidden test subset. This assessment uses benchmarking metrics that were purposely developed for this challenge, to verify the efficacy of unsupervised deep learning algorithms in tracking soft-tissue. The metric used for ranking the methods was the Expected Average Overlap (EAO) score, which measures the average overlap between a tracker's and the ground truth bounding boxes. Coming first in the challenge was the deep learning submission by ICVS-2Ai with a superior EAO score of 0.617. This method employs ARFlow to estimate unsupervised dense optical flow from cropped images, using photometric and regularization losses. Second, Jmees with an EAO of 0.583, uses deep learning for surgical tool segmentation on top of a non-deep learning baseline method: CSRT. CSRT by itself scores a similar EAO of 0.563. The results from this challenge show that currently, non-deep learning methods are still competitive. The dataset and benchmarking tool created for this challenge have been made publicly available at https://surgt.grand-challenge.org/.

We propose a new mathematical model for the interaction of skin cell populations with fibroblast growth factor and bone morphogenetic protein, occurring within deformable porous media. The equations for feather primordia pattering are based on the work by K.J. Painter et al. [J. Theoret. Biol., 437 (2018) 225--238]. We perform a linear stability analysis to identify relevant parameters in the coupling mechanisms, focusing in the regime of infinitesimal strains. We also extend the model to the case of nonlinear poroelasticity and include solid growth by means of Lee decompositions of the deformation gradient. We present a few illustrative computational examples in 2D and 3D, and briefly discuss the design of tailored efficient solvers., Comment: 25 pages, 9 figures

For optimizing production yield while limiting negative environmental impact, sustainable agriculture benefits greatly from real-time, on-the-spot analysis of soil at low cost. Colorimetric paper sensors are ideal candidates for cheap and rapid chemical spot testing. However, their field application requires previously unattained paper sensor reliability and automated readout and analysis by means of integrated mobile communication, artificial intelligence, and cloud computing technologies. Here, we report such a mobile chemical analysis system based on colorimetric paper sensors that operates under tropical field conditions. By mapping topsoil pH in a field with an area of 9 hectares, we have benchmarked the mobile system against precision agriculture standards following a protocol with reference analysis of compound soil samples. As compared with routine lab analysis, our mobile soil analysis system has correctly classified soil pH in 97% of cases while reducing the analysis turnaround time from days to minutes. Moreover, by performing on-the-spot analyses of individual compound sub-samples in the field, we have achieved a 9-fold increase of spatial resolution that reveals pH-variations not detectable in compound mapping mode. Our mobile system can be extended to perform multi-parameter chemical tests of soil nutrients for applications in environmental monitoring at marginal manufacturing cost., Comment: Main article: 23 pages and 4 figures. Supplementary Information: 12 pages and 12 figures. Includes DOI for accompanying data and images datasets and URL for github repository of accompanying python code

In-utero fetal MRI is emerging as an important tool in the diagnosis and analysis of the developing human brain. Automatic segmentation of the developing fetal brain is a vital step in the quantitative analysis of prenatal neurodevelopment both in the research and clinical context. However, manual segmentation of cerebral structures is time-consuming and prone to error and inter-observer variability. Therefore, we organized the Fetal Tissue Annotation (FeTA) Challenge in 2021 in order to encourage the development of automatic segmentation algorithms on an international level. The challenge utilized FeTA Dataset, an open dataset of fetal brain MRI reconstructions segmented into seven different tissues (external cerebrospinal fluid, grey matter, white matter, ventricles, cerebellum, brainstem, deep grey matter). 20 international teams participated in this challenge, submitting a total of 21 algorithms for evaluation. In this paper, we provide a detailed analysis of the results from both a technical and clinical perspective. All participants relied on deep learning methods, mainly U-Nets, with some variability present in the network architecture, optimization, and image pre- and post-processing. The majority of teams used existing medical imaging deep learning frameworks. The main differences between the submissions were the fine tuning done during training, and the specific pre- and post-processing steps performed. The challenge results showed that almost all submissions performed similarly. Four of the top five teams used ensemble learning methods. However, one team's algorithm performed significantly superior to the other submissions, and consisted of an asymmetrical U-Net network architecture. This paper provides a first of its kind benchmark for future automatic multi-tissue segmentation algorithms for the developing human brain in utero., Comment: Results from FeTA Challenge 2021, held at MICCAI; Manuscript submitted

Context-aware decision support in the operating room can foster surgical safety and efficiency by leveraging real-time feedback from surgical workflow analysis. Most existing works recognize surgical activities at a coarse-grained level, such as phases, steps or events, leaving out fine-grained interaction details about the surgical activity; yet those are needed for more helpful AI assistance in the operating room. Recognizing surgical actions as triplets of combination delivers comprehensive details about the activities taking place in surgical videos. This paper presents CholecTriplet2021: an endoscopic vision challenge organized at MICCAI 2021 for the recognition of surgical action triplets in laparoscopic videos. The challenge granted private access to the large-scale CholecT50 dataset, which is annotated with action triplet information. In this paper, we present the challenge setup and assessment of the state-of-the-art deep learning methods proposed by the participants during the challenge. A total of 4 baseline methods from the challenge organizers and 19 new deep learning algorithms by competing teams are presented to recognize surgical action triplets directly from surgical videos, achieving mean average precision (mAP) ranging from 4.2% to 38.1%. This study also analyzes the significance of the results obtained by the presented approaches, performs a thorough methodological comparison between them, in-depth result analysis, and proposes a novel ensemble method for enhanced recognition. Our analysis shows that surgical workflow analysis is not yet solved, and also highlights interesting directions for future research on fine-grained surgical activity recognition which is of utmost importance for the development of AI in surgery., Comment: CholecTriplet2021 challenge report. Paper accepted at Elsevier journal of Medical Image Analysis. 22 pages, 8 figures, 11 tables. Challenge website: https://cholectriplet2021.grand-challenge.org

Audio-visual correlation learning aims to capture essential correspondences and understand natural phenomena between audio and video. With the rapid growth of deep learning, an increasing amount of attention has been paid to this emerging research issue. Through the past few years, various methods and datasets have been proposed for audio-visual correlation learning, which motivate us to conclude a comprehensive survey. This survey paper focuses on state-of-the-art (SOTA) models used to learn correlations between audio and video, but also discusses some tasks of definition and paradigm applied in AI multimedia. In addition, we investigate some objective functions frequently used for optimizing audio-visual correlation learning models and discuss how audio-visual data is exploited in the optimization process. Most importantly, we provide an extensive comparison and summarization of the recent progress of SOTA audio-visual correlation learning and discuss future research directions., Comment: 8 pages, 1 figure

In the study of turbulence, intermittency is a measure of how much Kolmogorov's theory of 1941 deviates from experiments. It is quantified with the flatness of the velocity of the fluid, usually based on structure functions in the physical space. However, it can also be defined with Fourier high-pass filters. Experimental and numerical simulations suggest that the two approaches do not always give the same results. Our purpose is to compare them from the analytical point of view of functions. We do that by studying generalizations of Riemann's non-differentiable function, yielding computations that are related to some classical problems in Fourier analysis. The conclusion is that the result strongly depends on regularity. To visualize this, we establish an analogy between these generalizations and the influence of viscosity in turbulent flows. This article is motivated by the mathematical works on the multifractal formalism and the discovery of Riemann's non-differentiable function as a trajectory of polygonal vortex filaments., Comment: Accepted manuscript. To appear in SIAM Journal on Mathematical Analysis

We provide a game-theoretic analysis of consensus, assuming that processes are controlled by rational agents and may fail by crashing. We consider agents that \emph{care only about consensus}: that is, (a) an agent's utility depends only on the consensus value achieved (and not, for example, on the number of messages the agent sends) and (b) agents strictly prefer reaching consensus to not reaching consensus. We show that, under these assumptions, there is no \emph{ex post Nash Equilibrium}, even with only one failure. Roughly speaking, this means that there must always exist a \emph{failure pattern} (a description of who fails, when they fail, and which agents they do not send messages to in the round that they fail) and initial preferences for which an agent can gain by deviating. On the other hand, if we assume that there is a distribution $\pi$ on the failure patterns and initial preferences, then under minimal assumptions on $\pi$, there is a Nash equilibrium that tolerates $f$ failures (i.e., $\pi$ puts probability 1 on there being at most $f$ failures) if $f+1 < n$ (where $n$ is the total number of agents). Moreover, we show that a slight extension of the Nash equilibrium strategy is also a \emph{sequential} equilibrium (under the same assumptions about the distribution $\pi$)., Comment: Appears in Proceedings of the 35th Annual ACM Symposium on Principles of Distributed Computing, 2016

We perform the linear stability analysis for a new model for poromechanical processes with inertia (formulated in mixed form using the solid deformation, fluid pressure, and total pressure) interacting with diffusing and reacting solutes convected in the medium. We find parameter regions that lead to spatio-temporal instabilities of the coupled system. The mutual dependences between deformation and diffusive patterns are of substantial relevance in the study of morphoelastic changes in biomaterials. We provide a set of computational examples in 2D and 3D (related to brain mechanobiology) that can be used to form a better understanding on how, and up to which extent, the deformations of the porous structure dictate the generation and suppression of spatial patterning dynamics, also related to the onset of mechano-chemical waves.

Riemann's non-differentiable function is one of the most famous examples of continuous but nowhere differentiable functions, but it has also been shown to be relevant from a physical point of view. Indeed, it satisfies the Frisch-Parisi multifractal formalism, which establishes a relationship with turbulence and implies some intermittent nature. It also plays a surprising role as a physical trajectory in the evolution of regular polygonal vortices that follow the binormal flow. With this motivation, we focus on one more classic tool to measure intermittency, namely the fourth-order flatness, and we refine the results that can be deduced from the multifractal analysis to show that it diverges logarithmically. We approach the problem in two ways: with structure functions in the physical space and with high-pass filters in the Fourier space., Comment: 17 pages, 2 figures. v2: Major revision. v3: Accepted manuscript

We analyse a PDE system modelling poromechanical processes (formulated in mixed form using the solid deformation, fluid pressure, and total pressure) interacting with diffusing and reacting solutes in the medium. We investigate the well-posedness of the nonlinear set of equations using fixed-point theory, Fredholm's alternative, a priori estimates, and compactness arguments. We also propose a mixed finite element method and rigorously demonstrate the stability of the scheme. Error estimates are derived in suitable norms, and numerical experiments are conducted to illustrate the mechano-chemical coupling and to verify the theoretical rates of convergence.

We introduce a model for the mass transfer of molecular activators and inhibitors in two media separated by an interface, and study its interaction with the deformations exhibited by the two-layer skin tissue where they occur. The mathematical model results in a system of nonlinear advection-diffusion-reaction equations including cross-diffusion, and coupled with an interface elasticity problem. We propose a Galerkin method for the discretisation of the set of governing equations, involving also a suitable Newton linearisation, partitioned techniques, non-overlapping Schwarz alternating schemes, and high-order adaptive time stepping algorithms. The experimental accuracy and robustness of the proposed partitioned numerical methods is assessed, and some illustrating tests in 2D and 3D are provided to exemplify the coupling effects between the mechanical properties and the advection-diffusion-reaction interactions involving the two separate layers.

We prove that a system of coupled nonlinear Schr{\"o}dinger equations on the torus exhibits both stable and unstable small KAM tori. In particular the unstable tori are related to a beating phenomena which has been proved recently in [6]. This is the first example of unstable tori for a 1d PDE.