Search

Surgical scene perception via videos is critical for advancing robotic surgery, telesurgery, and AI-assisted surgery, particularly in ophthalmology. However, the scarcity of diverse and richly annotated video datasets has hindered the development of intelligent systems for surgical workflow analysis. Existing datasets face challenges such as small scale, lack of diversity in surgery and phase categories, and absence of time-localized annotations. These limitations impede action understanding and model generalization validation in complex and diverse real-world surgical scenarios. To address this gap, we introduce OphNet, a large-scale, expert-annotated video benchmark for ophthalmic surgical workflow understanding. OphNet features: 1) A diverse collection of 2,278 surgical videos spanning 66 types of cataract, glaucoma, and corneal surgeries, with detailed annotations for 102 unique surgical phases and 150 fine-grained operations. 2) Sequential and hierarchical annotations for each surgery, phase, and operation, enabling comprehensive understanding and improved interpretability. 3) Time-localized annotations, facilitating temporal localization and prediction tasks within surgical workflows. With approximately 285 hours of surgical videos, OphNet is about 20 times larger than the largest existing surgical workflow analysis benchmark. Code and dataset are available at: https://minghu0830.github.io/OphNet-benchmark/., Comment: Accepted by ECCV 2024

Diffusion probabilistic models (DPMs) which employ explicit likelihood characterization and a gradual sampling process to synthesize data, have gained increasing research interest. Despite their huge computational burdens due to the large number of steps involved during sampling, DPMs are widely appreciated in various medical imaging tasks for their high-quality and diversity of generation. Magnetic resonance imaging (MRI) is an important medical imaging modality with excellent soft tissue contrast and superb spatial resolution, which possesses unique opportunities for DPMs. Although there is a recent surge of studies exploring DPMs in MRI, a survey paper of DPMs specifically designed for MRI applications is still lacking. This review article aims to help researchers in the MRI community to grasp the advances of DPMs in different applications. We first introduce the theory of two dominant kinds of DPMs, categorized according to whether the diffusion time step is discrete or continuous, and then provide a comprehensive review of emerging DPMs in MRI, including reconstruction, image generation, image translation, segmentation, anomaly detection, and further research topics. Finally, we discuss the general limitations as well as limitations specific to the MRI tasks of DPMs and point out potential areas that are worth further exploration.

Estimating parameters from data is a fundamental problem in physics, customarily done by minimizing a loss function between a model and observed statistics. In scattering-based analysis, researchers often employ their domain expertise to select a specific range of wavevectors for analysis, a choice that can vary depending on the specific case. We introduce another paradigm that defines a probabilistic generative model from the beginning of data processing and propagates the uncertainty for parameter estimation, termed ab initio uncertainty quantification (AIUQ). As an illustrative example, we demonstrate this approach with differential dynamic microscopy (DDM) that extracts dynamical information through Fourier analysis at a selected range of wavevectors. We first show that DDM is equivalent to fitting a temporal variogram in the reciprocal space using a latent factor model as the generative model. Then we derive the maximum marginal likelihood estimator, which optimally weighs information at all wavevectors, therefore eliminating the need to select the range of wavevectors. Furthermore, we substantially reduce the computational cost by utilizing the generalized Schur algorithm for Toeplitz covariances without approximation. Simulated studies validate that AIUQ significantly improves estimation accuracy and enables model selection with automated analysis. The utility of AIUQ is also demonstrated by three distinct sets of experiments: first in an isotropic Newtonian fluid, pushing limits of optically dense systems compared to multiple particle tracking; next in a system undergoing a sol-gel transition, automating the determination of gelling points and critical exponent; and lastly, in discerning anisotropic diffusive behavior of colloids in a liquid crystal. These outcomes collectively underscore AIUQ's versatility to capture system dynamics in an efficient and automated manner., Comment: 23 pages, 9 figures

The dynamics of cellular pattern formation is crucial for understanding embryonic development and tissue morphogenesis. Recent studies have shown that human dermal fibroblasts cultured on liquid crystal elastomers can exhibit an increase in orientational alignment over time, accompanied by cell proliferation, under the influence of the weak guidance of a molecularly aligned substrate. However, a comprehensive understanding of how this order arises remains largely unknown. This knowledge gap may be attributed, in part, to a scarcity of mechanistic models that can capture the temporal progression of the complex nonequilibrium dynamics during the cellular alignment process. The orientational alignment occurs primarily when cells reach a high density near confluence. Therefore, for accurate modeling, it is crucial to take into account both the cell-cell interaction term and the influence from the substrate, acting as a one-body external potential term. To fill in this gap, we develop a hybrid procedure that utilizes statistical learning approaches to extend the state-of-the-art physics models for quantifying both effects. We develop a more efficient way to perform feature selection that avoids testing all feature combinations through simulation. The maximum likelihood estimator of the model was derived and implemented in computationally scalable algorithms for model calibration and simulation. By including these features, such as the non-Gaussian, anisotropic fluctuations, and limiting alignment interaction only to neighboring cells with the same velocity direction, this model quantitatively reproduce the key system-level parameters--the temporal progression of the velocity orientational order parameters and the variability of velocity vectors, whereas models missing any of the features fail to capture these temporally dependent parameters., Comment: 20 pages, 14 figures

The ability of cells to reorganize in response to external stimuli is important in areas ranging from morphogenesis to tissue engineering. Elongated cells can co-align due to steric effects, forming states with local order. We show that molecular-scale substrate anisotropy can direct cell organization, resulting in the emergence of nematic order on tissue scales. To quantitatively examine the disorder-order transition, we developed a high-throughput imaging platform to analyze velocity and orientational correlations for several thousand cells over days. The establishment of global, seemingly long-ranged order is facilitated by enhanced cell division along the substrate's nematic axis, and associated extensile stresses that restructure the cells' actomyosin networks. Our work, which connects to a class of systems known as active dry nematics, provides a new understanding of the dynamics of cellular remodeling and organization in weakly interacting cell collectives. This enables data-driven discovery of cell-cell interactions and points to strategies for tissue engineering., Comment: 29 pages, 7 figures

Physically intelligent micro-robotic systems exploit information embedded in micro-robots, their colloidal cargo, and their milieu to interact, assemble and form functional structures. Nonlinear anisotropic fluids like nematic liquid crystals (NLCs) provide untapped opportunities to embed interactions via their topological defects, complex elastic responses, and their ability to dramatically restructure in dynamic settings. Here we design and fabricate a 4-armed ferromagnetic micro-robot to embed and dynamically reconfigure information in the nematic director field, generating a suite of physical interactions for cargo manipulation. The micro-robot shape and surface chemistry are designed to generate a nemato-elastic energy landscape in the domain that defines multiple modes of emergent, bottom-up interactions with passive colloids. Micro-robot rotation expands the ability to sculpt interactions; the energy landscape around a rotating micro-robot is dynamically reconfigured by complex far-from-equilibrium dynamics of the micro-robot's companion topological defect. These defect dynamics allow transient information to be programmed into the domain and exploited for top-down cargo manipulation. We demonstrate robust micro-robotic manipulation strategies that exploit these diverse modes of nemato-elastic interaction to achieve cargo docking, transport, release, and assembly of complex reconfigurable structures at multi-stable sites. Such structures are of great interest to future developments of LC-based advanced optical device and micro-manufacturing in anisotropic environments., Comment: 23 pages, 8 figures in the main text and 4 figures in the SI

Non-equilibrium dynamics of topological defects can be used as a fundamental propulsion mechanism in microscopic active matter. Here, we demonstrate swimming of topological defect-propelled colloidal particles in (passive) nematic fluids through experiments and numerical simulations. Dynamic swim strokes of the topological defects are driven by colloidal rotation in an external magnetic field, causing periodic defect rearrangement which propels the particles. The swimming velocity is determined by the colloid's angular velocity, sense of rotation and defect polarity. By controlling them we can locomote the particles along different trajectories. We demonstrate scattering -- that is, the effective pair interactions -- of two of our defect-propelled swimmers, which we show is highly anisotropic and depends on the microscopic structure of the defect stroke, including the local defect topology and polarity. More generally, this work aims to develop biomimetic active matter based on the underlying relevance of topology.

Differential dynamic microscopy (DDM) is a form of video image analysis that combines the sensitivity of scattering and the direct visualization benefits of microscopy. DDM is broadly useful in determining dynamical properties including the intermediate scattering function for many spatiotemporally correlated systems. Despite its straightforward analysis, DDM has not been fully adopted as a routine characterization tool, largely due to computational cost and lack of algorithmic robustness. We present statistical analysis that quantifies the noise, reduces the computational order and enhances the robustness of DDM analysis. We propagate the image noise through the Fourier analysis, which allows us to comprehensively study the bias in different estimators of model parameters, and we derive a different way to detect whether the bias is negligible. Furthermore, through use of Gaussian process regression (GPR), we find that predictive samples of the image structure function require only around 0.5%-5% of the Fourier transforms of the observed quantities. This vastly reduces computational cost, while preserving information of the quantities of interest, such as quantiles of the image scattering function, for subsequent analysis. The approach, which we call DDM with uncertainty quantification (DDM-UQ), is validated using both simulations and experiments with respect to accuracy and computational efficiency, as compared with conventional DDM and multiple particle tracking. Overall, we propose that DDM-UQ lays the foundation for important new applications of DDM, as well as to high-throughput characterization. We implement the fast computation tool in a new, publicly available MATLAB software package., Comment: Published in Physical Review E. 24 pages, 12 figures. Typos in Section 2B are corrected

Rain streaks in the air appear in various blurring degrees and resolutions due to different distances from their positions to the camera. Similar rain patterns are visible in a rain image as well as its multi-scale (or multi-resolution) versions, which makes it possible to exploit such complementary information for rain streak representation. In this work, we explore the multi-scale collaborative representation for rain streaks from the perspective of input image scales and hierarchical deep features in a unified framework, termed multi-scale progressive fusion network (MSPFN) for single image rain streak removal. For similar rain streaks at different positions, we employ recurrent calculation to capture the global texture, thus allowing to explore the complementary and redundant information at the spatial dimension to characterize target rain streaks. Besides, we construct multi-scale pyramid structure, and further introduce the attention mechanism to guide the fine fusion of this correlated information from different scales. This multi-scale progressive fusion strategy not only promotes the cooperative representation, but also boosts the end-to-end training. Our proposed method is extensively evaluated on several benchmark datasets and achieves state-of-the-art results. Moreover, we conduct experiments on joint deraining, detection, and segmentation tasks, and inspire a new research direction of vision task-driven image deraining. The source code is available at \url{https://github.com/kuihua/MSPFN}., Comment: CVPR 2020

The ability to dictate colloid motion is an important challenge in fields ranging from materials science to living systems. Here, by embedding energy landscapes in confined nematic liquid crystals, we design a versatile platform to define colloidal migration. This is achieved by placing a wavy wall, with alternating hills and wells, in nematic liquid crystals, to impose a smooth elastic energy field with alternating splay and bend distortions. This domain generates (meta) stable loci that act as attractors and unstable loci that repel colloids over distances large compared to the colloid radius. Energy gradients in the vicinity of these loci are exploited to dictate colloid trajectories. We demonstrate several aspects of this control, by studying transitions between defect configurations, propelling particles along well defined paths and exploiting multistable systems to send particles to particular sites within the domain. Such tailored landscapes have promise in reconfigurable systems and in microrobotics applications, Comment: 13 pages, 8 figures

The ability to control the movement and assembly of particles in liquid crystals is not only an important route to design functional materials, but also sheds light on the mechanisms of colloidal interactions. In this study we place micron-sized particles with "Saturn ring" defects near a wall with hills and dales that impose perpendicular (homeotropic) molecular anchoring. The strong splay distortion at the wall interacts with the distortion around the particles in the near field and favors their migration towards the dales via the so-called "lock-and-key" mechanism. We demonstrate experimentally that the lock-and-key mechanism can robustly localize a particle at specific topographical features. We observe the complex trajectories traced by the particles as they dock on the dales, estimate the binding energy, and explore a range of parameters that favor or disfavor the docking event, thus exploiting the capabilities of our experimental system. We extend the study to colloids with homeotropic anchoring but with an associated point defect instead of a Saturn ring and show that they find a different preferred location, i.e. we can place otherwise identical particles at well defined sites according to their topological defect structure. Finally, for deep enough wells, confinement drives topological transitions of Saturn rings to dipoles. This ability to tailor wall geometry to guide colloids to well defined sites within nematic liquid crystals represents an important new tool in directed assembly., Comment: 18 pages, 6 figures

Confined smectic A liquid crystals (SmA LCs) form topological defects called focal conic domains (FCDs) that focus light as gradient-index lenses. Here, we exploit surface curvature to self-assemble FCDs in a single step into a hierarchical structure (coined "flower pattern") molded by the fluid interface that is pinned at the top of a micropillar. The structure resembles the compound eyes of some invertebrates, which consist of hundreds of microlenses on a curved interface, able to focus and construct images in three dimensions. Here we demonstrate that these flowers are indeed "compound eyes" with important features which have not been demonstrated previously in the literature. The eccentric FCDs gradually change in size with radial distance from the edge of the micropillar, resulting in a variable microlens focal length that ranges from a few microns to a few tens of microns within a single "flower". We show that the microlenses can construct a composite 3D image from different depth of field. Moreover, the smectic "compound eye" can be reconfigured by heating and cooling at the LC phase transition temperature; its field of view can be manipulated by tuning the curvature of the LC interface, and the lenses are sensitive to light polarization., Comment: 17 pages, 9 figures (4 supplementary), 3 supplementary movies. This is the pre-peer reviewed version of the article which has been published in its final form in Advanced Optical Materials and is available at http://onlinelibrary.wiley.com/doi/10.1002/adom.201500153/abstract