Search

Power Doppler ultrasound is in widespread clinical use for non-invasive vascular imaging but the most common current method - Delay and Sum (DAS) beamforming - suffers from limited resolution and high side-lobes. Here we propose the Sub-Aperture Angular Multiply and Sum (SAMAS) algorithm; it combines the advantages of two recent non-linear beamformers, Frame Multiply and Sum (FMAS) which uses signal temporal coherence and the acoustic sub-aperture (ASAP) algorithm, which uses signal spatial coherence, respectively. Following in vitro experiments to optimise the algorithm, particularly the use of phase information and sub-aperture pairing, it was evaluated in vivo, first in a rabbit kidney and then in human lymph node, using ultrafast ultrasound images obtained with intravenous contrast agents. The SAMAS algorithm improved the CNR and SNR across all tests, on average raising the CNR by 11 dB and the SNR by 18 dB over DAS in vivo. This work demonstrates a promising vascular imaging method that could have widespread clinical utility.

Levels of selection and multilevel evolutionary processes are essential concepts in evolutionary theory, and yet there is a lack of common mathematical models for these core ideas. Here, we propose a unified mathematical framework for formulating and optimizing multilevel evolutionary processes and genetic algorithms over arbitrarily many levels based on concepts from category theory and population genetics. We formulate a multilevel version of the Wright-Fisher process using this approach, and we show that this model can be analyzed to clarify key features of multilevel selection. Particularly, we derive an extended multilevel probabilistic version of Price's Equation via the Kantorovich Monad, and we use this to characterize regimes of parameter space within which selection acts antagonistically or cooperatively across levels. Finally, we show how our framework can provide a unified setting for learning genetic algorithms (GAs), and we show how we can use a Variational Optimization and a multi-level analogue of coalescent analysis to fit multilevel GAs to simulated data., Comment: 18 pages, 3 figures

This paper introduces FlowMap, an end-to-end differentiable method that solves for precise camera poses, camera intrinsics, and per-frame dense depth of a video sequence. Our method performs per-video gradient-descent minimization of a simple least-squares objective that compares the optical flow induced by depth, intrinsics, and poses against correspondences obtained via off-the-shelf optical flow and point tracking. Alongside the use of point tracks to encourage long-term geometric consistency, we introduce differentiable re-parameterizations of depth, intrinsics, and pose that are amenable to first-order optimization. We empirically show that camera parameters and dense depth recovered by our method enable photo-realistic novel view synthesis on 360-degree trajectories using Gaussian Splatting. Our method not only far outperforms prior gradient-descent based bundle adjustment methods, but surprisingly performs on par with COLMAP, the state-of-the-art SfM method, on the downstream task of 360-degree novel view synthesis (even though our method is purely gradient-descent based, fully differentiable, and presents a complete departure from conventional SfM)., Comment: Project website: https://cameronosmith.github.io/flowmap/

Mathematical models of infectious disease transmission typically neglect within-host dynamics. Yet within-host dynamics - including pathogen replication, host immune responses, and interactions with microbiota - are crucial not only for determining the progression of disease at the individual level, but also for driving within-host evolution and onwards transmission, and therefore shape dynamics at the population level. Various approaches have been proposed to model both within- and between-host dynamics, but these typically require considerable simplifying assumptions to couple processes at contrasting scales (e.g., the within-host dynamics quickly reach a steady state) or are computationally intensive. Here we propose a novel, readily adaptable and broadly applicable method for modelling both within- and between-host processes which can fully couple dynamics across scales and is both realistic and computationally efficient. By individually tracking the deterministic within-host dynamics of infected individuals, and stochastically coupling these to continuous host state variables at the population-level, we take advantage of fast numerical methods at both scales while still capturing individual transient within-host dynamics and stochasticity in transmission between hosts. Our approach closely agrees with full stochastic individual-based simulations and is especially useful when the within-host dynamics do not rapidly reach a steady state or over longer timescales to track pathogen evolution. By applying our method to different pathogen growth scenarios we show how common simplifying assumptions fundamentally change epidemiological and evolutionary dynamics., Comment: 34 pages, 5 figures

The field of generative image inpainting and object insertion has made significant progress with the recent advent of latent diffusion models. Utilizing a precise object mask can greatly enhance these applications. However, due to the challenges users encounter in creating high-fidelity masks, there is a tendency for these methods to rely on more coarse masks (e.g., bounding box) for these applications. This results in limited control and compromised background content preservation. To overcome these limitations, we introduce SmartMask, which allows any novice user to create detailed masks for precise object insertion. Combined with a ControlNet-Inpaint model, our experiments demonstrate that SmartMask achieves superior object insertion quality, preserving the background content more effectively than previous methods. Notably, unlike prior works the proposed approach can also be used even without user-mask guidance, which allows it to perform mask-free object insertion at diverse positions and scales. Furthermore, we find that when used iteratively with a novel instruction-tuning based planning model, SmartMask can be used to design detailed layouts from scratch. As compared with user-scribble based layout design, we observe that SmartMask allows for better quality outputs with layout-to-image generation methods. Project page is available at https://smartmask-gen.github.io

Reconstruction of 3D neural fields from posed images has emerged as a promising method for self-supervised representation learning. The key challenge preventing the deployment of these 3D scene learners on large-scale video data is their dependence on precise camera poses from structure-from-motion, which is prohibitively expensive to run at scale. We propose a method that jointly reconstructs camera poses and 3D neural scene representations online and in a single forward pass. We estimate poses by first lifting frame-to-frame optical flow to 3D scene flow via differentiable rendering, preserving locality and shift-equivariance of the image processing backbone. SE(3) camera pose estimation is then performed via a weighted least-squares fit to the scene flow field. This formulation enables us to jointly supervise pose estimation and a generalizable neural scene representation via re-rendering the input video, and thus, train end-to-end and fully self-supervised on real-world video datasets. We demonstrate that our method performs robustly on diverse, real-world video, notably on sequences traditionally challenging to optimization-based pose estimation techniques., Comment: Project website: http://cameronosmith.github.io/flowcam

We introduce a method for novel view synthesis given only a single wide-baseline stereo image pair. In this challenging regime, 3D scene points are regularly observed only once, requiring prior-based reconstruction of scene geometry and appearance. We find that existing approaches to novel view synthesis from sparse observations fail due to recovering incorrect 3D geometry and due to the high cost of differentiable rendering that precludes their scaling to large-scale training. We take a step towards resolving these shortcomings by formulating a multi-view transformer encoder, proposing an efficient, image-space epipolar line sampling scheme to assemble image features for a target ray, and a lightweight cross-attention-based renderer. Our contributions enable training of our method on a large-scale real-world dataset of indoor and outdoor scenes. We demonstrate that our method learns powerful multi-view geometry priors while reducing the rendering time. We conduct extensive comparisons on held-out test scenes across two real-world datasets, significantly outperforming prior work on novel view synthesis from sparse image observations and achieving multi-view-consistent novel view synthesis., Comment: CVPR 2023, Project Webpage: https://yilundu.github.io/wide_baseline/, Last Two Authors Equal Advising

3D-aware GANs offer new capabilities for view synthesis while preserving the editing functionalities of their 2D counterparts. GAN inversion is a crucial step that seeks the latent code to reconstruct input images or videos, subsequently enabling diverse editing tasks through manipulation of this latent code. However, a model pre-trained on a particular dataset (e.g., FFHQ) often has difficulty reconstructing images with out-of-distribution (OOD) objects such as faces with heavy make-up or occluding objects. We address this issue by explicitly modeling OOD objects from the input in 3D-aware GANs. Our core idea is to represent the image using two individual neural radiance fields: one for the in-distribution content and the other for the out-of-distribution object. The final reconstruction is achieved by optimizing the composition of these two radiance fields with carefully designed regularization. We demonstrate that our explicit decomposition alleviates the inherent trade-off between reconstruction fidelity and editability. We evaluate reconstruction accuracy and editability of our method on challenging real face images and videos and showcase favorable results against other baselines., Comment: Project page: https://in-n-out-3d.github.io/

In biology, stochastic branching processes with a two-stage, hierarchical structure arise in the study of population dynamics, gene expression, and phylogenetic inference. These models have been commonly analyzed using generating functions, the method of characteristics and various perturbative approximations. Here we describe a general method for analyzing hierarchic first-order reaction networks using Lie theory. Crucially, we identify the fact that the Lie group associated to hierarchic reaction networks decomposes as a wreath product of the groups associated to the subnetworks of the independent and dependent types. After explaining the general method, we illustrate it on a model of population dynamics and the so-called two-state or telegraph model of single-gene transcription. Solutions to such processes provide essential input to downstream methods designed to attempt to infer parameters of these and related models., Comment: 9 pages

Controllable image synthesis with user scribbles is a topic of keen interest in the computer vision community. In this paper, for the first time we study the problem of photorealistic image synthesis from incomplete and primitive human paintings. In particular, we propose a novel approach paint2pix, which learns to predict (and adapt) "what a user wants to draw" from rudimentary brushstroke inputs, by learning a mapping from the manifold of incomplete human paintings to their realistic renderings. When used in conjunction with recent works in autonomous painting agents, we show that paint2pix can be used for progressive image synthesis from scratch. During this process, paint2pix allows a novice user to progressively synthesize the desired image output, while requiring just few coarse user scribbles to accurately steer the trajectory of the synthesis process. Furthermore, we find that our approach also forms a surprisingly convenient approach for real image editing, and allows the user to perform a diverse range of custom fine-grained edits through the addition of only a few well-placed brushstrokes. Supplemental video and demo are available at https://1jsingh.github.io/paint2pix, Comment: ECCV 2022

Neural scene representations, both continuous and discrete, have recently emerged as a powerful new paradigm for 3D scene understanding. Recent efforts have tackled unsupervised discovery of object-centric neural scene representations. However, the high cost of ray-marching, exacerbated by the fact that each object representation has to be ray-marched separately, leads to insufficiently sampled radiance fields and thus, noisy renderings, poor framerates, and high memory and time complexity during training and rendering. Here, we propose to represent objects in an object-centric, compositional scene representation as light fields. We propose a novel light field compositor module that enables reconstructing the global light field from a set of object-centric light fields. Dubbed Compositional Object Light Fields (COLF), our method enables unsupervised learning of object-centric neural scene representations, state-of-the-art reconstruction and novel view synthesis performance on standard datasets, and rendering and training speeds at orders of magnitude faster than existing 3D approaches., Comment: Project website: https://cameronosmith.github.io/colf. TMLR 2023

The generation of well-designed artwork is often quite time-consuming and assumes a high degree of proficiency on part of the human painter. In order to facilitate the human painting process, substantial research efforts have been made on teaching machines how to "paint like a human", and then using the trained agent as a painting assistant tool for human users. However, current research in this direction is often reliant on a progressive grid-based division strategy wherein the agent divides the overall image into successively finer grids, and then proceeds to paint each of them in parallel. This inevitably leads to artificial painting sequences which are not easily intelligible to human users. To address this, we propose a novel painting approach which learns to generate output canvases while exhibiting a more human-like painting style. The proposed painting pipeline Intelli-Paint consists of 1) a progressive layering strategy which allows the agent to first paint a natural background scene representation before adding in each of the foreground objects in a progressive fashion. 2) We also introduce a novel sequential brushstroke guidance strategy which helps the painting agent to shift its attention between different image regions in a semantic-aware manner. 3) Finally, we propose a brushstroke regularization strategy which allows for ~60-80% reduction in the total number of required brushstrokes without any perceivable differences in the quality of the generated canvases. Through both quantitative and qualitative results, we show that the resulting agents not only show enhanced efficiency in output canvas generation but also exhibit a more natural-looking painting style which would better assist human users express their ideas through digital artwork.

The COVID-19 pandemic, caused by the coronavirus SARS-CoV-2, has led to a wide range of non-pharmaceutical interventions being implemented around the world to curb transmission. However, the economic and social costs of some of these measures, especially lockdowns, has been high. An alternative and widely discussed public health strategy for the COVID-19 pandemic would have been to 'shield' those most vulnerable to COVID-19 (minimising their contacts with others), while allowing infection to spread among lower risk individuals with the aim of reaching herd immunity. Here we retrospectively explore the effectiveness of this strategy using a stochastic SEIR framework, showing that even under the unrealistic assumption of perfect shielding, hospitals would have been rapidly overwhelmed with many avoidable deaths among lower risk individuals. Crucially, even a small (20%) reduction in the effectiveness of shielding would have likely led to a large increase (>150%) in the number of deaths compared to perfect shielding. Our findings demonstrate that shielding the vulnerable while allowing infections to spread among the wider population would not have been a viable public health strategy for COVID-19 and is unlikely to be effective for future pandemics.

Efficient exploitation of exascale architectures requires rethinking of the numerical algorithms used in many large-scale applications. These architectures favor algorithms that expose ultra fine-grain parallelism and maximize the ratio of floating point operations to energy intensive data movement. One of the few viable approaches to achieve high efficiency in the area of PDE discretizations on unstructured grids is to use matrix-free/partially-assembled high-order finite element methods, since these methods can increase the accuracy and/or lower the computational time due to reduced data motion. In this paper we provide an overview of the research and development activities in the Center for Efficient Exascale Discretizations (CEED), a co-design center in the Exascale Computing Project that is focused on the development of next-generation discretization software and algorithms to enable a wide range of finite element applications to run efficiently on future hardware. CEED is a research partnership involving more than 30 computational scientists from two US national labs and five universities, including members of the Nek5000, MFEM, MAGMA and PETSc projects. We discuss the CEED co-design activities based on targeted benchmarks, miniapps and discretization libraries and our work on performance optimizations for large-scale GPU architectures. We also provide a broad overview of research and development activities in areas such as unstructured adaptive mesh refinement algorithms, matrix-free linear solvers, high-order data visualization, and list examples of collaborations with several ECP and external applications., Comment: 22 pages, 18 figures

We propose an algorithm grounded in dynamical systems theory that generalizes manifold learning from a global state representation, to a network of local interacting manifolds termed a Generative Manifold Network (GMN). Manifolds are discovered using the convergent cross mapping (CCM) causal inference algorithm which are then compressed into a reduced redundancy network. The representation is a network of manifolds embedded from observational data where each orthogonal axis of a local manifold is an embedding of a individually identifiable neuron or brain area that has exact correspondence in the real world. As such these can be experimentally manipulated to test hypotheses derived from theory and data analysis. Here we demonstrate that this representation preserves the essential features of the brain of flies,larval zebrafish and humans. In addition to accurate near-term prediction, the GMN model can be used to synthesize realistic time series of whole brain neuronal activity and locomotion viewed over the long term. Thus, as a final validation of how well GMN captures essential dynamic information, we show that the artificially generated time series can be used as a training set to predict out-of-sample observed fly locomotion, as well as brain activity in out of sample withheld data not used in model building. Remarkably, the artificially generated time series show realistic novel behaviors that do not exist in the training data, but that do exist in the out-of-sample observational data. This suggests that GMN captures inherently emergent properties of the network. We suggest our approach may be a generic recipe for mapping time series observations of any complex nonlinear network into a model that is able to generate naturalistic system behaviors that identifies variables that have real world correspondence and can be experimentally manipulated., Comment: 20 pages, 15 figures; corresponding author: Gerald Pao geraldpao@gmail.com

Reaction--diffusion mechanism are a robust paradigm that can be used to represent many biological and physical phenomena over multiple spatial scales. Applications include intracellular dynamics, the migration of cells and the patterns formed by vegetation in semi-arid landscapes. Moreover, domain growth is an important process for embryonic growth and wound healing. There are many numerical modelling frameworks capable of simulating such systems on growing domains, however each of these may be well suited to different spatial scales and particle numbers. Recently, spatially extended hybrid methods on static domains have been produced in order to bridge the gap between these different modelling paradigms in order to represent multiscale phenomena. However, such methods have not been developed with domain growth in mind. In this paper, we develop three hybrid methods on growing domains, extending three of the prominent static domain hybrid methods. We also provide detailed algorithms to allow others to employ them. We demonstrate that the methods are able to accurately model three representative reaction-diffusion systems accurately and without bias., Comment: Main text: 22 pages, 6 figures. Supplementary material: 8 pages

The simulation of stochastic reaction-diffusion systems using fine-grained representations can become computationally prohibitive when particle numbers become large. If particle numbers are sufficiently high then it may be possible to ignore stochastic fluctuations and use a more efficient coarse-grained simulation approach. Nevertheless, for multiscale systems which exhibit significant spatial variation in concentration, a coarse-grained approach may not be appropriate throughout the simulation domain. Such scenarios suggest a hybrid paradigm in which a computationally cheap, coarse-grained model is coupled to a more expensive, but more detailed fine-grained model enabling the accurate simulation of the fine-scale dynamics at a reasonable computational cost. In this paper, in order to couple two representations of reaction-diffusion at distinct spatial scales, we allow them to overlap in a "blending region". Both modelling paradigms provide a valid representation of the particle density in this region. From one end of the blending region to the other, control of the implementation of diffusion is passed from one modelling paradigm to another through the use of complementary "blending functions" which scale up or down the contribution of each model to the overall diffusion. We establish the reliability of our novel hybrid paradigm by demonstrating its simulation on four exemplar reaction-diffusion scenarios., Comment: 36 pages, 30 figures

We address the problem of reconstructing spatially-varying BRDFs from a small set of image measurements. This is a fundamentally under-constrained problem, and previous work has relied on using various regularization priors or on capturing many images to produce plausible results. In this work, we present MaterialGAN, a deep generative convolutional network based on StyleGAN2, trained to synthesize realistic SVBRDF parameter maps. We show that MaterialGAN can be used as a powerful material prior in an inverse rendering framework: we optimize in its latent representation to generate material maps that match the appearance of the captured images when rendered. We demonstrate this framework on the task of reconstructing SVBRDFs from images captured under flash illumination using a hand-held mobile phone. Our method succeeds in producing plausible material maps that accurately reproduce the target images, and outperforms previous state-of-the-art material capture methods in evaluations on both synthetic and real data. Furthermore, our GAN-based latent space allows for high-level semantic material editing operations such as generating material variations and material morphing., Comment: 13 pages, 16 figures. Siggraph Asia 2020

World ships are hypothetical, large, self-contained spacecraft for crewed interstellar travel, taking centuries to reach other stars. Due to their crewed nature, size, and long trip times, the feasibility of world ships faces an additional set of challenges compared to interstellar probes. Despite their emergence in the 1980s, most of these topics remain unexplored. This article revisits some of the key feasibility issues of world ships. First, definitions of world ships from the literature are revisited and the notion of world ship positioned with respect to similar concepts such as generation ships. Second, the key question of population size is revisited in light of recent results from the literature. Third, socio-technical and economic feasibility issues are evaluated. Finally, world ships are compared to potential alternative modes of crewed interstellar travel. Key roadblocks for world ships are the considerable resources required, shifting its economic feasibility beyond the year 2300, and the development of a maintenance system capable of detecting, replacing, and repairing several components per second. The emergence of alternative, less costly modes of crewed interstellar travel at an earlier point in time might render world ships obsolete.

Domain growth is a key process in many areas of biology, including embryonic development, the growth of tissue, and limb regeneration. As a result, mechanisms for incorporating it into traditional models for cell movement, interaction, and proliferation are of great importance. A previously well-used method in order to incorporate domain growth into on-lattice reaction-diffusion models causes a build up of particles on the boundaries of the domain, which is particularly evident when diffusion is low in comparison to the rate of domain growth. Here, we present a new method which addresses this unphysical build up of particles at the boundaries, and demonstrate that it is accurate even for scenarios in which the previous method fails. Further, we discuss for which parameter regimes it is feasible to continue using the original method due to diffusion dominating the domain growth mechanism., Comment: 23 pages, 10 figures