Search

An agent's ability to leverage past experience is critical for efficiently solving new tasks. Prior work has focused on using value function estimates to obtain zero-shot approximations for solutions to a new task. In soft Q-learning, we show how any value function estimate can also be used to derive double-sided bounds on the optimal value function. The derived bounds lead to new approaches for boosting training performance which we validate experimentally. Notably, we find that the proposed framework suggests an alternative method for updating the Q-function, leading to boosted performance., Comment: To appear in the 1st Reinforcement Learning Conference

In the field of reinforcement learning (RL), agents are often tasked with solving a variety of problems differing only in their reward functions. In order to quickly obtain solutions to unseen problems with new reward functions, a popular approach involves functional composition of previously solved tasks. However, previous work using such functional composition has primarily focused on specific instances of composition functions whose limiting assumptions allow for exact zero-shot composition. Our work unifies these examples and provides a more general framework for compositionality in both standard and entropy-regularized RL. We find that, for a broad class of functions, the optimal solution for the composite task of interest can be related to the known primitive task solutions. Specifically, we present double-sided inequalities relating the optimal composite value function to the value functions for the primitive tasks. We also show that the regret of using a zero-shot policy can be bounded for this class of functions. The derived bounds can be used to develop clipping approaches for reducing uncertainty during training, allowing agents to quickly adapt to new tasks., Comment: Accepted in UAI 2023. (11+21 pages, 2+7 figures)

In reinforcement learning (RL), the ability to utilize prior knowledge from previously solved tasks can allow agents to quickly solve new problems. In some cases, these new problems may be approximately solved by composing the solutions of previously solved primitive tasks (task composition). Otherwise, prior knowledge can be used to adjust the reward function for a new problem, in a way that leaves the optimal policy unchanged but enables quicker learning (reward shaping). In this work, we develop a general framework for reward shaping and task composition in entropy-regularized RL. To do so, we derive an exact relation connecting the optimal soft value functions for two entropy-regularized RL problems with different reward functions and dynamics. We show how the derived relation leads to a general result for reward shaping in entropy-regularized RL. We then generalize this approach to derive an exact relation connecting optimal value functions for the composition of multiple tasks in entropy-regularized RL. We validate these theoretical contributions with experiments showing that reward shaping and task composition lead to faster learning in various settings., Comment: Conference paper accepted in the Main track for AAAI-2023

Reinforcement learning (RL) is an important field of research in machine learning that is increasingly being applied to complex optimization problems in physics. In parallel, concepts from physics have contributed to important advances in RL with developments such as entropy-regularized RL. While these developments have led to advances in both fields, obtaining analytical solutions for optimization in entropy-regularized RL is currently an open problem. In this paper, we establish a mapping between entropy-regularized RL and research in non-equilibrium statistical mechanics focusing on Markovian processes conditioned on rare events. In the long-time limit, we apply approaches from large deviation theory to derive exact analytical results for the optimal policy and optimal dynamics in Markov Decision Process (MDP) models of reinforcement learning. The results obtained lead to a novel analytical and computational framework for entropy-regularized RL which is validated by simulations. The mapping established in this work connects current research in reinforcement learning and non-equilibrium statistical mechanics, thereby opening new avenues for the application of analytical and computational approaches from one field to cutting-edge problems in the other.

Phenotypic heterogeneity in cancer cells is widely observed and is often linked to drug resistance. In several cases, such heterogeneity in drug sensitivity of tumors is driven by stochastic and reversible acquisition of a drug tolerant phenotype by individual cells even in an isogenic population. Accumulating evidence further suggests that cell-fate transitions such as the epithelial to mesenchymal transition (EMT) are associated with drug resistance. In this study, we analyze stochastic models of phenotypic switching to provide a framework for analyzing cell-fate transitions such as EMT as a source of phenotypic variability in drug sensitivity. Motivated by our cell-culture based experimental observations connecting phenotypic switching in EMT and drug resistance, we analyze a coarse-grained model of phenotypic switching between two states in the presence of cytotoxic stress from chemotherapy. We derive analytical results for time-dependent probability distributions that provide insights into the rates of phenotypic switching and characterize initial phenotypic heterogeneity of cancer cells. The results obtained can also shed light on fundamental questions relating to adaptation and selection scenarios in tumor response to cytotoxic therapy.

The intrinsic stochasticity of gene expression can give rise to large fluctuations and rare events that drive phenotypic variation in a population of genetically identical cells. Characterizing the fluctuations that give rise to such rare events motivates the analysis of large deviations in stochastic models of gene expression. Recent developments in non-equilibrium statistical mechanics have led to a framework for analyzing Markovian processes conditioned on rare events and for representing such processes by conditioning-free driven Markovian processes. We use this framework, in combination with approaches based on queueing theory, to analyze a general class of stochastic models of gene expression. Modeling gene expression as a Batch Markovian Arrival Process (BMAP), we derive exact analytical results quantifying large deviations of time-integrated random variables such as promoter activity fluctuations. We find that the conditioning-free driven process can also be represented by a BMAP that has the same form as the original process, but with renormalized parameters. The results obtained can be used to quantify the likelihood of large deviations, to characterize system fluctuations conditional on rare events and to identify combinations of model parameters that can give rise to dynamical phase transitions in system dynamics., Comment: 10 pages, 2 figures, to appear in Physical Biology

Gene expression in individual cells is highly variable and sporadic, often resulting in the synthesis of mRNAs and proteins in bursts. Bursting in gene expression is known to impact cell-fate in diverse systems ranging from latency in HIV-1 viral infections to cellular differentiation. It is generally assumed that bursts are geometrically distributed and that they arrive according to a Poisson process. On the other hand, recent single-cell experiments provide evidence for complex burst arrival processes, highlighting the need for more general stochastic models. To address this issue, we invoke a mapping between general models of gene expression and systems studied in queueing theory to derive exact analytical expressions for the moments associated with mRNA/protein steady-state distributions. These moments are then used to derive explicit conditions, based entirely on experimentally measurable quantities, that determine if the burst distributions deviate from the geometric distribution or if burst arrival deviates from a Poisson process. For non-Poisson arrivals, we develop approaches for accurate estimation of burst parameters.

Stochasticity in gene expression can give rise to fluctuations in protein levels and lead to phenotypic variation across a population of genetically identical cells. Recent experiments indicate that bursting and feedback mechanisms play important roles in controlling noise in gene expression and phenotypic variation. A quantitative understanding of the impact of these factors requires analysis of the corresponding stochastic models. However, for stochastic models of gene expression with feedback and bursting, exact analytical results for protein distributions have not been obtained so far. Here, we analyze a model of gene expression with bursting and feedback regulation and obtain exact results for the corresponding protein steady-state distribution. The results obtained provide new insights into the role of bursting and feedback in noise regulation and optimization. Furthermore, for a specific choice of parameters, the system studied maps on to a two-state biochemical switch driven by a bursty input noise source. The analytical results derived thus provide quantitative insights into diverse cellular processes involving noise in gene expression and biochemical switching., Comment: Accepted in Phys. Rev. Lett

Stochasticity in gene expression gives rise to fluctuations in protein levels across a population of genetically identical cells. Such fluctuations can lead to phenotypic variation in clonal populations, hence there is considerable interest in quantifying noise in gene expression using stochastic models. However, obtaining exact analytical results for protein distributions has been an intractable task for all but the simplest models. Here, we invoke the partitioning property of Poisson processes to develop a mapping that significantly simplifies the analysis of stochastic models of gene expression. The mapping leads to exact protein distributions using results for mRNA distributions in models with promoter-based regulation. Using this approach, we derive exact analytical results for steady-state and time-dependent distributions for the basic 2-stage model of gene expression. Furthermore, we show how the mapping leads to exact protein distributions for extensions of the basic model that include the effects of post-transcriptional and post-translational regulation. The approach developed in this work is widely applicable and can contribute to a quantitative understanding of stochasticity in gene expression and its regulation., Comment: 10 pages, 5 figures

We explore a new variant of Small-World Networks (SWNs), in which an additional parameter ($r$) sets the length scale over which shortcuts are uniformly distributed. When $r=0$ we have an ordered network, whereas $r=1$ corresponds to the original SWN model. These short-range SWNs have a similar degree distribution and scaling properties as the original SWN model. We observe the small-world phenomenon for $r \ll 1$ indicating that global shortcuts are not necessary for the small-world effect. For short-range SWNs, the average path length changes nonmonotonically with system size, whereas for the original SWN model it increases monotonically. We propose an expression for the average path length for short-range SWNs based on numerical simulations and analytical approximations.

The intrinsic stochasticity of gene expression can lead to large variability in protein levels for genetically identical cells. Such variability in protein levels can arise from infrequent synthesis of mRNAs which in turn give rise to bursts of protein expression. Protein expression occurring in bursts has indeed been observed experimentally and recent studies have also found evidence for transcriptional bursting, i.e. production of mRNAs in bursts. Given that there are distinct experimental techniques for quantifying the noise at different stages of gene expression, it is of interest to derive analytical results connecting experimental observations at different levels. In this work, we consider stochastic models of gene expression for which mRNA and protein production occurs in independent bursts. For such models, we derive analytical expressions connecting protein and mRNA burst distributions which show how the functional form of the mRNA burst distribution can be inferred from the protein burst distribution. Additionally, if gene expression is repressed such that observed protein bursts arise only from single mRNAs, we show how observations of protein burst distributions (repressed and unrepressed) can be used to completely determine the mRNA burst distribution. Assuming independent contributions from individual bursts, we derive analytical expressions connecting means and variances for burst and steady-state protein distributions. Finally, we validate our general analytical results by considering a specific reaction scheme involving regulation of protein bursts by small RNAs. For a range of parameters, we derive analytical expressions for regulated protein distributions that are validated using stochastic simulations. The analytical results obtained in this work can thus serve as useful inputs for a broad range of studies focusing on stochasticity in gene expression.

Regulatory genes called small RNAs (sRNAs) are known to play critical roles in cellular responses to changing environments. For several sRNAs, regulation is effected by coupled stoichiometric degradation with messenger RNAs (mRNAs). The nonlinearity inherent in this regulatory scheme indicates that exact analytical solutions for the corresponding stochastic models are intractable. Here, we present a variational approach to analyze a well-studied stochastic model for regulation by sRNAs via coupled degradation. The proposed approach is efficient and provides accurate estimates of mean mRNA levels as well as higher order terms. Results from the variational ansatz are in excellent agreement with data from stochastic simulations for a wide range of parameters, including regions of parameter space where mean-field approaches break down. The proposed approach can be applied to quantitatively model stochastic gene expression in complex regulatory networks., Comment: 4 pages, 3 figures

A wealth of new research has highlighted the critical roles of small RNAs (sRNAs) in diverse processes such as quorum sensing and cellular responses to stress. The pathways controlling these processes often have a central motif comprising of a master regulator protein whose expression is controlled by multiple sRNAs. However, the regulation of stochastic gene expression of a single target gene by multiple sRNAs is currently not well understood. To address this issue, we analyze a stochastic model of regulation of gene expression by multiple sRNAs. For this model, we derive exact analytic results for the regulated protein distribution including compact expressions for its mean and variance. The derived results provide novel insights into the roles of multiple sRNAs in fine-tuning the noise in gene expression. In particular, we show that, in contrast to regulation by a single sRNA, multiple sRNAs provide a mechanism for independently controlling the mean and variance of the regulated protein distribution.

\textit{Vibrio harveyi} and \textit{Vibrio cholerae} have quorum sensing pathways with similar design and highly homologous components including multiple small RNAs (sRNAs). However, the associated luminescence phenotypes of strains with sRNA deletions differ dramatically: in \textit{V. harveyi}, the sRNAs act additively; however, in \textit{V. cholerae}, the sRNAs act redundantly. Furthermore, there are striking differences in the luminescence phenotypes for different pathway mutants in \textit{V. harveyi} and \textit{V. cholerae}. However these differences have not been connected with the observed differences for the sRNA deletion strains in these bacteria. In this work, we present a model for quorum sensing induced luminescence phenotypes focusing on the interactions of multiple sRNAs with target mRNA. Within our model, we find that one key parameter -- the fold-change in protein concentration necessary for luminescence activation -- can control whether the sRNAs appear to act additively or redundantly. For specific parameter choices, we find that differences in this key parameter can also explain hitherto unconnected luminescence phenotypes differences for various pathway mutants in \textit{V. harveyi} and \textit{V. cholerae}. The model can thus provide a unifying explanation for observed differences in luminescence phenotypes and can also be used to make testable predictions for future experiments., Comment: 10 pages, 6 figures and 1 table

Regulation of intrinsic noise in gene expression is essential for many cellular functions. Correspondingly, there is considerable interest in understanding how different molecular mechanisms of gene expression impact variations in protein levels across a population of cells. In this work, we analyze a stochastic model of bursty gene expression which considers general waiting-time distributions governing arrival and decay of proteins. By mapping the system to models analyzed in queueing theory, we derive analytical expressions for the noise in steady-state protein distributions. The derived results extend previous work by including the effects of arbitrary probability distributions representing the effects of molecular memory and bursting. The analytical expressions obtained provide insight into the role of transcriptional, post-transcriptional and post-translational mechanisms in controlling the noise in gene expression., Comment: Accepted by Physical Review Letters

The intrinsic stochasticity of gene expression can lead to large variations in protein levels across a population of cells. To explain this variability, different sources of mRNA fluctuations ('Poisson' and 'Telegraph' processes) have been proposed in stochastic models of gene expression. Both Poisson and Telegraph scenario models explain experimental observations of noise in protein levels in terms of 'bursts' of protein expression. Correspondingly, there is considerable interest in establishing relations between burst and steady-state protein distributions for general stochastic models of gene expression. In this work, we address this issue by considering a mapping between stochastic models of gene expression and problems of interest in queueing theory. By applying a general theorem from queueing theory, Little's Law, we derive exact relations which connect burst and steady-state distribution means for models with arbitrary waiting-time distributions for arrival and degradation of mRNAs and proteins. The derived relations have implications for approaches to quantify the degree of transcriptional bursting and hence to discriminate between different sources of intrinsic noise in gene expression. To illustrate this, we consider a model for regulation of protein expression bursts by small RNAs. For a broad range of parameters, we derive analytical expressions (validated by stochastic simulations) for the mean protein levels as the levels of regulatory small RNAs are varied. The results obtained show that the degree of transcriptional bursting can, in principle, be determined from changes in mean steady-state protein levels for general stochastic models of gene expression., Comment: Accepted by Physical Review E

The intrinsic stochasticity of gene expression can lead to large variability of protein levels across a population of cells. Variability (or noise) in protein distributions can be modulated by cellular mechanisms of gene regulation; in particular, there is considerable interest in understanding the role of post-transcriptional regulation. To address this issue, we propose and analyze a stochastic model for post-transcriptional regulation of gene expression. The analytical solution of the model provides insight into the effects of different mechanisms of post-transcriptional regulation on the noise in protein distributions. The results obtained also demonstrate how different sources of intrinsic noise in gene expression can be discriminated based on observations of regulated protein distributions., Comment: Accepted by Physical Review Letters

We present a framework for analyzing luminescence regulation during quorum sensing in the bioluminescent bacterium \emph{Vibrio harveyi}. Using a simplified model for signal transduction in the quorum sensing pathway, we identify key dimensionless parameters that control the system's response. These parameters are estimated using experimental data on luminescence phenotypes for different mutant strains. The corresponding model predictions are consistent with results from other experiments which did not serve as inputs for determining model parameters. Furthermore, the proposed framework leads to novel testable predictions for luminescence phenotypes and for responses of the network to different perturbations., Comment: 18 pages, 5 figures, IOP style files included

In E. coli, accurate cell division depends upon the oscillation of Min proteins from pole to pole. We provide a model for the polar localization of MinD based only on diffusion, a delay for nucleotide exchange, and different rates of attachment to the bare membrane and the occupied membrane. We derive analytically the probability density, and correspondingly the length scale, for MinD attachment zones. Our simple analytical model illustrates the processes giving rise to the observed localization of cellular MinD zones., Comment: 4 pages, 3 figures, submitted to PRL

We extend our previous stochastic cellular automata based model for areal aggregation of prion proteins on neuronal surfaces. The new anisotropic model allow us to simulate both strong beta-sheet and weaker attachment bonds between proteins. Constraining binding directions allows us to generate aggregate structures with the hexagonal lattice symmetry found in recently observed in vitro experiments. We argue that these constraints on rules may correspond to underlying steric constraints on the aggregation process. We find that monomer dominated growth of the areal aggregate is too slow to account for some observed doubling time-to-incubation time ratios inferred from data, and so consider aggregation dominated by relatively stable but non-infectious oligomeric intermediates. We compare a kinetic theory analysis of oligomeric aggregation to spatially explicit simulations of the process. We find that with suitable rules for misfolding of oligomers, possibly due to water exclusion by the surrounding aggregate, the resulting oligomeric aggregation model maps onto our previous monomer aggregation model. Therefore it can produce some of the same attractive features for the description of prion incubation time data. We propose experiments to test the oligomeric aggregation model., Comment: 8 pages, 10 figures For larger versions of several figures, see http://asaph.ucdavis.edu/~dmobley and click on the prion paper link