Search

In large-scale learning algorithms, the momentum term is usually included in the stochastic sub-gradient method to improve the learning speed because it can navigate ravines efficiently to reach a local minimum. However, step-size and momentum weight hyper-parameters must be appropriately tuned to optimize convergence. We thus analyze the convergence rate using stochastic programming with Polyak's acceleration of two commonly used step-size learning rates: ``diminishing-to-zero" and ``constant-and-drop" (where the sequence is divided into stages and a constant step-size is applied at each stage) under strongly convex functions over a compact convex set with bounded sub-gradients. For the former, we show that the convergence rate can be written as a product of exponential in step-size and polynomial in momentum weight. Our analysis justifies the convergence of using the default momentum weight setting and the diminishing-to-zero step-size sequence in large-scale machine learning software. For the latter, we present the condition for the momentum weight sequence to converge at each stage.

Practical applications of compressed sensing often restrict the choice of its two main ingredients. They may (i) prescribe using particular redundant dictionaries for certain classes of signals to become sparsely represented, or (ii) dictate specific measurement mechanisms which exploit certain physical principles. On the problem of RIP measurement matrix design in compressed sensing with redundant dictionaries, we give a simple construction to derive sensing matrices whose compositions with a prescribed dictionary have a high probability of the RIP in the $k \log(n/k)$ regime. Our construction thus provides recovery guarantees usually only attainable for sensing matrices from random ensembles with sparsifying orthonormal bases. Moreover, we use the dictionary factorization idea that our construction rests on in the application of magnetic resonance imaging, in which also the sensing matrix is prescribed by quantum mechanical principles. We propose a recovery algorithm based on transforming the acquired measurements such that the compressed sensing theory for RIP embeddings can be utilized to recover wavelet coefficients of the target image, and show its performance on examples from the fastMRI dataset.

Our main focus is on the generalization bound, which serves as an upper limit for the generalization error. Our analysis delves into regression and classification tasks separately to ensure a thorough examination. We assume the target function is real-valued and Lipschitz continuous for regression tasks. We use the 2-norm and a root-mean-square-error (RMSE) variant to measure the disparities between predictions and actual values. In the case of classification tasks, we treat the target function as a one-hot classifier, representing a piece-wise constant function, and employ 0/1 loss for error measurement. Our analysis underscores the differing sample complexity required to achieve a concentration inequality of generalization bounds, highlighting the variation in learning efficiency for regression and classification tasks. Furthermore, we demonstrate that the generalization bounds for regression and classification functions are inversely proportional to a polynomial of the number of parameters in a network, with the degree depending on the hypothesis class and the network architecture. These findings emphasize the advantages of over-parameterized networks and elucidate the conditions for benign overfitting in such systems.

The use of proximal point operators for optimization can be computationally expensive when the dimensionality of a function (i.e., the number of variables) is high. In this study, we sought to reduce the cost of calculating proximal point operators by developing a directional operator in which the proximal regularization of a function along a specific direction is penalized. We used this operator in a novel approach to optimization, referred to as the directional proximal point method (Direction PPM). When using Direction PPM, the key to achieving convergence is the selection of direction sequences for directional proximal point operators. In this paper, we present the conditions/assumptions by which to derive directions capable of achieving global convergence for convex functions. Considered a light version of PPM, Direction PPM uses scalar optimization to derive a stable step-size via a direction envelope function and an auxiliary method to derive a direction sequence that satisfies the assumptions. This makes Direction PPM adaptable to a larger class of functions. Through applications to differentiable convex functions, we demonstrate that negative gradient directions at the current iterates could conceivably be used to achieve this end. We provide experimental results to illustrate the efficacy of Direction PPM in practice.

The conclusions provided by deep neural networks (DNNs) must be carefully scrutinized to determine whether they are universal or architecture dependent. The term DAG-DNN refers to a graphical representation of a DNN in which the architecture is expressed as a direct-acyclic graph (DAG), on which arcs are associated with functions. The level of a node denotes the maximum number of hops between the input node and the node of interest. In the current study, we demonstrate that DAG-DNNs can be used to derive all functions defined on various sub-architectures of the DNN. We also demonstrate that the functions defined in a DAG-DNN can be derived via a sequence of lower-triangular matrices, each of which provides the transition of functions defined in sub-graphs up to nodes at a specified level. The lifting structure associated with lower-triangular matrices makes it possible to perform the structural pruning of a network in a systematic manner. The fact that decomposition is universally applicable to all DNNs means that network pruning could theoretically be applied to any DNN, regardless of the underlying architecture. We demonstrate that it is possible to obtain the winning ticket (sub-network and initialization) for a weak version of the lottery ticket hypothesis, based on the fact that the sub-network with initialization can achieve training performance on par with that of the original network using the same number of iterations or fewer.

Compressive sensing involves the inversion of a mapping $SD \in \mathbb{R}^{m \times n}$, where $m < n$, $S$ is a sensing matrix, and $D$ is a sparisfying dictionary. The restricted isometry property is a powerful sufficient condition for the inversion that guarantees the recovery of high-dimensional sparse vectors from their low-dimensional embedding into a Euclidean space via convex optimization. However, determining whether $SD$ has the restricted isometry property for a given sparisfying dictionary is an NP-hard problem, hampering the application of compressive sensing. This paper provides a novel approach to resolving this problem. We demonstrate that it is possible to derive a sensing matrix for any sparsifying dictionary with a high probability of retaining the restricted isometry property. In numerical experiments with sensing matrices for K-SVD, Parseval K-SVD, and wavelets, our recovery performance was comparable to that of benchmarks obtained using Gaussian and Bernoulli random sensing matrices for sparse vectors.

A general lack of understanding pertaining to deep feedforward neural networks (DNNs) can be attributed partly to a lack of tools with which to analyze the composition of non-linear functions, and partly to a lack of mathematical models applicable to the diversity of DNN architectures. In this paper, we made a number of basic assumptions pertaining to activation functions, non-linear transformations, and DNN architectures in order to use the un-rectifying method to analyze DNNs via directed acyclic graphs (DAGs). DNNs that satisfy these assumptions are referred to as general DNNs. Our construction of an analytic graph was based on an axiomatic method in which DAGs are built from the bottom-up through the application of atomic operations to basic elements in accordance with regulatory rules. This approach allows us to derive the properties of general DNNs via mathematical induction. We show that using the proposed approach, some properties hold true for general DNNs can be derived. This analysis advances our understanding of network functions and could promote further theoretical insights if the host of analytical tools for graphs can be leveraged., Comment: 26 pages, 14 figures

This paper presents a directional proximal point method (DPPM) to derive the minimum of any C1-smooth function f. The proposed method requires a function persistent a local convex segment along the descent direction at any non-critical point (referred to a DLC direction at the point). The proposed DPPM can determine a DLC direction by solving a two-dimensional quadratic optimization problem, regardless of the dimensionally of the function variables. Along that direction, the DPPM then updates by solving a one-dimensional optimization problem. This gives the DPPM advantage over competitive methods when dealing with large-scale problems, involving a large number of variables. We show that the DPPM converges to critical points of f. We also provide conditions under which the entire DPPM sequence converges to a single critical point. For strongly convex quadratic functions, we demonstrate that the rate at which the error sequence converges to zero can be R-superlinear, regardless of the dimension of variables., Comment: 29 pages, 12 figures

The un-rectifying technique expresses a non-linear point-wise activation function as a data-dependent variable, which means that the activation variable along with its input and output can all be employed in optimization. The ReLU network in this study was un-rectified means that the activation functions could be replaced with data-dependent activation variables in the form of equations and constraints. The discrete nature of activation variables associated with un-rectifying ReLUs allows the reformulation of deep learning problems as problems of combinatorial optimization. However, we demonstrate that the optimal solution to a combinatorial optimization problem can be preserved by relaxing the discrete domains of activation variables to closed intervals. This makes it easier to learn a network using methods developed for real-domain constrained optimization. We also demonstrate that by introducing data-dependent slack variables as constraints, it is possible to optimize a network based on the augmented Lagrangian approach. This means that our method could theoretically achieve global convergence and all limit points are critical points of the learning problem. In experiments, our novel approach to solving the compressed sensing recovery problem achieved state-of-the-art performance when applied to the MNIST database and natural images., Comment: 35 pages, 6 figures

We consider deep feedforward neural networks with rectified linear units from a signal processing perspective. In this view, such representations mark the transition from using a single (data-driven) linear representation to utilizing a large collection of affine linear representations tailored to particular regions of the signal space. This paper provides a precise description of the individual affine linear representations and corresponding domain regions that the (data-driven) neural network associates to each signal of the input space. In particular, we describe atomic decompositions of the representations and, based on estimating their Lipschitz regularity, suggest some conditions that can stabilize learning independent of the network depth. Such an analysis may promote further theoretical insight from both the signal processing and machine learning communities.

When the scale of communication networks has been growing rapidly in the past decades, it becomes a critical challenge to extract fast and accurate estimation of key state parameters of network links, e.g., transmission delays and dropped packet rates, because such monitoring operations are usually time-consuming. Based on the sparse recovery technique reported in [Wang et al. (2015) IEEE Trans. Information Theory, 61(2):1028--1044], which can infer link delays from a limited number of measurements using compressed sensing, we particularly extend to networks with dynamic changes including link insertion and deletion. Moreover, we propose a more efficient algorithm with a better theoretical upper bound. The experimental result also demonstrates that our algorithm outperforms the previous work in running time while maintaining a similar recovery performance, which shows its capability to cope with large-scale networks.

Frames are the foundation of the linear operators used in the decomposition and reconstruction of signals, such as the discrete Fourier transform, Gabor, wavelets, and curvelet transforms. The emergence of sparse representation models has shifted of the emphasis in frame theory toward sparse l1-minimization problems. In this paper, we apply frame theory to the sparse representation of signals in which a synthesis dictionary is used for a frame and an analysis dictionary is used for a dual frame. We sought to formulate a novel dual frame design in which the sparse vector obtained through the decomposition of any signal is also the sparse solution representing signals based on a reconstruction frame. Our findings demonstrate that this type of dual frame cannot be constructed for over-complete frames, thereby precluding the use of any linear analysis operator in driving the sparse synthesis coefficient for signal representation. Nonetheless, the best approximation to the sparse synthesis solution can be derived from the analysis coefficient using the canonical dual frame. In this study, we developed a novel dictionary learning algorithm (called Parseval K-SVD) to learn a tight-frame dictionary. We then leveraged the analysis and synthesis perspectives of signal representation with frames to derive optimization formulations for problems pertaining to image recovery. Our preliminary, results demonstrate that the images recovered using this approach are correlated to the frame bounds of dictionaries, thereby demonstrating the importance of using different dictionaries for different applications., Comment: 32 pages, 31 figures

Frame is the corner stone for designing decomposition and reconstruction operations in signal processing. Famous frames include wavelets, curvelets,and Gabor. A celebrated result indicates that if a synthesis frame is chosen for reconstruction, then its canonical dual frame is the analysis frame that performs decomposition, yielding coefficients that minimizes l2-norm of all coefficients of all dual frames. This paper tries to extend this result by inves-tigating whether a dual frame can be constructed so that the corresponding coefficients yield the minimum l1-norm. We show that this mission cannot be achieved for any over-complete frame. However, we present some conditions on a dual frame so that the minimizer of the l1-norm can be derived from the coefficients of the dual frame. We show that, based on frame structure, various applications that seek to optimize sparse analysis and synthesis co-efficients are logically equivalent. Finally, we present a general construction method of frames to demonstrate the proposed conditions are feasible. In addition, two design approaches are proposed to derive frames that meet the proposed conditions and yield small non-zero elements of frame coefficients., Comment: This paper has been withdrawn by the author due to a crucial error in Section IV. The update version is in preparation

In the current work, we have formulated the optimal bit-allocation problem for a scalable codec of images or videos as a constrained vector-valued optimization problem and demonstrated that there can be many optimal solutions, called Pareto optimal points. In practice, the Pareto points are derived via the weighted sum scalarization approach. An important question which arises is whether all the Pareto optimal points can be derived using the scalarization approach? The present paper provides a sufficient condition on the rate-distortion function of each resolution of a scalable codec to address the above question. The result indicated that if the rate-distortion function of each resolution is strictly decreasing and convex and the Pareto points form a continuous curve, then all the optimal Pareto points can be derived by using the scalarization method.

Blind image restoration is a non-convex problem which involves restoration of images from an unknown blur kernel. The factors affecting the performance of this restoration are how much prior information about an image and a blur kernel are provided and what algorithm is used to perform the restoration task. Prior information on images is often employed to restore the sharpness of the edges of an image. By contrast, no consensus is still present regarding what prior information to use in restoring from a blur kernel due to complex image blurring processes. In this paper, we propose modelling of a blur kernel as a sparse linear combinations of basic 2-D patterns. Our approach has a competitive edge over the existing blur kernel modelling methods because our method has the flexibility to customize the dictionary design, which makes it well-adaptive to a variety of applications. As a demonstration, we construct a dictionary formed by basic patterns derived from the Kronecker product of Gaussian sequences. We also compare our results with those derived by other state-of-the-art methods, in terms of peak signal to noise ratio (PSNR)., Comment: 11 pages, 37 figures