1. A theory of capacity and sparse neural encoding
- Author
-
Roman Vershynin and Pierre Baldi
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,0209 industrial biotechnology ,Polynomial ,Statistical assumption ,Multivariate random variable ,Computer science ,Cognitive Neuroscience ,Polytope ,Machine Learning (stat.ML) ,02 engineering and technology ,Mathematical proof ,68Q32, 06E30, 92B20 ,Machine Learning (cs.LG) ,020901 industrial engineering & automation ,Statistics - Machine Learning ,Artificial Intelligence ,Simple (abstract algebra) ,Encoding (memory) ,0202 electrical engineering, electronic engineering, information engineering ,Learning ,Neural and Evolutionary Computing (cs.NE) ,Layer (object-oriented design) ,Computer Science - Neural and Evolutionary Computing ,020201 artificial intelligence & image processing ,Neural Networks, Computer ,Algorithm ,Algorithms - Abstract
Motivated by biological considerations, we study sparse neural maps from an input layer to a target layer with sparse activity, and specifically the problem of storing $K$ input-target associations $(x,y)$, or memories, when the target vectors $y$ are sparse. We mathematically prove that $K$ undergoes a phase transition and that in general, and somewhat paradoxically, sparsity in the target layers increases the storage capacity of the map. The target vectors can be chosen arbitrarily, including in random fashion, and the memories can be both encoded and decoded by networks trained using local learning rules, including the simple Hebb rule. These results are robust under a variety of statistical assumptions on the data. The proofs rely on elegant properties of random polytopes and sub-gaussian random vector variables. Open problems and connections to capacity theories and polynomial threshold maps are discussed., Comment: 31 pages
- Published
- 2021
- Full Text
- View/download PDF