What is sparse coding for?

While reading David Field’s seminal paper “What is the role of sensory coding?”, I became aware of compelling evidence supporting the hypothesis that our cortex represents sensory information using a ‘sparse’ code. Sparse coding has its roots in information theory and relates to the optimal coding principle whereby a system’s aim is to represent a signal using as few ‘bits’ as possible. In an optimal code, each bit must be as independent as possible from other bits since any information carried simultaneously by two bits is redundant and decreases the optimality of the code. In the context of Neuroscience, this translates to our cortex using as few neurons as possible to encode sensory events, with active neurons carrying mutually independent information. Several findings substantiate this claim, but perhaps the most remarkable one comes from computer simulation using unsupervised feature-learning algorithms. Similarly to our sensory cortices, the aim of such algorithms is to learn a representational basis to encode sensory stimuli. Simulations have shown that when these algorithms implement sparse coding strategies (i.e., when they are forced to use as few elements as possible to represent a stimulus), they develop ‘receptive fields’ that are strikingly similar to those of neurons in our cortex. The close correspondence between simulation and biology indicates that our cortex might indeed perform a sparse encoding of sensory information.

Towards the end of his paper, Field mentions several interesting reasons why a sparse code might be beneficial. However, I failed to develop a personal intuition as to why our cortex might implement such a sparse code. After finishing the paper, I looked up from my computer and I gazed through the window. At this moment, I saw a gigantic object appear in front of me. It was a very complex stimulus that filled up my whole visual field. It was made of hundreds, if not thousands, of small moving patches each containing infinite details. Yet, however intricate this stimulus was, a single percept came to my mind and I thought: “a tree”. It is at this point that I realized what the main advantage of sparse coding is: it summarizes our sensory environment to essential features. I then got up and walked around the Institute. As I looked around me, objects emerged from their background. I didn’t see keys, letters, buttons and cables but a computer; nor did I see a detailed fabric, wheels and armrests, but simply a chair. I could almost feel my cortex process visual information. It was efficiently summarizing my sensory environment so that I only perceived large-scale objects. In other words, my cortex was sparsely encoding my environment: with just a few concepts (‘chair’, ‘computer’, ‘desk’) it described the information that my retina perceived through millions of photoreceptors.

Is this then the role of sensory coding? To sparsely represent sensory information such that relevant components are readily accessible? One of the purposes of cortical circuitry should then be to guide the development of sensory neurons’ receptive field so that each captures the maximum statistical structure in the environment. During perception, neurons should compete so that solely those that carry the most information about a sensory stimulus are allowed to be active. In this way, only a few active neurons carrying mutually independent information can accurately summarize a complex sensory environment.

- R. Holca