Comments and reactions to the posts are welcome! Simply click on the "comment" line below each post to see previous comments or on the pencil icon to add a new one!

What is sparse coding for?

While reading David Field’s seminal paper “What is the role of sensory coding?”, I became aware of compelling evidence supporting the hypothesis that our cortex represents sensory information using a ‘sparse’ code. Sparse coding has its roots in information theory and relates to the optimal coding principle whereby a system’s aim is to represent a signal using as few ‘bits’ as possible. In an optimal code, each bit must be as independent as possible from other bits since any information carried simultaneously by two bits is redundant and decreases the optimality of the code. In the context of Neuroscience, this translates to our cortex using as few neurons as possible to encode sensory events, with active neurons carrying mutually independent information. Several findings substantiate this claim, but perhaps the most remarkable one comes from computer simulation using unsupervised feature-learning algorithms. Similarly to our sensory cortices, the aim of such algorithms is to learn a representational basis to encode sensory stimuli. Simulations have shown that when these algorithms implement sparse coding strategies (i.e., when they are forced to use as few elements as possible to represent a stimulus), they develop ‘receptive fields’ that are strikingly similar to those of neurons in our cortex. The close correspondence between simulation and biology indicates that our cortex might indeed perform a sparse encoding of sensory information.

Towards the end of his paper, Field mentions several interesting reasons why a sparse code might be beneficial. However, I failed to develop a personal intuition as to why our cortex might implement such a sparse code. After finishing the paper, I looked up from my computer and I gazed through the window. At this moment, I saw a gigantic object appear in front of me. It was a very complex stimulus that filled up my whole visual field. It was made of hundreds, if not thousands, of small moving patches each containing infinite details. Yet, however intricate this stimulus was, a single percept came to my mind and I thought: “a tree”. It is at this point that I realized what the main advantage of sparse coding is: it summarizes our sensory environment to essential features. I then got up and walked around the Institute. As I looked around me, objects emerged from their background. I didn’t see keys, letters, buttons and cables but a computer; nor did I see a detailed fabric, wheels and armrests, but simply a chair. I could almost feel my cortex process visual information. It was efficiently summarizing my sensory environment so that I only perceived large-scale objects. In other words, my cortex was sparsely encoding my environment: with just a few concepts (‘chair’, ‘computer’, ‘desk’) it described the information that my retina perceived through millions of photoreceptors.

Is this then the role of sensory coding? To sparsely represent sensory information such that relevant components are readily accessible? One of the purposes of cortical circuitry should then be to guide the development of sensory neurons’ receptive field so that each captures the maximum statistical structure in the environment. During perception, neurons should compete so that solely those that carry the most information about a sensory stimulus are allowed to be active. In this way, only a few active neurons carrying mutually independent information can accurately summarize a complex sensory environment.

- R. Holca


Unknown said...

Very informative and interesting, Prof Holca. Keep up the good work informing others, especially the incoming Masters students!

In relation to this very interesting article, I can now finally understand the benefits of sparse coding.

Something interesting to consider is the relation between sparse encoding and memory access. It seems to me that when parsing a scene we only retain information at a high level in active memory. However, after inspecting a scene, it is possible to retrieve some additional information about the objects that we did not think we knew (ie: not being able to recall a specific detail of an object, yet seeing a picture of it makes it look familiar and we can recover additional details). This seems to indicate that there are different degrees of 'sparcity' for different purposes.

Florian Jugini said...

Nice story.
Magic, by the way, is the art of performing actions in such a way that they do not find there way in the (sparse) perceptual code of the spectators mind.