What would the world look like after sight recovery?

The following are demos from the publication Pulse trains to percepts: The challenge of creating a perceptually intelligible world with sight recovery technologies, by Ione Fine and Geoffrey M. Boynton.

Retinal diseases affect more than 20 million individuals worldwide. An extraordinary variety of sight recovery therapies are either about to begin clinical trials, have begun clinical trials, or are currently being implanted in patients. However, as yet we have little insight into what will the world will look like to patients. Here we show some movies based on neuro-perceptual models that simulate what people with ‘restored vision’ are likely to see.

Here we are simulating vision after two different kinds of sight recovery therapies. Optogenetic proteins and small molecule photoswitches make cells responsive to light. If inserted into the remaining retinal cells of a diseased retina, they can enhance the ability of the retina to respond to light. Retinal and cortical prostheses directly elicit neural responses using electrical stimulation with small electrodes, similar to a cochlear implant.

The movies are much more informative than the pictures, especially for Simulations 3 and 4.


Simulation 1 - ON and OFF-pathways


The image that reaches the retina is carried by a variety of cells, which each carry different kinds of information to the brain. The movies shown here simulate the information thought to be carried by ON-pathways (which fire in response to bright spots of light against a darker background) and OFF-pathways (which fire in response to dark spots on a brighter background).

Download simulation 1 movie here

Top row:

Left: The original movie of a child scooting.

Middle: A simulation of the information carried by ON-centre pathways

Bottom: A simulation of the information carried by OFF-centre pathways

Bottom Row:

Left: The difference of the simulations from ON-centre and OFF-centre pathways restores a band-pass version of the original image that emphasises the outlines of objects. Standard models of visual perception treat this difference as the representation of the visual stimulus leaving the eye.

Right: The sum of the simulations in the top row represents the effect of stimulation both ON-centre and OFF-centre pathways, as happens with current retinal electrical prostheses.



Simulation 2

This simulation shows the perceptual effects of axonal stimulation. One difficulty for retinal electrical prostheses is that they stimulate ganglion cell axons as well as cell bodies. The simulation shows how stimulating axon bodies can lead to the percept of streaks that blur the image in the direction of the underlying axon fibers. Simulations are shown for three levels of severity, because the amount of axonal stimulation seems to differ across prosthesis type and across individuals. 

Figure 2

Download simulation 2 movie here




Simulation 3

Currently both optogenetics and small molecule photoswitches tend to respond to changes in the visual scene much more slowly than natural vision. The movie example shown here shows a simulation of what this slow timing does to a moving object. Notice that the scooting child almost disappears!

Figure 5

Download simulation 3 movie here




Simulation 4


These examples combine the effects above to show simulations of the perceptual experience of sight recovery.

Figure 6

Download simulation 4 movie here

Left: This is the ideal 'scoreboard model' – what we’re trying to achieve.

Middle: Simulation of the effects of electrical stimulation. We assume that electrical stimulation will simultaneously cause responses in both ON- and OFF pathways (see Simulation 1), and will also cause some axonal stimulation (see Simulation 2).

Right: Simulation of optogenetics and small molecule photoswitch stimulation. This movie is based on assuming that only ON-centre pathways are selectively stimulated (see Simulation 1), and the timing of responses is sluggish compared to natural vision (see Simulation 3)