This is the idea that each input to a system should be represented by many features, and each feature should be involved in the representation of many possible inputs.
For example, suppose we have a vision system that can recognize cars, trucks, and birds and these objects can each be red, green, or blue.
One way of representing these inputs would be to have a separate neuron or hidden unit that activates for each of the nine possible combinations: red truck, red car, red bird, green truck, and so on.
This requires nine different neurons, and each neuron must independently learn the concept of color and object identity.
One way to improve on this situation is to use a distributed representation, with three neurons describing the color and three neurons describing the object identity.
This requires only six neurons total instead of nine, and the neuron describing redness is able to learn about redness from images of cars, trucks and birds, not only from images of one specific category of objects.