You are here: Home / Research / Bernstein I / Multisensory Integration

Multisensory Integration

Leo van Hemmen, Harald Luksch

Project summary

In the struggle for survival in a complex and dynamic environment, animals have developed a multitude of sensory systems. These sensory systems form the link between events in the real world and the neuronal representation of the very same events in the brain of the animal. In order to realize an efficient view of the outside world, different senses--like vision, hearing, and touch, but also exotic senses such as the lateral line system of fish and amphibia or the infrared vision of snakes--have to be assembled to a coherent whole.

But how multisensory integration is achieved in the brain? First, it is necessary to understand how neuronal maps (topographic representations of the environment) are formed in the brain. Second, we need to understand how information from multiple sensory systems interacts in the brain to form an unambiguous representation of the environment.

The project currently focuses on the theoretical component of multisensory integration. We have concentrated on studying neuronal realizations of the general mechanisms that underlie sensory integration. That is, we study how a sensory modality dominates other sensory input and how such a dominance that has been suggested by several experimental studies can arise. This research is extended by the study of information processing in exotic sensory modalities. We currently put our knowledge to use in technical systems by an alliance with the TUM excellence cluster for cognition in technical systems, CoTeSys.


llustration of our model. On the left side, the mathematical model, where the input functions are taken as continuous and where the subsequent functions are mathematically calculated. On the right side, the neural model realized with spiking and noisy neural populations. In both cases, two Input Maps A and B are multiplicatively combined into the Integrated Multiplicative Map. The combined information can be used to intrinsically gauge the two input maps.

Related Publications

  • A. B. Sichert, R. Bamler & J.L. van Hemmen, Hydrodynamic Object Recognition: When Multipoles Count Phys. Rev. Lett. (2008 accepted)
  • H. J. A. Hagedorn, J.-M. P. Franosch, J. Goulet & J. L. van Hemmen Wake Tracking and Detection of Vortex Rings by the Canal Lateral Line of Fish, in preparation
  • A. B. Sichert, C. Vossen & J. L. van Hemmen Intrinsic Multimodal Integration, in preparation
  • P. Friedel, A. B. Sichert, M. Buerck, C. Vossen & J. L. van Hemmen MapsOptimality and Object Formation, in preparation
  • Engelmann J. Goulet J, Chagnaud B., Jung N., Scholze, B., Bleckmann  H.  and van Hemmen JL, Response to Noise, coding at high precision in  the velocity regime, in preparation
  • Goulet J, Engelmann J, Sichert A.B. and J.L van Hemmen Distance and Stimuli Encoding for the Fish Lateral-Line, in preparation
  • Goulet J, Franosch J.M.P and J.L. van Hemmen Water Imaging: what the lateral line is good for, in preparation
  • P. Friedel & J. L. van Hemmen Inhibition, not excitation, is the key to multimodal sensory integration Biol. Cybern  98:597–618 (2008)
  • J. Goulet, J. Engelmann, B. P. Chagnaud, J.-M. P. Franosch, M. D. Suttner & J. L. van Hemmen Object localization through the lateral line system of fish: Theory and experiment J. Comp. Physiol. A., 194:1-17 (2008)