Parallel Processing and Image Analysis in the Eyes of Mantis Shrimps.
JUSTIN MARSHALL 
Abstract. The compound eyes of mantis shrimps, a group of tropical marine crustaceans, incorporate principles of serial and parallel processing of visual information that may be applicable to artificial imaging systems. Their eyes include numerous specializations for analysis of the spectral and polarizational properties of light, and include more photoreceptor classes for analysis of ultraviolet light, color, and polarization than occur in any other known visual system. This is possible because receptors in different regions of the eye are anatomically diverse and incorporate unusual structural features, such as spectral filters, not seen in other compound eyes. Unlike eyes of most other animals, eyes of mantis shrimps must move to acquire some types of visual information and to integrate color and polarization with spatial vision. Information leaving the retina appears to be processed into numerous parallel data streams leading into the central nervous system, greatly reducing the analytical requirements at higher levels. Many of these unusual features of mantis shrimp vision may inspire new sensor designs for machine vision.
Of all the senses, vision provides animals with the most precise spatial registration of the external world and of objects in it. Animal vision operates in the spectral range between 300 and 750 nm, but technology has extended the "visual" senses of artificial sensors to cover the spectrum from X-rays to microwaves. Nevertheless, engineers can learn much from the mechanisms that animals use to image the world; to map it to self-centered coordinates; to segment the physical properties of light transmitted, scattered, or reflected by objects; and finally to extract the critical features necessary for survival. Like autonomous vehicles, animals must distinguish and recognize items of interest (e.g., landmarks, predators, prey, and conspecifics); they need to discriminate signals from noise; and they must find their way around a geometrically complicated and visually confusing world.
Biological systems are far more constrained than artificial ones in their choice of materials and in the means by which information is transduced and transmitted. More fundamentally, living visual systems are products both of evolution and development. Evolutionary history acts as a bottleneck, greatly limiting the options available in the design of a modern animal's eye. Also, the eye must pass through a biologically feasible developmental sequence. If the animal is one that undergoes metamorphosis (like many insects and crustaceans), the developing eye must also be functional for the larva's habitat and biological demands, which may affect the eye design available to the adult.
Here, we consider the highly evolved eyes of stomatopod crustaceans, commonly known as mantis shrimp. These marine invertebrates possess compound eyes both as adults and as larvae. Compound eyes are bulky, and they provide very poor spatial resolution for their size. These limits exist because each photoreceptor unit of a compound eye, termed a rhabdom, is paired with an individual optical system. Thus, the compound eye type demands an immense proliferation of optical units, increasing its size and complexity, and ultimately limiting the total number of receptors to a few thousand at best (compared to many millions in a vertebrate-type eye). In compound eyes, the unitary structure composed of optics, photoreceptors, and associated pigment cells is called an ommatidium.
Compound eyes, nevertheless, do offer certain possibilities not available to the much more compact camera eye design. Normally, all photoreceptors in a single ommatidium view an identical receptive field, which may be imaged onto several photoreceptor types for spectral or polarizational analysis. Furthermore, the geometry of the compound eye is very flexible, and it is even possible for ommatidia in well-separated regions of the eye to view the same spatial location. These options provide for considerable flexibility in the analysis of an image, and greatly multiply the possibilities for parallel processing. Indeed, much of the image analysis performed in the visual systems of mantis shrimps occurs very early in the visual process (sometimes at the level of single photoreceptor cells), producing a well-analyzed and much-simplified data stream flowing into higher visual centers. The compound eye design serves as an excellent model for artificial sensors in its modularity and in its ability to analyze image fe atures and characteristics independently in each unit.
In this paper, we consider some of the specializations found in mantis shrimp compound eyes that permit both serial and parallel analysis of visual stimuli. We also describe some other unusual features of these eyes that permit the decomposition of images into their polarizational, spectral, spatial, and depth-plane features. Wherever possible, we indicate potential applications of mantis shrimp visual design to the development of autonomous systems.
Mantis Shrimp Compound Eyes
As their common name implies, the stomatopods look much like other modern crustaceans. This appearance belies their distinctive evolutionary history, which has been separate from the crustacean mainline for about 400 million years. Unlike other crustaceans, they actively hunt prey and disable or kill it with a catapult-like blow of a specialized raptorial appendage. They also can severely injure other members of their own species, an ability that has encouraged the evolution of complicated signaling behavior, often involving color or polarization features. Among the most interesting adaptations are those involving the structure and function of their compound eyes.
Figure 1 shows the eyes of Neogonodactylus oerstedii, a mantis shrimp species from the Caribbean. Superficially, these ovoid eyes appear to be similar to those of other crustaceans, or even insects, but a closer inspection reveals some important differences. Although most of the eye consists of a standard hexagonal array of ommatidia, there is a band of six parallel rows of specialized ommatidia running around the equator of each eye, looking something like a tire tread in the figure. These divide the eye into two roughly equal groups of receptors, forming two approximately hemispherical halves (see also Manning et at., 1984). Most ommatidia throughout each of these two hemispheres share visual fields with ommatidia in the other, providing the possibility of stereoscopic vision in a single eye (Marshall and Land, 1993). The various ommatidial rows of the midband are individually specialized for spectral and polarizational analysis. We describe these specializations in detail later in the paper.
Another unusual feature is that the eyes are mounted on stalks. Stalked compound eyes occur throughout the Crustacea, but in mantis shrimps the movement of the eye on the stalk is unusually free, with the eye being driven in all possible axes of movement by six functional groups of muscles (Jones, 1994). Thus, whereas eye movements in other crustaceans are used to stabilize the eye in the visual field, mantis shrimp eyes are impressively and spontaneously active, giving the animals an air of curiosity and intelligence. However, the movements of the two eyes seem entirely uncoordinated, which is disconcerting to watch.
Specializations for the analysis of polarized light
Even though it is not recognized by our visual systems, polarized light is abundant in nature, being produced by reflection or scattering from almost any object. Analysis of partially linearly polarized light, the most common form, can be used to infer the surface texture or orientation of an object, and many animals use natural polarized-light fields for navigation. Machine-vision systems are beginning to incorporate polarization-vision concepts for material identification, surface-orientation detection, and other applications (see Wolff, 1997). The combination of spatial, spectral, and polarizational analysis in sensors may be simplified by resorting to the compound eye design as a model, and the analytical synthesis of these independent visual modalities may substantially increase the information content of artificial images. Technically, the addition of polarization analytical ability to artificial imagers is rather simple (Wolff, 1997).
The structure of arthropod photoreceptors naturally makes them sensitive to the plane of polarization of incoming light, because the visual pigments that absorb light are dichroic and are oriented mostly parallel in small, tubular membranes called microvilli. Most crustaceans take advantage of this sensitivity by producing ommatidia that contain two photoreceptor classes. Each class has microvilli that are all parallel but are perpendicular to those of the other receptor class, together producing a system with two analyzers, each maximally sensitive at an e-vector orientation orthogonal to the other (see Waterman, 1981). Mantis shrimps are no exception here, and ommatidia throughout the eye (except in parts of the midband, as described later) are all polarization-sensitive.
Figure 2 is a diagram of a typical ommatidium from the hemispherical regions of the eye outside the midband. The top half of this structure, including the refracting cornea and the underlying crystalline cone, is devoted to optics, focusing light on the top of the photosensitive structure, called the rhabdom. This rhabdom is constructed from eight receptor cells formed into a single light guide. It consists of a small upper tier, produced by receptor cell number 8 (R8), and a much longer underlying tier formed by contributions from receptor cells numbered 1 to 7 (Rl-7). R8 is sensitive only to ultraviolet light (Cronin et al., 1994b; Marshall and Oberwinkler, 1999), but the fused receptor produced by R1-7 is sensitive to middle-wavelength light, near 500 nm (see Cronin and Marshall, 1989a, b). The R8 receptor is polarization-insensitive, as its microvilli are not all parallel, but the two sets of receptors in the main rhabdom have orthogonal microvilli and are very sensitive to the plane of polarization (see also Marshall, 1988; Marshall et al., 1991a). Axons from these two receptor sets converge onto higher order interneurons (Horwood and Marshall, unpubl. obs.), which could explain how polarization opponency arises in mantis shrimp vision (see Yamaguchi et al., 1976). Thus, information leaving the outer regions of the retina has already undergone processing that emphasizes differences in the signals received by paired sets of receptors.
Ommatidia like those of Figure 2 are also typical of crabs and other "standard" crustaceans. In contrast, the ommatidia of the six rows of the midband are unique to the stomatopods. For convenience in discussing the function of midband ommatidia, we have grouped them into three types (Fig. 3), each of which normally appears in two of the six ommatidial rows of the midband. These types vary primarily in the construction of the rhabdom, so only this part of each ommatidium is diagrammed in Figure 3.
Type I (found in the two most ventral rows of the mid-band) is the simplest and is specialized for the analysis of polarized light, like the rhabdoms in ommatidia of the retinal hemispheres. Here, however, the R8 (ultraviolet) receptor is much enlarged and has microvilli that are all parallel and thus polarization-sensitive. The rest of the rhabdom, formed by receptors R1-7, is normal, providing two orthogonal receptor channels. Thus, the overall structure has a single ultraviolet-sensitive polarization analyzer on top and two orthogonal middle-wavelength (near 500 nm) analyzers in the main part of the rhabdom. A second ultraviolet system, perpendicular to the first, is available as well, because the two ommatidial rows with the Type I ommatidium are rotated 90[degrees] to each other. Together, these systems provide excellent polarization vision in two separate spectral regions.
The Type I ommatidium has an additional potential function. The overlying R8 receptor, with its parallel microvilli, may serve as a quarter-wavelength retarder. If so, it would convert circularly polarized light to linearly polarized light, which could then be analyzed by the polarization-sensitive cells of the main part of the rhabdom. Circular polarization may be produced naturally by reflection from metals or certain birefringent structures (such as crustacean cuticle), or by transmission through birefringent material or solutions of some complex molecules. The ability to detect and analyze circular polarization would add yet another information channel to the data stream flowing into the central nervous system. Note also that the geometrical arrangement of the receptors (i.e., their segregation into tiers in a single rhabdom) permits each level both to operate on its own and also to affect the operation of receptors placed more deeply in the overall retina.
Specializations for the spectral analysis of light
The spectrum of light reflected from an object gives rise to the psychophysical perception of color. Color is an inherent property of an object, useful for object recognition and identification. It also can be used to discriminate objects that have similar brightnesses but different reflectance spectra and to recognize borders. Many animals, including mantis shrimps, use colored markings to produce highly recognizable and unambiguous signals.
The two themes just discussed with reference to polarization vision--tiering of receptors to modify the responses of receptors at deeper levels and immediate processing of receptor output into opponent channels--continue with further elaborations in the spectral analysis systems of mantis shrimp compound eyes. These retinal specializations give mantis shrimps the potentially most complicated color vision systems of any animals. Ommatidia specialized for spectral analysis are those of Types II and III (Fig. 3; see also Marshall, 1988; Marshall et al., 1991a, b).
Type II ommatidia are located in two of the four most dorsal ommatidial rows of the midband. In these, the main rhabdom that formed a single tier in ommatidia specialized for polarizational analysis is divided into two tiers. Therefore, light entering the rhabdom transits first the ultraviolet-sensitive R8 receptor, then the distal tier of the main rhabdom, and finally the main rhabdom's proximal tier.
Visual pigments, the molecules that absorb light and trigger the biochemical processes of vision, modify incoming light because they absorb within particular spectral regions. Rhabdoms of Type II ommatidia thus operate as a series of spectral filters, each of which affects all receptors beneath. Unlike the situation in other ommatidia, in Type II main rhabdoms (as well as those of Type III, to be described shortly), the distal tier and proximal tiers of main rhabdoms contain different visual pigments. In every case, the pigment of the distal tier absorbs at shorter wavelengths than that of the proximal tier, thereby acting as a long-pass filter. Each tier of Type II rhabdoms sharpens and tunes the spectral sensitivity of the receptors below, producing a set of spectrally specialized, narrow-band photoreceptors (see Cronin and Marshall, 1989a, b).
The serial filtering of Type II ommatidia is further elaborated in Type III (Fig. 3), found in the remaining two ommatidial rows of the midbands of most mantis shrimp species. Type III rhabdoms intercalate colored filters, made of tightly packed vesicles containing strongly absorbing, photostable pigments, at each successive junction between tiers (see Marshall, 1988; Cronin and Marshall, 1989a, b; Marshall et al., 1991a, b; Cronin et al., 1994a). Light entering each tier is therefore altered not only by absorption by visual pigments at all higher levels, but also by the colored filters that it has transited. Type III rhabdoms are particularly suitable for producing middle-to-long-wave-length receptor classes. Visual pigments tend to absorb poorly at these wavelengths (thus making weak filters), but the photostable filters are very effective for such tuning.
The tiers of the main rhabdoms in Type II and Type III ommatidia are formed from the same receptor sets that produce the orthogonal polarization channels in the remainder of the retina. Thus, of the seven receptor cells that contribute to the main rhabdom, those numbered 1, 4, and 5 produce one polarization channel, while 2, 3, 6, and 7 produce the other. These same groupings are retained in the tiers of Type II and Type III ommatidia (see Fig. 3), and this converts what was originally a polarization-opponent organization into chromatic-opponent channels (see Cronin and Marshall, 1989a, b; Marshall et al., 1991a, b, 1996). To prevent confusion between polarization and spectral components, receptors in Type II and Type III rhabdoms are polarization-insensitive, having randomly oriented microvilli (Marshall et al., 1991b).
Since four rows of the midband include ommatidial Types II and III, eight varieties of receptor tiers exist in their main rhabdoms. Each of these contains a different visual pigment (Cronin and Marshall, 1989a, b), producing eight distinct classes of spectral receptor. Together, these sharply tuned receptors span the visual spectrum to peak at wavelengths from about 400 nm to nearly 700 nm in some species. To these are added a diversity of ultraviolet receptor types (in the R8 cells) and also the polarization classes, for a grand total of perhaps 16 spectral receptor classes (see Cronin et al., 1994c; Marshall and Oberwinkler, 1999).
With all these receptor types, one might conclude that no crustacean brain could possibly interpret such a complex, multivariate set of incoming data. Obviously, the polarization system would make things even worse. However, the actual situation is probably quite the opposite. By handling incoming information with a large number of individually specialized receptors (groups of which analyze the same location in visual space) and by processing receptor outputs immediately, sensory information leaving the retina is already streamed into a parallel series of data channels. Each of these may act as a single labeled line of visual information.
The division of the visual spectrum into a series of discrete channels has another, less obvious, advantage. It is difficult for most visual systems (and systems of artificial imaging) to achieve color constancy--the ability to recognize a given color unambiguously--underwater or in spectrally challenging environments. Different classes of color receptors normally respond to a broad spectrum of light, and this can cause them to adapt strongly to stimuli that are spectrally far from their wavelengths of peak sensitivity. By having narrow, sharply tuned spectral classes, a visual system reduces this adaptation to off-peak wavelengths, and thereby maintains excellent color constancy (Osorio et al., 1997). Mantis shrimps use color flamboyantly in their signals (Caldwell and Dingle, 1975), more so than any other aquatic invertebrate. Their unique system of color vision (see also Marshall et al., 1996; Chiao et al., 2000) makes this possible. When color constancy is critical, artificial designs should use narrow, well-chosen spectral channels for accuracy.
Spatial and motion analysis, and the integration of visual information
Being active predators that incapacitate their prey using a ballistic strike, mantis shrimps require a very high-quality sense of space. Again, their eyes are uniquely specialized for this purpose as well. The overlapping fields of view shared by the two halves of each eye probably form the basis of a monocular rangefinder, whereby the distance to a particular object determines which particular receptor pair converges onto it. Since the stalked eyes of mantis shrimps (Fig. 1) move freely in pitch, roll, and yaw, binocular stereopsis is difficult or impossible for these animals to achieve. So monocular rangefinding is particularly suitable (Milne and Milne, 1961).
The very mobility of the eyes of mantis shrimps can confound their interpretation both of spatial location and of relative motion. Like most animals with mobile eyes, the mantis shrimps solve this problem by immobilizing their eyes most of the time relative to the overall visual field, and making rapid eye movements (saccades) to object of interest (Cronin et al., 1991). They may also track individual objects moving within the visual background (Cronin et al., 1988, 1992). These stabilizing mechanisms, necessary to any visual system, whether biological or artificial, permit mantis shrimps to recognize self-motion or motion of other objects in their vicinity.
Mantis shrimps move their eyes for far more unusual purposes. In many species, including Neogonodactylus oerstedii, the ommatidia of the compound eye are located at the end of a relatively long stalk, the pivot of which is far from the visual region of the eye (see Fig. 1). Consequently, as the eye swings, its motion produces a visual flow field in which more distant objects move slowly relative to nearby objects. This relative motion provides an additional depth cue, operating at substantially greater range than the monocular rangefinders, that could be used to distinguish foreground from background. The basic design principle is that the nodal point of the optics is removed from the geometrical center of rotation; some vertebrates (e.g., the sandlance fish and chameleons) have hit on a similar solution to the problem of monocular rangefinding (Ott and Schaeffel, 1995; Pettigrew et al., 1999).
Equally unusual are the small scanning movements that mantis shrimps use, apparently to map their spatially restricted senses of color and polarization onto the extended visual fields of the hemispheric regions of the compound eye. The problem that these animals must solve is a consequence of the geometry of the midband. Since its six rows of receptors all lie in a single plane, together they sample only a planar slice through visual space. Yet this slice contains the most highly analyzed chromatic, ultraviolet, and polarization information. To "paint" color, ultraviolet intensity, and polarization onto the extended visual field, mantis shrimps use small and slow scanning movements. These sweep the planar visual fields of midband ommatidia over the broad and extended visual fields of ommatidia in the hemispherical regions of the compound eye (Land et al., 1990). Engineers often attempt to minimize sensor motion relative to the external world, but in mantis shrimps, motion is essential for visual function. As is also implied in the work of Srinivasan and his colleagues (2001), sensor motion can add information, and instrument design incorporating sensor movement deserves further study.
Summary and Conclusions
Mantis shrimps are among the most successful invertebrate predators in shallow, tropical waters. They are formidable hunters, either cruising about the bottom actively in search of prey or stealthily ambushing it from a concealed burrow. Their success is largely due to their extraordinary visual senses, which perform much of the required lower-level processing of incoming visual information before relaying it to higher visual centers. In doing so, they considerably reduce the computational overhead required in the central nervous system, permitting decisions to be made rapidly and accurately. Although the central nervous system of stomatopods is poorly studied at present, it seems likely that the extensive retinal pre-processing may guide appropriate motor output with relatively minor amounts of central processing. These principles should find excellent application in artificial autonomous constructs as well, and can serve as models for the design of systems of artificial vision.
We thank Christina King, Mike Land, Simon Laughlin, and Roger Hardie for discussions. Phil Rutledge and Tim Ford helped to prepare the figures. This work is based on research supported in the United States by the National Science Foundation under Grant Number IBN-9724028, in the United Kingdom by the BBSRC, and in Australia by the Australian Research Council.
(*.) E-mail: email@example.com
(1.) Department of Biological Sciences, University of Maryland Baltimore County, Baltimore, Maryland 21250
(2.) VTHRC, University of Queensland, Brisbane, Queensland 4072, Australia
Caldwell, R. L., and H. Dingle. 1975. Ecology and evolution of agonistic behavior in stomatopods. Naturwissenschaften 62: 214-222.
Chiao, C. C., T. W. Cronin, and N. J. Marshall. 2000. Eye design and color signaling in a stomatopod crustacean, Gonodactylus smithii. Brain Behav. Evol. 56: 107-122.
Cronin, T. W., and N. J. Marshall. 1989a. A retina with at least ten spectral types of photoreceptors in a mantis shrimp. Nature 339: 137-140.
Cronin, T. W., and N. J. Marshall. 1989b. Multiple spectral classes of photoreceptors in the retinas of gonodactyloid stomatopod crustaceans. J. Comp. Physiol. A 166: 261-275.
Cronin, T. W., J. N. Nair, R. D. Doyle, and R. L. Caldwell. 1988. Ocular tracking of rapidly moving visual targets by stomatopod crustaceans, J. Exp. Biol. 138: 155-179.
Cronin, T. W., N. J. Marshall, and M. F. Land. 1991. Optokinesis in gonodactyloid mantis shrimps (Crustacea; Stomatopoda: Gonodactylidae). J. Comp. Physiol. A 168: 233-240.
Cronin, T. W., H. Y. Yan, and K. D. Bidle, 1992. Regional specialization for control of ocular movements in the compound eyes of a stomatopod crustacean. J. Exp. Biol. 171: 373-393.
Cronin, T. W., N. J. Marshall, and R. L. Caldwell. 1994a. The intrarhabdomal filters in the retinas of mantis shrimps. Vision Res. 34: 279-291.
Cronin, T. W., N. J. Marshall, C. A. Quinn, and C. A. King. 1994b. Ultraviolet photoreception in mantis shrimp. Vision Res. 34: 1443-1452.
Cronin, T. W., N. J. Marshall, R. L. Caldwell, and N. Shashar. 1994c. Specialization of retinal function in the compound eyes of mantis shrimps. Vision Res. 34: 2639-2656.
Jones, J. P. 1994. Architecture and composition of the muscles that drive stomatopod eye movements. J. Exp. Biol. 188: 317-331.
Land, M. F., N. J. Marshall, D. Brownless, and T. W. Cronin. 1990. The eye-movements of the mantis shrimp Odontodactylus scyllarus (Crustacea: stomatopods). J. Comp. Physiol. A 167: 155-166.
Manning, R. B., H. Schiff, and B. C. Abbott. 1984. Eye structure and classification of stomatopod crustaceans. Zool. Scr. 13: 41-44.
Marshall, N. J. 1988. A unique colour and polarisation vision system in mantis shrimps. Nature 333: 557-560.
Marshall, N. J., and M. F. Land. 1993. Some optical features of the eyes of stomatopods. I. Eye shape, optical axes and resolution. J. Comp. Physiol. A 173: 565-582.
Marshall, J., and J. Oberwinkler. 1999. The colourful world of the mantis shrimp. Nature 401: 873-874.
Marshall, N. J., M. F. Land, C. A. King, and T. W. Cronin. 1991a. The compound eyes of mantis shrimps (Crustacca, Hoplocarida, Stomatopoda). I. Compound eye structure: the detection of polarised light. Phil. Trans. R. Soc. Ser. B 334: 33-56.
Marshall, N. J., M. F. Land, C. A. King, and T. W. Cronin. 1991b. The compound eyes of mantis shrimps (Crustacea, Hoplocarida, Stomatopoda). II. Colour pigments in the eyes of stomatopod crustaceans: polychromatic vision by serial and lateral filtering. Phil. Trans. R. Soc. Ser. B 334: 57-84.
Marshall, N. J., J. P. Jones, and T. W. Cronin. 1996. Behavioural evidence for color vision in stomatopod crustaceans, J. Comp. Physiol. A 179: 473-481.
Milne, L. J., and M. Milne. 1961. Scanning movements of the stalked compound eyes in crustaceans of the order Stomatopoda. Pp. 422-426 in Progress in Photobiology. B. C. Christensen and B. Buchmann, eds. Proceedings of the Third International congress on Photobiology, Elsevier, New York.
Osorio, D., N. J. Marshall, and T. W. Cronin. 1997. Stomatopod photoreceptor spectral tuning as an adaptation for colour constancy in water. Vision Res. 37: 3299-3309.
Ott, M., and F. Schaeffel. 1995. A negatively powered lens in the chameleon. Nature 373: 692-694.
Pettigrew, J. D., S. P. Collin, and M. Ott. 1999. Convergence of specialised behaviour, eye movements and visual optics in sandlance (Teleostei) and the chameleon. Curr. Biol. 9: 421-424.
Srinivasan, M. V., S. Zhang, and J. S. Chahl. 2001. Landing strategies in honeybees and possible applications to autonomous airborne vehicles. Biol. Bull. 200: 216-221.
Waterman, T. H. 1981. Polarization sensitivity. Pp. 281-469 in Handbook of Sensory Physiology, Vol. VII/6B, H. Autrum, ed. Springer. New York.
Wolff, L. B. 1997. Polarization vision: a new sensory approach to image understanding. Image Vision Comput. 15: 81-93.
Yamaguchi, T., Y. Katagiri, and K. Ochi. 1976. Polarized light responses from retinular cells and sustaining fibers of the mantis shrimp. Biol. J. Okayama Univ. 17: 61-66.
|Printer friendly Cite/link Email Feedback|
|Author:||CRONIN, THOMAS W.; MARSHALL, JUSTIN|
|Publication:||The Biological Bulletin|
|Date:||Apr 1, 2001|
|Previous Article:||Limulus Vision in the Marine Environment.|
|Next Article:||View From the Boundary.|