Receptor encounters; untangling the threads of the serotonin system.

Receptor Encounters Untangling the threads of the serotonin system

With the help of antidepressants, Ann finally conquered the guilt and despair that had overwhelmed her for two years after her husband's death in 1984. The 62-year-old woman said she felt "as if a cloud had been lifted." Then, in early 1988, Ann took part in a medical experiment. She and 20 other patients recently recovered from depression drank a chocolate-flavored concoction of amino acids that drastically lowers blood levels of a precursor for serotonin, a neurotransmitter involved in several mental disorders.

Three hours after drinking the liquid, Ann began sobbing uncontrollably. She slept little that night and mourned her husband's death as if it had just happened. The following evening, the episode ended as abruptly as it had begun. She stopped crying and told the doctors she felt back to normal.

Bewildered by her sudden, short relapse, Ann was not alone. Thirteen other participants experienced similar setbacks after drinking the fluid.

Researchers say this study, conducted by researchers at the Yale University School of Medicine and the Connecticut Mental Health Center, highlights serotonin's powerful and often confusing role in human behavior. More than a century after discovering it in blood serum and 40 years after characterizing its chemical makeup, scientists are finally succeeding in tracing the pathways through which serotonin stimulates nerve cells in the brain. They now liken its influence to a balancing act: Too little serotonin activity may trigger depression or anxiety, while too much may stimulate equally undesirable effects such as the nausea associated with cancer chemotherapy.

The new key to understanding serotonin is an old buzzword among brain researchers: receptors. Over the last decade, using radioactive compounds that bind to the tiny tips of nerve fibers, or axons, to illuminate them like stars in the night sky, researchers have discerned the shape and apparent function of at least six different serotonin receptors in the brains of rats, rhesus monkeys and humans. With these findings has emerged a battery of experimental drugs that bind to serotonin receptors, advancing the treatment of such serotonin-related disorders as depression and offering researchers novel tools for probing serotonin function.

Researchers exchanged their insights into serotonin receptors in July at a week-long conference sponsored by the New York Academy of Sciences in New York City. In one presentation, Stanford neuropharmacologist Stephen J. Peroutka displayed a graph showing a fourfold increase in journal articles on serotonin over the past decade. Peroutka's own work has added to this flurry of papers: At his lab, researchers computer-scan the structures of some 55,000 catalogued drugs per minute, searching for compounds with just the right shape to attach to a specific serotonin receptor.

"The field is exploding," says neurobiologist Efrain C. Azmitia of New York University. The Food and Drug Administration has approved new serotonin-active drugs to treat depression and anxiety, and has allowed the experimental treatment of obsessive-compulsive disorder with a serotonin-active drug marketed in Europe. In clinical trials, some serotonin-enhancing drugs show promise of relieving migraines and reducing anxiety. Other experimental drugs that block serotonin's action appear to prevent migraines and relieve chemotherapy-related nausea.

Yet nothing is black-and-white when it comes to the gray matter of the brain. Researchers still don't know exactly how serotonin-active drugs work or which might be most effective against a particular disorder. One complicating factor is the drugs' tendency to influence several neurotransmitter systems at the same time. Another is the complexity of the serotonin system itself -- a network of nerve fibers so convoluted that under the microscope it resembles a Jackson Pollock painting come to life.

Serotonin receptors, unlike those receiving signals from certain other neurotransmitters such as dopamine or acetylcholine, do not congregate in just a few areas of the brain, notes Dennis L. Murphy of the National Institute of Mental Health (NIMH) in Bethesda, Md. Fibers emanate from clusters of serotonin-producing nerve cells -- primarily regions called the raphe nuclei -- along the mid-line of the brain stem. From these sites they wend their way to a multitude of areas, including the emotion-regulating limbic region deep in the cerebrum. Such widespread distribution may help explain serotonin's involvement in disorders as diverse as migraine, schizophrenia and overeating.

Further complicating the picture, serotonin studies in animals have revealed a phenomenon common to many neurotransmitter systems: Serotonin receptors exist on nerve cells that produce the neurotransmitter as well as on those stimulated by it. Drugs that influence serotonin's action may affect both kinds of receptors, as well as regulate the clearing of serotonin released by cells.

When a serotonin-producing nerve cell "fires," sending an electrical impulse down the fiber, it triggers the release of serotonin stored in the fiber's end. Receptors perched there act as brakes, regulating the chemical's release into the tiny gap, or synapse, between the cell and an adjacent, or "postsynaptic," nerve cell. Once serotonin spills into the gap, receptors on the postsynaptic nerve cell recognize and bind to it, much as a lock "recognizes" its key. Serotonin activates postsynaptic receptors, prompting them to help open cellular ion channels or to trigger cell release of chemical "second messengers" critical to normal brain function. Its mission accomplished, the serotonin returns -- through a process called reuptake -- to storage compartments in the cell that released it. There it awaits the next round of firing.

Azmitia says this scenario normally occurs with such regularity, as demonstrated in rat studies, that he and others think serotonin may govern the pace at which several types of neurons release their neurotransmitters. But sometimes the system gets out of kilter. Receptors can become overly sensitive to stimulation, or nerve cells may not produce enough serotonin, causing or exacerbating a variety of psychological problems. That's when serotonin-active drugs can intervene.

Some of these compounds act selectively. Serotonin "agonists," for instance, mimic the neurotransmitter's effect as they chemically tweak certain receptors. Serotonin "antagonists" achieve the opposite effect, blocking selected receptors and thus preventing them from receiving serotonin stimulation.

Other drugs flood nerve pathways with serotonin either by blocking its reuptake or by stimulating its release. Comparing the selective and non-selective approaches, Peroutka says: "It's the difference between turning on all the lights in the house or just turning on one -- it depends on what effect you want."

Before 1979, scientists didn't know they had such an option, thinking instead that serotonin bound to only two general classes of postsynaptic receptors. Then biochemical studies and radioactive labeling of animal brain tissue revealed that serotonin -- known chemically as 5-hydroxytryptamine or 5-HT -- binds to three specific families of postsynaptic receptors. Peroutka and his colleagues, who conducted some of these early studies, went on to divide the largest of the three families, called [5-HT.sub.1], into four subtypes in animals: 1A, 1B, 1C and 1D. And ever since, the research papers have been pouring in.

"Theoretically, each receptor subtype provides a target site in the central nervous system that can be pharmacologically manipulated," Peroutka says.

In the alphabet soup of [5-HT.sub.1] receptors, 1A might well stand for anxiety. Some serotonin-mimicking drugs that attach to 1A receptors "might be considered a major breakthrough in the treatment of anxiety," Peroutka says, because they reduce anxiety without the fatigue caused by other anti-anxiety agents. In 1986, the FDA approved the 1A agonist buspirone for treating anxiety. Other 1A drugs, including gepirone and ipsapirone, have shown promise of reducing anxiety in clinical trials.

Intriguingly, rat and monkey studies indicate 1A agonists may also relieve depression, stimulate appetite and help lower high blood pressure. One suggestive finding, says neurologist Claude de Montigny of McGill University in Montreal, is that the number of 1A receptors in rats increases when the animals are exposed to serotonin-active anti-depressants. De Montigny says he plans to present this finding at the annual meeting of the Society for Neuroscience on Oct. 30. Notes Peroutka, "There are a large number of potential clinical uses for [5-HT.sub.1A] agents, and still more may be discovered."

A receptor classified as "[5-HT.sub.1]-like," first identified in 1987 in the blood vessel of a dog, appears the most likely candidate for the 1D receptor in humans and seems to play a role in the constriction of cerebral blood vessels.

Because vessel dilation may trigger migraine headaches, investigators say manipulating the 1D receptors to narrow dilated vessels might offer a new strategy for migraine relief. In fact, the experimental drug sumatriptan, which has provided dramatic migraine relief in clinical studies, may selectively interact with the 1D receptor, say Peroutka and Patrick P.A. Humphrey, a pharmacologist at Glaxo Group Research Ltd. in Hertfordshire, England, who helped identify and test the drug.

The 1C receptors, concentrated in cerebrospinal-fluid-producing blood vessels known as the choroid plexus, may regulate the production and absorption of this fluid, which helps maintain the brain's chemical and physical environment. But because scientists have yet to develop drugs that selectively target these receptors, they remain uncertain of 1C's role in psychiatric illness.

Researchers do know, however, that 1C receptors stimulate cells to activate the same class of second messengers -- chemical "middlemen" within cells that prompt other chemicals to act -- as those stimulated by [5-HT.sub.2] receptors. Some effects of the [5-HT.sub.2] receptors may add up to one big headache. That's because this class of receptor stimulates the brain to produce arachidonic acid, a fatty acid involved in inflammation and possibly in migraines. Some researchers think the migraine-preventing effects of certain drugs, including the [5-HT.sub.2] antagonist methysergide, may stem from their ability to shield these receptors from serotonin stimulation.

Drugs that bind the [5-HT.sub.2] receptors can also powerfully alter mood, notes Mark E. Molliver of the Johns Hopkins University School of Medicine in Baltimore. The now-illicit drug MDMA, also known as "ecstasy," causes nerve cells to release large surges of serotonin, he says. In high does, the drug kills serotonin nerve fibers. Rat research in Molliver's laboratory revealed that brain regions dense with [5-HT.sub.2] receptors also contain nerve cells highly responsive to MDMA's command to release serotonin. Molliver suggests the drug achieves its mood-elevating effect when serotonin released from MDMA-responsive cells stimulates nearly [5-HT.sub.2] receptors. Some psychiatrists contend the euphoria and empathic feelings associated with MDMA have potential in treating people with borderline personality disorders, psychotherapy patients grappling with deep-seated fears, and terminal cancer patients attempting to accept death with equanimity. This controversial prospect has led researchers to seek experimental compounds that elicit some of MDMA's mood-altering effects but do not kill nerve cells.

Researchers have also found a possible link between [5-HT.sub.2] receptors and mood changes in Alzheimer's patients. During the early stages of the illness, the dense bands of [5-HT.sub.2] receptors normally present in the middle layers of the cerebral cortex nearly disappear, notes Alan J. Cross of the Astra Neuroscience Research Unit in London, England. But the receptor loss does not correlate with memory decline, and Cross and others speculate that it may instead relate to the increased aggression and moodiness frequent in Alzheimer's patients. Cross points out that such personality changes cause enormous distress for caregivers, typically prompting families to hospitalize afflicted relatives. "It would seem essential that the large number of novel serotonergic drugs currently under investigation be examined for treatment in Alzheimer-type dementia," Cross says.

Scientists first identified [5-HT.sub.3] receptors in the human brain in 1987. Previous studies in rats, ferrets and rabbits had shown that a brain region called the area postrema, which regulates vomiting, packs a high density of these receptors. This finding dovetails with more recent studies indicating [5-HT.sub.3] antagonists may serve as anti-nausea drugs. Several of these drugs are now in clinical testing to determine whether they can reduce or eliminate the debilitating nausea suffered by cancer patients on chemotherapy.

Although the exact mechanism of such drug action remains unknown, "this is the only receptor we know of that may control vomiting," says Lawrence M. Pinkus of the A.H. Robins Co. in Richmond, Va., which manufactures one of the anti-nausea compounds. Pinkus notes that some of these drugs bind to [5-HT.sub.3] receptors on nerve cells in the gut, a factor that may contribute to their ability to quell nausea.

Many drugs stimulate the entire serotonin system rather than targeting specific receptor types. The controversy over how these drugs work highlights scientists' incomplete understanding of serotonin's role in the brain. Take the story of clomipramine, used widely as an anti-depressant in Europe and Canada and now approved in the United States as an experimental therapy for obsessive-compulsive disorder.

Researchers agree that clomipramine dramatically reduces the repetitious behaviors of many people with obsessive-compulsive disorder. They also agree thaet it stops serotonin-producing cells from reabsorbing the neurotransmitter, leaving more serotonin free to stimulate postsynpatic receptors.

It might stand to reason, then, that any drug that increases serotonin's stimulation of receptors would also help patients with obsessive-compulsive disorder. But when Murphy of NIMH administered such an experimental drug (m-chlorophenylpiperazine) to patients with obsessive-compulsive disorder, their symptoms only worsened. Moreover, he says his unpublished review of several previous clomipramine studies indicates that obsessive-compulsive patients on clomipramine often get worse before they get better.

Some researchers part company with Murphy's explanation of how clomipramine and some other serotonin-reuptake blockers interact with receptors. Murphy believes the findings support the notion that people develop obsessive-compulsive disorder because part of their serotonin system -- perhaps the receptors -- is unusually sensitive. After a few weeks of clomipramine treatment, the suggests, the pendulum swings the other way: Relentless stimulation causes serotonin receptors to decrease in number or sensitivity, ultimately damping the serotonin system. Symptoms fade as the effect of oversensitive receptors dwindles, he reasons.

"I don't buy it," declares McGill's de Montigny, who bases his argument mostly on studies of serotonin-reuptake blockers used to treat depression. While de Montigny agrees that receptor sensitivity serves as an important regulator of serotonin action, he says it's hard to imagine that a drug flooding hypersensitive serotonin receptors could cause them to develop below-normal sensitivity or density. If that were true, "our brain would not function very well at all," he contends.

Treating depression is "one of the most striking roles" of serotonin-influencing drugs, says Hopkins' Molliver. He calls fluoxetine, a serotonin-reuptake blocker approved by the FDA last year, "the most dramatically successful antidepressant drug developed in the last decade." Nonetheless, "it's still confusing how [such drugs] work."

Several studies suggest a link between depression and decreased serotonin activity. During 1987 and 1988, for instance, George R. Heninger, Pedro L. Delgado and their colleagues at the Yale University School of Medicine and the Connecticut Mental Health Center in New Haven studied the effect of L-tryptophan, a serotonin precursor, on 21 patients who had recovered from depression -- including Ann, the middle-aged widow described earlier. On some days the patients received a placebo drink; on others, they drank a chocolate-flavored concoction of amino acids that deplete L-tryptophan. They showed no significant change in behavior after drinking the placebo, but 60 percent suffered a short-lived relapse of depression after partaking of the amino acid mixture.

In previous, smaller studies, the researchers had observed that the drink eliicited only mild responses in healthy people who have no history of depression. Heninger says their more recent results, presented in preliminary form at three scientific conferences last year, will be published early next year.

"We have robust evidence that there is something wrong with the 5-HT system in depressed patients," says de Montigny. If a decrease in serotonin activity can indeed exacerbate depressive tendencies, that might explain why the recovered depressives in Heninger's study suffered a temporary relapse after drinking the fluid.

But work has really just started on what makes the serotonin system tick. At Stanford, Peroutka and his colleagues use a compact disk loaded with data on drug shapes to determine which might bind to the [5-HT.sub.3] receptor. In an analysis of the structure of 40 drugs already known to react with the [5-HT.sub.3] serotonin receptor, the computer calculated the correct reactivity for 38 compounds, he says.

Most tantalizing of all, says Peroutka, is the prospect of constructing hybrid compounds that target many different serotonin receptors simultaneously to elicit multiple desired effects. "Our goal is to combine the effects of different [receptor] subtypes," he says. "For example, some drugs that lower high blood pressure have depression as a major complication. We may be able to construct a hybrid drug that can [help patients] avoid depression and reduce high blood pressure."

With such a multipurpose approach, researchers untangling the serotonin system might someday turn its baffling complexity to their own advantage.