Soft science.

ONE of the grandest ambitions of the world's great minds is to unlock the mysteries of the financial markets--strip away our superstitions and lay bare the true inner workings of a mechanism like the stock market.

"The financial markets aren't mysterious, but rather physical systems that ought to be examined scientifically and engineered rationally," says the Wall Street Journal, summarizing the views of Yale University mathematician Benoit B. Mandelbrot.

The efficient market hypothesis says in effect that market prices at any given time already reflect all relevant information that can be known or suspected. Index funds are based on the premise that low-cost portfolios that mimic a market index hold a natural edge over the average higher-cost managed fund.

You need not own index funds to benefit from them. They bring intense, relentless competition to bear on active managers, pressing them to keep their performance up and their costs down.

Indexers have been around for 30 years. So why hasn't the advance of financial science made many more great strides since then?

One obstacle is the human factor in markets. Other sciences may rest on reliable truths: Ice floats in water; rolling friction is less than sliding friction. In the stock market, higher corporate profits mean higher stock prices only when fickle investors decide it should be so.

So academics look not only at the "hard" science of mathematical equations, but also at "soft" sciences such as psychology. Witness the rise of behavioral finance, which investigates the role of regret in investment decision-making.

Because of most humans' aversion to pain, regret theory says, they may make investment decisions that emphasize avoiding unpleasant experiences at the expense of logic.

For example, short-term uncertainties regularly dissuade people from making long-term investments in stock or bond mutual funds, even when their objectives are unquestionably long-term.

Another thorny problem is that once any law of market behavior is found and published, it naturally tends to vanish.

If we discover that certain stocks outperform another group, for example, investors will quickly adjust stock prices to account for the disparity, and in so doing neutralize it.

For all its stumbling blocks, academic research is a real and potent presence in today's markets. It is embodied in such firms as Dimensional Fund Advisers in Santa Monica, which manages $55 billion using an approach that, in its own words, "applies academic research to the practical world of investing."

Maybe someday the art of investing can be turned into a science, maybe it can't. Either way we stand to learn a lot.

--Chet Currier, Bloomberg News