Benoit Mandelbrot (1924 - 2010) and the "Mystery of Cotton"

Benoit B. Mandelbrot, 1924 - 2010

On Saturday, October 16th 2010 Tyler Cowen reported on his blog "Marginal Revolution" the death of Benoit B. Mandelbrot, mathematician, inventor of fractals, and analyst of financial markets. I checked the web site of Yale University, where M. had taught, for confirmation of the news but there was none. But later in the day the news was confirmed by the New York Times and by Wired.

I first heard of Mandelbrot in the early 1980s when I was working with Sanjeev Gupta, who now is with the IMF, on our papers on the temporal and spatial efficiency of hog markets in Germany. At the time, Sanjeev had recently arrived at the Institute of World Economics from the University of Vancouver where he had done his graduate research on black market exchange rates for the Indian Rupee. We thought it would be fun to apply cutting- edge statistical methods that had been developed for financial markets to markets for real commodities.

I had no idea what finance chaps meant when they talked about efficiency and I spend a considerable amount of time in the economics and business section of our university library, reading papers by Fama, Cootner, Alexander, and Mandelbrot. I didn't understand much of what I read and if I went back to that literature, I doubt that I would understand it now.

When Mandelbrot's "Apfelmännchen" graphics were all the rage, I didn't care much about them. They were pretty pictures, but not much more. Then sometime in 2005 I bought Mandelbrot's book with Richard L. Hudson "The (Mis)Behavior of Markets" (New York, NY: Basic Books, 2004) which I did not read for some time. What brought me back to the book was Nassim Taleb's "Black Swan", which I read during a holiday at Ramsau in 2007.

After reading Cowen's notice of Mandelbrot's demise I re-read Chapter 8, "The mystery of cotton". It is an enjoyable chapter of three interwoven stories: (i) statistical analysis of cotton exchange market prices, (ii) Mandelbrot's intellectual development leading to his idiosyncratic approach to the study of market prices, and (iii) the unenthusiastic reception he and his work received from the economics profession.

Cotton prices

In 1961 M. was invited by Hendrick S. Houthakker, a Harvard economist, to give a seminar on his work on income distribution that M. was doing at IBM. On the blackboard in Houthakker's office M. saw a graph that looked like graphs he that was using in his analysis of income data. But the graph did not show any properties of income data, is was on changes in cotton prices. Houthakker told M. that he had been studying cotton prices for some time but he admitted that he was getting nowhere. Houthakker complained, "I've had enough ...  I've done everything I could to make sense of these cotton prices. I try to measure the volatility. It changes all the time. Everything changes. Nothing is constant. It's a mess of the worst kind" (p. 149).

Houthakker abandoned his cotton price research and bequeathed all his cotton price data - stored on punch cards - to M. who took them back to his IBM lab at New York. M. began analyzing them in his way and added as many cotton price data as he could. The result of this effort was a working paper the was widely circulated among economist interested in financial market analysis. Agricultural economists appear to have been largely unaware of M.'s cotton price analysis.

The key result of this analysis is a log-log graph of the size and frequency of cotton price changes during a period of one hundred years. The changes seemingly show a power-law behavior and a stable distribution. Moreover, the distributions were the same for daily, monthly, and yearly cotton price changes. In short, the cotton price data showed a fractal pattern. M. concludes: "the very heart of finance is fractal". (p 165)

This graph from Mandelbrot 1963. The variation of certain speculative prices. Journal of Business 36(4):394-419. The graph is reproduced in Ch. 8 on p. 163.

Background: Zipf, Levy, and Pareto

Getting beyond Houthakker in the analysis of cotton exchange price data required three cognitive tools: power laws, Pareto's income distributions, and stable distributions.

M. had been introduced to power laws as a mathematics student in Paris in 1950. When he was looking for a thesis topic his uncle, a mathematics professor, gave him Zipf's book "Human behavior and the Principle of Least Effort" to read. M. "was spellbound". He could see how to expand the math behind Zipf's word frequencies and Zipf led M. to economics.

At the Ecole Polytechnique at Paris M. was introduced to stable distributions by the mathematician Paul Levy, who M. describes as "... now acknowledged as one of the greatest probabilists." (Apparently, M. had a rare fondness for Levy: at the end of a lecture series by Levy, M. was the only student left.) M. characterizes stable probability distributions as follows: "Now, stable means that you can do something to an object - for instance rotate it, shrink it, or add to it to something else - and its basic properties remain unaltered. A Gaussian bell curves is stable in this sense." (p. 160)

M. started studying economics, and Pareto's work on income distribution, in his thirties. From Pareto M. learned that "Society was not a social pyramid" but more of a "social arrow" where "... few people are outrageously rich, and the vast bulk of people are middling or poor." But M. was interested in the mathematical properties of income distribution. In particular, when income is plotted against the number of people having that income is plotted on log-log paper, a straight line obtains, indicating that a power law is in play. M. continued to pursue his interest in the analysis of income distributions at the IBM lab where he was hired to work on economics as a new use of computers.

Hostile reception by economists

In 1962 M. circulated results of his cotton price data analyses in a IBM research report in order for M. to become eligible for a temporary teaching position at Harvard. The report tickled the curiosity of economists and Zvi Griliches (an ag. economist) invited M. to present his cotton price work at a meeting of the Econometrics Society.

Paul Cootner of MIT, who was a discussant of M.'s paper, wanted to include M's paper in his compendium "The Random Character of Stock Market Prices". However, the book should assemble only papers that had already been printed in "proper academic sources". Cootner therefore asked M. to urgently arrange for the publication of his cotton price paper in an academic economics journal. This M. achieved with the help of his ex-student Eugene Fama and Merton Miller, the editor of the Journal of Business.

"And the reception was hostile" (p. 166). Even Cootner, who had invited M. to contribute to his book, had reservations: "... Cootner included a five- page critique in his book. He felt my graph-paper test too simplistic, the math intractable, the evidence insufficient, and cotton too peculiar a commodity from which too draw sweeping conclusions. The implications were great, he wrote. But 'surely before consigning centuries of work to the ash pile, we should like to have some assurance that all our work is truly useless." (p. 166)

Significance

What, in the words of M., is the significance of the "Mystery of Cotton"?:

"Let us mull the promise that science makes to society to win its support. The grand promise is to endeavor solving the great mysteries - to the list of which I have added one. But there is also a more practical promise. It consists in helping society to improve, to prevent it from acting on the basis of theories that sound nice but are not true to the facts, and to help it act on the basis of facts - even if those facts have yet to find a theory that fully explains them." (p. 170)