Stats Made Easy

Practical Tools for Effective Experimentation

Sunday, February 18, 2007

Overreacting to patterns generated at random – Part 2

Professor Gary Oehlert provided this heads-up as a postscript on this topic:

“You might want to look at Diaconsis, Persi, and Fredrick Moesteller, 1989, “Methods for Studying Coincidences” in the Journal of the American Statistical Association, 84:853-61. If you don't already know, Persi was a professional magician for years before he went back to school (he ran away from the circus to go to school). He is now at Stanford, but he was at Harvard for several years before that.”

I found an interesting writeup on Percy Diaconis and a bedazzling photo of him at Wikipedia. The article by him and Moesteller notes that “Coincidences abound in everyday life. They delight, confound, and amaze us. They are disturbing and annoying. Coincidences can point to new discoveries. They can alter the course of our lives; where we work and at what, whom we live with, and other basic features of daily existence often seem to rest on coincidence.”

However, they conclude that “Once we set aside coincidences having apparent causes, four principles account for large numbers of remaining coincidences: hidden cause; psychology, including memory and perception; multiplicity of endpoints, including the counting of "close" or nearly alike events as if they were identical; and the law of truly large numbers, which says that when enormous numbers of events and people and their interactions cumulate over time, almost any outrageous event is bound to occur. These sources account for much of the force of synchronicity.”

I agree with this skeptical point of view as evidenced by my writing in the May 2004 edition of the Stat-Ease "DOE FAQ Alert" on Littlewood’s Law of Miracles, which prompted Freeman Dyson to say "The paradoxical feature of the laws of probability is that they make unlikely events happen unexpectedly often."


  • At 12:56 PM, Anonymous Anonymous said…


    First of all let me say that I have enjoyed reading your DOE FAQ email newsletter for a while now and I really enjoy your blog. I have thought about doing a similar blog myself, I really enjoy thinking about and running stats on pop news headlines and stuff. So bravo.

    The issue of coincidence is one that I really have alot of interest in. Here is a link to a discussion thread I started on one of ASQ's discussion boards. I assume you have access to it. If not I can send you a copy of the whole discussion, it got pretty long and interesting. But I have an article planned on this example and subject in the future.

    Anyway, well done, love to read more in the future and converse with you more...

    Matt B.

  • At 1:39 PM, Anonymous Anonymous said…

    You may enjoy this too. - mr. mork

  • At 7:15 AM, Blogger Mark said…

    Mr. Mork points to an intriguing look at random (?) selection of music on now-ubiquitous portable players. I do not have one myself, but my wife does and, of course, both teenage daughters, so I feel really out of tune. (Ha!) On a similar vein, I keep watch on Microsoft's My Pictures Slideshow screen saver while doing my cardio work on my elliptical. See this thread from Ask MetaFilter. - Mr. Mark

  • At 6:26 AM, Anonymous Anonymous said…

    People are befuddled when they set their mp3 players to "random shuffle" and wind up hearing three songs in a row by the same artist.

    The thing is, if there are multiple songs by the same artist in the playlist, it's likely for a random shuffling to result in a block of consecutive songs by the same artist.

    An argument over whether the shuffling is random then ensues between users and the creators of the devices. Ultimately the engineering creators prevail, proving that the ordering is indeed random in the statistical sense of the word, but what seems to get lost in the [ahem] shuffle is what is actually desired by users.

    When users say they want their songs played in random order, what I think they really want is a subset of all possible permutations, a subset determined by constraints such as "if possible, don't play three songs in a row by the same artist."

    When mp3 player creators prove users harbor mistaken notions of randomness, they're only being pedantic, and they're failing, I believe, to provide users with what they want.

    -- mr. mork

    Competing for the Longest Comment Award. :)


Post a Comment

<< Home