Naked Statistics is the most interesting book about the most boring science

2024 Author: Malcolm Clapton | [email protected]. Last modified: 2023-12-17 03:44

Who said statistics is a dull and useless science? Charles Wheelan convincingly argues that this is far from the case. Today we publish an excerpt from his book on how to win a car, not a goat, using statistics, and understand that intuition can deceive you.

The Monty Hall Riddle

The Monty Hall Mystery is a famous problem in probability theory that baffled participants in a game show called Let’s Make a Deal, still popular in several countries, which premiered in the United States in 1963. (I remember every time I watched this show as a child, when I did not go to school due to illness.) In the introduction to the book, I already pointed out that this game show can be interesting for statisticians. At the end of each of its issues, the participant who reached the finals stood with Monty Hall in front of three large doors: Door No. 1, Door No. 2 and Door No. 3. Monty Hall explained to the finalist that behind one of these doors was a very valuable prize - for example a new car and a goat behind the other two. The finalist had to choose one of the doors and get what was behind it. (I don't know if there was at least one person among the participants in the show who wanted to get a goat, but for the sake of simplicity, we will assume that the vast majority of participants dreamed of a new car.)

The initial probability of winning is fairly easy to determine. There are three doors, two hides a goat, and the third hides a car. When a participant in the show stands in front of these doors with Monty Hall, he has one out of three chances to choose the door behind which the car is located. But, as noted above, there is a catch in Let’s Make a Deal that immortalized this TV program and its presenter in the literature on probability theory. After the finalist of the show points to one of the three doors, Monty Hall opens one of the two remaining doors, behind which there is always a goat. Then Monty Hall asks the finalist if he wants to change his mind, that is, to abandon the previously selected closed door in favor of another closed door.

Let's say, for the sake of example, that the participant pointed to Door # 1. Then Monty Hall opened Door # 3, behind which the goat was hiding. Two doors, Door # 1 and Door # 2, remain closed. If the valuable prize was behind Door No. 1, the finalist would have won it, and if it was behind Door No. 2, then he would have lost. It is at this point that Monty Hall asks the player if he wants to change his initial choice (in this case, abandon Door # 1 in favor of Door # 2). You will, of course, remember that both doors are still closed. The only new information the participant received was that the goat ended up behind one of two doors that he did not choose.

Should the finalist abandon the initial selection in favor of Door # 2?

I answer: yes, it should. If he sticks to the original choice, then the probability of winning a valuable prize will be ⅓; if he changes his mind and points to Door No. 2, then the probability of winning a valuable prize will be ⅔. If you don't believe me, read on.

I admit that this answer is far from obvious at first glance. It seems that whichever of the remaining two doors the finalist chooses, the probability of receiving a valuable prize in both cases is ⅓. There are three closed doors. At first, the probability that a valuable prize is hidden behind any of them is ⅓. Does the finalist's decision to change his choice in favor of another closed door make any difference?

Of course, because the catch is that Monty Hall knows what's behind every door. If the finalist chooses Door # 1 and there is indeed a car behind it, Monty Hall can open either Door # 2 or Door # 3 to reveal the goat lurking behind it.

If the finalist selects Door 1 and the car is behind Door 2, then Monty Hall will open Door 3.

If the finalist points to Door 1 and the car is behind Door 3, then Monty Hall will open Door 2.

By changing his mind after the presenter opens one of the doors, the finalist gains the advantage of choosing two doors instead of one. I will try to convince you of the correctness of this analysis in three different ways.

The first is empirical. In 2008, New York Times columnist John Tyerney wrote about the Monty Hall Phenomenon. After that, the staff of the publication developed an interactive program that allows you to play this game and independently decide whether to change your initial choice or not. (The program even provides for little goats and little cars that appear from behind the doors.) The program records your winnings in the event that you change your initial choice, and in the case when you remain unconvinced. I paid one of my daughters to play this game 100 times, changing her original choice each time. I also paid her brother to play the game 100 times too, keeping the original decision each time. The daughter won 72 times; her brother 33 times. Each effort was rewarded with two dollars.

Evidence from episodes of the game Let’s Make a Deal shows the same pattern. According to Leonard Mlodinov, author of The Drunkard's Walk, those finalists who changed their initial choice were about twice as likely to win as those who were unconvinced.

My second explanation for this phenomenon is based on intuition. Let's say the rules of the game have changed slightly. For example, the finalist starts by choosing one of three doors: Door # 1, Door # 2, and Door # 3, as originally planned. However, then, before opening any of the doors, behind which the goat is hiding, Monty Hall asks: "Do you agree to give up your choice in exchange for opening the two remaining doors?" So, if you chose Door # 1, you can change your mind in favor of Door # 2 and Door # 3. If you pointed to Door # 3 first, you can select Door # 1 and Door # 2. And so on.

For you, this would not be a particularly difficult decision: it is quite clear that you should abandon the initial choice in favor of the two remaining doors, as this increases the chances of winning from ⅓ to ⅔. The most interesting thing is that it is this, in essence, that Monty Hall offers you in a real game, after opening the door behind which the goat is hiding. The fundamental fact is that if you were given the opportunity to choose two doors, a goat would be hidden behind one of them anyway. When Monty Hall opens the door behind which the goat is, and only then asks you if you agree to change your initial choice, it significantly increases your chances of winning a valuable prize! Basically, Monty Hall is telling you, "The chances of a valuable prize hiding behind one of the two doors that you didn't choose the first time are ⅔, which is still more than ⅓!"

You can imagine it like this. Let's say you pointed to Door # 1. After that, Monty Hall gives you the opportunity to abandon the original decision in favor of Door # 2 and Door # 3. You agree and you have two doors at your disposal, which means that you have every reason expect to win a valuable prize with a probability of ⅔, not ⅓. What would have happened if at this moment Monty Hall had opened Door 3 - one of "your" doors - and there was a goat behind it? Would this fact shake your confidence in your decision? Of course not. If the car was hiding behind Door 3, Monty Hall would open Door 2! He wouldn't show you anything.

When the game goes through a knock-off scenario, Monty Hall really gives you a choice between the door you specified at the beginning and the two remaining doors, one of which could be a car. When Monty Hall opens the door behind which the goat is hiding, he is simply doing you a favor by showing you which of the other two doors is not the car. You have the same probabilities of winning in both of the following scenarios.

1. Selecting Door # 1, then agreeing to "switch" to Door # 2 and Door # 3 even before any door is opened.
2. Selecting Door # 1, then agreeing to "switch" to Door # 2 after Monty Hall shows you the goat behind Door # 3 (or choosing Door # 3 after Monty Hall shows you the goat behind Door # 2).

In both cases, abandoning the original decision gives you the advantage of two doors over one, and you can thus double your chances of winning from ⅓ to ⅔.

My third option is a more radical version of the same basic intuition. Let's say Monty Hall asks you to choose one of 100 doors (instead of one of three). After you do this, say by pointing to Door # 47, he opens the 98 remaining doors, which are followed by the goats. Now only two doors remain closed: your Door No. 47 and another, for example, Door No. 61. Should you give up your initial choice?

Of course yes! There is a 99 percent chance that the car is behind one of the doors that you did not choose at first. Monty Hall did you the courtesy by opening 98 of these doors, there was no car behind them. Thus, there is only a 1 in 100 chance that your initial choice (Door # 47) will be correct. At the same time, there is a 99 out of 100 chance that your initial choice was wrong. And if so, then the car is located behind the remaining door, that is, Door No. 61. If you want to play with the probability of winning 99 times out of 100, then you should "switch" to Door No. 61.

In short, if you ever have to play Let’s Make a Deal, you will definitely need to backtrack on your original decision when Monty Hall (or whoever will replace him) gives you a choice. A more universal conclusion from this example is that your intuitive guesses about the likelihood of certain events can sometimes mislead you.