Previous Month | Archives | Current | Next Month

October 30th, 2006 (Permalink)

The Prosecutor's Fallacy

Ben Goldacre has a new Bad Science column discussing a court case involving two distinct mistakes in reasoning about probabilities:

1. The mistake of treating probabilities that may not be independent as if they were. To determine the joint probability of two independent events, you multiply together the probabilities of the individual events. For example, suppose that you toss two coins, what is the probability that both come up heads? Of course, the probability of one coin coming up heads is 1/2. The two events are independent because whether one comes up heads doesn't affect whether the other does, so the probability of both coming up heads is 1/2 X 1/2 = 1/4. However, in the case discussed by Goldacre, the probability of a family having one case of SIDS may well not be independent of the probability of having another, so it is a mistake to use the multiplication rule to determine the joint probability of two such cases in a single family. This type of error doesn't have a name that I'm aware of, though it probably ought to.
2. The "prosecutor's fallacy" is the mistake of confusing the direction of a conditional probability. For example, the probability that you will get lung cancer if you smoke cigarettes is a conditional probability; so is the probability that you smoke given that you have lung cancer. But these are two distinct probabilities, the latter being a much higher probability than the former. Similarly, in the case in question, the probability of an accused person's family having two unexplained infant deaths if they are innocent of murder is different from the probability that the accused person is innocent if there are two such deaths in their family. Suppose, for instance, that there are ten cases of families with two unexplained infant deaths, nine of which are due to SIDS and one is a case of murder. Then, the chance of having two such deaths in a family is very low, namely, ten out of however many total families there are. In contrast, given that a family has two such deaths, the probability that they are due to murder is only 1/10.

   Confusing these different probabilities is an error similar to the propositional fallacy of commuting a conditional―as well as all of the other formal fallacies involving conditional statements, such as affirming the consequent―and perhaps has the same psychological origin. I suppose that it got the name "prosecutor's fallacy" by occurring in some prominent court cases, such as the one discussed by Goldacre, but it might mislead people into thinking that it is only committed by prosecutors.

If anyone knows of other names for these two mistakes, please let me know.

Source: Ben Goldacre, "Prosecuting and Defending by Numbers", Bad Science, 10/28/2006

October 24th, 2006 (Permalink)

An Infinitely Frightening Puzzle

The Boojum is a frightening monster. It is the most frightening monster of all. In fact, it is infinitely frightening. One of the reasons why the Boojum is so frightening is that it attacks people in their sleep, frightening them to death. The only way to keep the Boojum away is to pray to it just before you go to sleep.

I won't blame you if you're skeptical about the Boojum's existence, but you must agree that either it exists or it doesn't. Supposing that the Boojum really does exist, and you don't pray to it, then you risk an infinitely awful Boojum attack. Now, the chance of a Boojum attack may be very small, but it's not zero, and a small chance of an infinite loss is still an infinite loss―this is because an infinite value multiplied by a very small probability is still infinite. On the other hand, if the Boojum does exist and you pray to it, then you lose very little: just a minute or two of your time. Also, if the Boojum doesn't exist and you pray to it, then you'll still lose very little: in addition to some time, maybe a little embarrassment. Finally, if it doesn't exist and you don't pray to it, then you'll come out even.

These possibilities are summarized in the following table:

So, you're slightly better off if you don't pray to a nonexistent Boojum, but infinitely worse off if you don't pray to it and it does exist. However, if you do pray to it, then all you suffer is a small loss in either case. Therefore, for a small loss in time and self-respect, you can avoid the risk of an infinitely frightening Boojum attack. Think of it as insurance: for a small premium, you can be protected from an infinite loss. The only reasonable conclusion is: You should pray to the Boojum!

What's wrong with this argument?

Solution

October 22nd, 2006 (Permalink)

Logic Check

It's that time of year when political ads pile up as fast as the dead leaves, and Annenberg Political Fact Check is putting out a report a day on deceptive ads. An old political trick is to attribute a quote to a media source with no further details than the name of the newspaper, magazine, etc., it appeared in. This suggests that the quote represents the editorial opinion of the source, or that it comes from a news story reported by it. However, what if the quote comes from a letter to the editor or is itself quoted within a story? In that case, it may represent an opinion not endorsed by the source quoted. In this way, a politician can give a false impression that an opinion or unsubstantiated claim is supported by a reputable news agency. That makes it a type of appeal to misleading authority, since the authority of the source is misappropriated by the politician. It's also a good way to smear a political opponent, as you can see in the following report from Fact Check. Check it out.

Source: Justin Bank, "Talent For Deception", Annenberg Political Fact Check, 10/21/2006

October 21st, 2006 (Permalink)

Name That Fallacy!

"One economist quoted prominently in the 'no' [on California's Proposition 87] side's ads is William Hamm, identified as a former California State Legislative Analyst. It's true that Hamm's job used to be one of a neutral analyst for California voters. However, he left that job 20 years ago and currently is one of several managing directors of LECG, a global firm offering expert analysis for hire to corporations, lawyers and others. Hamm's firm is being paid by the 'no' campaign, a total of $94,718 as of Oct. 19 according to reports filed with the California Secretary of State's office. That certainly doesn't make Hamm's statements wrong. However, identifying him by a position he held two decades ago invites voters to give what he says more weight than they might when they discover he is a consultant being paid by one side in the debate."

Fallacy

Source: Brooks Jackson, "$122 Million Worth Of Hype", Annenberg Political Fact Check, 10/19/2006

October 20th, 2006 (Permalink)

What's New?

Since it's the election season, and the political pollsters are working overtime to turn out new polls of public opinion, I decided to dust off my old "How to Read a Poll" article and repost it. It's extensively revised, especially in terms of the external articles it used to link to, most of which have in the meantime disappeared from the web or become outdated. I've also brought up to date the links to relevant weblog entries.

Source: How to Read a Poll, Fallacy Watch

October 13th, 2006 (Permalink)

Check it Out

A common criticism of health care in the United States is based on unfavorable comparisons of the U.S. infant mortality rate with the rates of other countries. However, as Bernadine Healy explains, this is comparing apples to oranges, that is, equivocating on "infant mortality":

…[I]t's shaky ground to compare U.S. infant mortality with reports from other countries. The United States counts all births as live if they show any sign of life, regardless of prematurity or size. This includes what many other countries report as stillbirths. In Austria and Germany, fetal weight must be at least 500 grams (1 pound) to count as a live birth; in other parts of Europe, such as Switzerland, the fetus must be at least 30 centimeters (12 inches) long. In Belgium and France, births at less than 26 weeks of pregnancy are registered as lifeless. And some countries don't reliably register babies who die within the first 24 hours of birth. Thus, the United States is sure to report higher infant mortality rates. For this very reason, the Organization for Economic Cooperation and Development, which collects the European numbers, warns of head-to-head comparisons by country.

Source: Bernadine Healy, "Behind the Baby Count", U.S. News & World Report, 10/2/2006

October 12th, 2006 (Permalink)

October Surprise II: The Lancet Strikes Back

The same group of researchers who two years ago came out with the controversial "100,000" Iraq death toll study published in The Lancet, which I and many others criticized at the time, has come out with a sequel. There were two main problems with the previous study: fake precision, because its confidence interval was so large; and politicizing science, because the authors of the study and the journal's editor were attempting to influence the election by releasing the study shortly beforehand. These facts warranted skepticism about the study's claims, and subsequent research supported that skepticism, indicating that the 100,000 number was probably exaggerated.

The current study is an improvement on the first in that, though it is still wide, the confidence interval is not so wide as to be practically meaningless. There is still some lingering fake precision in the use of 601,027 as the number of Iraqis who suffered violent deaths over the period covered by the study, which is simply an estimate within a broad confidence interval. However, the timing―two years after the original study and, once again, in October before the elections― suggests that the researchers are still grinding a political axe.

Another reason for skepticism about both of these studies is that their numbers are much higher than other estimates: 600,000 is an order of magnitude higher than even the highest of other estimates. Of course, it's possible that the other estimates are the ones in error, or that the true number is somewhere in between. However, when evaluating a poll or survey, it's a good idea to compare it to others done at about the same time and on the same subject. If a poll or survey differs significantly from other comparable ones, that's good reason for caution.

Ultimately, what is needed is an actual count, as opposed to yet another estimate based on a sample. If cluster sample studies are systematically overestimating the death toll, then only an actual count will prove it and give an idea of the degree of overestimation.

Source: Gilbert Burnham, Riyadh Lafta, Shannon Doocy & Les Roberts, "Mortality after the 2003 invasion of Iraq: a cross-sectional cluster sample survey", The Lancet, 10/11/2006

Resources:

• Hamit Dardagan, John Sloboda & Josh Dougherty, "Reality Checks: Some Responses to the Latest Lancet Estimates", Iraq Body Count Press Release, 10/16/2006. A detailed critique of the new Lancet study by the Iraq Body Count group, using "back of the envelope" calculations and comparisons with other sources of information to make a strong case that the study vastly overstates the death toll. (Added: 10/16/2006)
• Sabrina Tavernise & Donald G. McNeil, Jr., "Iraqi Dead May Total 600,000, Study Says", New York Times, 10/11/2006

Update (10/17/2006): Rebecca Goldin of STATS, which I often link to because they usually do good work, has a lengthy article defending the Lancet study against the criticisms of a few bloggers that I've never heard of before. Now, it's perfectly reasonable for her to criticize some weak arguments against the study―after all, criticizing weak arguments is what I do here. But the article suggests that these are the only "scientific" arguments available. She doesn't mention the IBC critique linked above, though that just came out yesterday, so she may not have had time to assess it. However, the biggest problem with Goldin's article is summed up in its conclusion:

The methods used by this study are the only scientific methods we have for discovering death rates in war torn countries without the infrastructure to report all deaths through central means. Instead of dismissing over half a million dead people as a political ploy…we ought to embrace science as opening our eyes to a tragedy whose death scale has been vastly underestimated until now.

Certainly, dismissing the study is unwarranted, but an order of magnitude estimate is a common scientific technique for checking a result for plausibility. Also, subjecting an implausible claim to skeptical scrutiny, and expecting an extraordinary claim to be supported by extraordinary evidence, are standard exercises of "critical thinking", not dismissals.

Goldin seems to be arguing that since this cluster sample survey is the only one we have, we just have to accept its results, no matter how implausible. However, this study is not the only cluster sample study to have been done in Iraq. Here's what the IBC group has to say:

…[O]ur view is that there is considerable cause for scepticism regarding the estimates in the latest study, not least because of a very different conclusion reached by another random household survey, the ILCS, using a comparable method but a considerably better-distributed and much larger sample. This latter study gave a much lower estimate for violent deaths up until April 2004, despite that period being associated with the smallest number of observed deaths in the latest Lancet study.

Goldin does mention the ILCS study in passing, but doesn't address the fact that the results appear to be inconsistent with the Lancet study. So, which is right: the ILCS or the Lancet? Which study is more plausible? Which coheres better with other available sources of information? These are reasonable criticisms which should not be so easily dismissed.

Source: Rebecca Goldin, "The Science of Counting the Dead", Statistical Assessment Service, 10/17/2006

Resource: Update on the Lancet 100,000, 5/14/2005

Fallacies:

• Straw Man
• Suppressed Evidence

October 9th, 2006 (Permalink)

Poll Watch: Record-Setting Hype

Newsweek has a new poll out that has attracted some media attention, especially for its claim of a record low presidential approval rating. I've previously warned that the media tends to hype meaningless "records" in polls, and this is no exception.

Strangely, the report doesn't include standard poll information, such as sample size and margin of error, nor is there a link that I can find to the poll results. I'm under the impression that most media outlets have a policy of including such standard information in poll reports, even though the reporter often proceeds to ignore it in the rest of the story. In this case, it is missing altogether, though a press release put out by Newsweek contains the information.

Here is the supposed new record: "…[T]he president’s approval rating has fallen to a new all-time low for the Newsweek poll: 33 percent, down from an already anemic 36 percent in August." According to the press release, the margin of error is ±4%, so that the difference between the former record and the new one is within the margin of error.

Sources:

• Marcus Mabry, "A Political Limbo", Newsweek, 10/7/2006
• Newsweek Poll Press Release, 10/8/2006

Via: Charles Franklin, "Bush Approval: Newsweek 33%", Pollster Blog, 10/8/2006

October 6th, 2006 (Permalink)

Fact Check it Out

It's that scary season, again. No, I don't mean Hallowe'en; I mean campaign season. The candidates are back attacking each other with fallacies. Fact Check reports on an Iowa congressional race in which each major party has an ad fallaciously attacking the other side's candidate:

• The Democrats quote the Republican out of context:

  The Democratic Congressional Campaign Committee spot, which aired on Sept 30…quotes Republican House candidate Mike Whalen saying, "we don’t have an energy problem." … But by taking Whalen’s quote out of context, the ad misleadingly makes it appear as if Whalen is insensitive to consumers’ energy woes. The quote is part of a lengthy response Whalen gave during a debate with his primary challengers on the May 19 episode of the public television show "Iowa Press." Here is Whalen’s full response to the question…:

  Whalen: Well, we don’t have an energy problem. We have a government problem. Government has stopped American oil and gas producers from drilling in large parts of this country. We have governmental policies that have stopped us from building clean nuclear energy. We have governmental policies that have stopped us from adopting clean coal technology. And we’ve had a government that hasn’t done enough, in my opinion, to really boost renewable fuels. So what we need to do in a lot of ways is recognize that our energy problem is a government problem.

  It is clear from the full exchange that Whalen believes there's a problem with the nation's current energy situation, and that government is at the root of that problem.

• The Republicans are using McCarthyism against the Democrat:

  The National Republican Congressional Committee's Sept 29 ad uses the guilt-by-distant-association tack to go after Democratic candidate Bruce Braley. … The NRCC ad…tells us that Braley has been called a "peace candidate" by the Communist Party. That's true. The Party's newspaper, the People's Weekly World, gave the label to several Democratic candidates in an August issue. However, Braley didn't seek the Communists' support, and it's not even clear whether being called a "peace candidate" is the equivalent of an endorsement.

Source: Viveca Novak & Emi Kolawole, "Midwestern Mythmaking in Iowa", Annenberg Political Fact Check, 10/6/2006

October 5th, 2006 (Permalink)

The Paradox Files: Galileo's Paradox

A perfect square is an integer whose square root is also an integer. For instance, 16 is a perfect square because its square root is 4; and since 6 squared is 36, 36 is a perfect square. Compare the following two arguments:

1. The set of perfect squares is a proper subset of the set of positive integers, that is, it is a subset of the positive integers and it is not equal to the positive integers. In other words, every perfect square is a positive integer, but not every positive integer is a perfect square. One can see this clearly by listing the positive integers while putting the perfect squares in bold:

   1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16….

   One can see that there are more positive integers than there are perfect squares.

2. For every positive integer there is a perfect square―namely, the result of squaring the positive integer―and for every perfect square there is a positive integer―namely, the result of taking the square root of the perfect square. In other words, there is a one-to-one correspondence between the positive integers and the perfect squares, which can be visualized in the following way:

   Positive Integers:	1	2	3	4	5	6	…
   Perfect Squares:	1	4	9	16	25	36	…

   Thus, as we can see, the positive integers and the perfect squares are equinumerous, that is, there are just as many of one as the other. Therefore, there are no more positive integers than perfect squares.

These two arguments appear to have contradictory conclusions: the first shows that there are more positive integers than perfect squares, but the second proves that there are just as many of one as the other. Obviously, something has gone wrong in the reasoning, but what is it?

There are two different ways of comparing the sizes of sets: In the first argument, the proper subset relation was used to compare the sizes of the two sets. In the second argument, a one-to-one correspondence between the two sets was used to conclude that they are the same size. The contradictory conclusions of these arguments show that these are two different notions of size.

When applied to finite sets, the two notions give the same results. No finite set can be put into a one-to-one correspondence with any of its proper subsets, but every infinite set can. In fact, this unusual characteristic is often used to define "infinite set", that is, an infinite set is one that is equinumerous with a proper subset.

So, the moral of the paradox is that the set of positive integers is both the same size as its subset of perfect squares―in the sense that the two sets are equinumerous―and it is larger―in the sense that the latter is a proper subset of the former. This sounds like a contradiction but is not, because these are two distinct senses of "size". It is not intuitively obvious that these senses differ, because they agree on finite sets, which are the only sets that we can experience.

Fallacies:

• Accident
• Ambiguity

Source: Galileo Galilei, Dialogues Concerning Two New Sciences (1638). The original source of the paradox, hence its name.

Resource: Peter Suber, "Infinite Reflections", St. John's Review, XLIV, 2 (1998), pp. 1-59. Excellent detailed discussion of infinity by a philosopher taking Galileo's paradox as the starting point.

October 1st, 2006 (Permalink)

A Philosopher's Fallacy: "All or Nothing"

Philosopher Jerry Fodor has an entertaining review of playwright and novelist Michael Frayn's new book, which is not fiction but a work of philosophy. I haven't read Frayn's book, so I can't comment on the accuracy of Fodor's criticisms, but the following passage is of interest:

Another thing Frayn gets wrong (and here too he doesn’t lack for company) is his persistence in what I’ll call "all or nothing" arguments. So, for example:

What, for example, could have more gratifyingly distinct spatial frontiers than a car? … But now follow it through time, from its beginnings in vague discussions between designers and sales directors [to when it] undergoes compression, meltdown, absorption into the fabric of other cars, into tin cans and bicycles. When did it start being a car…? Somewhere this side of the preliminary discussions, certainly. When did it cease? Somewhere before its transmutation into cans of baked beans.

The implication is that, since there’s no fact of the matter about when a thing starts to be a car (or ceases to be one), there is likewise no fact of the matter about whether a thing is a car; it may be a car according to your story but not according to mine and, in principle, there’s nothing to choose between the stories. So, it’s all or nothing: if there’s no matter of fact at the margins, there’s none in the middle either.

I look out of the window…I tell you that the sun is setting… But, even here, in this simple factual report of what is before my eyes…there is also a performative element… I am deciding that the sun is setting…even though we have no agreement on what precise relationship between sun and horizon constitutes the sun’s setting… All narration and description…is indissolubly subjective because it involves selection.

I’m not saying the bridge is open because it is; it’s open because I say it is.

And finally, with a flourish: "The story is the paradigm. Factual statements are specialised derivatives of fictitious ones."

Piffle. Much of what we know is organised around clear cases, so what’s indeterminate about the marginal Xs can be a plain matter of fact about the paradigms. How many legs can Bossie have before she becomes not a funny kind of cow but a funny kind of centipede? How small can Bossie be before she’s too small to be a cow? (As small as a bread box? As small as an atom?) How big can she be before she’s too big to be a cow? (As big as Australia? As big as the universe?) And what if she walks upright and serves tea to friends? And what if she speaks Latin? What if she turns on and off on Tuesdays? Search me. Or search a biologist; if he doesn’t know, nobody does. But that Bossie as she stands, in full sunlight with four legs and flies, is a cow: that isn’t up to me, or to you, or to anybody else. Bossie is a cow without caveats, a cow sans phrase, a cow tout court. Nor is her being such merely the asymptote that the indeterminate cases converge towards; that gets things backwards: cows grade off from Bossie, not the other way around. That Bossie is a cow is story-independent.

Fodor's "all or nothing" argument is what is sometimes called a "slippery slope" argument, specifically, the semantic type as opposed to the more familiar causal type. "All or nothing" is, perhaps, a better name for the fallacy, since "slippery slope" is more commonly used to refer to the causal form of argument. Frayn is apparently an amateur philosopher, but this type of argument is unfortunately common among the pros―as Fodor notes in parenthesis; in fact, that's probably where Frayn learned to argue this way.

Source: Jerry Fodor, "Who ate the salted peanuts?", London Review of Books, 9/21/2006

Fallacy: Slippery Slope Fallacy, Semantic Version

Solution to the Puzzle (11/1/2006): Several puzzlers pointed out that the puzzle is similar to an argument known as "Pascal's wager". The philosopher and mathematician Blaise Pascal argued that you should "bet" on God's existence because you stand to gain an infinite value if He exists, while only losing a finite amount if He doesn't. The Boojum argument presents a negative version of this argument, where you stand to lose an infinite amount if the Boojum exists and you don't pray to him, but only lose a finite amount if you do pray. It wasn't part of the puzzle proper, so the following readers deserve extra credit for pointing out this similarity: Dan Adams, Lee Randolph, Matt Turner, and Kyle Wilkinson.

So, what's wrong with the argument? Christopher S. Moore identified the fallacy committed:

The logical fallacy of false dilemma―also known as falsified dilemma, fallacy of the excluded middle, black and white thinking, false dichotomy, false correlative, either/or fallacy and bifurcation―involves a situation in which two alternative points of view are held to be the only options, when in reality there exist one or more other options which have not been considered.

Christopher also deserves extra credit for giving the fallacy some aliases that I hadn't heard before!

John Congdon explains why the argument commits the fallacy:

"…[Y]ou must agree that either it [the Boojum] exists or it doesn't" assumes that, if the Boojum exists, it exists only in the manner described. This ignores a number of possibilities: that the Boojum exists but that praying to it won't help you, that the Boojum exists but is purely benign, that the Boojum exists but praying to it will only help if you pray to it sincerely and not merely out of fear or calculation, or―irrespective of the Boojum's existence―that praying to it will certainly offend the Snark, against whose retribution the anger of the Boojum is a mere bagatelle. Since the calculation of possible outcomes does not indicate the full range of possibilities, it is therefore useless as a guide to what action to take.

Dan Adams sums it up nice and concise:

The problem is that only two possibilities are presented: Boojum exists and the claims are true; Boojum doesn't exist. There could be a third option: Gorjack exists and detests prayers to Boojum and will torture all who pray to Boojum, which changes the face of the whole gambit. There could also be this option: Boojum exists and he hates being prayed to and will torture all who pray to him.

In other words, the chart oversimplifies the situation in the following ways: The "Boojum exists" column conceals many alternatives in addition to the one listed, including that the Boojum exists but doesn't like to be prayed to, the Boojum exists but an anti-Boojum monster also exists who hates people who pray to the Boojum, etc. Similarly, the "Boojum Doesn't Exist" column should include the alternatives of an anti-Boojum existing, or a jealous Yahweh who punishes all those who pray to false gods, etc. One needs to take into account the full range of possibilities since they affect the bottom line value of the decision. The possibilities of a Boojum and an anti-Boojum cancel each other out, leading to the correct conclusion that we really have no good reason to pray to the Boojum. When applied to Pascal's Wager, this is known as the "many gods" objection.

Other readers who correctly identified the mistake are: James Knobbs, Mike McKay, and Wade Wesolowsky. Congratulations to them all!

A few readers suggested, not surprisingly, that the argument was an appeal to fear, or more generally, an appeal to consequences. However, it isn't a fallacious appeal, because it doesn't conclude that the Boojum exists, or even that you should believe in it. Rather, the conclusion is that you should pray to the Boojum. It isn't necessarily fallacious to use fear of bad consequences to motivate people to change their behavior, though it may often be better to use a carrot rather than a stick.

Resource: Alan Hájek, "Pascal's Wager", Stanford Encyclopedia of Philosophy

Previous Month | Weblog Archives | Current Weblog | Next Month