A Lie in a Lab Coat
Posted by Shannon Love on March 21st, 2005 (All posts by Shannon Love)
So my old nemesis the Bogus Lancet study of Iraqi casualties is showing up again here and there so I thought I would revisit it.
What grabbed my attention this time around is the intentionally inconsistent use of the Falluja cluster data. The study produces radically different results depending on whether the Falluja data is included or excluded. With the Falluja data, most of the excess deaths result from violence, without it most excess deaths result from accident or disease. With the Falluja data, most of deaths from violence were of woman and children, without it, most of the deaths from violence were of military aged males. With the Falluja data, well over 250,000 Iraqi, over 1% of the entire population, have died largely from Coalition helicopter airstrikes, without it, that number is in the more plausible tens of thousands.
The highly selective inclusion or exclusion of the Falluja data in various statements are clearly finely tuned for maximum political impact while still conveying a plausible number of Iraqi deaths. The paper is clearly written to maximize the damage to the Coalition war effort and for political impact in the US presidential election.
It is an act of outright scientific corruption, a lie in a lab coat.
(If you’re unfamiliar with the Lancet Study and my critiques of the same, look at this post.)
The study defined 33 clusters each statistically representing 1/33rd or ~%3 of the entire Iraqi population. That comes to 739,000 individuals.[p2 pg2] Each cluster contained 30 households assumed to contain 7 individuals each (total of 210 per cluster). So, simplistically, each individual in the study represented 3,519 fellow Iraqi. (Cluster sampling explained.)
The Falluja cluster caused problems because it returned results so extreme that it destroyed the plausibility of the entire study. The Falluja cluster reported 71 deaths from violence [table 2] out of population of 210 individual so a full third of the clusters pre-war population was supposedly killed by violence (remember this was before the invasion of Falluja in Nov 2004). Extrapolated to the rest of the population that means that 1% of the total Iraqi population or ~243,870 individuals were killed in Iraq over an 18 month period. Adding in the results from the other clusters (21*3519=73,899) raises the toll to ~317,769. (Its not that simple as the actual number would result from heavy statistical processing but the magnitude would be the same and the actual number varying at most by +-50,000 I would think.)
This is obviously a non-sensical number. It would mean ~1,745 people per day were dying largely (according to the study) as a result of Coalition air-strikes with most of those airstrikes delivered by helicopters. Since there have been no reports of any airstrikes were more than a few dozen of people where killed (an only a handful of those) it would mean that hundreds of airstikes occured every day that killed several people each. It would mean only 1 in 15 civilians deaths were ever reported in any media at any time. It would mean a nation dotted with fresh mass graves. By comparison recall that Japan in WWII, where every major city save two were saturation bombed or nuked, suffered only 500,000-600,000 civilian deaths (almost all due to air raids) out of a population of 78 million. That is just under 1% of the population of Japan. That means that Iraq suffered proportionally as many or more deaths from airstrikes than did the nation that suffered the harshest air bombardment in history. Clearly this is not the case. The shear logistics of burying that many people would be impossible to conceal.
The fact that a Falluja cluster even got studied raises red flags. The studies authors bemoan the lack of security in the nation and explain that it caused them move the original site of other clusters yet they managed to get one cluster in downtown bandit central, in the one city in the entire country that had absolutely no Coalition presence at the time the study was conducted. Riiiiiiiiiight, I’ll by that. The best explanation is that the Baathist and foreign Jihadist who controlled Falluja at the time realized that the study could only help their cause and so they let the researchers in but tried to control the results. Unfortunately, they couldn’t resist the urge to inflate the damage and they over did it.
With the Falluja data obviously compromised the researchers now had a dilemma. If they clearly published the data the study actually returned they would get laughed at. On the other hand, they had no methodological reason to exclude the Falluja data. The cluster was supposedly chosen based on objective standards. Why exclude the Falluja cluster but keep others? The study’s design defined no criteria for excluding clusters before hand, save for reasons of safety. It defined none for excluding clusters after the survey. The Falluja data was selectively excluded merely because it was extreme. This is a textbook example of bad science.
The honest approach would have been to either completely include or exclude (preferably the later) the Falluja cluster completely. However, without the Falluja data the most headline grabbing findings in the study aren’t supported. In order to accomplish their political goals, the researchers decided to lie by purposely obscuring when they were using the Falluja data and when they were not.
Don’t believe me? Below is the Findings and Interpretation sections of the summary, the most widely reported parts of the study and the parts which have the most practical political effect. First read the section in its original widely quoted form:
Findings The risk of death was estimated to be 2·5-fold (95% CI 1·6–4·2) higher after the invasion when compared with the preinvasion period. Two-thirds of all violent deaths were reported in one cluster in the city of Falluja. If we exclude the Falluja data, the risk of death is 1·5-fold (1·1–2·3) higher after the invasion. We estimate that 98000 more deaths than expected (8000–194000) happened after the invasion outside of Falluja and far more if the outlier Falluja cluster is included. The major causes of death before the invasion were myocardial infarction, cerebrovascular accidents, and other chronic disorders whereas after the invasion violence was the primary cause of death. Violent deaths were widespread, reported in 15 of 33 clusters, and were mainly attributed to coalition forces. Most individuals reportedly killed by coalition forces were women and children. The risk of death from violence in the period after the invasion was 58 times higher (95% CI 8·1–419) than in the period before the war.
Interpretation Making conservative assumptions, we think that about 100000 excess deaths, or more have happened since the 2003 invasion of Iraq. Violence accounted for most of the excess deaths and air strikes from coalition forces accounted for most violent deaths. We have shown that collection of public-health information is possible even during periods of extreme violence. Our results need further verification and should lead to changes to reduce noncombatant deaths from air strikes.
Can you easily pick out the statements that are only supported if the Falluja data is included? Here’s the same text with the Falluja dependent statements bolded.
[The struck sections were originally considered unsupported without the Falluja data but after posting I decided they are supported. See update3 below]
Findings The risk of death was estimated to be 2·5-fold (95% CI 1·6–4·2) higher after the invasion when compared with the preinvasion period. Two-thirds of all violent deaths were reported in one cluster in the city of Falluja. If we exclude the Falluja data, the risk of death is 1·5-fold (1·1–2·3) higher after the invasion. We estimate that 98000 more deaths than expected (8000–194000) happened after the invasion outside of Falluja and far more if the outlier Falluja cluster is included. The major causes of death before the invasion were myocardial infarction, cerebrovascular accidents, and other chronic disorders whereas after the invasion violence was the primary cause of death. Violent deaths were widespread, reported in 15 of 33 clusters, and were mainly attributed to coalition forces. Most individuals reportedly killed by coalition forces were women and children. The risk of death from violence in the period after the invasion was 58 times higher (95% CI 8·1–419) than in the period before the war.
Interpretation Making conservative assumptions, we think that about 100000 excess deaths, or more have happened since the 2003 invasion of Iraq. Violence accounted for most of the excess deaths and air strikes from coalition forces accounted for most violent deaths. We have shown that collection of public-health information is possible even during periods of extreme violence. Our results need further verification and should lead to changes to reduce noncombatant deaths from air strikes.
Surprised? I sure as hell was. I read this study several times back in Nov 2004 and missed it almost completely. Only when I went back over it today did it leap out at me.
Based on Table 2 [p4] in study, if violent deaths in Falluja are excluded then none of the bolded statements are supported. Excluding Falluja, the total delta in deaths before and after the war is +44. Of those deaths 21 (47%) resulted from violence. Of those deaths 4 (19%) were children =60 years and 13 (62%) were males age 15-59.
So an honest report of the Findings and Interpretations would read something like:
Findings: After excluding a cluster in Falluja that appeared to have been compromised, the risk of death was estimated to be 1·5-fold (1·1–2·3) higher after the invasion. We estimate that 98000 more deaths than expected (8000–194000) happened after the invasion. The major cause of death both before and after the invasion remained illness and accident. Violent deaths were reported in 14 of 32 (44%) of all clusters and were mainly attributed to coalition forces. Most individuals (62%) reportedly killed by coalition forces were males of military age. The risk of death from violence in the period after the invasion was 24 times higher than in the period before the war.
Interpretation: Making conservative assumptions, we think that about 100000 excess deaths, or more have happened since the 2003 invasion of Iraq. Accident and illness accounted for most of the excess deaths. We have shown that collection of public-health information is possible even during periods of extreme violence but that compromise of data by political actors remains a problem that must be accounted for. Our results need further verification and should lead to changes to improve delivery of medical services to non-combatants in war zones.
Of course, the honest version would have had nearly zero political impact and indeed could have rebounded to Bush’s favor. Reports that more people died of accident and illness than from violence and that of those who did die of violence, most where military aged males would not have the same resonance.
When you realize that without the Falluja data the study tells a very different story than the one widely reported and that the Falluja data could only have been collected with active collusion of the Baathist and the Jihadist who ruled Falluja at the time, the publication of this study assumes a very sinister cast. Either through intention or willful disregard, the researchers and publisher acted as a propaganda tool for the Fascist elements in Iraq. Given the degree to which they carefully spun their results, I conclude the effect was intended.
I have watched all my adult life with sadness the increasing whoring out of scientist and scientific institutions to politics. In the past, politicization of science was viewed as a moral flaw but the post-modernist morality encourages every person to regard politics as the highest good and to evaluate the morality of each decision on the basis of its political utility. A generation of scientist has grown up with this idea and have abandoned the concept that they should consciously seek to shield their work from the contamination brought by politics. Instead, they embrace politicized science and seek to use their positions of scientific authority to advances their preconceived political agendas.
Lancet has long since fallen prey to this corruption. This paper is just the latest example of its long spiral into darkness.
(Update: The Lancet site seems to be down. If you want a PDF of the study email me at shannon-at-chicagoboyz-dot-net and I’ll email you a copy)
(Update2: Here a jpeg of Table 2 referenced above. It got a little blurry but I think it should be readable. Click the image to display a larger version.)
Unless I muffed some really basic addition and subtraction it is clear that excluding the Falluja cluster undermines the statements in the Summary, as I said above.
Update3: In the comments, Tim Lambert points out that I neglected to correct for the differing time frames of the pre- and post-war samples, so that the delta is larger than it should be.
Increasing the pre-war time time span from 14.6 months to 17.8 would presumably add something like 8 more deaths to the study’s pre-war total of 46, bringing it to 54. In addition, the Falluja cluster apparently contained 5 deaths from non-violent causes that are not broken out separately in Table 2. [p5, pg3]. Both of these factors will increase the ratio of violent to non-violent deaths. With Falluja excluded, the delta becomes +32 (89 post, 54 pre). 21 deaths is 66% of 32 so it far to say that “most” of the “excess” deaths were caused by violence.
I have corrected the post to reflect this. However, this does not alter my main point, the selective inclusion and exclusion of Falluja data throughout the paper. I will add some additional examples latter today.
Note that what we are talking about is the ratio between excess violent to non-violent deaths. Perhaps ironically, anything done to maintain or improve the general health of the population will make this ratio look worse. If mortality from other causes had remained perfectly flat, then 100% of the deaths would have been due to violence even if only one person in the entire country died of violence.
Upon further refection, I am at a loss to understand the practical use of this ratio. What information is it supposed to convey to decision makers? Just because the ratio is high doesn’t mean that military tactics should be changed. What point does this observation accomplish beyond being a propaganda tool?
ATTENTION!: I broke down and just Fisked the entire damn study paper if anybody is still interested.
March 21st, 2005 at 8:57 pm
Falluja was excluded because it was an outlier. This is perfectly standard practice. Even excluding Falluja, the majority of the excess deaths were from violence — you screwed up your calculations.
March 21st, 2005 at 8:58 pm
Stay on these guys, baby.
March 21st, 2005 at 9:02 pm
You think that `corruption’ is evident because in the totals they omitted a cluster that would have shown much higher death rates?
March 21st, 2005 at 10:29 pm
Tim, What role might researcher bias be playing in the Lancet study?
-Steve
March 21st, 2005 at 10:42 pm
Tim Lambert,
“Falluja was excluded because it was an outlier. This is perfectly standard practice”
But the exclusion should have been total. It wasn’t, The summary in particular flips back in forth from including the Falluja data to excluding it and then back again. If the Falluja cluster is in fact an outlier then it has no place in study period at all.
Cluster sampling was a poor design choice expressly because it was likely to produce just such an outlier. They should have created criteria for excluding a cluster a priori but I see no evidence they did so. Deciding to selectively exclude evidence after it is measured because it doesn’t fit is indicative of bad design.
“Even excluding Falluja, the majority of the excess deaths were from violence — you screwed up your calculations.”
Including Falluja, the study measured a delta of +96 deaths (146 deaths post invasion - 46 deaths pre-invasion) Of those deaths, 73 (52 in Falluja + 21 elsewhere) or 76% were the result of violence.
Excluding Falluja, the study produced a delta of +44 (146 deaths post invasion - 52 deaths in Falluja - 46 deaths pre-invasion). Of those deaths 21 or 48% resulted from violence which means that 52% did not. Last time I looked 48% did not define “most.”
Based on the data in table 2, most of the excess deaths resulted from violence only when the Falluja cluster is counted.
March 21st, 2005 at 11:05 pm
“You think that `corruption’ is evident because in the totals they omitted a cluster that would have shown much higher death rates?”
No I think that corruption is evident because they selective include or exclude the cluster based solely on whether doing so makes the Coalition look worse.
They exclude the cluster when its extreme numbers would obviously invalidate the study but they include it in order to make the majority of excess deaths result from violence or to skew the age and sex ratios towards women and children.
The cluster should be in or out. The authors shouldn’t literally switch from including and excluding the Falluja from one sentence to the next.
Scientist shouldn’t pick and choose their data. That they did so is dishonest.
March 21st, 2005 at 11:17 pm
You need to adjust for the fact that the time periods before and after the war were unequal.
March 21st, 2005 at 11:17 pm
Shannon,
Steel yourself.
Your typos will be used to anti-intellectually attack your incredibly wise responses.
-Steve
March 22nd, 2005 at 1:25 am
“You need to adjust for the fact that the time periods before and after the war were unequal.”
You maybe right now that I look at it. Increasing the number of deaths pre-invasion from non-violence means will skew the percentage of excess deaths due to violence post-invasion upward. I will rerun my numbers tomorrow and correct the post if necessary.
But
That still does not explain the misstating of the gender and age ratio based on the Falluja cluster, nor does it explain the statement that “after the invasion violence was the primary cause of death.” When the study clearly shows that, absent the Falluja cluster, the major cause of death is still illness and accident.
This is supposed to be scientific paper in a major publication. It is supposed to be clear and concise not riven with spin.
March 22nd, 2005 at 2:14 am
God this is weak. I would advise anyone following Shannon’s links to “Cluster Sampling, explained” to read the comments thread in detail, where it is explained that Shannon’s past critique is entirely incorrect. This post, however, is not just incorrect but actually dishonest.
The fact that a Falluja cluster even got studied raises red flags. The studies authors bemoan the lack of security in the nation and explain that it caused them move the original site of other clusters yet they managed to get one cluster in downtown bandit central, in the one city in the entire country that had absolutely no Coalition presence at the time the study was conducted. Riiiiiiiiiight, I’ll by that
This is stupid. The clusters were chosen at random. There was no “moving of other clusters” and a simple reading of page 2 of the study makes it clear that this is the case. In fact, the Lancet team also took a cluster from Sadr City, the other main area of high violence, and as it happened found zero violent deaths there. This accusation of dishonesty in the selection of clusters is, in my non-professional opinion, very seriously defamatory.
The Lancet study does a good job of summarising the following facts:
1) In the Kurdish North, the death rate fell
2) In most of the rest of Iraq, the death rate rose by somewhat more than 50%, the average being pulled down by the Kurdish governorates
3) There were also isolated areas like Fallujah in which the death rate increased by much more than 50%, these additional deaths being entirely the result of violence.
The table is presented so that anyone who wants to can see exactly what happened, and it is downright stupid as well as dishonest to claim that the decision to give figures with and without the Fallujah cluster, saying at every stage that this is what was being done, constitutes “whoring out of science”. An example of “whoring out of science” might be someone pretending that cluster sampling tended to overestimate rare effects when they knew it didn’t. This is what Shannon Love did, and that makes Shannon Love, by the standard of this post, a liar.
The bottom line here is that when bombs are dropped in clusters, they tend to land in clusters, and therefore it is stupid to entirely exclude high-violence “outliers”, even if it is also wrong to incldue them in the computation of summary statistics. Anyone with an ounce of sense, or with any experience at all in practical work, can see that the sensible thing to do is to report all the numbers so that an intelligent reader can see the underlying facts; an increase in the baseline death rate of around 50%, plus the deaths in the high-violence clusters.
March 22nd, 2005 at 5:00 am
“Cluster sampling was a poor design choice expressly because it was likely to produce just such an outlier.”
Leaving aside the fact that this is simply wrong (cluster sampling is likely to miss the small neighbourhoods where the carnage is most heavily concentrated), what design would you have chosen in the difficult circumstances faced by the researchers?
March 22nd, 2005 at 6:51 am
Kevin. I don’t know if you are asking a rhetorical question, so please pardon the response with that in mind.
If the survey can’t be done other than in a flawed fashion because of security concerns, then it ought not be done at all, or its results listed as something other than the Lancet did.
Or, to approach the question from another direction, that the study was difficult does not retroactively make flawed numbers valid.
Without going into the methodology, averaging one of the worst battlefields into the mix as if it represents some kind of standard is suspect.
Outliers ought to be “out”, described and studied as single events.
March 22nd, 2005 at 7:36 am
I don’t fully buy Shannon’s analysis but she does raise several good points that critics like DSquared simply ignore in their haste to call her a “liar.”
DSquared claims that it is “stupid as well as dishonest” to claim that selective inclusions of the Fallujah data were misleading. I disagree and I also disagree that the authors were “saying at every stage that this is what was being done.” The point isn’t that they shouldn’t have analyzed the data both ways. The point is that they often began their interpretation by presenting results *without* the Fallujah data and then slid immediately into conclusions that *did* include this data without mentioning this shift. By their interpretation of death rates, one could easily come to believe that the Americans were rattling about the countryside indiscriminately shooting women and children! Indeed, this is *exactly* how some opponents of the war characterize American behavior. And it is just dead wrong.
It is not “stupid as well as dishonest” to point this out and DSquared is obviously smart enough to know better.
The two biggest issues that I see with this study - one of which isn’t dealt with here at all - are (A) the reliance on circumspect data on pre-war death rates (not discussed here) and (B) reliance on data from subjective interviews of a population made up largely of people who wanted to see the Sunni Baathists returned to power. It is simply incomrehensible that anyone can take the Fallujah data seriously when there is zero control for alternative explanations of the data (such as political manipulation). Hell, I’ve had to run entire experiments over from scratch due to less serious data collection issues.
This is like interviewing the Third Reich in their bunkers in the days before the fall of Germany in WWII and asking them what they thought of the moral standing of the Allies. What the hell do you expect them to say?
We should all recognize that any attempt to topple a ruthless dictator will cost innocent lives. It has been true since the beginning of human civilization that one must employ violence to beat back those who would destroy that civilization. This is a very important debate: just how long a leash is a despot given before he must be removed, knowing that this will cause grievous suffering and innocent lives lost? It is an excruciatingly difficult question but one that a mature and responsible intellectual class should be able to debate. Unfortunately the Left appears only capable of posturing and posing. Apparently, in their view, there is *no* justification for war and any number of innocent lives lost discredits the entire enterprise. Of course, to convince the rest of us, they stoop to the obvious and clumsy distortions introduced in the Lancet’s conclusions to inflate the number. But, truly, what kind of “idealism” lets the thug and the tyrant flourish and the people be crushed underfoot? What kind of “peace” movement protests the horrible war while absolving the criminal gang of its responsibility for starting it? What kind of people refuse to soften their hearts at the prospects of millions of Iraqis choosing their own future and instead hails the Jihadist fanatics as “modern patriots and minutemen”?
Not the Left that I grew up with and not a Left that I’m willing to side with ever again.
March 22nd, 2005 at 7:46 am
I do not withdraw the epithets “stupid” or “dishonest” and will not do so while Shannon continues to pretend that the cluster sampling critique is valid. I also stand by my assessment of the summary and conclusions; it is perfectly obvious to any reader which conclusions are based on the ex-Fallujah results and which are based on the cum-Fallujah results.
reliance on data from subjective interviews of a population made up largely of people who wanted to see the Sunni Baathists returned to power
Two errors here. First, the population of Iraq was not and is not made up largely of people who wanted to see the Ba’ath party returned to power. Second, the data was not “from subjective interviews”; it was checked against death certificates on a sample basis (two certificates per cluster, except in those clusters which did not have two deaths). When asked, 81% of respondents were able to produce death certificates.
This is like interviewing the Third Reich in their bunkers in the days before the fall of Germany in WWII and asking them what they thought of the moral standing of the Allies
No it isn’t. It’s like interviewing the Third Reich and asking them if a member of their household had died in the last eighteen months. There were simply no questions about “moral standing” asked in the Lancet survey, and I respectfully request that you read the damn thing before continuing to comment on it.
And finally, Richard Aubrey:
Outliers ought to be “out”, described and studied as single events.
It is not at all obvious that Fallujah was a “single event”; it stands proxy for Sadr City, Mosul, Samarra and Najaf, none of which were sampled.
March 22nd, 2005 at 7:47 am
(erratum: of course Sadr City was sampled, however by sheer fluke the survey picked a cluster which had seen zero deaths despite being in the middle of the fighting, a fine example of the tendency of cluster sampling to underestimate the frequency of rare events).
March 22nd, 2005 at 7:51 am
Dsquared said:
“The bottom line here is that when bombs are dropped in clusters, they tend to land in clusters, and therefore it is stupid to entirely exclude high-violence “outliers”, even if it is also wrong to incldue them in the computation of summary statistics.”
I don’t understand this argument. OFI and the aftermath have seen the most concentrated use of smart weapons in history. This has the effect of greatly reducing collateral damage - innocent civilian deaths. Given the weaponry, I become suspicious of claims of large numbers of innocent civilians being killing by American bombing. It doesn’t mean that it isn’t happening but if it is there could be other reasons. For example, if the terrorist-insurgents are in a mosque and have taken a group of hostage/human shields with them then a bomb that takes out the terrorists will probably take out the hostages.
If Dsquared is claiming that the U.S. engages in indescriminate bombing - that’s what I infer from his statement - then I’d say he’s wrong. Of course, in a fog-of-war situation in which CAS missions are carried out in an urban environment, as was the case in Falluja, mistakes will be made. Still innocent civilian casualties will be lower for urban combat than ever before in history.
What is really significant about the battle against the terrorist-insurgents is that it has shown that the old cannards about urban combat are wrong. Unlike the Russians, who leveled Grozny and really did kill innocent civilians by the bushel, U.S. urban warfare doctrine has kept civilian casualties very low.
I think the Lancet study needed to be more discriminating in determining exactly what it was trying to show. A more interesting study would be to compare civilian casualties in historical urban combat situtations with current U.S. doctrine.
March 22nd, 2005 at 7:56 am
“The bottom line here is that when bombs are dropped in clusters, they tend to land in clusters”
9 out of 10 bombs dropped in Iraq are laser guided, precision weapons. Artillery shells, main battle tank shells and mortar shells are also precision weapons (especially if you’re to avoid “blue on blue”).
All the talk about the statistical study will never explain is why there are no mass graves digged today, why Iraqis voted “en masse” (pardon the french), why “resistance” has not picked up (if it were a massacre, it should have) etc etc…
March 22nd, 2005 at 8:03 am
By the way, if the body count ratio is 1 to 10 in favor of the US Army it means that 15.000 terrorists/thugs were killed in combat/bombing/etc. (US casualties: 1500 => 15.000 terrorists. Excluding accidental death and IEDs, maybe it’s only 10.000 terrorists killed)
Of course they’re all “civilians” because none wore a uniform.
March 22nd, 2005 at 8:27 am
Richard Aubrey,
I am not asking a rhetorical question. If critics of the Lancet study consider cluster sampling inappropriate then it is hardly unreasonable to ask just what methodology they favour. Your only response here is:
“If the survey can’t be done other than in a flawed fashion because of security concerns, then it ought not be done at all, or its results listed as something other than the Lancet did.”
I don’t know what you mean by listing of results in this context. On my reading, the main purpose of the Lancet study was to determine whether Iraq’s mortality rate was raised or lowered as a result of the forcible regime-change. The result is unequivocal.
Of course security concerns meant that the study would be flawed, which for you means it should not have been done. Perhaps you would argue that studies which attempted to quantify the loss of life in the Rwanda genocide or the war in DR Congo should not have been done. I don’t agree and you present no reason why I should.
March 22nd, 2005 at 9:21 am
It’s tommorow, where’s the correction RE: your inability to correct for time periods?
March 22nd, 2005 at 9:33 am
Kevin, you clearly (accidentally, of course) missed my caveat. If one is going to present a study whose results could be flawed by the inability (due to security or any other reason) to do a first-class job of sampling, then the results should be presented as at the very least, coming from a flawed sampling process and thus should be taken as substantially different from other studies whose credibility is not so damaged.
You implication that I would oppose other studies is absurd, resulting from a non-accidental misreading of my statement and adds a retroactive flavor to your objections.
March 22nd, 2005 at 9:36 am
Dsquared:
As to Fallujah standing proxy….
I’m tempted to say “prove it”, but of course you can prove Fallujah is used to stand proxy. My point is the implication, supported by little, that there is any useful or significant similarity between Fallujah and the other cities you mention. IMO, that needs to be demonstrated, not asserted.
March 22nd, 2005 at 9:40 am
I am learning a lot just by reading the back and forth here. I hope all involved can keep up the informed debate and limit the insults. This kind of fact-checking is what blogging is all about.
March 22nd, 2005 at 10:01 am
Shannon, You’re trying to herd cats!
Researcher bias kills the Lancet report in its crib, as far as this American is concerned.
Debating the minutiae of its sampling errors, use of clusters and selective in/exclusion of data only dignifies its promulgators with your valuable reason, time and verbage.
Good luck!
-Steve
March 22nd, 2005 at 10:09 am
Can anyone tell me if the Lancet ever published a response to the criticism leveled at this report? If so, what was the gist of it?
March 22nd, 2005 at 10:19 am
It’s tommorow, where’s the correction RE: your inability to correct for time periods?
C’mon, guys, this is a hobby. If you want to pay us a few thousand per post then we’ll whip up a schedule and stick to it. Otherwise you’re just going to have to wait.
James
March 22nd, 2005 at 10:26 am
Richard,
I am not deliberately misreading you, if I am accidentally misreading you then I respectfully submit that this may be the fault of your writing rather than my reading. A straightforward reading of your earlier comment is that the study should not have been carried out in view of the obstacles presented by the security situation. It now appears that is not your view; so much the better.
When you now say “the results should be presented as at the very least, coming from a flawed sampling process” do you actually mean to imply that the study does not alert the reader to the flaws? For my part, having read the study I am quite satisfied that it treats the reader fairly in that regard.
My question to Shannon Love is a simple one: if cluster sampling is the wrong approach, what is the right one?
March 22nd, 2005 at 10:45 am
If cluster sampling was an appropriate way to measure excess deaths, then why not publish the result that your sampling technique actually gave you?
It’s extremely tricky logic to argue that a study was flawed for unknown and uncontrolled-for reasons, but that exclusion of a single data point has corrected for these errors, although they remain unknown and uncontrolled-for. It’s much more likely that you consciously or subconsciously massaged the data until a ludicrous result entered the high side of believability.
It’s extremely tricky to exclude data after you’ve collected it. Unless you have a very good idea of why (and how) the data isn’t measuring what you wanted to measure, the only real answer to outliers is to conduct a better study and publish that.
March 22nd, 2005 at 10:47 am
Shannon Love and Andy S take swipes at the Lancet study
Andy S, last seen criticizing the Lancet study without
reading it, has
March 22nd, 2005 at 10:52 am
There are 4 very serious problems with this study:
1) The study says nothing useful. The only meaningful (i.e. supported by a 95% confidence level) statement that can be made based on the data is that between 8,000 and 194,000 people were probably killed.
2) The pre-war death rate the study uses seems unusually low, lower than Iran’s or the EUs. Iran has a similar population age with a standard of living 3 times higher. When a more reasonable pre-war death rate is used, the already hard-to-detect effect disappears completely, or suggests the war saved lives.
3) Published reports say the authors only agreed to do the study on the condition it would be published before the U.S. elections. Political motivations lead to bad science.
4) The authors failed to survey some of the heavily Kurd and Shia provinces, which benefitted the most from the war. This is a very serious flaw which would tend to produce misleading results, and one suspects this is not unrelated to point 3.
March 22nd, 2005 at 10:59 am
When I was in the Army, I had the disagreeable duty of notifying next of kin on several occasions. I kept up with two of the families (one mine) and discovered that both parents of one guy killed had heart attacks within a year of his death, and in another, the mother had one, but not the father.
In a country with substandard health care, these might have been deaths.
My question regarding the Lancet’s inclusion of non-combat deaths is whether this has ever been done before? Did anybody ever count the number of parents who died of grief after Pearl Harbor?
Of course not. How about those who died because so many physicians were in the military during WW II that homefront health care declined? Of course not.
This inclusion might seem scholarly, but it reeks of an attempt to punch up numbers which, without it, could be anticipated to be too small to be useful.
March 22nd, 2005 at 11:00 am
Zach, everyone knows that outliers cause headaches for researchers. But doing a whole new survey isn’t a sensible response. There is nothing shameful about reporting the results with and without the outlier and letting readers draw their own conclusions.
The answer to your “why not” question is, they did publish the result their sampling gave them. Don’t take my word for it; read the study.
March 22nd, 2005 at 11:22 am
For me, the real significance of the Falluja results is not merely that they were absurd on the surface - and I feel safe in saying that, considering that they pointed to hundreds of thousands of death in a city where the rebels themselves had claimed only hundreds of innocent deaths. It’s also that there really was no more reason to exclude those results than any of the results. Methodology-wise, that is. They looked ridiculous so - heck, must be an outlier. But methodology-wise, the other results were arrived at the same way. That is, the absurdity of the Falluja results implies very strongly that the other results might be just as absurd.
I aslo agree with someone up above that the “excess deaths” ratio depends on accepting their calculation of pre-war deaths. I’ve heard a lot of people say - what, wasn’t Hussein killing a lot of people? Where were those violent deaths? But that’s not the point for me; it may well be true that Hussein was not actually killing thousands of people a year before the war. The point, from what I’ve read, is that numbers from pre-war Iraq were incredibly unreliable; and that the death rates reported compare favorably to the USA, in a land that was still suffering under sanctions and a dead economy.
March 22nd, 2005 at 11:31 am
J. Bennett asks:
“Can anyone tell me if the Lancet ever published a response to the criticism leveled at this report?”
The lead author has responded to e-mails requesting clarification of some points. If you click on Tim Lambert’s name in the comments above, you will find that he has examined the criticisms in some detail; he also provides numerous links.
You should note that much of the criticism has consisted of name-calling which no sensible person would bother responding to. Also, many of the critics haven’t read the study and/or are are innumerates posting gibberish about sampling, confidence intervals etc. which they have picked up from other innumerates.
March 22nd, 2005 at 11:34 am
Although I have read some persuasive arguments about why this study is correct, and I accept them, I have yet to see a persuasive argument about why the lead researcher on this project retroactively lowered the pre-war mortality rates, after having argued for years that they were incredibly high during the sanctions of the 1990s.
The methodology of this study may be perfect, but if the basic premise (that prewar rates were as low or lower than Iraq’s neighbours) is wrong, then no amount of scientific rigour *during* the survey will save it.
Anyone care to address this?
March 22nd, 2005 at 11:47 am
I love how Kevin, with no hint of irony, complains about namecalling and then does it himself. I guess as a sensible person I shouldn’t respond to it.
March 22nd, 2005 at 11:51 am
Philip, this issue has been addressed; I can’t go into all the relevant points, but the main point to note is that the survey examines both pre-war and post-war mortality. Even if the pre-war figure is biased downwards, which is certainly possible, that does not affect the results if there is a similar bias in the post-war figures. That is part of the reason for designing the study in this way.
March 22nd, 2005 at 11:57 am
While I’m taking the risk Kevin will call me names again, on the results I’d just like to point out that had they found a 95% confidence interval of say, 50,000 to 150,000, then we could say with 95% confidence that at least 50,000 people were killed. That would be both meaningful and useful to their cause. Saying at least 8,000 were killed isn’t all that useful to their cause, which is why they gave the point estimate to which no confidence level is or can be attached.
I think the pro-war folks can take great comfort in the fact that even a politically biased study using terrible methodology that oversampled Sunnis and used a an incorrect pre-war death rate, they still only proved 8,000 people were killed.
March 22nd, 2005 at 11:58 am