A Challenge to Lancet Defenders
Posted by Shannon Love on March 24th, 2005 (All posts by Shannon Love)
A lot of people have taken me to task for calling the Lancet Iraqi Mortality Survey [PDF] an example of scientific corruption. I still stand by this claim.
Many seem to equate scientific corruption with falsification of data but there are many ways to create a false impression even if the underlying data is sound. (I will expand on this in a subsequent post).
One easily graspable example in the Lancet study’s dishonesty is the key sentence in the Summary, the one repeated in the media world wide, that pegs the “conservative” estimate at 100,000 excess deaths. The actual given estimate is 98,000. What pure scientific purpose is served by rounding the number up to 100,000? There is no technical reason for doing so. They chose that number because a big, round numbers stick in people’s minds. Its a number chosen only for its marketing value.
More damning is the utter practical uselessness of the study’s findings. The cover-story for the study is that it is a medical epidemology study intended to provide decision makers with information they can use to reduce the mortality rate in Iraq from all causes.
When one looks at the study as actually published, however, it provides no solid information on which a decision maker could act in Oct 2004 or later. Indeed, the study obscures such data as the age, gender, combat status and means of death of those reported killed. It doesn’t report any kind of detailed time series that would let decision makers determine whether the people reported killed died in the major combat phase or not and it produces widely different scales and causes of death dependent on whether the outlier Falluja cluster is included or not.
But I could be wrong, so let me issue this challenge:
Can anybody point out information contained in the study, as published in Oct 2004, that would let a real world decision maker make changes to policy, strategy or tactics that would have or will save lives in Iraq?
Please be as specific as possible.
I think the answer is “No.” I think this proves this study was designed, conducted, written up and published purely for its hoped for political impact in America and the rest of the Western world. Trying to save Iraqi lives was never a major consideration.
To me that is scientifically dishonest and represents corruption of our scientific institutions.
(Update:) let me rephrase the question slightly
Given that: (1) The study is universally recognized as being carried out under very difficult conditions. (2) Difficult conditions seriously raised the risk of significant error in the study. (3) The consequences of acting on erroneous information in a war zone could be catastrophic.
What data provided by the study was so important that it was worth the risk of conducting the study under adverse conditions and getting an widely inaccurate result? What data had to be provided in Oct 2004. What data couldn’t wait?
March 24th, 2005 at 11:57 am
Oh Shannon!
1. You’re repeating a mistake I’ve already pointed out to you:
One easily graspable example in the Lancet study is the key sentence in the Summary reproduced world wide that pegs the “conservative” estimate at 100,000 excess deaths. The actual estimate is 98,000. What pure scientific purpose is served by rounding the number up 100,000? Of course, there is none. They chose that number because big round numbers stick in people’s minds. Its a number chosen only for its marketing value.
No. The number 100,000 is chosen because the figure of 98,000 deaths refers to “the 97% of Iraq represented by all the clusters except Falluja” (page 5). 98,000 divided by 0.97 is 101030, which rounds down to 100,000. Yet another correction of a calculation error needed, I’m afraid.
2. My criticism of you in particular has been that you have made, and continue to make, methodological criticisms which are invalid, and that you are not prepared to either defend them or stop making them. In particular, your “cluster sampling critique” from your original “Bogus Lancet Study” post. I’ve asked you three times now whether you are going to stop claiming that “a cluster sample has a very high chance of exaggerating the number of deaths” and I think I deserve an answer.
3. The answer to your sophomoric “challenge” is simple. The simple information that “The postinvasion management of Iraq is going badly wrong and currently there are more people dying than there were when Saddam was in power” is valuable to a decision maker, because it gives him the information that there is a serious problem to be dealt with. Such a decision maker could then commission more information and start to make a plan. The value of such “alarm bell” studies is, frankly, well known in epidemiological studies; Medecins Sans Frontieres have produced many excellent ones, including a recent one which estimated mortality in Western Darfur.
4. The charge of your posts never changes; it is always that the Lancet team were “scientifically dishonest” and “represent corruption of our scientific institutions”. This is the fourth time you’ve made this charge. But the grounds always change. First it was that cluster sampling was “bogus”. Then it was that the summary was confusing to you. Then it was that they had reported an outlier in a way that we later established was the correct way to do so. Now it’s because the study, in your unsupported opinion “wouldn’t be useful”. You’ve been wrong every time, as you changed your story, but the one constant has been that you are angry that a study was published which had the effect of making George W Bush’s policy in Iraq look like a failure. Can I be forgiven for suspecting that this is actually what you are angry about?
The Lancet team are a credit to science; you, on the other hand, are fast giving political “sound science” hackery a bad name.
March 24th, 2005 at 12:04 pm
Yes,
The results of the “study” could be used to elect a weasily “surrender monkey” to the American Presidency that would cut our military budget and tie our hands with a “global test”.
Once global Dhimmitude was complete and the caliphate extended beyond the New Jersey Mosques and Toronto Ghettos, the “White Satan” would no longer be able to kill innocent “minutemen” indiscrimately without suffering “collateral damage” in its restaurants, shopping malls and unholy gambling centers.
The Baath party, newly restored with Soros’ funding, would then return to its holy, humanitarian subsidization of baby formula for all of Saddam’s needy children with the help of Kojo Annan, who the caliphate will elevate to General Secretary of the U.N. for his “undying supplication to the justice and love of global Islam.”
Any “regrettable dissappearances of “jewish-pig sympathizers” into “accidental excavations” in the desert would pale in comparison to the awful killing by Bush-Hitler and the neocons so clearly demonstrated by the Lancet study.
Saddam’s media handlers would be sure to explain to Eason Jordan’’s “journalists” that the new sand mounds are newly discovered Pre-Iron Age ruins deserving demarcation on a map of tourist sites in the “Greater Arabic” region of the global caliphate.
This is how the Lancet study could “make changes to policy, strategy or tactics that would have saved lives in Iraq.”
I’m surprised you couldn’t see it, Shannon.
-Steve
March 24th, 2005 at 12:13 pm
Minor comment on dsquared’s 11:57am #4:
> Then it was that they had reported an outlier in a way that we later established was the correct way to do so. [bolds added]
If the bolded phrase means “dsquared & his allies have claimed,” then all is well. If the missing antecedent to “we” is to be taken as “dsquared and other ChicagoBoyz commenters,” then this recent entry and comment thread stand as a rebuttal.
March 24th, 2005 at 1:10 pm
I’ve replied in that thread. I am not aware of anyone having made any specific complaint about the handling of that outlier other than ones which proved unsustainable.
March 24th, 2005 at 1:18 pm
ahhh, now we’re on to the fifth grounds for accusing the authors of scientific dishonesty!! Productivity or what? A pedant would say that you would be better off correcting your serious calculation error, but hey, it’s Thursday!
What data provided by the study was so important that it was worth the risk of conducting the study under adverse conditions and getting an widely inaccurate result? What data had to be provided in Oct 2004. What data couldn’t wait?
The data that couldn’t wait was that there was a prima facie case that “not doing body counts” was resulting in the coalition missing a potentially very serious problem. That’s why the authors recommended, at length and with (IMO erroneous) references to the Geneva Convention, that the coalition should start “doing body counts” immediately.
If you find that 100,000 people have died unnecessarily in 18 months, then you can assume that 5,555 people are dying unnecessarily every month. Very few things “can wait” when the cost of waiting is so high. The US elections do not constitute a reason to delay.
And in any case, this is pure ad hominem. The study could have been carried out by John Kerry and Howard Dean, but it would not have changed a single standard deviation in the data. Shannon appears to have given up the losing battle on the science, and switched to pure and simple attacks on the authors. It really takes the biscuit to look at evidence that 100,000 people have died and say “well obviously this has to be delayed or else it might effect the elections!”
March 24th, 2005 at 1:43 pm
dsqaured,
Thank you for demonstrating my main point so elegantly! (see#3)
1) So they either rounded up from 98,000(2,000) or down from 101,030(-1,030). This in no way changes the validity of my criticism. I am pointing out that they rounded to the more marketable number of 100,000 instead of providing the actual number, whatever that was. There was no technical reason to round at all.
More over, if they did calculate the figure as you suggest the researchers were incredibly sloppy. Remember this is a cluster study. Each cluster represents a unique area of Iraq calculated by the demographic software. If a cluster is removed, the remaining clusters must be recalculated to stretch over the area of the missing cluster.
I suppose it is possible that they recalculated everything and ended up at exactly 100,000 but what are the odds of that?
2) I am working on a post explaining the Cluster sampling critique. This time I’m going to use pictures! In any case, my most recent post have not concerned methodology i.e. how the data was collected and processed, but just presentation. Honest communication of results is just as important to the integrity of scientific institution as methodology. So I haven’t pounced on any issues of methodology you have raised.
3) “The post-invasion management of Iraq is going badly wrong…”
Honestly, was there any possibility that the study would return a finding that you personally would not use to confirm your preexisting bias that that the post-invasion was mismanaged? Again, what specific information supplied by the study did you have to get to reach your conclusion. If the study had not been done, how would your current view differ?
More to the point, “the post-invasion management of Iraq is going badly wrong” compared to what other real world event? How does this study tell us what could have been done better and how?
“there are more people dying than there were when Saddam was in power”
Again, you needed the study to tell you this? Everybody who has studied armed conflict at all knows that you will see a spike in deaths. Did we need the study to tell us this.
Frankly, this is just the kind of vague response I expected. You provide no concrete example how this study might be used in a real world fashion. This study provided you personal with no new information and changed your assessment not at all. Rather, if confirmed your prejudices and gave you a rhetorical stick to beat people up with. Which I contend was the gaol of the study in the first place. This study was always about politics. Its corrupt.
4) “But the grounds always change”
No, I keep adding new grounds. The base methodology was risky. The on the ground conditions made the study prone to error and subversion. The presentation was deceitful. Now, I have learned that key data was left out of the published paper. I’m sure something else will turn up tomorrow.
I am not shifting the grounds of my critique. Instead the reasons to question the study keep piling up.
Again, thanks a lot for proving my point. The study has no practical utility.
March 24th, 2005 at 2:10 pm
dsqaured,
So the practical utility that you think the study proves is the need to do body counts?
Given that in the “conservative estimate” most people died of illness, accident or non-Coalition violence, how would body counts help? The remainder of he deaths occurred from airstrikes the results of which are already accessed as standard protocol.
Moreover, what evidence does the study present that the military is not already taking the maximum amount care humanly possible to prevent the deaths of non-combatants. Given that the conservative estimate says most deaths from violence are due to non-Coalition actors couldn’t making changes to Coalition tactics that make those non-Coalition actors harder to neutralize actual cost lives in the long run.
The fact that the researchers chose to conduct the study under difficult conditions (which would degrade their data) and rush it in to print, even though the study produced no time-dependent results, strongly supports the idea that they did created the study for purely political reasons. They never expected to have any other positive effects.
March 24th, 2005 at 3:07 pm
So they either rounded up from 98,000(2,000) or down from 101,030(-1,030). This in no way changes the validity of my criticism. I am pointing out that they rounded to the more marketable number of 100,000 instead of providing the actual number
Has anyone ever read any criticism more idiotic than this one?
Although it is a perfect match for such an imbecile challenge. According to Love the Lancet study is good or bad depending on whether politicians would implement any changes based on its conclusion. Accordingly, Dr. Love will soon propose that quantuum mechanics is wrong beyond repair, likely a product of some corrupt physicists.
Unfuckingbelievable.
March 24th, 2005 at 3:22 pm
Palo,
Loose analogies and obscenities say more about your position than you might wish.
S. Love,
I am untroubled by Roberts’ use of “100,000″ instead of 98,000, given the 95% confidence interval (8,000-194,000). This estimate has zero significant figures (E5), or one significant figure (100,000) at most.
March 24th, 2005 at 3:25 pm
Everybody who has studied armed conflict at all knows that you will see a spike in deaths
On the other hand, anyone who has studied a chart of deaths in Iraq over the eighteen months between the invasion and September 2004 will know that there was no such “spike”, in the ex-Fallujah clusters, just a gradual increase in the death rate which got worse toward the middle of 2004. Luckily, there is exactly such a chart on page 5 of the Lancet report; perhaps you have forgotten as it must be at least an hour since you last read it looking for a new argument.
March 24th, 2005 at 3:28 pm
By the way, here is a source of a picture that you might want to include in this wonderful forthcoming “cluster sampling post”.
March 24th, 2005 at 4:09 pm
I have 2 questions for the stats gurus, if anyone can help out here. Since the 95% confidence interval refers to the 8,000-194,000 band, what confidence could be assigned to a somewhat narrower band surrounding 100,000, like 90-110k?
Also, what would the inclusion of Falluja data do to the confidence interval? Any ideas?
March 24th, 2005 at 4:35 pm
Telluride,
See dsquared’s answer to your (and my) including-Fallujah confidence interval question in the comments to the “Fisking Falluja” post. It is one-third down the thread, at 3/23 2:53pm.
March 24th, 2005 at 4:46 pm
Okay, because you called me a statistical guru, I’ll answer your questions.
1.) If the estimate is 98,000 and you want a confidence interval from 88,000 to 108,000 the CI is going to be about 16%.
2.) Including Fallujah should make the point estimate (the 98,000 number) more extreme and the confidence interval broader.
March 24th, 2005 at 4:47 pm
That was an interesting response from dquared. It sounded very much like “I have no idea and I’m frankly scared to ask.” Of course we know SOMEWHAT what it would do to the confidence interval, since it would massively blow out the standard deviation of the results used to compute it.
But if we’re treating Falluja and ex-Falluja as completely distinct sets, why are they being conflated again and again? If “there is no very good way to make general statements about percentile confidence levels of this sort of distribution,” why is the study useful?
March 24th, 2005 at 4:52 pm
If the estimate is 98,000 and you want a confidence interval from 88,000 to 108,000 the CI is going to be about 16%.
Right, so referring to 100,000 as the “most likely” value means next to nothing. Just checking.
March 24th, 2005 at 5:16 pm
Telluride said:
“Right, so referring to 100,000 as the “most likely” value means next to nothing. Just checking.”
Well this stuff complicated and cannot be boiled down to one simple thing. It’s a rough estimate with a range. And that’s informative. I’ll quote one of the study authors, Dr. Gilbert Burnham, from his interview in The New Republic one more time below:
“Now, you can argue, is this increased mortality rate 70,000? Is it 60,000, is it 150,000, is it 200,000? Our best guess, on a conservative side, is 100,000. But it could be less and it could be more. Because just by the statistical nature of this thing, the kind of zone around this number where we are sure this answer truly lies is fairly broad. It’s a national survey, it’s a massive survey, but it’s not a national census.”
It is what it is. Not perfect and also not useless.
March 24th, 2005 at 5:23 pm
Well so far my challenge isn’t netting much input. dsqaured seems to think that the study has practical utility because body counting will miraculously reduce non-combantant deaths, at least, I think that is what he is arguing but nobody else is biting.
Anybody else, please?
March 24th, 2005 at 5:28 pm
Amac,
“I am untroubled by Roberts’ use of “100,000″ instead of 98,000, given the 95% confidence interval (8,000-194,000). This estimate has zero significant figures (E5), or one significant figure (100,000) at most.”
Yes and if they had not made sweeping claims of mass murder and had not spun like used car salesmen in the presentation of the paper (including apparently hiding data) I would give them a pass but since it appears that the entire paper and the publicity effort around are anchored on the 100,000 claim I think I will choose to call them to account.
March 24th, 2005 at 5:44 pm
Well, I don’t think that you can expect the scientists to come up with perfect policy answers. Their thing is to collect and analyze data, which they did.
And that’s really important. I mean how can you possibly know that there aren’t many civilian deaths when you are not even attempting to count them? The main conclusion of this paper is that there is a need for more research into what’s going on, research that may answer Shannon’s additional questions about person, place and time.
This preliminary data make it appear that there’s a good chance that more civilians are being killed than our government assumed were being killed. I really don’t know what number of civilian deaths the administration deams acceptable (for all I know it could be way, way higher than the 100,000 number.) But if it turns out that your weapons and tactics are killing more civilians than is acceptable then you’ve got to reevaluate your tactics.
Do you think America is fighting the perfect war and that there’s no room for improvement? In everything, whether it be business, technology, warfare, personal fitness goals, etc. there’s room for improvement.
March 24th, 2005 at 5:49 pm
dsqaured seems to think that the study has practical utility because body counting will miraculously reduce non-combantant deaths
This is, of course, untrue, though I doubt any neutral observers were fooled. If Shannon decides to declare victory on this account he/she is a bigger fool than I thought, which is quite some fool.
Telluride: I can’t at this late hour be bothered calculating the 90-110K confidence interval but the Excel spreadsheet function NORMSDIST will do it for you. The cum-Fallujah confidence interval would give a negative number of excess deaths if calculated crudely using a normal distribution, but this is clearly an indication that the calculation of the CI is wrong; taking a distribution with a positive 95% confidence level and adding an observation which is much higher shouldn’t make you think that the true value is lower (for example, would you say that the 1987 crash is a good reason to believe that the London market often goes up by 20% in a day?). To be honest, confidence intervals aren’t the be-all and end-all; sometimes you just have to accept that some things can’t be summarised with a single number.
By the way, can we get the charge clear; Shannon is still saying (falsely) that the authors made “sweeping claims of mass murder”, but appears to have dropped the accusation that they did so specifically in order to provide propaganda for Iraqi fascists? I only ask for the benefit of the libel lawyers who I still hope will take an interest in this series.
March 24th, 2005 at 5:55 pm
Right, so referring to 100,000 as the “most likely” value means next to nothing
No, it means that, ex-Fallujah and speaking loosely, the true value is equally likely to be higher than 100K or lower than 100K. It is 58% likely to be higher than 90k, etc etc up to the result that it is 97.5% likely to be more than 8000 and extremely unlikely to be less than zero.
With Fallujah, the distribution is bimodal (a fifty cent word meaning that there are two peaks; one associated with the middle of the ex-Fallujah distribution and one wth the Fallujah distribution, rather like a distribution of heights has a male peak and a female peak), so the confidence interval is difficult to calculate, but you can tell that, because you’re adding more probability mass on the higher numbers, the 50/50 figure is going to be higher than 100K.
March 24th, 2005 at 6:03 pm
” taking a distribution with a positive 95% confidence level and adding an observation which is much higher” results in a statistically meaningless amalgam, as you know quite well. You cannot selectively introduce or allude to the falluja cluster while ignoring its likely effects on the confidence interval.
March 24th, 2005 at 6:12 pm
With Fallujah, the distribution is bimodal
from one outlier you conclude the distribution is bimodal? The authors do not claim it is bimodal, or asymmetrically leptokurtotic, or heavily skewed. i would think this assumption would make the confidence intervals considerably more complicated to calculate.
March 24th, 2005 at 6:14 pm
Give over Telluride. If you have 32 observations which are reasonably normally distributed between 30 and 50, then one observation of 200, then does that make you think that the true value might be minus 20? I don’t want to start appealing to authority, but “common sense” points in the same direction as the underlying mathematics in this case.
March 24th, 2005 at 6:17 pm
from one outlier you conclude the distribution is bimodal?
No, that’s a modelling assumption made by me, not a conclusion. The underlying DGP might be a fat-tailed distribution, or it might be a genuine low-p event, or whatever. But given the non-data information that we have that the majority of Iraq saw a small rise in deaths, but some part of Iraq saw very heavy violence, I think the bimodal assumption is defensible, not that anything in the Lancet paper depends on anything like it.
March 24th, 2005 at 6:24 pm
“I don’t want to start appealing to authority, but “common sense” points in the same direction as the underlying mathematics in this case.”
Well here’s an example that might appeal to your authority. An option market with heavily skewed outlier calls says one thing about the underlying market: it is more likely to go down than up. Crazy, huh? Counterintuitive but true.
The Lancet editors are not claiming there is a bimodal or skewed or leptokurtotic distribution. They (and you) are selectively adding and subtracting the outlier value to the ex-outlier confidence interval based on a normal symmetrical distribution, resulting in a conceptual mishmash.
March 24th, 2005 at 6:28 pm
Is there some reason to restrict the discussion to lessons which relate to “policy, strategy or tactics that would have or will save lives in Iraq?”
Is it not of some importance to avoid making horrible mistakes in Iran, Syria and other countries where regime change may be contemplated?
Frankly, if you really can’t see why there is merit in studies which seek to quantify the consequences of waging wars of choice, it is hard to figure out how to communicate with you.
I hope for your sake that you are just changing the subject out of (understandable) embarassment at your failure to find some significant flaw in the Lancet study. To bite off more than you can chew in a technical debate merely makes you a hothead. To really believe that a matter of life and death isn’t worth study is something else. Can you really not see why it matters?
March 24th, 2005 at 6:40 pm
BTW, assuming either an asymmetrically leptokurtotic or a positively skewed distribution shifts the mode to the left, and I think the mode is what is interesting in this instance, is it not? You’re “the authority” here so do fill us in.
March 24th, 2005 at 6:57 pm
Telluride, as I say, give over. You’re not responding to my main example; if you have a dataset and then add a large positive outlier, then there is no summary statistic of the central tendency of your data which moves downward. I don’t quite understand what you’re saying about options markets, but since there is no way to hedge death rates in an underlying market, I suspect that this is a disanalogy.
Clearing up, your charge:
The Lancet editors are not claiming there is a bimodal or skewed or leptokurtotic distribution. They (and you) are selectively adding and subtracting the outlier value to the ex-outlier confidence interval based on a normal symmetrical distribution, resulting in a conceptual mishmash.
is wrong. The study’s authors (not the Lancet editors, who are a different group of people) are not “selectively” doing anything. They are reporting the crude numbers including the outlier and the extrapolated numbers excluding it. This is a sensible way to report the data.
and
BTW, assuming either an asymmetrically leptokurtotic or a positively skewed distribution shifts the mode to the left, and I think the mode is what is interesting in this instance, is it not?
looks like logorrhea to me. Different types of distribution can have higher or lower skew and all manner of kurtosis and all manner of funny things can happen to the mode. For any statement about kurtosis and skew, I can come up with a distribution for which the opposite of what you say about the mode is true. However, it is equally obvious that if you have made a distributional assumption under which the addition of a large positive observation shifts your measure of the the central tendency downward, then you have made the wrong assumption.
As I say again; you have 32 observations between 30 and 50, then observation number 33 comes in at 200; would this make you say that the true distribution is centred around zero?
March 24th, 2005 at 6:59 pm
By the way, the mode is not what we are interested in in this instance - for any sensible loss function over deaths, we are interested in the expectation of the number of excess deaths, not the single most likely point estimate.
March 24th, 2005 at 7:14 pm
Kevin Donoghue,
“Is there some reason to restrict the discussion to lessons which relate to “policy, strategy or tactics that would have or will save lives in Iraq?” “
I am attempting to falsify my own contention that the study has no practical value beyond the crudely political i.e. giving political advantage to one party over the other. For that reason I requested that people try to focus on providing examples of practical i.e. non=political uses for the study.
“I hope for your sake that you are just changing the subject…”
This is a large subject with lots of point for debate. When I bring up a new point that doesn’t automatically mean that I have abandoned all others. By nature I tend to chop of arguments in to subcomponents when possible and try to test each sub-argument. Each blog post tends to address one particular sub-argument.
My original criticism of the study, that Cluster sampling was a poor methodology, has never changed. In addition, when rereading the study just recently the inconsistent use of the Falluja data leapt out at me so I did a post on that. I’m adding bricks to the structure of my argument not thrashing around.
I’m really serious about attempting to uncover some utility beyond the grossly political. If you can provide me with a real use I will back down on my contention that the study is merely a political ploy.
March 24th, 2005 at 7:16 pm
“For any statement about kurtosis and skew, I can come up with a distribution for which the opposite of what you say about the mode is true.”
without exception, for a positively skewed distribution, the mode is to the left of the mean. The more positively skewed is the distribution, the further leftward is the mode. The mode is the relevant measure since “likelihood” is exactly what is being expressed.
March 24th, 2005 at 7:26 pm
The question has been updated to ask “What data couldn’t wait?” (Maybe the update was there before my previous comment; if so, my bad.)
That gives me a bit more to think about but since it is late here I will just note briefly:
If we could be sure (a) that the US had no plans in the works for further adventures and (b) that any lessons from the study would be inapplicable to operations in Iraq, then off-hand I can see no reason to rush.
Knock out either or both of those assumptions and it is obvious that the findings, if heeded, might lead to better management of operations in Iraq and (especially) elsewhere, thereby saving lives. Simple example: it might abort an invasion of Iran, whose cost in excess deaths would surely be higher.
Good night.
March 24th, 2005 at 7:49 pm
“For that reason I requested that people try to focus on providing examples of practical i.e. non-political uses for the study… I’m really serious about attempting to uncover some utility beyond the grossly political. ”
I answered this on my comment March 24, 2005 05:44 PM. Sorry to be squeaky here, Shannon, but I thought I had a good answer to your question however I posted it during a flurry of activity here and it might have gotten lost.
March 24th, 2005 at 7:55 pm
Kevin Donoghue,
The points you raise are beyond the scope of this thread but I would point out that for people actually interested in scientific accuracy, a single study is only suggestive.
One of the disturbing things about the sociology of this study is the degree to which many have embraced its findings as revealed truth even though, just as any other scientific study, it must be verified through replication before we can confirm its accuracy. (Our arguments over methodology are so vociferous because we don’t have any other means of evaluating the accuracy of the study. Solid science ends arguments, it doesn’t start them.)
If we use a single, unverified study to direct our policy we are not actually basing our decisions on good science.
March 24th, 2005 at 7:56 pm
“we concluded that the civilian death toll was at least around 100,000 and probably higher”
-Garfield & Roberts, in The Independent.
“Probably higher” from the mouth of a statistician, sounds to me like a positive scientific assessment of probability. Either their assumption is a normal distribution without the falluja data, or a normal distribution WITH the falluja data, and the concomitant increase in variability. it is altogether conceivable that zero is encompassed by this vastly larger confidence interval.
March 24th, 2005 at 9:46 pm
Chel,
Your 3/24 5:44pm answer to S. Love’s question struck this reader as honest and frank. I supported OIF as the least-bad of a bad batch of alternative policies, and, yes, any policy modifications that reduce noncombatant casualties are worth striving for in the future. And, yes (as commented on in “Fisking Fallujah”), more research will be needed on the number of Iraqis who died directly and indirectly as a result of the war. And the accompanying number of those who lived as a result.
Telluride,
See Mike’s comments on author interviews near the end of the “Fisking Fallujah” thread.
March 25th, 2005 at 4:06 am
That would assume that the civilian deaths aren’t investigated under current policy and that no attempts to improve medical and sanitation are made. And that these issue aren’t considered for future planning.
The data doesn’t tell us where we can better focus our efforts beyond what we already do.
March 25th, 2005 at 4:11 am
To summarize:
ARTHUR: Look, you stupid bastard, you’ve got no arms left.
BLACK KNIGHT: Yes I have.
At least the Black Knight, unlike Shannon, was sufficiently honest only to try and pretend the outcome was a draw.
March 25th, 2005 at 4:48 am
I think I’ve got it. It doesn’t quite answer Shannon’s question of how the study can help Iraqis, but the study tells us that a small study conducted during a war doesn’t provide us information useful in making decisions during the war or numbers that are better than what we would get from studies that will be conducted later. It also raises the question that perhaps a larger study could be conducted safely and cost-effectively during a conflict to provide real time data for decision making.
March 25th, 2005 at 5:52 am
The reason why you want to count the casualties is explained here
March 25th, 2005 at 7:01 am
That doesn’t address the Lancet study. The value of having good casuallty data is appartent, the Lancet study doesn’t provide this.
Urgency, having this data now, is asserted, but not demonstrated.
The Lancet study doesn’t tell us anything we don’t already know.
It doesn’t seem to be in interest of the military to make these policy changes now. It may want to consider making changes in the future when they can be better evaluated and implemented, the lancet study doesn’t help in this.
Better data will be availible in the future and other parties will do the work. Why would the government do the work at great cost to itself when others will do it eventually? This sense of urgency is unfounded, it seem more like they’re lobbying the military to do the work they want done for them.
March 25th, 2005 at 7:06 am
To sum up, you are arguing that in addition to being an effective propaganda tool for our enemies, the study is a valuable political tool for pressing the military into making rushed policy decisions.
March 25th, 2005 at 8:22 am
Why thank you AMac!
March 25th, 2005 at 9:30 am
Tim Lambert,
Thank you for comment, however, the page you linked to does not answer the specific question of how this particular study was useful and why it was so useful (see update in parent) that it had to be done under difficult conditions that conceivably could lead to significant error.
More generally, the comments you link to comprise a mere unsupported argument from authority by some arbitrary clump of “health experts” asserting without proof or even example that body counts would help save lives. For example, do we have study comparing a conflict where there was a body count versus one where there was not? I don’t find such unsupported arguments from authority compelling.
I realize that it would seem common sense that body counts would save lives but nobody has demonstrated that the current tactics currently used by the military do not represent the best possible practices that result in the minimum number of causalities. Warfare is to fluid and each conflict to unique for the tools developed for epidemology to provide much help for those trying to minimize causalities.
I think my point still stands. This study has no utility beyond trying influence whether one politician gets elected over the other.
March 25th, 2005 at 10:01 am
Amac, thanks for the reference. My comment addresses the authors’ misstatement, the Lancet’s deceptive synopsis, and dquared’s persistent partial-use of falluja by way of “intuitive” appeals that are actually incoherent.
If the distribution is presumed to be normal, the enclosure of the outlier data set would certainly widen the confidence interval. Since they dare not represent that new [ridiculously wide] confidence interval, the authors/editors allude darkly to the excluded figure, appealing as dsquared does above to “common sense” instead of statistical rigor. This approach is incoherent: if the outlier is excluded, it adds no information to the probability or confidence interval of the “true” results. If it included (assuming a normal distribution) it should blow the confidence interval out beyond repair, and that new confidence interval needs to presented as well. Assuming a skewed distribution, accomodating falluja would raise the mean, but it may alter the mode and median not at all as dsquared acknowledges, and would also have counterintuitive implications for the position of the confidence interval [the mode representing the "most likely" individual value, not the raw average.]
To take dsquared’s example, adding 200 to his hypothetical distribution changes the likelihood of randomly selecting “41″ not very much at all. This is the figure people mean when they say “most likely” - NOT simply the mean.
By grafting the outlier value to the framework of the ex-falluja data, “probably higher” is probabilistically incoherent and deceptive. It smuggles the outlier value in front of the jury without acknowledging any new blown-out standard deviation. Under all scenarios, alluding to the “true result” of the Lancet survey as “probably higher” due to Falluja represents a FALSE quantitative, statistical judgment, especially when voiced ex cathedra by a scientist or statistician.
March 25th, 2005 at 10:36 am
Shannon, Let me get this straight, you admit that it is common sense that body counts save lives but this is not sufficient — you want a controlled study comparing a conflict with a count with one without. But to do such a comparison you need to do a body count in both conflicts because otherwise you can’t measure whether lives are saved or not.
Anyway, at a minimum your own suggestion that a controlled study would be needed satisfies your challenge since conducting such as study would, according to you, serve a useful purpose.
March 25th, 2005 at 10:57 am
telluride (10:01am),
I don’t see how one can know the nature of the distribution of violent deaths in postwar Iraq, unless one has examples of simi