Monday, September 10, 2007

Lies, Damned Lies, and Statistics


Cynicism On Iraq

September 10, 2007


(Photo: Doug Mills, NY Times)

General Petraeus today delivered his much anticipated report to congress concerning the status of the war in Iraq.

According to his prepared statement (available at
http://media.washingtonpost.com/wp-srv/politics/documents/petraeus_preparedtext_091007.pdf ), here are the important points:

BEGIN QUOTES (chart and map labels have been added).

I wrote this testimony myself. It has not been cleared by, nor shared with, anyone int he Pentagon, the White House, or Congress.

The military objectives of the surge are in large measure being met.

The number of security incidents in Iraq has declined in 8 of the past 12 weeks.

Coalition and Iraqi operations have helped reduce ethno-sectarian violence, as well, bringing down the number of ethno-sectarian deaths substantially in Baghdad and across Iraq... The number of overall civilian deaths has also declined.

The military objectives of the surge are in large measure being met.

Progress has been achieved “in the security arena.”

“The number of security incidents in Iraq has declined in 8 of the past 12 weeks… with the numbers in the last 2 weeks the lowest levels seen since June 2006.”

“One reason for the decline in incidents is that Coalition and Iraqi forces have dealt significant blows to Al Qaeda-Iraq.”

Coalition and Iraqi operations have helped reduce ethno-sectarian violence, as well, bringing down the number of ethno-sectarian deaths substantially in Baghdad, and across Iraq...

Iraqi security forces have also continued to grow.

In what may be the most significant development of the post 8 months, the tribal rejection of Al Qaeda that started in Anbar Province and helped produce such significant change there has now spread to a number of other locations as well.

The fundamental source of the conflict in Iraq is competition among ethnic and sectarian communities for power and resources. This competition will take place, and its resolution is key to producing long term stability in the new Iraq. The question is whether the competition takes place more or less violently.

Map 1




Two US intelligence agencies recently reviewed our methodology, and they concluded that the data we produce is the most accurate and authoritative in Iraq.





Chart 1: Iraq Attacks

The level of security incidents has decreased significantly since the start of the surge of offensive operations in mid-June, declining in 8 of the past 12 weeks, with the level of incidents in the past two weeks the lowest since June, 2006.













Chart 2: Iraq Deaths






















As the next chart shows, the number of ethno-sectarian deaths, an important subset of the overall civilian casualty figures, has also declined significantly since the height of the sectarian violence.

Chart 3: Death Density



















Chart 4: Death trends




To be sure, trends have not been uniformly positive across Iraq, as shown by this char depicting violence levels in several key Iraqi provinces.










Chart 5: Iraq Troops

As I noted earlier, Iraqi Security Forces have continued to grow, to develop their capabilities, and to shoulder more of the burden of providing security for their country.












To summarize, the security situation in Iraq is improving, and Iraqi elements are slowly taking on more of the responsibility for protecting their citizens. Innumerable challenges lie ahead; however, Coalition and Iraqi Security Forces have made progress toward achieving sustainable security. As a result, the United States will be in a position ro reduce its forces in Iraq in the months ahead.

Chart 6: Recommended strategy:


















END QUOTES

ASSESSMENT:

This is an overall positive, persuasive report that documents mission progress since the surge began but paradoxically recommends troop withdrawals anyway.

The Opening Lie:

The explanation for that paradox, I believe, is that this is a White-house shaped, politically motivated report. In a carefully worded disclaimer, the general states that the written text has not been edited by, or even shared with anyone in the pentagon or white house. OK, that's probably literally true -- he did not share the final written copy with anyone. So what?

Written vs oral is a trivial distinction for a report whose content has been under intense discussion with both Pentagon and White House for weeks if not months. Why couldn’t he have told the truth, that he developed the report with the advice of his commander-in-chief and others? If he did not do that, then the commander-in-chief is seriously and dangerously negligent in his duties. It is an unnecessary misdirection that casts significant doubt on the credibility of the entire report.

Second Lie: Surge Effectiveness
In January of 07, the president ordered an additional 20 to 30K troops into Iraq to “stabilize” the situation and reduce the bombings and killings, in order to give the political process some time to work.

Has the surge been effective? General Petraeus thinks so. What is his evidence?

“The number of security incidents in Iraq has declined in 8 of the past 12 weeks… with the numbers in the last 2 weeks the lowest levels seen since June 2006”

Chart 1 plainly shows a downward trend in number of attacks since the 16th of June, 2007, to September 7th, a period of 2.5 months. No statistical tests are provided, but the eyeball analysis looks good, especially if you overlook the fact that the timeline on the horizontal axis is not uniform. In some months, there are two data points per month, while in others there is only one. Yet all data points are shown equidistant.

If a student turned in such a misleading graph in one of my classes, they would flunk. The effect of this misleading technique is to stretch and flatten the long tail of data to the left, and, since the last two months have only one data point each, to compress that period and make it look more dramatic than it is. Furthermore, July of 07 is represented twice, to make the peak of violence before the surge look stronger than it should.

Finally, we should note that there are no data for 2003, when the invasion took place and the civil war began. Omitting those data without explanation is arbitrary and misleading. What if the violence levels for that period were extremely low? Then the surge wouldn’t look so good, would it? Especially since there seems to be a down-tick in violence every August, at the peak of the sweltering summer. Maybe the current drop is just part of an annual cycle.

These are small graphical tricks or errors, but two factors make me believe they are intentional deceptions.
1. I can’t believe that a five star general does not have anybody on his staff that knows how to make a proper graph.
2. All the errors favor the administration’s desired outcome that “the surge has worked.”

The Third Lie: Iraqi Civilian Deaths

Chart 2 shows a clear hump and recent decline in Iraqi civilian deaths since November 06. Further proof of the surge effectiveness? Maybe.

But why is the timeline of the chart only the last year, when we have been at war there four years? Maybe if we could see the whole picture it would tell a different story. The overall trend might be flat.

What’s special about November 06, the peak of civilian deaths? Probably nothing except that it was a high point selected for the center of the graph to make the subsequent decline look more dramatic.

Without seeing a computed trendline showing that there is an overall down trend in the data shown it is not obvious that there is a downward trend. Overall, the chart looks more cyclical than down-trending to me.

Imagine a modified Chart 2 that only showed the data since June, 07 . That wouldn’t be such an impressive trendline, would it? Selecting arbitrary chunks of data to show, without explanation, means we cannot trust any of it.

Fourth Lie: Ethno-Sectarian Deaths

Chart three is complex, but its data source is the line graph in the lower left. That graph, of “ethno-sectarian” deaths suffers from the same flaws as the previous chart, namely, arbitrarily truncated data. For the same reasons then, the graph cannot be properly taken as evidence for anything in particular.

Nevertheless, I really like the look of the density charts shown in the other four sections of the graph. They seem to show a gradual diminution of deaths over time, almost like a wound healing up. It’s a great visual, and it’s almost convincing.

Again however, it is an arbitrary data set taken over an arbitrary time period, so it means nothing in itself. Furthermore, it is misleading in a new way, since representing the data on a geographical area means that any decline seems to be much more dramatic, since an area will reduce geometrically as the data upon which it is based drop arithmetically.

It’s like the relationship between a square’s width and its area. To get the area, you multiply the width times itself. That results in a much larger number, which, when displayed on a map, looks really big. It’s not illegitimate to represent data that way, but it is misleading for people who do not understand how graphs work.

Finally, we should note that the graph only shows “ethno-sectarian” deaths, not total deaths or US troop deaths or anything else. What the heck are “ethno-sectarian” deaths? It’s hard to say exactly, as this is custom Pentagon-White House jargon.

One gathers from news reports (e.g., Washington Post.com, September 2, 2006) that the term refers primarily to the activity of Sunni and Shiite death squads engaged in civil war.

“Ethno-sectarian violence” then would exclude all attacks on US and Coalition forces, attacks on Iraqi government troops or Iraqi government targets, or attacks by the US or Coalition forces on anyone.

We therefore have no idea from this graph what proportion of “ethno-sectarian” violence is of the total amount of mayhem going on in Iraq. To that extent, Chart 3 is uninterpretable.


Fifth Lie: Provincial Trends:
Chart 4 shows “violence trends” in four provinces of Iraq. Whatever “kind” of violence these data are supposed to represent, the key feature of these four graphs is the black arrows added, presumably to help you “see” the trends in case you couldn’t make them out.

Those are not legitimate, computed trend lines, or probably not, because if they are genuine, they would not be accurately positioned with their data. Therefore, I conclude they are just bogus propaganda overlaid on the data.

They also commit the same old error of being arbitrarily about the data they highlight. For example, I think a computed trend line for all the data of Salah ad Din Province would probably be about flat, showing no change overall. The two heavy black arrows are designed to divert your attention from that fact. It is classic misdirection.

The same could be said for the Baghdad chart. The other two, based on eyeball analysis, do seem to show trends, one down, one up.

The equivocal data overall and the heavy-handed attempt to manipulate it, make none of it credible to me, especially since we don’t even really know what the data represents, precisely (what kind of violence).


Sixth Lie: Iraqi Troops

Chart 5 shows Iraqi Security forces strength and capabilities. It is what is called a “stacked graph,” since each column is a stack of different data and the height of the column represents their sum.

The only dataset worth looking at, in my opinion, is the green set across the bottom, which represents fully independent Iraqi security forces. That data set shows essentially a flat trend since startup in April of 06 (data before then being negligible), and a high degree of variance. Essentially it says there are very few fully independent Iraqi security forces and what do exist, are unstable. Not an optimistic report.

However, the rest of the graph is made out to obfuscate that fact and look like an inexorable rising trend. The yellow bars, for example, are clearly expanding over time. What are they? “Iraqi lead with coalition support.”

What does that mean, exactly? Does it mean that the Iraqi troops lead in all phases of security operations and the US troops just take orders from them? I seriously doubt that.

It could mean only that when there is a security sweep, the Iraqi troops have to break down the door first and get shot first.

We have no idea what the yellow bars mean. It may be possible to research that question further, but this is supposed to be an informative document on its own. Which it isn’t.

What about the orange bars in the graph? They were large, shrunk up severely in the middle, but now are growing again.

They represent “Fighting Side by Side.” Again, it is difficult to know what that means in terms of performance and effectiveness. It might mean only that American and Iraqi troops ride in the same trucks when they go out on patrols. The extremely high variability is not explained. The data are uninformative.

The red bars on the top of each stack represent Iraqi units that are “forming.” Basically that means they do not yet exist as functional units, so they are on this graph just to make the columns look like they trend up more than they do. They had no blue ink? The red is almost indiscriminable from the orange, another trick to manage the overall visual impression of the chart. Take off the red bars and you would have a very different impression from this chart. They are an overt manipulation of the visual appearance of the graph, and as such, dishonest, in my opinion. Another lie.

Seventh Lie: Recommended Strategy.
Chart 6 is so incomprehensible that barely rises to the level of a lie. What is the meaning of the five pointed stars? They are undefined.

What are the tiny numbers inside the stars? Also undefined.

What is the unit of the vertical axis? Undefined. (A student graph with an unlabeled axis gets an automatic flunk for the assignment, since that omission renders the graph totally uninterpretable). The horizontal axis, which did not copy clearly on my picture here, is just a single long arrow to the right labeled “Time,” no dates given.

The visual impression is clearly one of a stair-step downward over time. One can surmise that it is intended to show troop reductions over some indefinite amount of time, which could be anywhere from one year to twenty years. In any case, there will apparently always be at least 5,000 troops in Iraq, which suggests establishment of a permanent military base there, for blue and green activities, which are strategic and operational “Overwatch” functions; no leading, no partnering, no tactical ops.

To the extent that such a strategy is comprehensible, it is not believable. These remaining troops would essentially be “advisors” apparently, and probably would not even have to carry guns, since they would only be “watching over.” Not very likely. If the country did not need fully armored US combat troops, there would be no reason for any to be there.

The stacked bars show a large and rapid (within 6 months) reduction in US military leadership and combat partnering. In light of all the previous data shown, that is just wishful thinking, or acquiescence to the American public’s demand for an immediate end to the war.

Is there any reason to believe, as suggested, that Sunni tribal leaders will continue to cooperate with US forces in opposition to Al Qaeda? No.

In the first place, it is clear that the tribal leaders want only to re-establish their fiefdoms that
have been disrupted. Any “cooperation” with the US is purely tactical, probably bolstered by the fact that while cooperating, we give them guns and ammunition. Is there any reason to think that that dynamic will spread after the US pulls out or draws down? It might for a little while as we arm all tribes to the teeth in preparation for the forthcoming all out civil war. But eventually, that powder keg has to blow.

Conclusion: The Big Lie:
Nowhere in his prepared text did General Petraeus say exactly what the mission of the Surge originally was. He just jumped in and started talking about success and progress. Of course those terms are undefined and the whole report is meaningless in the absence of some criterion of success and some measure of progress against the desired goal state.

If the desired goal was, as stated repeatedly by the White House, to allow the Iraqi politicians sort out their governance problems, then judging from other recently released reports, this Surge has not been a success, and has not made any progress.

If we take the much narrower view that the purpose of the Surge was merely to suppress the level of violence in Iraq, then this report seems to document progress, but upon close inspection, actually documents nothing at all. It is not an honest report and so must be taken principally as an exercise of political propaganda. As such it is irrelevant to the truth of the matter, whatever that might be.

As for the recommendations at the end. (Chart 6):
You have to wonder, if the surge is so tremendously successful, as this report would have us believe, why pull out the troops and end it? Why not continue it for another year? Why not double its strength? Triple it? It makes no sense at all to argue that the Surge is just starting to be successful AND that it will be terminated within 6 months. How does that compute? Mission Accomplished all over again?

It computes only because we know from other sources that there are no more troops to send, the troops we have now are exhausted, and the political will to keep fighting is rapidly waning in the American public. Therefore, for political, not military reasons, the wonderful, successful Surge must come to an end quickly.

Interesting, too, that the first large force reduction will be completed by the summer of 08, just weeks before the Republican National Convention is held in Minneapolis. But surely that is mere coincidence.

Why couldn’t the General own up to the real reasons for the troop drawdown? Why does he have to pretend the situation is something other than it is? The built-in lies, omissions, misdirections and evasions, are why this whole report must be taken with a very large grain of
salt. It’s a political document not an honest report.

What about the early disclaimer, that two intelligence agencies vouch for the general’s data accuracy? That is probably true. But it says nothing about how the data are presented. As we know, there are lies, damned lies, and statistics…

As for the timeline for the pullout, Petraeus is adamant that no fixed timeline should be set. That sounds like a White-House echo to me. Yet against that politically dictated rule, Chart 6 shows visually a timeline of a 95% pullout within 18 months, almost exactly the amount of time that G.W. Bush has remaining in office. What an amazing coincidence!

If we are supposed to interpret Chart 6 as an honest representation instead of a lie, then that is what the visual data show. However, since the general denies that there is any fixed timetable, then we must take the final chart as uninterpretable nonsense, as he may have intended anyway.

Since everything else seems to be a lie in this report, I would put the lie to the final big lie and say that the general is hinting here that there is, in fact a timetable and it is 18 months for 95% pullout.

At least that IS some good news!