If the 20th century really began with the assassination of Archduke Ferdinand in August 1914, which set in motion the start of a series of intrastate wars so brutal they killed tens of millions, then surely the beginning of this century was announced by the attacks of September 11, the harbinger of a new kind of war waged with spectacular acts of terrorism by non-state groups that seem likely to be a defining feature of the century to come.

On the beautiful morning of September 11, 2001 nineteen men armed only with knives attacked the preeminent symbols of American military and economic might, killing some three thousand people and inflicting hundreds of billions of dollars of damage on the American economy in the first serious attack on the American mainland since 1814.

Al Qa’eda’s assaults on New York and Washington were a shock in more ways than one: the world’s only superpower was bloodied not by a rival state, but a rogue organisation. But it should have been less surprising: Osama bin Laden declared war on the United States in an interview with CNN in 1997, and delivered on that promise with attacks on two American embassies in Africa in 1998 and the bombing the USS Cole in Yemen in 2000. The September 11 attacks were not the beginning of al Qa’eda’s campaign against the United States but its climax.

There was little question that al Qa’eda was at war with the United States. The critical question in the months that followed was: What kind of war was the United States going to fight against al Qa’eda? Clausewitz explains the importance of such decision-making in his treatise On War: “The first, the supreme, the most decisive act of judgment that the statesman and commander have to make is to establish…the kind of war on which they are embarking; neither mistaking it for, nor trying to turn it into something that is alien to its nature.”

Contrary to a common view among Europeans, who have lived through the bombing campaigns of various terrorist groups for decades, al Qa’eda is not just another criminal group that can be dealt with by police action. After all a terrorist organisation like the Irish Republican Army would call in warnings before its attacks and its single largest massacre killed only 29 people. Al Qa’eda struck at American government buildings and killed thousands of civilians without warning; acts of war by any standard.

Twelve days after the attacks, President Bush addressed Congress in a speech that would lay out the strategic doctrines of what he would term the “War on Terror” and the Pentagon later dubbed “the Long War”. It’s worth examining that speech in some detail, as it would set the course of the foreign policy of the United States for the rest of the Bush administration and reshape the Middle East. Before a packed congressional chamber and tens of millions of Americans watching at home, Bush explained that “our war begins with al Qa’eda but it does not end there. It will not end until every terrorist group of global reach has been defeated.” This war, then, would extend not only to the perpetrators of September 11, but to any other group that might potentially threaten the US – and, furthermore, it could last forever: “Americans should not expect one battle, but a lengthy campaign.”

Then Bush turned to the reasons the US was targeted: “Why do they hate us?…They hate our freedoms.” This was, in fact, a gross distortion of the rationale bin Laden laid out for his war on the United States, which was based on his religious critique of American foreign policy in the Muslim world and had nothing to do with western freedoms. Any consideration of what American policies in the Islamic world might be provoking Muslim anger was going to receive short shrift from the Bush administration.

Bush suggested that al Qa’eda followed in the footsteps “of the murderous ideologies of the 20th century… They follow in the path of fascism, Nazism and totalitarianism,” implying that the fight against al Qa’eda would be similar to World War II or the Cold War. This too was nonsense: the threat from al Qa’eda is drastically smaller than that posed by Mutually Assured Destruction or the triumph of Nazism in Europe. (Painting the conflict in such existential terms had the side benefit of casting the president as Winston Churchill – and anyone who had the temerity to question this framing as the reincarnation of Neville Chamberlain.)

The subsequent American war against the Taliban was backed by the world. On September 12, 2001, the UN Security Council passed an unusually forceful and unambiguous resolution “to combat by all means…terrorist acts” and recognised its members’ “inherent right of individual or collective self-defence.” Nato invoked Article 5 for the first time in its history, declaring the attack on America an attack on all the members of the alliance.

The New York Times correspondent Dexter Filkins has been an eyewitness to every phase of the subsequent “war on terror”. What he nicely terms “The Forever War” has already lasted longer than any American war save the one in Vietnam. Filkins’ journey through that war begins in Afghanistan while the Taliban are still in power, then turns to the American campaign that toppled them and then moves on to what is the heart of the book – an impressionistic narrative of the Iraq war from the invasion to the rise of the Sunni insurgency and its Shia analogue, the Mahdi Army.

The Forever War is not a work of history or analysis but a you-are-there description of what it feels like to taste cordite in battle. It stands in the grand journalistic tradition of Dispatches, Michael Herr’s narrative of his Vietnam experiences that did so much to inform the acid-tinged vibe of Apocalypse Now, and My War Gone By, I Miss It So, Anthony Loyd’s account of the destruction of Bosnia in the 1990s and his own descent into heroin addiction.

There are no drugs in Filkins’ stories, nor even any alcohol: the American army now goes into battle zones dry. Nor does Filkins give us much of a sense of his own feelings and emotions as he witnesses some of toughest battles Americans have waged since Vietnam. Rather his I-am-a-camera prose paints word pictures that resonate like a Hopper painting.

Here he recounts a scene of “collateral damage”: “Omar, a 15 year old boy, sat on the roadside weeping, drenched in the blood of his father, who had been shot dead by American marines when he ran a roadblock…’We yelled at them to stop,’ Cpl. Eric Jewell told me. ‘Everybody knows the word stop. It’s universal.’” As Filkins observes, “For many Iraqis, the typical nineteen year old corporal from South Dakota was not a youthful innocent carrying America’s goodwill to the people of Iraq; he was a terrifying combination of firepower and ignorance.”

But The Forever War is far from an anti-American screed. Filkins helps us understand how Saddam ruined the lives of so many Iraqis with the most wanton cruelty. Yakob Yusef, the headmaster of a Jesuit college in Baghdad, explains how his brother suddenly disappeared in 1998. Three weeks later Yusef received a call from a government official saying his brother had been executed and he could now recover the body. The official said “You are very lucky. Most people never get the body. You should be very grateful to us.” When he received his brother’s body, Yusef had to pay for the two bullets that had killed him. His brother’s executioners gave him a receipt for the money.

And Filkins shows how war drove Iraqis to the brink of madness and hell. Abu Marwa, a Sunni insurgent who, like many other Sunnis, eventually turned against al Qa’eda’s affiliate in Iraq, explains how he discovered his uncle in the local morgue; his legs had been drilled by electric power tools; burn marks ranged across his body. Abu Marwa ambushed the Syrian members of al Qa’eda who had tortured and killed his uncle and then presented his aunt with a vial of the killers’ blood, which she then drank. Abu Marwa explains, “You see. We were for revenge. She was full of rage.”

A particularly powerful aspect of The Forever War is the rebuke, never made explicit by Filkins, to those in the Bush administration who constantly complained that the media was ignoring the “good news” in Iraq. Defense Secretary Donald Rumsfeld, for instance, lamented in August 2004 that positive news stories “are apparently not as newsworthy, and they seem not to make the press,” bizarrely citing Iraq’s fielding of an Olympic team as evidence of progress at a time when Iraq was already one of the most dangerous places on the planet.

Filkins recounts what it was actually like in Baghdad as the war accelerated: “There was no law anymore, no courts, nothing – there was nothing at all. They kidnapped children now; they killed them and dumped them in the street. The kidnapping gangs bought and sold people; it was like its own terrible ecosystem.” And the good news, what little existed, was almost impossible to cover. “One the favourite targets of the suicide bombers were American ribbon-cuttings – a pump station, for instance, or a new school, because of the crowds they brought. It got so bad that the Americans sometimes kept the unveilings of new projects a secret. Which kind of defeated the purpose.”

Filkins also helps us understand how the war changed from a conflict that was relatively easy to cover in 2003 to the most dangerous war the media has ever covered. More than 130 journalists have been killed in Iraq including Khalid Al-Hassan and Fakar Al-Haider, two of Filkins’s Iraqi colleagues to whom he dedicates his book. Over the course of the war, the Baghdad bureau of The New York Times has gradually morphed into “A fortress... we brought in a crane to erect concrete blast walls, a foot thick and 20 foot high. We hired armed guards, twenty of them, then thirty, then forty…We put searchlights on the roof and then machines guns, 7.62 millimetre, belt-fed.” That “belt-fed” is a nice touch.

Filkins does not draw any grand conclusions in his book. But he does remind us of a very old truth: war turns everything it touches into dust. The final scene in The Forever War finds Filkins visiting the family of Lance Corporal William Miller, an American soldier who died in front of him in Iraq. “The Millers joked and smiled, they talked of Billy and his life, almost as if he were still there. Their cheerfulness was relentless. They did not flinch. I told them that I thought about Billy every day, about how he had taken a bullet for me... ‘He was just doing his job’, Susie [Miller] said. ‘He died doing what he wanted to do.’ She was ready for that one. I gathered it was a construction, the cheerfulness was, a Potemkin thing, and one whose construction came at no small effort. Still, it made me sad, even a little frustrated.”

Today the violence in Iraq is slowing dramatically. Yet it remains one of the most dangerous places in the world, and – with some four million refuges inside and outside the country, more than four thousand American war dead, tens of thousands of Iraqis killed, the United States’ moral capital around the world at an all-time low and a price tag that could rise to two trillion dollars – the cost has been enormous. The progress in Iraq made over the past year does not post facto justify all the blood and treasure spilt over the previous four years.

It is clear today that the Bush administration ignored Clausewitz’s admonition; nor, for that matter, did it think through many of the policies it adopted to fight al Qa’eda, from the use of coercive measures on prisoners to allowing the Taliban and bin Laden to fight another day on the Afghan-Pakistan border.

But now, belatedly, the Bush administration is adopting some of the policies that make sense to help defeat al Qa’eda and to lower the temperature in the Muslim world, such as restarting the Israeli-Palestinian peace process, engaging with Iran and co-opting Sunni militants in Iraq.

And there the next administration has an opening: To set a course that is based not on an ideological interpretation of the threat from Islamist militants, but instead approaches it with the kind of realism that the Bush administration has finally begun to adopt. That way, perhaps, the forever war may finally end.