Media organizations diverge in defining impact

By Catherine Bond

‘They listen, and that means you have an impact,’ says Jean-Marie Etter of listeners to Radio Agatashya, Fondation Hirondelle’s first project, an FM station set-up in mid-1994 as the genocide inside Rwanda ended and bloodshed across its border in Zaire (now the Democratic Republic of the Congo) began.

‘Measuring’ the impact of Agatashya – the ‘little swallow’ in Kinyarwanda – and other projects would come later, and is something Etter insists he is ‘personally very attentive to’,  but amid the trauma that followed the genocide, broadcasting accurate news to millions of people disorientated by flight and confused by propaganda, was impact enough.

‘Impact’ has since become both a mantra and a scorecard for humanitarian agencies and development organizations, as well as the donors that fund them. Yet it remains, for those whose task it is to understand and measure it, an imperfect art.

Hirondelle’s objectives are closer to those of broadcasters dealing with the intangibles of news, than to aid agencies offering the tangibles of food or health care, though Anne Bennett of Hirondelle USA says there are cases when information has measurable benefits too: in South Sudan, the UN-Hirondelle media, Radio Miraya, broadcast messages which helped families, separated for decade by war, find each other.

But in some development circles, providing information alone is losing its allure, and the argument for changing not just people’s minds – once the driving force behind Western funding for newly independent media in fledgling democracies – but their habits too, is gathering pace.

BBC Media Action, for example, seeks to change not just attitudes but behavior, and in vast and varied arenas – humanitarian development, governance, and public health. For the people controlling the purse strings – the donors –  it’s an attractive notion: information that you can quantify into ‘results’.

‘It’s an interesting time for media development,’ says Susan Abbott of Internews, a US-based non-profit which works in scores of countries shoring up local media. Abbott describes a time of ‘transition which mirrors media transitioning’ – one throwing-up other questions, such as how to measure media’s contribution to a complex process of societal change.

The global journey of IREX, also a US-based non-profit, charts the history of Western-backed, media development: it went into East Asia in 2001, the Middle East in 2005, and Africa in 2007. Its annual index has become a tool donors use to assess how free a country’s media is.

This Media Sustainability Index sets out to measure how effectively a country’s media serves its public as ‘the fourth estate’, a check on government. Panels of twelve score five different aspects of a country’s press: freedom of speech, professional journalism, news plurality, the media’s management and financing, and the strength of local institutions supporting it.

Critics say it’s too simplistic; shouldn’t research be done on how people communicate now,  on channels they might trust more than the media, in order that organizations can work with those? Supporters say it provides a starting point.

Leon Morse of IREX says these studies are more nuanced than the data they provide suggests. In them, IREX looks for ‘robust development’ resistant to reverses in progress already made. He gives the example of how ten years of data from Serbia shows mixed results: at times, media business management might improve while freedom of speech backtracked. Nationwide, while the media’s strengths would rise and fall, individual outlets – like radio stations in small, Serbian towns – performed well.

The paucity of funding available for monitoring and evaluating, Morse says, limits its scope. ‘It’s an inexperienced way to address a complex problem.’

Not everyone lacks funds for research. BBC Media Action, an international development charity of the BBC, has received a British grant of £90 million (US$135 million) that’s made it possible to hire teams of local researchers to help hone themes on governance, health and humanitarian assistance, for target audiences in 14 countries.

If what Hirondelle believes in is ‘good journalism … information without moral judgment’,  in contrast, BBC Media Action overtly seeks to change people’s behavior, grounding it in a long domestic tradition of audience research.

Moving ‘beyond the logframe’, says Kavita Abraham-Dowsing of BBC Media Action, requires not just monitoring the data but evaluating it – two very different things: ‘The world is full of data, but we need to analyze it.’

Measuring impact

The impact of journalism should be ‘measured carefully, objectively, but also roughly’, says Richard Tofel in a white paper on non-profit journalism, published by the online investigative site, Propublica. He argues impact can be tracked with rigor and described with precision, but shouldn’t be thought of as ‘subject to mathematical proof or even, in some cases, statistical reliability’.

Tofel says as ‘successful a story from the perspective of impact as one could imagine’ – the 2009 sacking of most  of California’s nurses’ board by their boss, the then Governor of California, Arnold Schwarzenegger, followed a story by Propublica and the Los Angeles Times – is relatively rare.

Had ‘the alchemy of impact’ (a term Tofel borrows) been different – had Schwarzenegger not seen the story, had he been by temperament less inclined to act swiftly, had other factors stopped him or slowed him down – it may have little or no impact. ‘None of these factors,’ says Tofel, ‘would have lessened the purely journalistic value of the story.’

More often the impact of investigative reporting is measured in months and years. Insisting on ‘ quick results would ‘distort behavior,’ he writes, ‘and reward thinking that focuses on smaller but more visible objectives over longer-term, larger targets.’

Philippe Dahinden was the Swiss journalist who teamed up with Etter to found Hirondelle shortly after covering the 1994 Rwanda genocide. The killing was organized by ethnic extremists from Rwanda’s then majority Hutu government, which targeted the Tutsi minority it felt sympathized with Tutsi-led rebels fighting it, as well as political moderates (both Hutu and Tutsi) who wanted a peace deal between the two.

‘We did our job as well as we could, informing the rest of the world,’ he wrote later of the foreign journalists there, ‘but the victims of the events, who didn’t see our reports, also deserved information … a right that is as vital as the right to food or medical care. People have a right to know.’

Inside Rwanda, Dahinden had noticed extremist militias listening to the ‘hate radio’ that played a role in inciting the killings of Tutsi and their allies. Hirondelle decided to steer clear of advocacy, ‘even,’ says Etter, ‘for very honorable causes’. Instead, it would serve everybody’s news interests, those of ‘the victims and the torturers’.

Remaining neutral was, however, unpopular with the rebels whose victory brought the state-controlled genocide inside Rwanda to an end. Radio Agatashya never received the authorization to work and broadcast from Rwanda; it was thus based in Bukavu, Eastern Congo (DRC) and was forced off air, forever, when the AFDL forces from Laurent Desire Kabila took power in the region and looted the radio station in late 1996.

Catherine Bond is an award winning news journalist with 25 years experience, largely in Africa with CNN and the BBC. As CNN’s Nairobi-based correspondent and Bureau Chief from 1996-2004, she produced, wrote and reported live on key political transitions and humanitarian crises. Prior to this, Bond reported from Rwanda in the early 1990s, including in-depth reporting of the 1994 genocide as it unfolded.