Giving by numbers – how measuring social impact stacks up

These include local and central government, which is the single biggest funder of third sector activity. Its need to demonstrate value for money and show accountability and transparency is creating, often onerous, demands on service providers to demonstrate clear evidence of the impact they are having.

The changing profile of philanthropists over the last decade is also a factor in the push for quantifying impact – many of today’s philanthropists are successful business people who understand the world in commercial terms and actively seek data to demonstrate the value of their social investment.

However, it is not only the ‘business’ of philanthropy that is propelling the third sector towards a more metric approach – there remains a demand from all those involved to know the interventions they are making are changing the world for the better and are working to close the gap between rich and poor.

Fiona Halton, chief executive of Pilotlight, which recruits senior business people to coach and mentor charities, says: “We need to know we are making a difference and moving in the right direction, and we can only do that through measurement. If we don’t know where we are are going – how do we get there?”

Ken Berger, president and CEO of Charity Navigator, a US-based charity rating service, says: “My concern is that some people do not appreciate the life and death nature of what we are dealing with. I have been teased that I am an ‘evangelist’ for outcome measurement, but I think we ALL should be. Without outcome measurement to tell us what works and what doesn’t, we may well be wasting billions of pounds on useless or even harmful interventions.”

Yet the central aim of measurement, to establish a causal link between activity and long-term impact, is proving extremely difficult. For example, how do you show beyond a doubt that providing advocacy services for victims of domestic abuse can lead to them becoming independent, happy individuals 10 years on? 

Colin Nee, chief executive of  of the Charity Evaluation Service (CES), which has worked with thousands of voluntary organisations and their funders, feels that often, “the more grandiose the aim, the greater the claim.”

An added complexity is that some sectors of society are harder to study in the long-term than others. For example, homeless people are difficult to track and often ‘disappear’; ex-offenders are difficult to monitor as they want to leave that period of their lives behind them.

Some posture that measuring shorter-term outcomes is a better way forward, particularly as longer-term impacts don’t fit conveniently into three-year funding streams anyway.

That said, the adoption of the principle of measurement across third sector organisations is happening, albeit in a patchy fashion. Some passionately embrace it and some may be forced to ignore it because of cost, culture or lack of capacity, while most are somewhere in between.

Sally Cupitt, senior consultant at CES, commented, “Our experience over the last 20 years, combined with recent research we have done, would show that many are keen and do undertake measurement. Most do some monitoring of outputs, and many do at least some monitoring of outcomes.”

And independent consultant David Carrington, who draws on 25 years experience of senior management positions in charities, confirms that the concept is not new: “training courses on measuring impact have been running since the 1980s,” he says.

Funding organisations are equally passionate about embedding measurement into their organisations.

Paul Bernstein, CEO of Absolute Return for Kids (ARK), a philanthropy fund, says measurability and accountability is “in our DNA” and explains that ARK was started in response to the lack of philanthropic organisations demonstrating the impact of their funding.

Brian Trelstad, CIO of the Acumen Fund, a New York-based venture philanthropy fund working to alleviate global poverty, says in the Summer 2008 issue of Innovations ,: “For all the innovation in social enterprise, the question remains: So what? Have these new approaches led to enduring change? Are people drinking cleaner water, living healthier lives, or moving out of poverty because of the new products designed to be affordable and accessible to the poor?

“We think so, but the evidence is nascent and mostly anecdotal.”

Frameworks are being put in place to help gather that evidence more extensively.

For example, the SROI Network is a membership organisation which is leading a consortium managing a Cabinet Office-supported project called Measuring Social Value. The consortium has just published a guide to social return on investment (SROI) (see SROI: government and third sector extend reach of measurement tool) – a framework which expresses impact in cost benefit terms. The framework and soon-to-be online tool are at the heart of the government’s plans in supporting third sector organisations to demonstrate added social value.
 
But whether measurement will – or even should – ever be mandatory is up for debate.

There are calls for legislation and heavy regulation, while others believe in a lighter touch approach to measurement; where self-regulation, codes of conduct and best working practices are the order of the day.

The thinking is that ultimately market forces will demand the publishing of such information anyway as organisations compete for donors and funds.

And the hope is that the publication of outcomes will lead to wider learnings across charity sectors – though not generalisations across the third sector.

But before such hopes can be harvested, there are a huge number of challenges to overcome.

The issues that hamper the progress of impact measurement are a long way from being solved. The first is fundamental: definition.

Defining social impact

Nee of CES says: “Every discussion on social impact is lost in a swirl of discussion about semantics and definitions.”

According to CES’ discussion paper ‘Assessing Impact’, there is broad agreement that impacts are “changes resulting from an organisation’s activities” and are often viewed as “the real reason for the existence of many voluntary sector organisations and the justification for the personal effort and money that go into the organisation.”

Yet there are as many definitions as there are organisations.

Fortunately there are numerous resources that can help with defining social impact, as this table shows.

But while defining the term may be a stumbling block for some, CES’ Sally Cupitt, and author of ‘Your Project and Its Outcomes’, says making a distinction between outcomes and impact does not always matter, as both terms are about change.

“However, outcomes and impact refer to different levels of change, requiring different methodologies to measure them. ‘Impact’ describes longer term, broader change than outcomes, may affect a wider group of people and be affected by a range of organisations. As such, true impact assessment takes time, skills and money, and it can be harder to attribute change.”

Identifying indicators

The difficulty of definition continues in identifying what to measure. In recent years there has been a refining of indicators from those that measure absolutes to more sophisticated ones that attempt to capture change – not just how many clients used a workshop for unemployed people, for example, but how many more clients applied for jobs and whether they felt more confident as a result.

Brian Mellon, managing director of Frontline Consultants, who work with UK’s public sector organisations to increase efficacy, sums up the problem: “What if people improving their performance ‘as measured’ are not actually achieving the outcome needed?  Hitting the target, but missing the point.

“Defining ‘the point’ – the intended impact is an essential part of initial project design and measures should be chosen at that stage.  Inputs are easy – the initial funds.  Processes are popular – number of computers for a school.  Output counts reassure that the activity was delivered – numbers of trainee completion certificates.  However, if the desired outcome was fewer young people not in employment, then none of these measures give funders a sense of what is being achieved – good or bad.”

And who should define indicators – funders or fundees?

Jamie Cooper-Hohn, co-founder and CEO of The Children’s Investment Foundation Fund (CIFF), says ‘both’:  “It is essential that what defines success is jointly agreed by the funder and implementer at the outset of the project.  Key to this being a productive exercise is de-mystifying the fuzzy aspirational jargon most entities use to describe their work and establishing clear targets for change that are directly measurable.  For example, “improving the lives of children” needs to be translated into concrete impacts such as reduction in child mortality or increased percentages of children transitioning from primary to secondary school or fewer teenage pregnancies.  There is no issue around which this clarity cannot be achieved if both parties engage with an honest willingness and desire to understand their impact.”

Carrington explains: “I believe that it should be universal practice for all organisations to set and measure goals, outcomes or impact in ways that are appropriate and proportionate for them – that funders should encourage (and, if necessary, help to fund) them to do so, and that funders should identify what data they need and negotiate with the organisations if the grantee’s or investee’s systems do not already deliver that data or proxies for it.”
 
Defining indicators gets murky once again when programmes focus on preventative measures as they often do – how do you measure what hasn’t happened with any accuracy?

And what do you measure outcomes against? Expensive research to establish a baseline or benchmark is beyond the purse strings of many of the small organisations doing so much good work.

Shared evidence and data may well help – and there is a call for more openness and transparency.

For example the Scottish Government is supporting an SROI project led by Forth Sector Development and the SROI-Network, to create a database that would publish indicators and their relevant proxies.

But how far should transparency go?

Dierdre Mortell, co-founder and CEO of the One Foundation, a philanthropic fund based in Dublin, says: “We need to be honest about findings and report the successes as well as the failures in order that we can learn from our actions. As long as we learn from failings then we should not be penalised.”

There is a concern that publicising ‘failures’ in a highly competitive world will shoot those brave enough to do so in the foot as funders plump for organisations that demonstrate higher impact.

However, Brian Lamb of RNID believes that ‘warts-and-all’ publishing has given RNID greater credibility. “When we published outcomes showing we hadn’t hit targets we waited for the avalanche of negative calls – but those that came were positive. Our honesty earned us respect and credibility.”

Counting the cost of measurement

Many organisations discount measurement as being too costly and time consuming to be worth doing.

CES’s Sally Cupitt says: “Many smaller organisations are having trouble finding the resources and capacity to measure outputs and outcomes, let alone impacts. They need lighter touch measurement and evaluation systems, and appropriate support in implementing these.”

Funders too are grappling with the ‘proportionate cost’ issue. Many are happy to accept the measures being used by the organisations they fund, while others actively work to help them clearly identify more meaningful indicators within their scope.

Amy Philip, deputy director of the Pears Foundation, explains: “Impact assessment has to be proportionate – you can't undertake a quarter of a million pound research exercise for a grant of £10,000.”
 
Victoria Warne, head of grants at London’s Capital Community Foundation (CCF), says: “Gathering information can be difficult when funding very small community organisations. They repeatedly tell us, ‘we know this works and we know this makes a difference’ but often find it difficult to produce the evidence.”

CCF now asks each group funded to give a short case study stating what an individual gained as a result of being part of their project. This is submitted alongside a short report detailing their achievements and any learning they have gained through their work. “Whilst this is relatively basic, we feel this is in proportion to the size of grants we give.”

New technology is however being harnessed to help cut costs.

Jeremy Nicholls, CEO of SROI Network, points to the proliferation in ‘off the shelf’ impact measurement software programmes as a good sign.

But existing technology is also being pressed into action; Oxfam is using mobile phones to collect data from the frontline across continents, reports their Senior Advisor of Monitoring Evaluation and Accountability Dr James Stevenson, while the internet provides analytics such as ‘visitor numbers’ and other data that could also be useful.

Skills, capacity and training

A lack of capacity, skills and training is also hindering data-collection.

But even those that do have the capacity, don’t necessarily have the skills. Training will be crucial in the implementation of measuring impact, as is building strong relationships with frontline workers collecting data and engaging organisations from the ‘bottom up’.

Motivation to collect data is also highlighted as an issue, though Halton of Pilotlight disagrees: “Once people see the results and can clearly define the difference they are making; they become very enthusiastic about measurement.”

So, there are a multitude of challenges facing those measuring impact. Yet these are outweighed by the benefits it will bring to all those involved – the riches of greater efficiencies, effectiveness, profit and, ultimately, positive social and environmental change.

More effective organisations

Organisations that measure and evaluate learn a lot about themselves, about how to deliver more effectively, and as a result build expertise and knowledge that can lead to sustainable futures and policy change: ‘If you can’t measure it, you can’t manage it’, is a phrase that is increasingly popular in the non-profit world.

The Pears Foundation’s Philip says: “We believe that being able to quantify impact should increase a charity's chance of being considered experts in their field and being offered a seat at the policy table, as well as leading to improvements in their practice. Impact assessment is not just about keeping funders happy; charities themselves should want evidence that they are achieving their aims.”

Measurement is only the starting point on the road to more effective organisations and governance however. Evaluation and learning join up this virtuous circle, and is widely accepted as best practice.

Elizabeth Harper, performance director of St Mungo’s, London’s largest charity for the homeless, explains how implementing their ‘Outcomes Star’ measurement (see case study): across a wide range of services has brought both expected and unexpected benefits.

“Not only have we been able to measure and plot individual clients’ journeys but also we have been able to aggregate ‘outcomes star’ data to help us identify strengths and weaknesses in our services, and this supports our performance improvement.”

Innovation and development

Impact measurement helps organisations take calculated risks, innovate and create scalable programmes that help more people and help themselves to become more sustainable organisations.

For example, applying clearly defined key performance indicators (KPIs) as ARK does has helped it identify innovative programmes that have been scaled up. Its Antiretroviral Therapy (ART) Programme in South Africa, which ensures that mothers and children infected with HIV receive life-saving drugs, exceeded its 2010 aims by 21 months. The charity is now rolling out this programme to Mozambique with the aim of enrolling 5,000 HIV+ mothers, caregivers and children onto ART by 2010, thus preventing a further 22,500 children from becoming orphans.

Victoria Hornby, of The Sainsbury’s Family Charitable Trusts (SFCT), which helps a wide range of causes, cites the Every Child a Reader programme, part funded by its JJ Charitable Trust, as an example of how evaluation led to development. “By measuring the impact the pilot achieved in cost benefit terms, they learned that for every £1 spent, £18 would be saved in future spending. The scheme went on to win government support and is being widely implemented across the UK – it’s the ideal scenario.”

Impetus Trust’s Barone Soares adds: “Looking at potentially transformational programmes and extrapolating from the findings can ultimately lead to policy change.”

Inspiring donors

By expressing philanthropy in numbers, value, cost benefits and ratios, the belief is it will inspire more people to join in: “Knowing what you are paying for and what you will get in return will give more people confidence to invest in social change so that giving will become a bigger part of their spending plans. Philanthropy will no longer be just for the super rich,” says Martin Brookes, Chief Executive of NPC.

Mapping outcomes also helps philanthropists navigate their way round the world of need. “Having a clear idea of the way different organisations are changing the world makes it easier for donors to choose between different causes and helps the causes compete better for scarce resources,” says Brookes.

Building trust, integrity and legitimacy

In his business ethics blog, Professor Chris McDonald of Saint Mary's University in Halifax, Canada, explains how measurement is being used as a way for charities to build legitimacy: “Rather than grounding their legitimacy in terms of the number of people they represent, [they] gain legitimacy from their track record of successfully using donor money to promote their stated goals. Others lack legitimacy precisely because they're so bad at doing what they claim to do.”

Charity Navigator’s Berger adds: “I believe that an outcome-driven culture is vitally important for a charity to be at its best and to be trusted. Objective data will become more and more important for the public's perception of a charity's ongoing legitimacy.”