Charity impact evaluation

What do we mean by ‘impact’?

Impact measures are useful information for donors to understand how their contribution has made a difference.  In this article, we have created ‘AnyService’ as an example to illustrate the range of information which can be provided through evaluation and performance management data on an organisation’s services.

Information on how and what AnyService is intending to achieve can be collected. This is often referred to as a logic model or theory of change. Input and output data can be included, for example the services offered and how they are staffed, the number of people who use AnyService and the sequencing of different parts of the service.

Information can also be collected on the wider context in which AnyService operates, for example where it is located, the people who use the service and how it links to other services in the area. If the data is available, the outcomes for people who use AnyService can be monitored, both the short-term consequences from using the service and longer-term impacts.

Finally, understanding how much it costs to deliver AnyService, combined with data on the impact it is having, can inform an assessment of the value for money it offers.

The benefits of measuring performance and impact

There are two stages when performance and impact measurements can benefit donors.

The first stage is when funds are being allocated. Anecdotal evidence, visits to services and the stories of service users will undoubtedly remain powerful influences on giving. Evaluation or performance data can provide an additional source of information to support or question these experiences.  The existing evidence base can be reviewed, through a transparent and systematic rapid evidence assessment, to identify what is already known about effectiveness. It is not uncommon to have no published evidence of impact; however this does not mean a particular type of service or intervention is necessarily ineffective. This is a particularly important point for philanthropic giving, which often seeks to support and incubate new and innovative work, which by its very nature will not have an established evidence base. Qualitative evidence, for example from an interview programme with experts in the area, can also be a useful contribution to decision-making.

The second stage is once funds have been allocated. Measuring performance and impact offers benefits on a number of levels:

• For donors: to assess the impact of their giving and provide information for the wider donor community.
• For the recipient organisation: to inform their internal development and growth. Evidence of impact can also be used to support further fundraising and bids for service contracts.
• For the end beneficiaries: measuring impact is an important check that the individuals using a service are benefiting.

Key principles of impact evaluation

Before exploring the detail of how to measure impact, three principles should always be borne in mind, regardless of the method employed.

1. Attribution risk: It is important to assess and clearly report the extent to which the subject of the evaluation, the service or intervention, is responsible for the impact found. For example, if AnyService supports people to find jobs, to what extent does can we be sure that the reported impact on the employment rate among the people they support is to do with AnyService, and that it would not have happened anyway? 
2. Passion for improvement: Impact evaluation can identify when things do not work and why, thereby providing useful information to improve services or funding decisions. It is important to secure the support of all stakeholders for a process which is not about identifying failure but about finding ways to improve the services and interventions which they are passionate about.
3. Adding value: Donors can add value through impact evaluation by building on existing evidence and ensuring that new impact evaluation includes the contextual information needed for results to inform practical decisions about replication and scaling up. Collaboration among donors to develop evidence on a particular area can ensure that efforts are not duplicated and that charities providing data do not have unnecessary burdens on their time and processes.

How to measure impact

There is no one method for measuring impact. The final choice will need to take into account:

• The purpose of the research 
• The data which is available or can be accessed 
• The resources and timescales for the research 
• The stage of implementation of the service to be evaluated

The research will also need to be informed by an understanding of what it is that the service is trying to achieve, that is, their intended outcomes. It is important to be clear about the purpose of the research at the outset, before commissioning.

A brief overview of some options is provided below.

Experimental or quasi-experimental impact evaluation

While these research designs are regarded as the most rigorous or ‘scientific’ in terms of demonstrating the impact of an intervention, they are resource intensive and not always appropriate. For example, the objectives of ‘developmental’ programmes typically cannot be sufficiently specified in advance, so intended outcomes may not be immediately identifiable.

Individual tracer studies

Following up a cohort of service users to see what happens in their lives and relating these to the inputs of a service. Picking out a few exemplars from this cohort to use as individual 'stories' is also possible.

Self evaluation / performance monitoring

Encouraging the project staff to review, keep records, set targets, and analyse what happens with some guidance supports self learning and continuous improvement, while also providing a system which is embedded and owned by the organisation and is not time limited.

Case studies

Well designed case studies can provide in-depth qualitative data, which can be very useful in understanding whether, how and why an intervention is working from the perspectives of those involved.

Theory-based evaluation

This approach is useful when the true impacts extend beyond the timescales available to apply the methods suggested above and/or when our knowledge base only supports general hypotheses of likely change. The outcome of such models is

a) direction of travel (what needs to be happening at AnyService to indicate that true impacts will happen in the future); and
b) distance travelled (how far AnyService has gone in the relevant direction).

Next steps: practical information to get started

A number of questions need to be considered to get started on impact evaluation:

• What do we want to know about, for what purpose and for what audience?
• What evidence already exists?
• What data is available?
• What resources are available?
• When do we need to have results by?
• Who else wants to know about this? Can we work together?
• What support do we need?

Getting answers to some of these questions may require support, for example through a rapid evidence assessment or data gap analysis. Once these questions have been answered it is possible to commission or undertake the most appropriate research with a clear expectation of what it can deliver.

Recommended resources

• Knowledge Briefings">Matrix Knowledge briefings
• Policy Hub Magenta Book, ‘Guidance Notes on Policy Evaluation’
• UK Evaluation Society
• ‘Think Research: Using Research Evidence to inform service development for vulnerable groups’: Cabinet Office

_________________________________