When profit is not available it becomes difficult to measure effectiveness, so most charities don’t bother; at best they report on their outputs. But that can be highly misleading.
For example, a charity might tell you that they supply two additional teachers to a school in Africa. That sounds like a good project but perhaps it isn’t. It might encourage a cash restrained government to reallocate some of the existing teachers they supply to other schools, or perhaps the more urgent need is better infrastructure, not more staff. Unless you measure Impact you can’t tell what works best. That’s a problem for donors. Arguably it’s an even bigger issue for management of charities.
Why is there so little impact reporting?
Impact measurement is still in a formative phase and only used by a small minority of charities because:
- In many sectors impact is hard to measure, in some it is impossible
- The relevant time period can often be years
- Other factors may have contributed to the impact
- It involves cost and time
Despite these issues, impact measurement is gradually becoming accepted as the most reliable basis for evaluating interventions.
What to look for
In an ideal scenario you will be able to see a report produced by a respected independent organisation. Ideally this should include the results of a Randomised Control Trial (RCT) that will compare results of two cohorts of beneficiaries, one with and one without the intervention. However, these are still a rarity. In all cases what you should be trying to identify are the outcomes from the programme, not the outputs.
For an example of a detailed cost/benefit analysis look at this one produced by GiveWell.
If you need guidance on producing an Impact Report check out this work by New Philanthropy Capital (NPC).
What is Theory of Change?
ToC is a relatively new concept that helps to define how a programme or intervention will achieve the desired result. It involves a careful examination of each step in the process and of the evidence supporting its effectiveness. It can be a useful indicator of how well an organisation has considered the implications of its activities. The absence of one may indicate a lack of rigour in the organisation.
A ToC usually starts by considering:
- the Outcomes that you are seeking and these should be quite specific,
- the Outputs that will be needed to achieve this,
- the Inputs required to make the outputs happen
Let’s assume that the main objective is to improve the life chances of children by ensuring they are better educated. Following a consultation with all of the stakeholders (especially the locals) you would list all of the factors that could contribute to the objective, such as:
- increasing the number of pupils attending, especially in the final years
- improving the quality of teaching
- improving the facilities
That would lead to an action plan of Inputs, which might look like this:
- build three new classrooms over the next four years,
- pay for additional teachers commencing in two years,
- build a dormitory for girls in the second year
- collect rain water to supply freshwater,
- Install solar panels and satellite aerial to provide internet access
- plant trees to provide shade.
From this plan, budgeted costs can be established together with key indicators of progress. In the short run those indicators are likely to be mainly the outputs but as times goes by they will be replaced by outcomes such as final year pupil numbers and exam results.
If you are interested in learning more about Theory of Change there is a considerable amount of useful content at: The Center for Theory of Change.
