SROI limitation findings should not hold back learning to use the tool properly
January 2011
Social Return on Investment tool (SROI) as a tool to demonstrate impact is still in its infancy with much needing to be done to educate organisations on how to use it properly.
The Third Sector Research Centre published a report on 10 January highlighting some of the challenges associated with using SROI to evaluate the impact of third sector organisations [i]. With the huge growth of interest in measuring social impact, this research identifies how new evaluation approaches such as this might be made more rigorous and trustworthy. ‘The ambitions and challenges of SROI’ was written by Dr Malin Arvidson, Professor Fergus Lyon, Professor Stephen McKay and Dr Domenico Moro.
What is SROI?
Social Return on Investment is one approach being developed to measure the social, environmental and economic impact of third sector activities. Whilst there are a range of approaches, SROI has received particular attention and is being promoted by some third sector organisations, public and private bodies. It involves attributing a financial value to inputs and outcomes, and calculating these as a ratio. For example, if the SROI is 3:1, it means that every £ invested in the organisation generates a social value worth £3.
The report notes that, while aiming for rigour, the method leaves a great deal of space for personal judgement. This makes it possible to inflate the value created and may also lead to misunderstandings about how to interpret the SROI ratio. Although auditing tools and procedures to help standardise the way SROI is calculated have been developed, attention needs to be paid to how the results of SROI are used.
Lack of comparators
One of the main limitations of SROI, as with other types of evaluation, is that it is difficult to compare results between organisations. While recent SROI guidance warns against using it for comparative purposes, evidence shows that this is the main motivation for many organisations to use SROI. There are considerable risks to the reputation of the method as it is increasingly used as a comparative tool for organisations competing for contracts or philanthropic funds.
The process of producing an SROI ratio is specific to every organisation. The analysis that accompanies the ratio makes it possible to see some of the choices that have been made, about what to measure or how to value an impact for example. However, there is a tendency for the focus to be placed on the overall SROI ratio. While it aims to help organisations understand and manage the outcomes they create, SROI is better at demonstrating impact than explaining how or why these impacts occur. As such, it may not help organisations to replicate and improve interventions.
Other limitations
There are also obvious problems with attributing monetary values to outcomes. How, for example, the report argues, do you accurately measure improvements in quality of life, or feelings? SROI seeks to value both the benefit to the wider economy and the individual. But while we may be able to calculate, for example, the average cost of treating depression on the NHS, valuing personal benefit in monetary terms may be more complex.
Many of these challenges are by no means exclusive to the SROI model of evaluation, but its growing popularity makes it important to investigate how SROI will deal with them. As a framework, SROI provides useful guidelines to help organisations collect evidence and map their impact. But there are areas where more research is required, for example, on how stakeholders are involved in determining outcome measures and on the use of procedures that aim to standardise the way SROI is calculated.
Malin Arvidson from TSRC said ‘SROI is likely to become an increasingly dominant approach to measuring the impact of voluntary organisations due to support from various parts of the public sector. But analysis of SROI raises important questions about why and how we measure impact. SROI arguably provides a powerful tool to help organisations illustrate the value they create in a language that those outside the sector understand. But we need to pay attention to how results are used, especially as there is a tendency to adopt it as a comparative tool. Furthermore, if it doesn’t help us to understand why change happens then it may not help organisations to improve or replicate interventions.’
Not just a magic recipe
Jim Clifford, head of charity advisory at Baker Tilly, as well as undertaking research into the interaction of social impact, brand and conventional valuation at Cass Business School believes that if the research is constructed properly, SROI is still a useful tool. He told Caritas:
“This commentary recognises that the conceptual basis of SROI is as economic cost-benefit analysis. As such it is part of a long tradition of analyses of the effects of policies and social interventions. With the branding that the New Economics Foundation[ii] and others have endeavoured to put around SROI, it is easy to take its process as a magic recipe for value, but it is more than that. Moving above the process-driven presentation of some of the published guides, and unfortunately some of the examples of its use, we find the ‘incredibly useful tool’ that it really can be.
Don’t throw the baby out with the bathwater
It suffers from instances of poor application, as indeed do some commercial valuation methodologies, and it is upon this aspect that Arvidson and his co-authors focus. I agree with them: this is an issue, and one that is at risk of devaluing a sound and sensible methodology that can add much to our view from outside or inside an organisation. They identify the poor use of financial proxies, deriving a ratio which gives a foundation for uninformed comparisons between organisations, and inadequate research as to how outcomes arise from the work done. However these are not faults of a poor approach, but poorly constructed research. It works well if applied carefully and intelligently: don’t throw the baby out with the bathwater.”
Comments
There are no comments on this article. Be the first to comment.