Outcomes Measurement

Hi Debra:

I'd like, first off, thank you for hosting this event - on a very pertinent topic, I think - and then to ask you a fairly simple question, which I believe you hint at in passing in your introduction.

I see the force of major investors requesting already successful organizations to provide metrics that support their requests for funds, but I am concerned that the closer we get to the individual entrepreneur taking an innovative stab at an intractable problem, the harder such metrics will be to provide.

Some of the people I am thinking of run very tight ships with minimal staff - I guess my question is how we make sure that funding requirements and metrics "scale" appropriately from very large and established enterprises to very small and new ones…

Thanks.

Charles,

Thanks for starting the feedback on this topic! I'm not sure your question is so simple, but I will offer my thoughts. As far as metrics requirements being appropriate to the resources available for measurement, data collection and reporting, I think it's important to understand the outcomes that lead to social change, including early-stage (perhaps the easiest to measure), intermediate and end outcomes (long-term and the most difficult to systematically measure).

That way, not every organization is measuring all outcomes possible but they are aware of the range or continuum of outcomes. Within the solid set of researched outcomes, organizations can choose the ones most appropriate to their goals and their resources. I imagine that overtime, there may be additional pressure from funders to measure more toward the long-term side of the range, but my hope is that funders might also recognize their responsibility in supporting that effort (with additional funding).

I absolutely agree about the need for good outcome measurement that shows some degree of consistency across the Social Enterprise sector. Achieving that though is, as you point out, very complex. In Scotland where I work an organsiation was recently launched called "Inspiring Scotland". This is a Venture Philanthropy body made up from a variety of organisations including financial backing from the Scottish Government. They are very committed to helping the sector to develop relevant systems of measuring impact and this is a positive sign which I'm sure will be positive in the medium term. At YMCA in Perth, where I work, we recently had a Social Return on Investment Report done on two of our programmes - a training programme for unemployed young people and a full time volunteering programme for young people. One result of the report was to show that for every £1 invested in the programmes there is a social return on investment of £4.35 and that the social added value for each partcipant was just over £14,000. I am not suggesting that measurment should be entirely moneraty becuase you have to measure value against your mission statement as well as the bottom line, but monetary measurements are very easily understandable for everyone. The Social Return on Investment is a social audit tool developed by the New Economics Foundation and well worth a look. One last comment - in all this measuring to attract effective investment we need to be careful that our sector does not become just another competition field in exactly the same way as the private sector. Being more business like and better at measuring our impact is important but so is holding on to our distinctive ethos - even if that is complex!

The cost of not evaluating impact and outcomes is higher than that of doing so, potentially undermining the results for the beneficiaries and the non-profit sector’s reputation as a whole.

The best advice is to simplify indicators and follow a small number of the most quantitative measures for the most objective and replicable assessment. Keeping indicators to a minimum would reduce cost and increase the efficiency of data collection and reporting. Ensuring that they are as quantitative as possible would result in clear, comparable measures. The smaller the task is kept, the more likely the non-profit programme managers will be able to include data collection as a routine part of their regular activities. And yet, impact evaluations tend instead to become major events with multiple criteria, often essentially qualitative, and expensive to perform and report.

To ensure a set of guideline indicators is appropriate to smaller organisations would mean the ideal authority to draw up such a list might be charity forum of some kind (trying to avoid committee consensus mediocrity), not the multi-laterals with their different criteria and huge budgets. They, too, are often guilty of weak identification of the essential deficiency to be addressed and of attempting to objectively evaluate qualitative indicators.

The problem of public reporting means minor variations in published results for any one year would take on larger significance when comparing non-profits. Performance history records and other criteria could over-ride the variations, however, to find the essential quality of the organisation, such as independent assessment of the non-profit by an agency such as ours, Development Ratings, based on quantitative measures and management interviews.

I strongly advise non-profits to keep it simple – the proof of the effectiveness of interventions is the measurable impact on the beneficiary.

Thanks.

Helen,

Thank you for joining the conversation and for your thoughtful comments! Clearly, I couldn't agree more about the importance of measuring outcomes (perhaps as a proxy for impact/effectiveness) and that in order to do that across this divergent and complex sector, the indicators need to accomplish the delicate balance of simple and meaningful. I like to recommend that organizations spend the greatest amount of time figuring out what success looks like in their given field and then determining no more than 3 key outcomes they need to accomplish to get there. Within those outcomes, there needs to be at least one way to indicate or measure how close they've come, but I do not think any more than 3 metrics are necessary.

As far as determining what to measure, how to collect data and when to report (and for who), these are central components to the research work we have done and continue to do with the Urban Institute. I am curious what you meant when referring to a "charity forum" to draw up the list of indicators. Clearly, there would need to be a neutral body, ideally well-versed in outcomes measurement and intimately aware of nonprofit challenges.

What do others think on the question of establishing key outcomes and indicators?

Debra

Hi Debra, nice to meet you on Social Edge. For those who don't know Debra and I are connected via our mutual work with tutor/mentor programs in Chicago and Debra frequently does workshops in the conferences we host. She'll be a speaker in the May 2008 conference, so if you'd like to meet her, come to the conference. The web site is http://www.tutormentorconference.org

Think you've correctly pointed out how difficult it is to define outcomes that are mutually agreed upon by multiple donors, multiple non profits and multiple stakeholders.

I'll look forward to learning from the ideas shared in this discussion.

The SROI approach mentioned by Ian has developed to put the question of what outcome indicators to use in the hands of the stakeholders, and so recognises that relevant outcome indicators will often be different even in what appears to be, on first sight, the same sector. This makes the consistency of the process important. The European and UK SROI networks (www.sroi-europe.org)have been set up in part to help people answer the questions being raised here.

Nice to see Social Edge having a robust discussion on SROI. =)

If y'all don't mind, I'd like to play a little devil's advocate first. To Helen and Debra: I agree that it would be nice to have simple indicators to simplify measurement (expensive SROI measurement systems may not be the best use of philanthropic dollars). But can you find indicators that provide a comprehensive enough picture of an organization to be meaningful? For example, what organizations should The Center for What Works be compared to and what metrics could we apply to The Center that help us compare it to other organizations? I'm skeptical of whether we can find just "3 metrics" in many of the different nonprofit sectors (from charter management organizations to workforce development organizations to programs like Social Edge) and use them to make meaningful comparisons.

Helen, I do like your idea though of some neutral intermediary creating standards - though I think ultimately it must be a collaborative effort by funders (foundations) and the organizations funded (nonprofits). Furthermore, in order for nonprofits to have an incentive to measure, the measurement must be applicable to many funders (which is why a group of foundations and not only one, have to agree to use the metric in their evaluation). The key will be getting all the relevant players, and I think on this point we can learn from the the mission related investing movement. Just how MRI investing and SRI investing more broadly is becoming more and more widespread, by virtue of certain groups like FSG and Rockefeller Philanthropy Advisors publishing more on the subject followed by conferences and events that promote dialogue on the subject with the relevant players, it would probably help to have SROI metrics follow a similar path.

To Charles' point, I agree that the use of measurement has to be very different depending on what stage a venture is in - and I think to some extent that has been addressed. Venture philanthropies, compared to traditional foundations, require different metrics at different stages and are less focused on certain metrics since they have such intimate knowledge of a nonprofit's goals and performance. Furthermore, it might not be terribly useful to always have standard metrics fo ventures in the early stage (contrary to Helen's point that the cost of not evaluating impact and outcomes is higher than that of doing so - since I can certainly think of situations, like early stage ventures, where that is not the case). So perhaps in moving forward the discussion, the metrics that we should be discussing are for organizations that have the means to create a relatively more comprehensive SROI measurement system.

Looking forward to hearing everyone's thoughts.

Tony,

Thank you for your thoughts! First, I find it interesting that this discussion has turned toward SROI, although I suppose I should not be surprised. My opening comments (and current research) is focused on the social sector as a whole and it seems to me that SROI calculations have been largely utilized by social entrepreneurs. Given this discussion is on SocialEdge, that makes sense. Having said that, although I recognize the potential benefits of leaning toward monitizing social good (policy implications, for one thing), I am hesitant to recommend the process or the results across nonprofits (NGOs). Beyond the complexity of the calculations themselves, I think there is a danger in viewing social impact from a purely economic standpoint. The inherent issues with the program to administrative overhead ratio enter. For example, a homeless shelter may be able to provide many more beds for a lower cost if they simply lower the quality of service, but this is not exactly the goal of anyone interested in effectiveness.

As far as comparing organizations effectively based on a set number of outcomes/metrics, you may be familiar with the two defining categorizing taxonomies in existance: NPC and NTEE codes. Although not perfect, they do allow for some groupings of nonprofits based on program area and focus. There are many organizations providing similar services toward similar outcomes and for sure, some are doing it better than others. At the same time, there are organizations providing quite unique services and those will be more difficult to find easy or appropriate comparisons.

I do not envision a benchmarking system to encompass the entire million plus US-based organizations, let alone the global number of NGOs for many years, if ever, but I do think there exists the potential for significant levels of comparison among many types of organizations today.

I agree with your point about funders being the main driver for standards and perhaps the most logical, however, they may not be the most feasible. We have been in touch with local funder consortiums and we hope to reach groups of funders who seem to philosophically support this concept.

I look forward to further comments on this topic!

Thanks,

Debra

To Debra, here in the UK, we have a couple of organisations that promote the exchange of experiences, ideas and information amongst their non-profit members. One has taken steps on accountability research and analysis but so far I don’t think has made inroads into the practical formulation of indicators. The specifics on this, whether using an SROI approach or otherwise, are the real problem, particularly across different sectors. I wish I had the time to do this work myself.

To Tony, my suggestion was actually for a pool of indicators to choose from rather than a ‘strictly limited number covers everything’ approach (not sure Debra meant that either?). I understand the need for comparables but there are major problems in evaluating different sectors’ outcomes, not least the campaigning-type areas where metrics highlight the inherent weaknesses of approaches that are purely advocacy. But this indicator pool should be a limited and well thought out set of options so that non-profits in similar sectors would choose the same indicators. There is little point in comparing apples and pears from the funders’ perspective, either.

Start-up/venture/smaller non-profits have the issue of accurate situation analysis and appropriate identification of approach to deal with. But this is intrinsically tied up with the ideal outcome expected and they, too, can measure baselines and set ideal targets for their chosen set of outcomes indicators, I would suggest.

I hope that clarifies things.

Some of the non-profits’ management that I meet tell me they ‘would like’ to evaluate how well they are doing but ‘haven’t got around to it yet’. To me, this is the danger and where the cost is – continuing to do work, and use donors’ funds, with no real idea of the impact. So, interesting debate, thanks.

As someone deploying a profit for social purpose model, in the belief that it would be more effective, I've wondered whether the outcome could be measured/compared in any way. By and large, most of what we do is to research opportunities for social investment and then leverage funding to do it. But then part of the social output might be the activism we engage in, or even participating here.

As an illustration my colleague and founders work to leverage a project in Russia was an investment of $4000, to attract an initial $6m in microcredit. Others managed it and made a success.

http://www.mixmarket.org/en/demand/demand.show.profile.asp?ett=2236

The next project, for the Tatars in Crimea which is currently on hold, was around the same ball park figure to leverage $40m and we learned that UNDP came in behind with a budget 200 times our cost to see it through.

The Marshall Plan which we are still hoping for progress on represents a significant scaling up. the cost of this I know, and are in the order of $30k to date.

I guess I'm asking if advocacy can be measured?

Jeff

Hi Jeff,

Thanks for joining the conversation. There are several efforts out there to measure the process and results of advocacy work. You may also be interested in some of the measurement work around community organizing:

Here are a few links to get you started:

http://www.urban.org/center/met/projects/upload/Community_Orgs.pdf

http://www.urban.org/center/met/projects/upload/Advocacy.pdf

http://www.innonet.org/?section_id=3&content_id=601

Hope that helps and best of luck to you!

Debra

Its great to see this very relevant discussion on board. When dealing with foundations - corporate foundations, community foundaitons and family foundations have very deviant views on common metrics and ROI. Community foundations do a fairly good job, and understandably so as they are using similar methodologies with similar visions.

Corporate foundations have a great potential for reaching common grounds on metrics. But the dialogue needs to be stronger, and that is where improvements are needed. There is lot of advantage there too. I don't want to divert this discussion to corporate foundations, but as far as metrics go, I encourage a look at Michael E. Porters article in the Harvard Business Review, "Strategy and Society: the Link between Competitive Advantage and Corporate SOcial Responsibility". According to Porter the fact is, prevailing approaches to corporate social responsibility (CSR) is so fragmented as to obscure the many opportunities for companies to benefit society. I refer to Ian's point about the risk of creating competition in the not-for-profit sector. Michael's article recommends a new way of looking at CSR that does not treat social welfare as a zero sum game.

As far as family foundations go, the dialogue needs to be more open. Trustees of family foundations are tied to a "what to do manual" the limits the scope of discussions with grant seekers. The dilemma with this group, is who has the authority to establish common grounds? Do we convert organizations like the philanthropic foundations of Canada into a governing body with a rotating board of directors? How do we ensure that the motives don't change with each family sitting on the board? Foundations also tend to grant on a "who-knows-who" basis, so the idea of sharing data might become cumbersome.

The call for metrics should come from funders to create an essence of accountability and efficiency.

Jyotsana,

Thank you for joining the discussion and weighing in on the question of setting the standards by which to measure effectiveness. Based on your description of the various funding groups, perhaps you would suggest that common outcomes should be driven by community foundations and United Ways, since a similar methodology may exist. Although corporate foundations and family foundations seem to pose opportunities for various reasons, the barriers you mentioned may disallow them from being early adopters.

What do others think?

Debra

In my intro, I posed a question about privacy:

Who will share what data under what circumstances? What do nonprofits feel about that degree of sharing? What data should be shared

information describing the metrics they track? The data points / metrics themselves? Both? What will be publicly accessible?

I'd love some feedback and input from nonprofit practitioners out there.

Thanks, in advance, for joining the conversation!

Debra

Dear Colleagues,

Glad to see there is a discussion of this. I am amazing by how little progress we have collectively made in this area. Although, I guess I shouldn't be surprised if the majority of our capital sources don't understand, measure, or value such efforts themselves.

Financial investors have set metrics they value. Government programs have enough paperwork as it is and foundations (with a few exceptions) seem to be led by their beneficiaries in this area, not vice-versa.

We don't need to build a complicated effort with years of university scholarship that proves impact. We could begin with simple indicators for program that we can be fairly confident in their accuracy.

I personally like the idea of using a simplified balanced scorecard approach to generate an exec sum. dashboard of the indicators most important for an organization. The best simpliest management tool I've seen is Grameen Bank's 10 questions which are answered Y/N for every one of their 5 million plus clients annually. It is a simple scoresheet. (Disclaimer: I used to work for Grameen Foundation).

Such dashboard approaches can also readily link to a United Way logic model, if you want an SROI then take things a step further and cost outputs or outcomes towards monetization. That way, you see the true cost for service and program activities. Of course, inefficient, ineffective colleagues won't be interested in such effort...

We've written some on this topic, largely for microfinance, and welcome visitors to Social Enterprise Associates' website, www.socialenterprise.net (click on Publications). We also welcome others' writings for our own information and for potential inclusion as an Emerging Topics Paper, one of our series.

Best, Drew Tulchin drew@socialenterprise.net 202-256-2692

Social Enterprise Associates - www.socialenterprise.net New Ventures Non-profit Earned Income Microfinance * Economic Development

Drew, I agree with you - it's amazing what little progress has been done in this area. While many organizations have done great work in this area, metrics of the sort we've been discussing so far have yet to be adopted. And I think it goes back to Jyotsana's point that there are barriers to adoption by foundations, since many foundations lack the proper incentives to adopt outcome-centric approaches (especially corporate and family foundations). And as Helen points out, nonprofits may fail to adopt metric measurement because they don't consider it a priority - even if it's as simple as the dashboard system Drew mentioned - though I think the dashboard idea is a good one.

So I'm curious - what can be done to get the sector to adopt better performance measurement? Debra, I'm curious what your experiences with funders are - and why you say they might not be the most feasible in adopting. Foundations do seem to be the most logical adopter of metric standards, but to date, very few have adopted robust standards and even fewer have shared their analysis with others (why, I'm not sure - I'd like to know the answer to the question you posed about privacy too!).

Also, if anyone has a more concrete example about the challenges of an effort in creating metrics across programs and organizations for a specific area (whether it's one particular NPC or NTEE code), I'd love to hear it.

Tony,

I can tell you from my experience that there are many institutional foundations that support this work and the general effort of moving toward a common outcomes/effectiveness definition PHILOSOPHICALLY. The issue seems to be finding forward-thinking and risk-taking program officers who not only understand the future benefits, but will find a way to support an effort, FINANCIALLY. I think some of the barriers and concerns are real (i.e. foundations are concerned about a top-down approach, don't think it can be done without causing harm to small nonprofits, or can't find a way to make it fit into their funding mix of minorities, children and global warming, for example). Still, there are a few who seem close to funding a multi-year effort to finally move the sector beyond the seriously problematic program/overhead ratio.

The effort to develop a common language for nonprofit effectiveness is critical to organizational and program comparison and improvement and hopefully, it's just a matter of time before the idea gains appropriate partners and financial backing.

As far as dashboards, although I respect the methodology and seeming simplicity, they can be challenging to design and even harder to gain organizational buy-in to remain populated with relevant and accurate data. We are leading a project currently for a large nonprofit with national reach and I continue to concluse that readiness and an open organizational culture are key factors to the success of any type of ongoing measurement.

As far as your call for lessons learned about metrics across programs, let's hope that someone chimes in...there are efforts out there!

Thanks again,

Debra

Ian,

I think that the most useful criteria would be be life expectancy and workforce participation in the 55-65 age range. the latter is particularly sensitive to thje wellbeing of an economy as many older workers are forced to leave the workforce on disability pensions aa a transition to retirement if the economy is not healthy.

In 2004 an undertaker's assistant in Shettleston (east Glasgow) reported that there were several young people who had suffered sudden, unexplained deaths. The standard explanation seems to be that Shettleston's diet of chips, fags and booze means that life expectancy is actually falling in this most deprived area (David Smith, 2004). A veritable Andy Capp syndrome!

In another deprived inner city area of Glasgow the chance of surviving to old age is lowest in UK. In the Calton ward of Glasgow the average life expectancy of a male is just 53.9 years; by comparison in the Gaza Strip it is 70.5. A massive 44% are on incapacity benefit (Gillan, 2006). Thinks made me wonder if this really an Andy Capp lifestyle choice scenrio or a case of social submergence (and if so whether it is work related).

One report that I tracked down subsequently ("Health and Wealth Inequalities in the UK") mentions that men living in poverty are more likely to "die prematurely from occupational disease and injuries" (George Clarke, 2006, page 40). However, they are also more likely to be socially pressured into smoking, excessive drinking and dangerous driving".

This report provides a table (page 34) that compares a rich area of Scotland with a poor area for "Adults unable to work due to disability". In the poor area the figure is 28% while in the rich area the figure is only 7%.

Regards,

David McFarlane

Sources

"You'll be lucky to live to 60 here. But it's not the third world ... it's Glasgow's East End" by David Smith (The Observer, Sunday March 14 2004) http://www.guardian.co.uk/society/2004/mar/14/medicineandhealth.lifeandhealth

"In Iraq, life expectancy is 67. Minutes from Glasgow city centre, it's 54" by Audrey Gillan (The Guardian, Saturday January 21 2006) http://209.85.173.104/search?q=cache:X3UjZoJRWYUJ:www.guardian.co.uk/society/2006/jan/21/health.politics+glasgow+%22life+expectancy%22&hl=en&ct=clnk&cd=1&gl=au&lr=lang_en

"Health and Wealth Inequalities in the UK" "Modern Studies. Social Issues: Health and Wealth Inequalities in the UK" by George Clarke [Learning and Teaching Scotland, 2006]. http://www.ltscotland.org.uk/nq/images/HealthWealthH_tcm4-338995.doc.

We have designed an approach which ensures active involvement of the community where the project is activated. The measuring yardstick, of procedures or steps which could be designed by any one of the Activists or Associates, followed by the project is - How many miilions lives will it touch. Believe me the young men of the villages have quickly understood the value of Indicators and the relevance of Accountability. Once the systems are readied, it could be available to the world at large to view and comment on the work being done by the project. And as far as money is concerned, with a little initial help from local business folks, active participation by the community through the system of what we term as Micro-Investments is gradually taking over. Each micro-investment will have an assured return which will not be less than double the amount invested. the entire fund that is generated from various activities of the project are fully banked details of which would be available to each investor. This could in turn activate the imagination of such investors for launching projects measured against the above indicator - How many millions of lives will the product / service of his/her project touch ? The entire system is being designed as an Open Book

Regards

SBR

Hi Debra

I just logged in again today and noticed that the SROI discussion has been picked up. You expressed the view that there is a danger in viewing social impact from a primarily economic stand point. I would entirely agree with you. The reality is though that funders and policy makers like to see the kind of monetary measures that SROI provides. That said what they are looking for generally is "best value" to use a Scottish Government term. Part of measuring best value is about monetary value and SROI allows us to do that kind of measurement but I think it is then our responsibility when passing the monetary information back to the funders and policy makers to point out that best value is measured partly against bottom line but also singificantly it has to be measured against the vision or mission statement of the organisation involved and that of the funder / policy maker. We need to be ready to answer the questions about financial value but also highlight the importance of measuring value by reference to the vision / mission of the organisation. That is part of the comlexity of the environment in which we operate and part of the ongoing learning experience of all those involved, funder, policy makers and social enterprises / NGOs.

Hi I have been part of producing quite a few independently audited social accounts the most recent at the end of January. Check out this review : http://www.guardian.co.uk/society/2008/feb/20/socialenterprise.jamieoliverfifteen A pdf version is available from angela.morris@fifteenrestaurant.com

Before Fifteen I led the FRC Group in northern England which pioneered social reporting and continues to win awards for the quality and transparency of its reporting.

Question is though is it worth the time, money and effort? Philosophically I feel it is. Non-profits as you lovely Americans call them receive direct and indirect subsidy (or compensation as I prefer) from the tax payer. There is surely a moral obligation to prove that this free money is being used effectively for the intended purpose. No doubt that social accounting helps everyone understand better what's going on throughout the business. at Fifteen the whole company were intrigued and sometimes appalled by what we found out out. Will it make us a better business and trainer of young people? Too early to tell but it has led to some radical re-organising of the way we do things. We considered getting into the SROI part but we didn't have the money or time. Also I'm unconvinced that any of the numbers which we could come up with would mean much really given there are no agreed benchmarks around , for example, how much it "should' cost to get a young scallywag off the streets, away from drugs and into a tax paying, settled lifestyle.

Debra asks a great question about just how honest should we be and who has access to the data. Here are some of the challenges I have faced about being too honest! If you are involved in a social enterprise which bids for public contracts (waste management, welfare provision etc) your honesty can be used against you by private sector competitors who have no obligation to transparency. I know that competitors would make a point of telling the purchaser about the ways in which we were missing the mark which they had gleaned from our published social report. For example, one of them would higlight where our diversity strategy wasn't working as a way of undermining us. I have no evidence that any of this lost us work - the way public bodies decide what they buy is far to complex and opaque - but being honest publicly is not without risk.

Also on the SROI issue, say you come up with the figure that it costs us £20,000 per young person to get them from A to B. For funders driven by volume and "value for money" (which always in my experience means the "cheapest") your honesty will count against you in the line up for cash. I could give you example after example.

The assessors of Foundations too are - with some excellent exceptions - not the sharpest tools in the box and the complexity and nuanced results one gets from social reporting confuse them! And when no-one else is doing it, the question is again why bother if you dont have to in order to get the cheque?

Over the last decade in loads of speeches to hundreds of non-profit ceo's, foundation heads, public sector procurers et al , in books, magazine articles and submissions to government, I have been going on about the importance of independently verified social reporting. I think not a penny of tax payer's moolah should be given to any organisation that doesn't come clean! But in the UK there are hardly any being done. None of the best known social enterprise brands or big household name charities. Honestly, Debra in the Uk at least you are flogging the deadest of dead horses! I can't see social accounting ever becoming a core activity. It's too complex, too expensive (if you want to do it right), and the culture of entitlement and "trust us our hearts are in the right place, dont look too close at what we do just at our intentions" too rooted.

Perhaps you have some sway with the glitterati at Skoll, Schwab and Ashoka? If they started demanding a higher standard of proof from their social entrepreneur A Listers we might start getting somewhere.

Liam,

I so appreciate your comments, especially with all of the experience you bring to social accounting. I respect your thoughts about the importance of the work of measurement and accountability and truly believe the time to move it forward is now...so, if the horse is dead, in the sprit of social entrepreneurship (and idealism), let's develop a new animal! End of pun.

I agree with your closing statements as well. It is time for the philanthropy leaders, the funders, to demand solid measurement processes from grantees. This does not necessarily mean 0-60 in the first year, requiring complex, expensive, or time-intensive methods. It does mean taking the best research frameworks and bringing them to practice in a methodological and cohesive way.

The folks who are starting new ventures are some of the best, brightest, and most caring in the world: they are perhaps the most perfect group to incorporate and refine a set of metrics to encompass the complexity of the social ills they aim to eradicate. That is NOT to say that the funders should be determining what the metrics or measurement frameworks should be, but it is time for them to fund solid research efforts combined with forward-thinking technology and training to bring the research to practice.

From there, we would at least be one step closer to a nonprofit marketplace, where donors would have comparable information and data on the effectiveness of their funding and the organizations doing the work would have some idea of how they were doing, toward their mission, beyond anecdotal evidence and erroneous financial ratios. They could be part of a powerful community of practice, to learn and improve and perhaps move this whole sector to the next level.

Debra

Ian,

I think the term "best value" may start to move toward the need for the effectiveness metrics I believe in, in addition perhaps to the monetary value. My concern is that monetizing social impact may be catchy for investors and easier to understand and for sure to quantify, but does not present a full picture of a social venture or nonprofit/NGO.

Additionally, it is interesting to me that most of the responses to this topic have been from Europe...

What do Americans think?!

Thanks for your participation,

Debra

Hi all, I believe that if we are going to take outcome measurement seriously a taxonomy not unlike the private sector will need to be created. While this will be a difficult task - if we can get consensus about what to call sectors and subsectors as well as agreed upon both program and outcome indicators are - we will be able to have great transparency as well allow capital to flow more efficiently.

I believe this work could be done by forming a coalition that agrees to work through this. This is not much different then a 10 year effort by the private sector.

Hi Debra and all,

You asked for input from the US and from nonprofits,so here goes. I think that offering tested indicators in select fields is a great way to advance evaluation efforts, especially for the many groups that don't have specialized evaluation capacity or funding. As a funder who reads many proposals from small NGOs, I see that the measures many groups propose won't provide useful information to them (or us) on what changes they are effecting. Evaluation is hard, and it would be enormously beneficial to go to a menu of indicators to consider as measures of effectiveness--the simpler the better.

That said, I think it will be a hard slog to get nonprofits to commit to using/reporting on common indicators, and I am not sure how meaningful the results will be. This is both because of the complexities in the sector--which are likely to result in mixing apples with figs--but also I suspect that what will happen is what now occurs with the required categorization of funding expenses (for 990 tax forms) by program, fundraising, or administration. People have gotten so creative with their definitions, to appear maximally cost-effective, that these portrayals and comparisons among nonprofits on such expenditures have little meaning.

Chiming in well after the fact, but here goes. I should mention that I work with a very small (300k annual revenue) organization that does community development work in Nepal (www.etc-nepal.org). I guess I should also preface by saying that I don't believe that society or nation-wide solutions for poverty and other social ills are possible. Rather, it is through disciplined, focused yet locally specific interventions that lasting social structural change is possible. Enfranchising women, for example, doesn't give them power. Working with them to develop organizational capacity and accumulate resources so that they garner respect and deference does. Enfranchisement can be done with the flick of a pen. Respect, that requires a lot of basic social work which takes time and dedication.

The problem with standardized metrics is that they presume standardized interventions. Wouldn't it be great if we could apply one basic model to all social situations? Well, it has been tried. The Green Revolution for example. Or IMF structural adjustment loans. The kindest thing to say is that these led to mixed results (even Mr. Shock Therapy, Jeffrey Sachs, has changed his mind on the latter example). The point is that across the world we are dealing with a limitless amount a variation. Intervention strategies that work in one place often don't work as well or are flat out disasters in other places.

The most effective interventions, ultimately, are the ones that work closely with the communities, provide resources and guidance for poor people to exercise some measure of power over their own socio-economic circumstances (and, I hasten to add, such power must never be reduced to a mere question of market access; such a presumption, which is the default ideology of Social Edge unfortunately, is, to put it nicely, debatable). But as soon as you get into local level exercises of power, things become idiosyncratic very fast, which militates against uniform metrics. We have learned many times over that the urge to simplify and homogenize does great damage to the world and, frankly, paints a false portrait of what is going on.

So, it seems to me, while I agree that we need rigor in evaluating what organizations do (and the IRS 990 is a joke as far as that goes), more attention needs to be paid to longer term impact, which is inherently qualitative in character, against the short term, input-output metrics brought over from business schools. I'm not arguing that we shouldn't do input-output measures (which already dominate evaluation research as it is), but rather that such measures need to be assessed within the context of the intervention site.

This means that evaluation takes time and it isn't simple. Simple analyses lead simplistic solutions which ultimately don't really serve the poor. The developing world is littered with failed technologies that were supposed to be simple solutions to intractable problems. The real problems though were social structural not technical; input-output metrics are biased towards such solutions though, so they continue to be trumpeted. Look at the spectacular and public failure of the XO laptop. So much time was spent on the hardware and hardly anything on what could actually be done with it.

What I would like to see is mid-term (3-5 years) investment by funders in interventions that approach social problems as social problems rather than as technical ones. Such mid-term funding should require benchmarks towards the end goals of a project, but the benchmarks should be defined, not through some off-the-shelf set of metrics, but in terms of the organization's mission and capacity and the specific context of the intervention (that is, the local communities). I think it would be possible to have some shared repository of evaluative practices (indicators, data collection methods, etc.) and the onus would be on the organizations to work those into a reasonable approach to assessing the specific communities they're working with.

What would funders have to do? Well, pay for the staff time for one thing. They need to get away from being scared to pay people for doing the hard work intervention AND measurement. If I get one more foundation telling me I have to jump through a million hoops and, by the way, they won't pay for any staff time, I'll scream. Oh, wait, I'm already screaming and it doesn't help. Oh well. But they would also have to become more sophisticated in how they think about social change. Short term, technical interventions have a limited range of effective applications. If social change (which encompasses buzzphrases such as community participation, empowerment, bottom-up development and so on) and not mere market growth, is truly the goal, then they have to be willing to accept that a world of diversity requires, diverse and, yes, idiosyncratic solutions. I would like to see funders work with us, instead of trying to make us work for them.

I realize that this must sound grating to most here. I am not advocating a laissez faire NGO world. But I am advocating that were fixate less on the get rich quick schemes of the MBAs, which inevitably, and falsely, reduce complex social situations to inputs and outputs, and more on developing longer term strategies to measure progress toward contextually defined socially agreed upon goals in local communities. I do think we can work toward a common language for evaluation but primarily at the level of inputs. Outcomes/impact require something more nuanced than a cookie cutter.