There is little consensus on how best to approach the creation of a single, commonly endorsed set of metrics by which to assess the performance of non-financial aspects of both organizations and funds. Even if we can measure these aspects, it does not mean we can accurately value what has been measured. Beyond the challenge of assessing the social value of capital and returns is the challenge of creating effective frameworks for assessing how this value affects the financial performance of firms. Suffice it to say that as the dust begins to settle regarding how to count, the discussions must then begin with regard to how to value what it is we’ve tracked through our efforts.

In the Blended Value Map, we discussed issues such as the lack of consistently effective approaches and tools for measuring and reporting social value, the lack of consensus regarding the focus of measurement and the issue of disbelief and lack of confidence or trust in what is measured. For this session, we have invited several special guests to speak about these issues and give us their perspectives on how they might be addressed. The event is being moderated by David Martin and Sara Olsen.

To spark the discussion, we recommend you read the following section from the Blended Value Map Paper: Click here to read: “Ideas for Improving Social Measurement and Performance Metrics.”

For a quick overview of the Blended Value concept, check out www.blendedvalue.org or the special interview by Mario Morino. Click here to read the interview.

To begin participating in this event now, just post a message introducing yourself and share with us your interest in the topic. Here are some questions to consider when posting a message in the blue box below:

Sara Olsen - May 4, 2004 10:51 am (#2 Total: 78)
Founding Partner, SVT Consulting


Hello! I'm honored to be co-moderating this discussion as the founder of SVT, which measures and does valuation of social impact for investors and companies on both sides of the for-profit/nonprofit line.

I would like to throw out there a straw man "theory of change" for the social metrics field: by developing metrics and data reporting frameworks that inform enterprise managers about the social value they create on a daily basis, meaningful performance assessment frameworks for investors/funders, and standards for the field, will follow.

The converse of this: if funders determine the units of measurement and reporting standards without regard for whether these units are of value to managers on the ground, social performance measurement will never thrive.

peter scholten - May 4, 2004 12:02 pm (#3 Total: 78)
scholten&franssen


hello all! I'm happy to participate in this discussion from the European continent (more specific: the Netherlands) as a special guest. Regarding the third question mentioned above, and the remarks made by Sara and David: most things only develop if there is a strong incentive (let's just assume: money) and/or people are forced (by government-rules). So we need a marketing strategy to create a market for our product. We could do this in different ways. 1. Get the governments to demand for performance measurement analyses as a condition for subsidies (this method is more likely to be useful in countries like the Netherlands and Europe) 2. make investors and enterprises (both for-profit and nonprofit) interested in our tools and products. I don't think things will develop as easy as Sara says. Assessment frameworks will need to be 'sold' to the market and we should act like entrepreneurs selling this product.

And since we're not alone, competing consultants will develop their own copyrighted tools (like ACCESS ??) if there really turns out to be a profitable market for these products.

So the big question is: how can we get as much as possible relevant support (gvrnmnt,major funds, enterprises) for the framework we're building to get into the position of claiming some kind of standard.

Here is (a. o.) were Jed's silo-thing can be helpfull; if we can develop this theory towards communication between the actors in the silo's.

kford - May 4, 2004 12:03 pm (#4 Total: 78)
Community Reinvestment Fund


Hi. I am a newly registered member, casting about for knowledge on thie issue of how to measure success. My job here at the Community Reinvestment Fund is to measure the effect our efforts have on the communities we serve and, most importantly, to engage in activities to ensure that we do better. That is, part of my assignment here is to dig into a project that CRF funds and maximize its community impact.

This is a new task for me. I've been in this position for a few months and, while I have acquired all manner of reading materials, I'm still looking for something beyond number of jobs and number of affordable housing units.

We'll see what I can learn here.

kford - May 4, 2004 12:17 pm (#5 Total: 78)
Community Reinvestment Fund


One of the considerations I am struggling with is the issue of how much of a burden to place on our partners. My organization, the Community Reinvestment Fund, buys loans from local, non-profit and governmental lenders. Our local lending partners are, in one very real sense, our customers. They range in size from small development funds on an Indian reservation to major city government agencies. The sophistication and capacity of these lenders varies widely. But not their tolerance for being hassled.

So, if I bug them too much for information on loan recipients and impacts of their lending, they can just blow us off. Then there's no community impact. So finding a balance or finding less intrusive ways to gather information is crucial.

sjordan - May 4, 2004 1:09 pm (#6 Total: 78)
US Chamber of Commerce CCC


Thanks for kicking this conversation off. I'm also pleased to be able to participate. At this point though, it's easier to ask questions than to provide answers:

(1) Can we specify which silos we are particularly interested in measuring? It's a lot easier to talk about adding value in terms of education, health, housing, transportation services and other Generally Accepted Social Objectives (GASOs), than it is talking about BV in the abstract. In this context, it might also be useful to talk about developing a GASO architecture & classification system and developing a Quality of Life index based on it, but I think we'll be helped at this stage by talking as concretely as possible about specific cases.

(2) Can we recognize or stipulate that we are not likely to get absolute answers, but that we will most likely get consensus or "generally accepted" answers -- assuming this discussion parallels the development of the accounting profession as a whole?

(3) Should we think in terms of "monetizing" all of the value? Or should we accept that there are benefits that we can't adequately monetize or foresee (eg psychological satisfaction, or the long-term effects of certain current practices)?

And (4) picking up on David's point about accessibility, maybe we should stipulate that different people are going to have different values, and that we should strive toward a measurement architecture that can be used by widely different schools of thought. In this case, maybe we're looking at not one statement of value, but several interlocking ones. To build on the findings from Cathy Clark, her research team, and some of the other East Coast folks -- maybe we're looking at three statements (if not more) to capture process metrics, impact metrics, and "monetized" metrics, and maybe even a "QL" contribution.

Sidebar note: It would be great if we could have a tally sheet of:

-- issues raised by this discussion, -- issues resolved (by definition, we're optimists) -- next steps required to make progress on unresolved issues, and -- volunteers willing to work on them.

I look forward to seeing how this discussion unfolds.

jem nicholls - May 4, 2004 1:30 pm (#7 Total: 78)
New Economics Foundation


Hello and looking forward to joining the discussion - with suggestions to a couple of the questions we have been posed and a couple of responses to the submissions so far.

Firstly I think that whilst there are costs in measuring social value, standard measures that let us start to compare these values is a source of ideas, innovation and competitive advantage - an investment. The costs should be covered by revenues.

Since organisations have different capacities, values and access to opportunities for investment it is more than likely that they will take different paths to measuring value. I would agree with the suggestion that a measurement structure could have interlocking statements of value. Finding a way to balance demands for comparability between organisation's measurements of social value and the need for accessibility will be critical.

But I think its important we don't expect too much at a relatively early stage in the development of this field. Relatively early that is compared to financial accounting, where there are still debates about measures of value and where issues of comparability of measures were being debated over 100 years ago. What is measurable will increase as specific tools are developed and this is alright so long as organisaitons are clear about what is and what is not included.

David Bonbright - May 4, 2004 1:52 pm (#8 Total: 78)
Managing Partner, Keystone


OK, we are off and running. I agree with Sara that essentially metrics and reporting systems must drive performance management. Let's call this the internal dimension. But I also agree with Peter that the internal incentives alone are not likely to be enough. External incentives -- like access to funding and legal compliance -- also need to be lined up. At ACCESS, we have said that the key may be in manifesting the mutually reinforcing links between performance, resourcing and accountability. We see the challenge of building an entirely new informational basis for tracking social value creation as one of a solving a simultaneous equation.

sjordan - May 4, 2004 2:22 pm (#9 Total: 78)
US Chamber of Commerce CCC


In the Capital Challenge conversation, I put in a little piece on funder motivations -- to recap -- different investors are looking to (a) recoup and gain additional resources for themselves, (b) find other donors willing to match their gifts to the recipient -- ie get more inputs working toward the desired social outcome, (c) create rewards to "solve" the social issue that they are investing in (could lend itself to a closed-end mutual fund model), (d) create annuities for intransigent issues that can't be "solved" -- eg endowment for a university or continued support for a mentally ill patient, and (e) receive nothing at all (the charity mentality), and I'm sure there are a few other motivations out there -- but I agree that perhaps we need to look at creating markets based on donor/contributor expectations as well as, or rather than just focusing on social needs and desired social impacts.

David Martin - May 4, 2004 3:53 pm (#10 Total: 78)
Researching & designing financial services for low-to-moderate income Americans


Hello, all. I'm honored to be co-moderating this event. I look forward to getting to know you all better during this event.

I've got a couple of thoughts about our topic thus far:

WHY METRICS? We all have an answer or two. Why do we need blended value metrics? In other words, which stakeholders/ constituencies will derive value from metrics? Thus far, we've identified four: (1) Management--as David B said, for "internal" use. This is very much the case in corporate business--where I reside. (2) Investors/Foundations--As "proof" that there hard earned dollars are being well-utilized. (3) Government--compliance issues, 501c3 status audits, etc. (4) Community--accountibility towards purpose (5) ??

If metrics will serve different purposes for the different stakeholder groups (and I believe they will), should we think about designing them with this in mind?

WHAT IS MEASURED? The money part is easy. Like Jem said (see post # 7), we've got 100 years worth of theory.

How about the smile factor? Jed Emerson had a great quote in one of his speaches given at Stanford University last year. He spoke about counting the number of smiles that resulted from a social enterprise's efforts. His point (I think) was that blended value can take on almost any form--some forms being much harder to measure than others.

PERSUASIVENESS AND CREDIBILITY Ultimately, I believe we want to establish industry-wide guidelines similar to what I believe Steven Jordan is suggesting in post #9. Generally Accepted Social Objectives sounds an awful lot like Generally Accepted Accounting Standards, which are codified accounting standards for businesses in the US. Perhaps we could start with guidelines (I assume ACCESS is such a set of guidelines). The bottom line, in my opinion, is that the guidelines/ standards for blended value metrics must be persuasive outside of the social enterprise realm and must be based, at least partially, on leading business metrics theory. Can we quantify everything? Probably not. But that doesn't stop us from developing good qualitative measures that serve the same purpose.

Have fun and happy typing,

David

Sheila Bonini - May 4, 2004 4:25 pm (#11 Total: 78)
Hewlett Foundation


We are off to a great start here and I'm excited to read on over the coming weeks! A couple interesting issues have already been raised that I want to comment on.

David asks - why metrics and who derives value from them?

I would posit that we need better metrics to provide managers (both for-profit and nonprofit) with information to create more effective and efficient strategies to pursue social and environmental value creation. We also need better metrics so that investors (market-rate, philanthropic and government) can invest their capital more efficiently to drive greater social and environmental value creation. Better metrics will also help us pursue more effective policy, regulation and tax strategies. Finally, better metrics would help us to understand the interplay between economic value creation and social and environmental value creation which in turn give investors a better array of investment choices.

For me the issue is how do we get there and what kind of metrics are we talking about - which is where the what is measured? question is relevant.

Ultimately we need to move from the big picture and turn to the specifics. When we talk about metrics, we need to talk about the specific area - education, environment, population, etc.. Then we need to understand the goals for that area and develop a robust strategy or theory of change for how to get there. Then metrics branch into two areas - one has to do with proving that the underlying theory of change is effective in reaching our goals (typically research in the social science areas) and the other has to do with a specific organization's effectiveness in pursuing that theory of change. When we overlay the investors point of view, we tend to branch off into yet another area.

I would love to hear others' perspectives - what kind of metrics should we be talking about?

Sheila

Eli - May 4, 2004 4:35 pm (#12 Total: 78)
GlobalGiving


hi folks. enjoying the start of good conversation. Regarding promoting consistent approaches to measuring social value -- there is be a third motivation (in addition to return for investors and return for constituents) -- that is enabling mechanisms for participation in new value chains. I'm looking at my own supply chain at GlobalGiving, which is dependent on partnering organizations who supply projects. By working with numerous partners, globalgiving is able to offer a wide geographic and thematic scope. But in representing projects to donors (through our Web site or otherwise), the lack of consistent standards raises questions, e.g. why does it cost one group 3X more per unit for what is seemingly the same programming?

I think there will be increasing numbers of organizations dependent on similar value chains and one motivation for consistent approaches for measuring social value could be the ability to participate in (or benefit from) them.

Cynthia Gair - May 4, 2004 5:24 pm (#13 Total: 78)
REDF, Director, Portfolio and Field Advancement (www.redf.org)


Hello all! Wow, I step out of the office for a day and the conversation’s already off to a roaring start!

Since 1997 REDF has developed ways to measure the outcome of the work we support. We’ve encountered numerous challenges in all aspects of what to measure and how to measure it. We’ve devised creative solutions to many of the challenges and are still looking for solutions to others. Descriptions of our work in measurement of social outcomes, as well as in other areas of social purpose enterprise and engaged philanthropy can be found at www.redf.org. I’ll describe what we do in a little more detail below – easily skippable for those of you who want just want to focus on the core of this conversation.

A number of interesting points have been raised here. Sheila mentioned something that has come up again and again at REDF: if we’re trying to assess degrees of success in accomplishing a social mission, we need to know what an organization is trying to accomplish and how it fits in to their theory of change. What do they want to change and how do they propose to change it? Once that’s clear we can decide a) do we believe in this theory of change enough to support it? and then b) how well is this particular organization (or project) meeting its goals.

In the case of our portfolio of nonprofits, each operates with a theory that its supported employment in social purpose enterprises will help people with severe barriers to employment change their lives for the better.

It has been possible, though not without challenge, to design a consistent approach to data gathering and analysis across organizations within our portfolio. It would be much more challenging to do so across organizations in different fields, with different missions and widely varying theories of change.I’m eager to hear others’ ideas on how this might work!

(And below, a little more about what we do and what we measure) REDF generates and distributes resources to help people move out of poverty. We partner with a portfolio of Bay Area nonprofit organizations. Each of the organizations in our portfolio runs one or more social purpose enterprises that create jobs and training for people with a wide mix of barriers to employment, including homelessness, mental illness, developmental disabilities, substance abuse, and criminal conviction. REDF assists our portfolio organizations in a variety of ways, most notably by providing financing for organizational and enterprise infrastructure, access to additional funds for capital expenses, strategic business assistance, organizational development support, and access to business networking opportunities, social outcome measurement and technological tools and training. We assess the effectiveness of our portfolio’s social enterprises on the basis of a combination of their financial and social outcomes.

We created a Social Return on Investment (SROI) model that has influenced many others to examine ways to gauge their effectiveness and create their own measurement tools. As we implemented SROI, we realized that most nonprofits lack the funding and infrastructure required to track a key element in assessing return: the amount of change or “social outcome” that has been realized. To address this lack and to broaden our analysis of value beyond a community’s financial savings, we developed OASIS, a customized social management information system housed within the nonprofits we serve.

This system has allowed our portfolio nonprofits (and us) to measure the changes in the lives of the 2000+ disadvantaged individuals they they have employed, almost 80% of whom are still employed two years after hire. Our extensive measurement efforts have enabled us to know with certainty that employees of REDF Portfolio organizations are making dramatic improvements in their lives.

REDF continues to track social outcomes and refine our measurement practices as we and our portfolio social enterprises mature.

Sara Olsen - May 4, 2004 5:29 pm (#14 Total: 78)
Founding Partner, SVT Consulting


I think Eli nailed one of the fundamental reasons metrics are needed: to answer why it costs one organization 3x more than another per unit for seemingly the same thing.

As soon as we decide to answer that question, the need for standards or protocols, and probably external auditing of results, comes into focus-- because although on the surface it might seem like the organization that costs 3x as much is just not as good, the reality may be that: a) one organization may be doing different cost accounting or other procedural techniques than the other, or b) the "difficulty of the dive" may be greater in one context than another, to use a phrase from Kim Smith of New Schools-- meaning that although the programming seems the same, to achieve a single unit of outcome for one set of constituents (say recovering drug addicts) may be a lot harder than a unit with another group (say typical high school seniors). There may be other reasons.

I was recently made aware of the way standards are developed in the IT community. Every time there is a new technology (say electricity, or wireless internet) that a lot of different players could develop products for (like lamps, or palm pilots that can get online wirelessly), a standard needs to be developed (like standard voltage or a 3-pronged wall socket, and I dunno what for wireless internet) that dictates how the product should be configured so it can interact with the technology (the lamp cord and plug). It's not FASB's equivalent that establishes what the standard should be-- it's self-organizing bodies of interested players, who either set up nonprofits to facilitate the discussions and debates that lead ultimately to agreement, or forprofits funded by industry heavy hitters who want to advance their version of the standard to be. This is what is starting to happen in our field.

Stephen Jordan, I love you idea that someone keep a record of the issues resolved, points for further exploration, etc. Does this mean you've volunteered?

jeffreysteen - May 4, 2004 7:14 pm (#15 Total: 78)
The Barneson Group / Stanford University


The discussion of standards is an interesting one. In IT, standards get developed in one of two ways. De jure standards (like you describe) are those set by a governing body of some type, These bodies can be non-profit industry associations, like the IEEE, government bodies, like the FCC, or multi-laterals, like the UN or WTO. Television and radio frequencies are an example of a government enforced de jure standard. Wireless ethernet (IEEE 802.11x BTW) is an example of de jure standards set by a non-profit industry association.

On the other hand are de facto standards. De facto standards are those that evolve via market forces. Microsoft operating systems are the de facto standard for PC operating systems, despite no governing body issuing an order.

So that leads to another interesting question. Should the market determine the standards for measuing social value, or does it require intervention by an overseer? If it does require intervention, the key questions are ones of accpetance and credibility. If the organization setting the standards is not seen as credible and accepted as a standard setter by a significant majority of the participants, competing standards will be set by different bodies (this happened early on with the European engineering body and the IEEE on wireless ethernet).

The IEEE, while a voluntary membership organization, has a good model for acceptance and credibility. They are inclusive, democratic (Robert's Rules of Order being run by engineers!) and exhaustive in their approach. The outcomes, therefore, tend to be accepted by the participants (for the most part) on a global basis. Contrast that electricity standards, which are generally accepted in the US, but are different around the globe as any frequent traveller can tell by the number of adapters they need to carry.

Of course, it might make sense to simply allow the market to dictate the standards it needs in terms of measurement. Those different types of participants, mentioned in another thread of the discussion might want to compete with each other in order to sell their ideas on measurement and become the Microsoft of measuring social value... I don't advocate for either side yet, but it is certainly worth hearing others' persepctives on the issue.

Paul Epstein - May 4, 2004 9:57 pm (#16 Total: 78)
Epstein & Fass Associates


From reading the 1st 15 posts of this conversation, I see some great expertise represented and valuable ideas to get the dialog going, but only a brief mention or two in passing to the people who should have at least as important a role--perhaps more important--than us "experts," or various investors or government or "industry" regulators or standard setters.

I mean the citizens (or residents & other stakeholders) of the communities that are supposed to benefit from the social value created. What good is having a whole set of metrics if none of them matter to the people in communities whose outcomes we expect to improve?

This is my first post to the "blened value" discussion, so I'll introduce myself a bit. I do consulting mainly to North American governments (at all levels) and non-profits, and occasionally to the UN, and I do some foundation funded research. Almost all of my work involves some form of public, non-profit, or community "performance measurement." I've got 30+ years experience in this field, but in the last 7 years part of my focus (in part with Sloan Foundation grants, and in part bootstrapped with like-minded colleagues) has been on the nexus of citizen engagement and community results measurement. I'm lead author of a forthcoming book on the subject (Jossey-Bass, expected late '04 or early '05) with the working title RESULTS THAT MATTER.

On a related topic I'll probably say more about later, one of my clients is the Governmental Accounting Standards Board (GASB)--FASB's younger sister--which sets U.S. state & local gov't accounting standards, and which has been working for years toward potentially developing reporting standards for performance measures. I'm a conculting member of the GASB's performance reporting research team. To bring the GASB's work back to my original point, a couple of years ago the GASB held discussion groups of active citizens across the country on what they'd want to see in public performance reports by state & local gov'ts. A major finding in the Research Report on those discussions was: "Perhaps the strongest common theme across the discussion groups was participants' desire to have citizens involved in determining what performance information to report." (Fountain, Campbell, Epstein, & Robinson, REPORT ON THE GASB CITIZEN DISCUSSION GROUPS ON PERFORMANCE REPORTING, Norwalk, CT, USA: The GASB, pp. v & 12, available at www.seagov.org )

From a non-profit/community perspective, I'll end this post with a quote from Tom Kingsley, Director of the Urban Institute's National Neighborhood Indicators Partnership, who was asked about the role of experts vs. citizens or residents of communities who were supposed to benefit from having community indicator data available. Tom answered, "The experts should make the community their CEO."

jeffreysteen - May 4, 2004 10:03 pm (#17 Total: 78)
The Barneson Group / Stanford University


Thanks for taking this conversation in the direction I was hoping it would go from my post on de jure and de facto standards. My point in raising de facto standards was to try to show there are alternatives to forcing things from above.

You did a much better job of stating it than I did! Thanks.

Paul Epstein - May 4, 2004 10:32 pm (#18 Total: 78)
Epstein & Fass Associates


Glad you agree. I tend to agree with those who say multiple perspectives are needed in determining measures or standards, including investors, government, & service providers. But I really believe in the centrality of community citizens' role in determining what is measured and targeted for improvement. For that to work, we need to have some comfort level with a degree of local (or regional) variation of at least part of what is measured, as values and priorities will vary from community to community.

BTW, when I say "citizen," I mean citizen in 1/2 the classical Athenian sense of anyone who takes responsibility to be engaged in their community. I do NOT mean the other, restrictive sense of someone of a given legal status. Undocumented aliens who become engaged in their communities are "citizens," from my perspective.

(I'm on East Coast time, so I'd better check out for now. I hope to see more interesting posts along these lines from some of you out West when I check in tomorrow.)

Eli - May 5, 2004 7:52 am (#19 Total: 78)
GlobalGiving


Sheila raises a question in my mind about developing metrix around a specific area. Can we do that? In my experience, especially with entrepreneurial ideas, even taxonomies are difficult (given that so many good ideas cut across silos to form a 'portfolio solution.')

But then again, to what extent should we be trying to recreate the wheel when goals exist that we could measure ourselves against (and happen to be rooted in silos), e.g. Millennium Development Goals?

And (to Jeffreysteen's point) to what extent should goals be defined top down?

when it comes down to the plumbing, we have to start somewhere...

sjordan - May 5, 2004 9:14 am (#20 Total: 78)
US Chamber of Commerce CCC


I'd like to throw this hypothetical case study out there and see how folks would answer:

A media company says that it is interested in building up its relationships with organizations that matter to its employees in the community. They received a $10,000 request to support an inner city transportation service serving 120 people for one month (10 employees live in the neighborhood), a $10,000 request to sponsor a race event for a national breast cancer organization (10 employees signed up for the race), and a $10,000 request to support an after school reading program for 10 kids for a semester (1 employee volunteers for the program).

Which program should the media company invest in assuming it only has $10,000 set aside for these purposes? What metrics matter?

Paul Epstein - May 5, 2004 10:48 am (#21 Total: 78)
Epstein & Fass Associates


Great idea for advancing this conversation. Permit me to broaden the context of the hypothetical case to demonstrate the power of engaging community citizens in determining metrics, while still answering your question. If the media company is contributing its $10k in a vacuum, whatever they do will be nice but the impact will likely be limited, metrics or no metrics.

But let's assume there has been a broader, highly participative effort in the community to develop citizen-driven priority goals with measurable outcome indicators attached. The goals & outcome measures could have resulted from a government-led visioning or strategic planning process, as they do, for example, in Prince William County, VA. Or they could have emerged from non-profit led efforts to develop community or regional quality of life indicators, as has been done by the Jacksonville Community Council, Inc. (JCCI) in FL, or Truckee Meadows Tomorrow (TMT) in the Reno-Sparks region in NV. Then, the company--or, say, an employee committee--can decide which of the community goals or desired outcomes it most wants to get behind, and which of the alternatives considered can make the best contribution to improving one of those outcomes, as measured by the community's outcome indicators. For example, the community may have public health, transportation, and education goals that any of the suggested alternative contributions, and company employees, would support. But perhaps the breast cancer race & the transportation service would not be strongly aligned with the specific communbity outcome indicators. However, the after school program is targeted at a middle school grades reading score drop-off the community really wants to improve. So that may emerge as the program most aligned with the community's measurable desired outcomes, and the one to support with the $10k.

It becomes even more powerful if other for-profit & non-profit organizations get behind the same deisred outcome and support a range of programs well-aligned at measurably progressing toward the same goal. Then the impact will be larger, and they may really "move the needle" on their middle school grades reading achievement indicator--or whatever outcome indicator they get behind.

What I described is similar to TMT's bottom-up strategy of getting people and organizations to "Adopt an Indicator," and organizations & collaboratives to make bigger commitments to "Quality of Life Compacts."

sribatli - May 5, 2004 11:56 am (#22 Total: 78)


My name is Srilatha Batliwala, and I am a Research Fellow at the Hauser Center for Nonprofit Organizations, Harvard University. I am from India, and came to the Hauser Center via the Ford Foundation in New York, and after twenty years of grassroots activism with very poor urban and rural communities in India. During those years, and then in my role as a donor and "pracademic", I have always been intrigued, if not obsessed, with the question of measuring change.

I just read the Blended Value Map Paper, and was prompted to weigh into this discussion with the following thoughts.

I think it's really important to surface the assumptions we're making in this whole process, and interrogate them critically. Assumptions about social value, measurement, and so on.

For example, what assumptions are we making about the mobility of language and the concepts developed in particular languages which are in fact particular social and intellectual contexts? Are we assuming, for example, that everyone interested in "social measurement" and "performance metrics" thinks and works in English, and comes from the Western Liberal Democratic world view?

As a feminist activist deeply influenced by ideas about gender, liberation, and empowerment, I was constantly humbled by the fact that I couldn't readily translate most of these terms, let alone the concepts, into the Indian vernacular languages I worked in at the grassroots . Gender became sex (any feminist theoretician would shudder at that!), and empowerment had to be deliberately separated from the vernacular words that connote spiritual power (shakti) to denote political and social power, which are in turn words about authority and power over!

Worse, I found the concept of liberation as increasing individual autonomy was very problematic for the Dalit women I worked with, who thought of it far more collectively, for their entire caste and class.

Why am I going on about this? Because if we are going to develop consistent approaches to measuring social value, we have to understand how social value is defined in different societies and by different parts of the social spectrum - something like the Commonwealth Foundation's study of how ordinary grassroots people around the world think about the role of civil society, and the role of governments. And we'll have to build a map of such definitions, of their compatibility as well as difference, before we can locate the universal ideas contained therein.

I'm not trying to glorify or essentialize the grassroots, but I do think the definition of social value - and the best ways to measure it - cannot exclude the voices and views of those who have experienced the greatest devaluation and marginalization in societies, and who know its meaning in a way that we cannot.

This also goes to costs and validity questions - I've noticed many times how funders and development investors are far more impressed with the legitimacy of assessment done by the poorest beneficiaries, than by the data provided by NGOs (who may have gathered it from the very same source!). I also found grassroots people very good at finding quantifiable indicators for measuring changes in their lives, and far less resistant to quantification than social change actors sometimes are!

And now, here at Hauser, I am studying several transnational grassroots movements that have linked up across borders, and are finding ways of creating collective grassroots ideas about social value and collective measures for this. There is a network, for instance, of groups working in post-disaster zones (earthquakes and floods and such) that have developed a collective "check-list" for post-disaster reconstruction processes, and how these can be designed to transform communities and governance. So we need to look at how they go about it as well....

Sorry for this rather rambling submission, and I hope my points are not too obscure....

David Bonbright - May 5, 2004 12:44 pm (#23 Total: 78)
Managing Partner, Keystone


Wonderful narrative Cynthia. Wish I could clone it. Is OASIS available in some way for wider use?

David Bonbright - May 5, 2004 1:15 pm (#24 Total: 78)
Managing Partner, Keystone


Anything but obscure! At ACCESS, given our concentration on the "developing world", we are in fact tethered to the concerns you raise. I will be very interested to hear how others are thinking about the challenge of rendering this exploration a multi-cultural one.

David Bonbright - May 5, 2004 1:33 pm (#25 Total: 78)
Managing Partner, Keystone


To your assumption that ACCESS is a set of guidelines for measuring and communicating social value creation, the answer is yes. And linking my initial post to Paul's point (message 16 I think) about who sets the measures, the ACCESS framework also displays organizational capabilities. One important capability turns on the responsivness of the organization to its stakeholders. ACCESS reports will make it clear to third parties the procedures and values the organization has in this regard. It will then allow Paul to value this capability very highly and make his investment decisions accordingly. But it would allow someone interested in plumbing that works, by God, to look harder at product than process.

Eric Weaver - May 5, 2004 2:43 pm (#26 Total: 78)
Lenders for Community Development


Hello All. I’m Eric Weaver, Executive Director of Lenders for Community Development in San Jose, California. We are a community development financial institution that invests in affordable housing, non-profit facilities, small businesses, and individual families. As a practitioner on the ground, my feelings about social measurement and performance metrics are conflicted. I love to be able to see how we compare against other organizations in terms of performance, and I think that metrics which allow us to do that, and allow donors or investors to do that, are very useful. I think most of that kind of comparison measurement, to be meaningful, best happens within “silos,” and it gets much harder to come up with those kinds of useful metrics across all social enterprises.

On the other hand, as a practitioner, I get scared by the idea of funders and investors continually ramping up their demands for data from us—especially if they each ask for different data in a different format. That said, I think selecting high-performing social enterprises and contracting with them (and compensating them) to do “Cadillac” data collection is a great idea, but the reason for doing it would be to prove a certain concept, i.e., “Individual Development Accounts, if delivered in such and such a way, help low-income people move out of poverty. Here are the statistics.” But at some point, after the efficacy of a program has been shown, future practitioners shouldn’t be held to the Cadillac data collection standard. Instead, they should just be measured on how many people they can serve and at what cost—so long as they aren’t deviating too far from the basic model. And then to go from there to trying to figure out, “Ok, are Individual Development Accounts a better investment than planting trees in the Amazon?” My head spins!

I frankly don’t believe the data is ever going to be there to really evaluate social return on investment, other than in very carefully controlled laboratory conditions, yet I think those experiments, if structured well, can be enormously valuable. Their greatest value, I believe, should be in informing public policy, where the real action has to be if we want to bring about big changes.

One of my chief worries with the mania for measurement is that it will lead funders away from investing in efforts to change public policy, and even more critically, to changing the balance of power in the US. How do you measure the social return on investment on funding efforts at voter registration among the poor, or efforts to lobby for long-term changes to election law, housing policy or tax policy? Yet those are potentially the highest-impact investments a funder can make. I worry sometimes that we are all dithering around (and I am very much guilty of this) with quasi-market-based approaches to helping the poor, or cleaning up the environment, when the only way we are really going to make a really big impact is by changing the political equation—and this holds true both for the US domestic situation and for the US role in developing countries.

I know in my own work, the single biggest factor in the growth of community development finance in the US was not philanthropy, and it was not a truly market-driven recognition of an investment opportunity, it was the passage and enforcement of the Community Reinvestment Act—a federal regulation requiring banks to do meaningful business in low-income communities.

So, that’s a bit rambling . . . I think we should measure what we can, but with an eye first and foremost to figuring out which types of activities/interventions should best be the target of public spending—once we figure out how to get the government spending more on the things that matter to us.

jimfruchterman - May 5, 2004 3:12 pm (#27 Total: 78)
Benetech


Since we run social enterprises, we're interested in metrics for several reasons.

1. Resources. We need to do the accounting to tell how we're doing financially. Have we reached break-even? If we haven't, when will we and how much money will we spend before that?

2. Management. How do we know if the project is successful? What are our goals and how will we know if we're attaining them?

3. Choices. Which of these ten projects should be the next one funded? What metrics do we use to choose one over another? If one of those metrics is a measure of social impact (which it is, for Benetech), how are we measuring it? Is it an SROI analysis ala REDF? Is it benchmarking against alternative solutions?

4. Gaining investment. I think everybody recognizes that raising certain kinds of capital is easier with a pitch that includes metrics.

sjordan - May 5, 2004 3:18 pm (#28 Total: 78)
US Chamber of Commerce CCC


Okay:

so metric #1: how the project scores against desired community outcomes.

What should the top metrics for the media company be?

Possibilities: -- Community Impact -- Employee Involvement -- Branding/Marketing potential -- Social infrastructure support for employees and future employees -- Others?

David Martin - May 5, 2004 3:26 pm (#29 Total: 78)
Researching & designing financial services for low-to-moderate income Americans


David,

It might be a bit early in this discussion to suggest this, but I'll push the envelope anyway.

David B., if you are willing, I suggest that you post the general operating principles of ACCESS either in this thread or we'll start a new one. We can use your guidelines to start a tangible discussion about metrics. If it works well, you'll get great feedback and hopefully we'll all get a since of general operating standards for metrics that serve our !global! needs for !all! stakeholder groups. What do you say????

David

David Martin - May 5, 2004 3:39 pm (#30 Total: 78)
Researching & designing financial services for low-to-moderate income Americans


Eric, great post and good to hear from you again (I attended one of your SV2 luncheons last year).

You make great arguments for the need for practicality of metrics. You mention that one of your worries is "the mania for measurement...will lead funders away from investing in efforts to change public policy, and even more critically, to changing the balance of power in the US." You then mention several examples of things that you believe are hard to measure. My comment is this: let's not get confused between QUANTIFICATION and MEASUREMENT. Is it hard to quantify voter registration? No. Is it hard to measure its effects on public policy? Yes. Is it hard to quantify lobbying for long term changes to election law? Yes. Is it hard to measure the effects of such legislation? No.

My point is that we need to de-couple metrics equating directly to either quantification or measurement or qualitative analysis or any other specific thing. Measures and Metrics should intend to provide feedback on the operations of business or social enterprise or nonprofit. The HOW of how to create them should really be dealt with after we decide specifically WHAT is being measured.

I have faith that we can come up with many, many ways to get around the fact that quantification isn't always an option. I think it will take a while and a considerable amount of experimentation to get to where we want to be.

Thanks

David

Paul Epstein - May 5, 2004 4:55 pm (#31 Total: 78)
Epstein & Fass Associates


David, it sounds like Access has qualities similar to the "macro" level of a multi-dimensional measurement approach used by Greater Kansas City LISC and its investor group that supports CDCs in that bi-state region. The macro level consists of "weighted grades" for six performance dimensions, one of which is how well each CDC delivers their "products" as you put it (mostly affordable housing), another dimension is "Community Connection" which includes how well they organize and engage their neighborhoods. Resident engagement also counts toward selected indicators of several other dimensions, including planning and CDC Board development. This "Performance Metric," as they call it, was developed by GKC LISC & CDC Exec Directors, and approved by investors, not the community, but it reinforces community engagement.

Forming a virtuous cycle with with that macro-level metric in Greater KC is a requirement for all CDCs to have "strategic work plans" with measurable targets and implementation milestones. The resident engagement needed to get good grades on the macro performance metric clicks in at micro level planning & measurement, as residents then become engaged in details of CDC plans, what specific projects are attempted, what the goals of neighborhood organizing will be, etc. So, at the neighborhood-by-neighborhood level, community members ARE involved in setting criteria for success, so their priorities will be reflected by specific projects and measurable project success will be their success.

Sheila Bonini - May 5, 2004 5:08 pm (#32 Total: 78)
Hewlett Foundation


Eric brings up some great issues that relate to Jim's insights from Benetech. To me it means that metrics have a variety of different uses.

In the for-profit world, it is common to think of using different systems and metrics to guide management choices versus those used for reporting or those used for investing (think of the difference between cost accounting and financial accounting). I don't see why we should assume there is only ONE set of metrics, or more accurately, data, in the Social Value space.

Also, Jim's list highlights the fact that, because we are talking about Social Metrics doesn't mean we can ignore the financial metrics...or the environmental for that matter.

But, coming back to different metrics for different purposes, let me suggest a few levels.



1. Across organizations doing similar things, we need to have data and metrics that test the underlying Theory of Change - this is what Eric referred to as "Cadillac" data and what I referred to in my earlier post. This is the kind of scientific data that proves a point, for example that IDA account do indeed help low income people, or that Microcredit helps get people out of poverty or whatever the case may be.

2. Within a particular organization or program, we need data to track how we are doing relative to our goals, relative to other similar organizations pursuing these goals and relative to how we have done over time. So, using Eric's example, this would be tracking how well you deliver IDAs (once you have proved in 1. that these do indeed align with your underlying goal of helping low income people). It also helps us see that one organization may have a much more efficient delivery model versus another (the point that Eli made about why it costs one group 3X more per unit to deliver what seems to be the same service).

3. At the next level we have something that relates to Jim’s third point regarding making choices. If you have a portfolio of similar projects, how do you decide which to fund? This may be something like an IRR or hurdle rate for Social Value creation. This is similar to the funding decisions a foundation would make within a particular program area. So – if I want to help low income people, would I invest in Eric’s organization or another model of helping low income people?

4. Finally, you get to the more diversified social investor, perhaps akin to the for-profit holding company, which is making the apples to oranges comparison. Should I help low income people in Northern California or plant trees in the Amazon? This would be like large foundations trying to determine how much funding to allocate to each program area.

I know I could take each of these apart into more levels, but the basic point is that we need to start unpacking what we mean by data and metrics…

Hope this makes sense.

Sheila

Paul Epstein - May 5, 2004 5:17 pm (#33 Total: 78)
Epstein & Fass Associates


I suggest that a variation on multi-dimensional measurement may help get at the issue raised by Eric. Not quite the multi-dimensional measurement I described for Greater KC in my last post, but more along the lines of a "multi-perspective balanced scorecard."

Someone raised the idea of "value chain" earlier. Think of a value chain of cause-and-effect assumptions from registering poor voters and getting them to the polls and community civic leadership to ensure the new voters are heard not just on election day, to changes in public policy and spending in response, to design of programs with stakeholder engagement that better meet social needs, to actual performance of those programs in products and services they deliver, to desired changes in community outcomes. Each of those "perspectives" in every "to" statement potentially has something strategic to measure, not just the end results. In some cases, the measures may be simply "did it happen or did it not?," in other cases it may be quantifiable in a more traditional metric sense. The idea would be to keep track of what's going on at each level, and not just of the service delivery and end result, all the while testing assumptions concerning whether if the desired things happen at the beginning and middle parts of the value chain, do we get the improved services for people and ultimately improved community outcomes at the macro level that we want.

Paul Epstein - May 6, 2004 12:19 pm (#34 Total: 78)
Epstein & Fass Associates


Isn't it interesting that many of us have cited or implied the need for measures across multiple dimensions for different uses. It strikes me that an exercise I used to use with clients, "Users and Uses of Measurement" would be helpful here. I'll start it by listing "Users" I remember from earlier posts, with some additions, then listing "Uses" by building on Sheila's helpful list in Post 32, and finally suggesting "Next Steps" should anyone want to take this further.

USERS (not in priority order; not mutually exclusive):

- Citizens or stakeholders of the community or region affected

- Citizens' elected representatives & policy officials

- Users ("customers," "clients," "beneficiaries") and of the social good (service, product, ouctome) produced by a program or project, and other stakeholders potentially affected

- Managers and staff of a program or project

- Investors (public & private)



USES (not in priority order, not mutually exclusive):

- In-depth evaluation (e.g., "Test the Theory of Change")

- Stimulate action, investment, or focusing of resources by many players toward community or investor goals or desired outcomes

- Monitor progress toward community or investor goals

- Monitor progress toward internal organizational goals

- Compare (benchmark) performance against other organizations or programs with similar goals or processes

- Operational management

- Development or management of strategy

- Resource allocation (e.g., budgeting) across different programs or units in an organization

- Resource allocation (e.g., managing investments) among different organizations or programs aimed at the same major desired outcome

- Resource allocation (e.g., managing diverse portfolios) among programs or projects with different major desired outcomes



SUGGESTED NEXT STEPS (by anyone who thinks this will be useful):

- Add to either or both lists

- Match up users with their uses

- Describe the nature or types of performance measures or data needed for specific users and uses

Please jump in even with only a partial addition to this analysis approach, or with your comments on it, if you think it will be useful.

David Bonbright - May 6, 2004 2:40 pm (#35 Total: 78)
Managing Partner, Keystone


It does indeed! I clearly need to learn more about this Greater Kansas City LISC experience. Thanks for the lead.

David Bonbright - May 6, 2004 2:58 pm (#36 Total: 78)
Managing Partner, Keystone


Well...you asked. ...and for what it may contribute to the discussion. Here are the initial specifications that we are using to develop the ACCESS Reporting Standards Framework. N.B., as a reporting standard it goes beyond metrics. I hope that this proves to be useful to what is already a rich and dense conversation. I am taking away a great deal from all of you. I am curious -- and a bit nervous -- about what this group of metrics hounds is going to make of our specs!

------------------------

Initial Specifications. In order to design the ACCESS Reporting Standards Framework, ACCESS has developed a set of initial specifications that reporting organizations will elaborate upon and test in the field through “action research”.

ACCESS is oriented toward predicting future performance of reporting organizations. It does this by describing two things: capabilities and the results of past performance.

Capability-based analysis. To describe capabilities, ACCESS looks to two categories of information: - Strategies (including mission, vision, activities and underlying theories of change). - Organisation structure, culture and systems (including people, governance, management, commitment, entrepreneurial quality, resource base). Domestic and international legal and tax compliance issues will be covered here.

Performance metrics geared to proximate results. As is explained in detail in the ACCESS inception report (at www.accountability.co.uk), ACCESS does not seek to measure long-term impact (e.g., changes in income or well-being in an entire community), but asks the organization to measure results that, in law, it could be deemed to have “proximately caused”. In other words, ACCESS will measure the changes in the behaviour, relationships, activities or actions of the people, groups and other organizations that would not have occurred but for the work of the reporting organization.

A number of additional specifications shape the ACCESS reporting approach.

1. Non-evaluative. A reporting standard is not a rating system. It does, however, yield information from organizations that can be meaningfully compared and assessed. The ACC