The fetishization of metrics

Hi Charles,

Trying not to be a quant, I guess. This statement has been on the home page of p-ced.com since January 1997:

"Enterprise profitability and economic success cannot be fairly measured in terms of gains of money capital alone. Profit is redefined in human terms rather than pure quantitative analyses that remove human and social concerns in the name of profit."

To emphasise the point, peoplenotnumbers.com is our alternate domain.

Our manifesto reasons the case for this statement:

http://www.p-ced.com/1/about/background/

Best impacts were before I cam on board in sourcing the Tomsk initiative and after in the influence in Ukraine. This interview from 2004 relates the aims and success in Tomsk.

http://www.iccrimea.org/scholarly/economicdev.html

Jeff

 

Oops, I should have typed peoplenotmumbers.org for that alternate domain.

Hi Jeff:

I think peoplenotnumbers.org is a terrific domain name!

To digress, doesn't part of this jumble begin with the very term "metrics?" I have found no definition of the term that makes a particle of sense in the quant vs. qualit context. Isn't the term itself representative of the a contrived perspective in the nonprofit/social entrepreneurial sector that seeks to ascribe something impressively technical to what is nothing more than jargon?

I recently gathered with some folks who struggle with the importance of telling the story of how young people, who do not thrive when left to their own devices, do thrive when assisted. I suggested in our rambling discussion that we needed to make certain that counting heads counts for something other than numbers.

Harvey:

I think "the importance of telling the story" is key here -- real stories may, I think, include pie charts and graphs, but it has to be authentic human interest that drives them, no? Otherwise, they're not really stories.

Dear Social Edge

First, I would like to make a personal testimony for social edge.

Social edge is an outstanding and unselfish online reference and knowledge sharing center I have ever encountered. It is part of my life as a social entrepreneur and changemaker. Everyday, I should not miss reviewing discussions and blogs and reflecting with other's comments and knowledge sharing as well as updating my participation at the GSBI. Social edge is my online text book and mentor as I spend almost 16-18 hours a day in my small non-profit office here in southern Philippines. In fact, when I got my brain drained and stressed with my work to the extent of having a burn-out, I just read articles at Social Edge. Its a relief at the same time increasing my understanding of the intricacies of being a social entrepreneur. My peers too are exposed to social edge but they need the understanding why they should learn. Knowledge is free and connecting local knowledge to contribute to the global body of wisdom and greater principles of life is indeed a great service to humanity. Great job!

For Charles,

The way you wrote this discussion is quite very technical and I do not know if my comments here would be relevant to your thoughts. But I will try since i feel very attached to the essence of this article.

Our organization is a youth-led non-profit social enterprise and because we are very passionate we have mustered our youthful energies in creating positive change. A number of funding agencies has come to us for aid that assist us in pursuing our efforts.

For ten years now, we realised that out of the external support we have and despite reaching 60 communities, identified more than 800 peer leaders, trained approximately 400 peer educators and directly served 8,000 at-risk youth since and almost 400,000 indirect audiences since 1998, we realize that only 10% sustained the change we want to happen.

However these numbers had achieved donor's requirements for the money they granted but measuring outcomes has been made for the use of external donors leaving communities (like young people) with unsustainable behaviour change interventions without any tools of their own to assess their own risk, benchmark their own life goals, measure their own progress and celebrate own change. Our own mandates, causes for being and dreams were not achieved and seemingly we are running out from our own tracks to meet somebody's else goals. Somewhat like a modern form of slavery where beneficiaries depends on NGOs and government subsidies and NGOs and Government depends on grants, and donor countries (while money from tax payers goes to corruption) while communities are left without the necessary skill to generate their own income to subsidize own needs (especially among impoverished communities) - of course to keep them poor in order for them to keep electing corrupt officials and keep them dependent on rich entities..

Their (funders) metrics are numbers: how many condoms distributed, how many peer educators trained, how many trainings conducted, how much money spent, how many peers reached and etc. How about the communities? do they measure their own progress or manage to sustain the change they want to be?. After project ends, these funders will wave their call for proposals (for someone to achieve their goals because they have no manpower to meet their mandates).

But look, millions of condoms produced and distributed but HIV still escalates, many peer educators trained but same clients still return to their service providers with same needs, hundreds of thousands of peers being reached but new cases that emerge and those who cannot access peer education are left due to non-expansion of efforts to other areas while resources continue being poured to the needs of almost same clients, billions of money spend for aid and yet same countries needs more than the same amount each years they need. The need for external aid increases while the problems continue to increase. Where is the return of investments?

Thats why I am an emerging social entrepreneur: Transforming youth beneficiaries into social entrepreneurs who share their responsibility to generate financial and social returns while leading and sustaining change that gives them the power over their lives and make a difference. And my metric is the "self-measurement of change"

Since my metrics is still under media embargo policy in one conference, I will share that after my presentation and will continue to update my self in this e-froum.

Read you all soon

JP

PS. I do not want to be a quant, I'm a qualit because a qualit is someone who is honest and transparent, who demonstrate that the impact of a development intervention is not uniform to all target audience. A quant is fond of demonstrating achieved intended outcomes to please funders but not the community he/she serves. A qualit is a risk taker that includes demonstration of both intended and unintended outcomes, the failures and success that sometimes critique the way funders impose their policies that are not relevant to the needs and priorities of the communities being served (as per Paris Declaration of Aid Effectiveness). A qualit put funders, co-performers of a development intervention and beneficiaries into the same level partnering to achieve a common goal.

Hi John:

Your comment, "seemingly we are running out from our own tracks to meet somebody's else goals" is one of the big reasons I wanted to start this discussion, and your comments are right on target.

Would you, or anyone else here, like to tell us more about the way quantification and funders' goals can get in the way of an organization's goals and quality?

Numbers are unavoidable. But they should remain our servants and not our masters. Measured outputs should not be confused with outcomes. The former may or may not contain indications of quality or effectiveness of effort. The latter is ultimately what we are in (social) business for.

Excellent.

I must say I'm a bit of both: qualit/quant...quanlit? :) I think that if we can eventually get past the superficial terms of "numbers" and "metrics," then what we're left with is our human desire to measure if we are really helping. A lot of us need proof, which in the traditional sense, means numbers and data. I work in a University where assumptions and hypothesis of change or impact also require research, data, trials, and validity. However, I can see how one might get caught up in the frenzy of a systematic approach. In the end, I believe having a system and infrastructure to work within can be a wonderful foundation for human beings to bring about change and also be confident that they are making a difference.

My two cents.

Hi Autumn:

In don't see any harm in being both -- I think it's great. But I suspect that when one tries to be both, others who lean heavily on the quant side and disregard the qualit will make decisions on that basis -- so there's the risk of a deadening effect whereby only the quantifiables really get the attention of (eg) funders.

The opposite risk is of an org being so happy with its social purpose that it doesn't notice how ineffective it is, I suppose.

IMHO,
This is the single biggest challenge that we as a movement will face, and must answer.
The reality is that you need to show that the venture works, that it is firing on all eight cylinders.
You must be able to demonstrate positive, lasting social impact as well as sustainable revenue and efficient operations. If not, you will never be taken seriously.
And, yes, qualitative factors can be measured and analyzed (ask any sociologist).
In reading and studying Greg Dees for a number of years now, I don’t think that this quote means that we should not be trying to measure.
I think it means that we measure differently, we measure holistically, and we balance the measurements against each other to develop a scorecard of overall effectiveness.
Since the members of this movement come from a number of different perspectives, each of us views metrics slightly differently. I’m hopeful that we can struggle together to build a dashboard that is relevant for all.

Hi Jim:

And thanks! Wow, this conversation is positively blazing!

I certainly think a dashboard is better than any single metric, and that different projects could use a variety of metrics, so maybe, maybe some sort of common dashboard would be possible -- but I'm still inclined to think it needs to be embedded in story, ie the quantities need to be presented within a qualitative framework.

Does that sound about right?

I suppose I should say that I personally am hugely biased in favor of qualit -- and that I fear the way so much of the world is run is no less strongly biased towards quant. That's why I think this conversation is so important -- and here's why I personally think that in *real* terms, qualit is so much more important than quant.

The person I've known personally who had the greatest impact on the world around him was a priest and school teacher in South Africa during the apartheid era -- an Englishman who was known to his parishioners and students in Sophiatown, the shanty town where he served, as "Makhalipile" -- the dauntless one.

Let me tell you two stories about him and them (and please forgive me if you've heard them before):

One young black kid in South Africa remembered him years later: "I was standing with my mother and I must have been about maybe nine or so, when this white person in a flowing cassock swept past and doffed his black hat to my mother. ... What? A white man raising his hat to a black woman, an uneducated domestic worker at that? Quite unheard of!" But the priest in question was Fr. Trevor Huddleston, and it was a natural courtesy for him to lift his hat in greeting a lady... The young boy never quite recovered from this encounter: we know him now as Nobel Peace laureate Desmond Tutu.

Another young black kid, Hugh, aged 12 or 13, fell ill and was taken to hospital, where Trevor Huddleston visited him. Trevor asked him what he would like more than anything in this world, what would so thrill and please him that he would have the greatest possible motive for getting himself out of the hospital and back to school. Hugh said, "a trumpet, Father", so Trevor got hold of a trumpet to give to the boy -- now known the world over as the internationally renowned jazz trumpeter, Hugh Masekela.

You can count the number of heads Trevor had in his classes over the course of a decade, and compare them with the numbers of kids that other teachers teach. You can compare the exam records of his students with the exam records in other schools. And you can do the same with his sermons, the number of parishioners o whom he preached, the number of times he said Mass.

But how can you possibly quantify the impact of this man who raised his hat to a lady and brought a schoolboy a trumpet?

I suspect that Trevor's lifting his hat to a lady that day may outweigh all the classes he ever taught, that the impact of that trumpet may re4ach farther than all his sermons combined.

My basic premise is that there's a spark, Trevor had it, and he transmitted it to Tutu and Masekela -- and that that's the secret of education. But how do you quantify that?

Hi Charles,

Another important discussion. Thanks for leading it.

You wrote, "Even in war, "morale" can mean more than guns and ammo, and the pen can be mightier than the sword."

I agree. I think numbers can help give more precise understanding to define what we're doing and progress we're making toward goals. However, I think they are more useful to the CEO of an enterprise than to the evaluator, donor, or customer. That's because as CEO, I understand the context of the number. I know what the challenges were, what resources I could bring to bear, and what I'm trying to achieve. If my numbers are put in a column of numbers comparing me to organizations doing similar work, they don't mean much because the context for each org is different, and the donor, volunteer, customer, probably has not spent nearly as much time as I have in understanding what I'm doing, what resources I have, and what my challenges are.

I and constantly frustrated by people asking for the "elevator speech" as if in 30 seconds I could define a complex social problem, describe how I'm a cog in the wheel trying to contribute to a solution, and how they might help. To me metrics and similar talk are just ways to say "I don't understand, and I don't have time to invest in understanding."

I agree with the posts of Harvey, John, Brian and since we all seem to work with youth, I encourage you to join me at http://tutormentorconnection.ning.com where our combined voices might cause more people to spend time looking at what we do, why we do it, why they should be involved and where we are doing our work. At the same time you can build relationships with others doing similar work, with is the "morale" that we need to stay involved in th is insane business with so little consistent support.

I'm also happy that Autum posted her comment. When people ask what a graduate learned at the University he/she will say "I got a degree in xxx". Or I got a Masters, or PhD. That's the elevator speech. Everyone understands the concept. Everyone understand that this person spent four, six, or 8 years in constant learning and reflection, so that degree means they know something.

Yet, the student and the leaders at the university know that this represents thousands of hours of work, millions of dollars of funding, etc.

I feel that we need to constantly be looking for numbers to define what we do, so we can try to get better each year. At the same time we need to create universities and faith groups of "learning" where people who are not directly involved with what the SE is doing can be engaging in on-going learning so they become more sophisticated over time in understanding what needs to be done and how organizations differ in what they accomplish, based on what resources they have, and what degree of difficulty they face in the problem they are trying to solve.

My work with Tutor/Mentor Connection is an "alternative" university. By hosting information on our web site, anyone can come and learn from that information. Anyone can use it. If we can find ways to connect our story telling with others doing similar work, we can attract more "customers" and "student" to the information on our web sites, and overtime, create the type of understanding that enables people to know exactly what we're doing when we say "tutor mentor" or "fast food".

Thus, I'm not a Qualt or a Quant. I'm a story teller, an aggregator, a network builder, a dreamer and an innovator. I'm just trying to find ways to remove the barriers that prevent me from having as much impact as I need to have in the world I'm now living in.

Daniel:

Always a pleasure. I think your comment about numbers serving those who know the work, the effort and care and love that's present in an organization, far better than they serve people from outside making comparisons between one org and another is really important:

QUOTE: I think numbers can help give more precise understanding to define what we're doing and progress we're making toward goals. However, I think they are more useful to the CEO of an enterprise than to the evaluator, donor, or customer. That's because as CEO, I understand the context of the number. I know what the challenges were, what resources I could bring to bear, and what I'm trying to achieve. If my numbers are put in a column of numbers comparing me to organizations doing similar work, they don't mean much because the context for each org is different, and the donor, volunteer, customer, probably has not spent nearly as much time as I have in understanding what I'm doing, what resources I have, and what my challenges are. :UNQUOTE

I very much appreciate your sense of Tutor/Mentor as an alternative university.

And I think your self-description -- "I'm a story teller, an aggregator, a network builder, a dreamer and an innovator. I'm just trying to find ways to remove the barriers that prevent me from having as much impact as I need to have in the world I'm now living in" -- is very much to the point. You're a tutor and a mentor to us all.

You're right Charles it's a great conversation, I agree with Kjerstin in her blog too that "the subtleties of word choice deeply influence our subconscious" as much as our perceptions so fetish could be seen as a scenario of, say being obsessed about going round in circles without actually getting anywhere, (maybe a bit like the framework of measuring the world's entire aid over the last 50 years) perhaps it's time for new metrics?

I agree too that education is the metric of social value that stimulates economic progress but that it's hard to quantify if the framework for measurement is too narrow. If numbers are driving the system, people tend to get overlooked and the depth of impact hidden. Putting social value back on the dashboard would have great transformative value for society at large of course, if social value gets lost metrics would definitely then become a fetish.

That's why story, is such a good place to start in bringing human interest back into the picture. There's an old saying in education - you can't light dead wood but if you've got nothing left to burn, you've got to set yourself on fire.

Also you know, creating something out of nothing is a form of true value creation, (anything that's measured after that is just variations on that spark) so how do you measure that?

take two: I guess the question is how do you measure the spark passed between people? That's why stories are so great, in stories we start with a character (or entrepreneur) and as they are put in situations and progress step by step towards change or shift in psychological state, etc, the reader/audience share in the feeling of the experience together. So from my perspective, whether it's quants (quantitative) of qualits (qualitative)it helps to have a common metric for access and inclusion or in changing consumer behavior, etc, the measurement is in the meaning, - with an awareness of a wider sense of community - that's when quantity(scale)can become quality.

Hello again, Jo:

It's always a pleasure to see you here. I suspect the situation we're looking at here is a bit of a conundrum -- whether you ask how quality can be measured, or how to balance head and heart, or what the interaction between quantity and quality is, the answer doesn't seem to be in words, it's more like riding a bicycle perhaps.

You throw yourself into it -- with all your wits about you, heart first

...looking at holistically turning risk into opportunity (+ when the numbers fit the context) as you well know Charles, there's a holiness to the heart's affections - the heart opens and continues opening - addressing the spirit of the matter, connecting and interacting (with trust, etc,) in that, it binds the team (quants vs qualits) together - whether they like it or not...

I forget who said this phrase but, sometimes the heart sees what's invisible to the eye, and also as the hand can never execute anything higher than the heart can imagine, there's no doubt the heart has the lead. It already knows what the mind can only dream of, and because it has the goal of oneness, the heart is a healing art.

Measurement here seems to depend on intentionality compared to actuality and can realistically only be performed by, and be relevant to, those who make the decisions about the desired future in question. If a future state turns out to have farther reaching effects than intended then we need to be able to appeal to the process and start the process of visualising a better future state again with more factors and more complexity. When complexity increases and we don't have good systems to deal with it people tend to become blinded to 'soft data' like social impact and rely instead on metrics. People collectively reduce a problem until they can comprehend it in terms that they able to understand each others position with. It's harsh to judge or measure someone or some group according to factors they are incapable of or can't comprehend effectively during decision making. I think if you want to measure these things then you need transparent processes and to engage everyone who's consideration of your performance is of value to you early on. This need not be at one level of engagement.
 
A tool that works well to factor soft data is 1000 minds. It is a multi criteria decision making tool which uses an approach similar to conjoint analysis. Its uses complex mathematical models to gather data from potentially large numbers of stakeholders about their preferred future but the difference is that the process is transparent and accessible so people can collectively understand how they came to end up preferring one scenario or concept over another without reducing the complexity. The tool allows the complexity to be dealt with more methodically and incrementally.

Measuring these things directly is an oxymoron in my opinion but if we can better compare future states and agree on them - then uncover better, more effective ways to obtain those future states. That would be nice.

That is very helpful, Hugh, both for all of us at this event, and for my own personal thinking -- this whole business is a nut I have been trying to crack for a long time! I'll investigate 1000 Minds with keen interest!

Many thanks!

For what it's worth this is how I would approach it:

At the start of an initiative I would engage all of my stakeholders to build a preference model based on key success criteria. Then use the preference model to rank the initiative as if it had met all of it's desired outcomes. The preference model gives weightings to criteria through the 1000 minds process which preserves the importance of soft data. The initiative will gain a percentage score against the preference model. This can then be used as the yardstick of success. Then at whatever intervals desired asses the initiatives progress against the model again and it will give it's current state as a percentage score against the preference model. The difference between the current score and the score the initiative might obtain as if it had reached it's desired outcomes will give a meaningful comparison of success and progress.

Due to the processes accessibility and deconstructibility it becomes possible to adapt the model if need be by re-engaging the stakeholders. Issues should be easy to communicate because of the transparency the model affords.

Hope it helps. I have huge faith in this approach but have never had the opportunity to apply it myself.

Hugh,

In my experience, many of the innovators who launch a new idea don't have many stakeholders surrounding them to give feedback, or provide input. Just because we think we have a good idea does not mean others think it is good. Even if it is good, most of us don't have the database of "friends and stakeholders" or the advertising/communications ability, to reach out with invitations to participate, over and over, for as long as it takes to build a group large enough to do what you are suggesting.

Those who have reach a point where they can draw a crowd with one invitation are far beyond the playing field that many social entrepreneurs are involved in trying to bring their ideas to reality.

Some people can spend decades trying to get attention for an idea.

Thanks to you both for keeping up the conversation here...

I think Daniel's work with Tutor/Mentor is an excellent example of innovation. To really appreciate what he';s doing, you need to have two sets of interest up and running, it seems to me -- you need a lively interest in kids and how they can he helped to gain access to influences that are both practical and inspirational -- and you need to understand how social media / web 2.0 can be used to provoke and provide authentic community.

There are people who are deeply into one of those camps, and mildly familiar with the other -- but it will only be when people see *both* aspects as crucial that the real value of Dan's work will be apparent.

At which point, many others will benefit from it, borrow from it, and put it to good use.

And if there's something I can do to nudge that idea along...

One of the limitations when measuring soft data is you need people to agree on evaluations, they cannot be reliably decided by individuals. The reliability of the measurement is roughly proportional to the number of people involved in making it.

Beauty of using good tools is that engaging people is virtually effortless.

Hi --

Do either of you have a URL for that?

< racing through, trying to set things up for a fire time poster >

Thank you for this discussion on qualitative vs. quantitative considerations. I don't think we need to necessarily choose to view life, our business or the world from just one perspective--ideally, the two should be in balance, each given their appropriate due.

Another way to describe it is to balance the head and the heart. In my own life, I chose to balance the two by becoming a social entrepreneur—bringing together my passion for changing the world with my love of good business principles. As the founder and CEO of UniversalGiving™, I use my head and my heart all the time. Some of what we do is quantitative; as we connect people to opportunities to give or volunteer all over the world, we use our rigorous, step-by-step Quality Model™ to vet our projects. Other aspects are more qualitative; forming relationships, with partners and with team members, and working to achieve each other’s goals.

Metrics and quantitative views have their place in life and in philanthropy, but so too do qualitative considerations--and for those we should lead with the heart.

Sincerely,
Pamela Hawley
Founder and CEO
UniversalGiving

phawley@universalgiving.org
www.universalgiving.org

Living and Giving blog
www.pamelahawley.wordpress.com

Hi again, Pamela:

Yup. I think the reason I was so excited to run across the Skoll Foundation and SocialEdge in the first place is that as social entrepreneurs, we're about the business of balancing head and heart...

I guess my concern is that we don't lose that balance because of an emphasis on quantitative measures, just because they're so "convenient" for others, in terms of making comparisons and allocating resources.

How do we avoid that trap, and still make sure we're effective at what we set out to do?

Dear Charles,

This is such an important statement:

"I guess my concern is that we don't lose that balance because of an emphasis on quantitative measures, just because they're so "convenient" for others, in terms of making comparisons and allocating resources."

It's interesting that we see these quanitative measures as convenient and tangible. And yet, they truly do not capture the entire value. They just don't.

I am going to step out on a limb here and bring up something "touchy-feely" that may not resonate with some of you. Please keep in mind UniversalGiving generates revenue; that we want companies' bottom lines affected by their CSR, philanthropy and volunteerism programs. We want to be able to track business effectiveness across global brand; employee retention; product adoption, etc. So here then comes my comment:

How do you measure Love? Or better put, loving relationships in your life?

I was thinking about this walking on my way home from work the other day. There are just so many intangible, immeasurable benefits to companies giving and volunteering. How do you measure, truly, employee satisfaction and where it comes from? How can you track that x% comes from their benefits; y% comes from a positive relationship with their manager; z% comes from colleagues; a% comes from the volunteering benefits; b% comes from the childcare facilities; c% comes from feeling good that the company foundation gives grants;... the list goes on and on... How do you measure all of it and tie it back specifically?

The surveys, yes, can get more specific as noted above. And some of them are. However, some of the quantitative is still subjective. On one day I might be feeling really good about the childcare program, perhaps especially grateful, because that day I had a tremendous report due and needed that childcare support. If I were surveyed that day, my percentage loyalty to the company might jump by 12%. Or by 32%. Or would it be 6%? Then go down all the other variables noted above (and more) and apply the same thinking...

So let's then get back to this concept of Love. How do you measure it? Is it by how many meals your partner makes for you? The kind note? Consistent kindness? The warm feeling and respect you get when you see how they treat others? Loyalty?

How... do you measure love?

How... do you measure giving?

Again, we can get certain metrics. Number of donation; average donation; number of donors; NGOs affected and with what outputs and outcomes. And we should measure this.

However, there is an extremely important, extra-tangible benefit (benefits) that simply cannot ever be measured fully. And they should not be taken for granted. They supercede, in some ways, the quantitative. They build upon, enhance, enlargen the quantitative.

I think in answer to your question, Charles... when we look at what we value most -- love, relationships, purpose... we would feel a bit shallow if we took it only to metrics. It just loses the import of what it really means to us. How would we feel if our spouse or partner were measuring us on # of meals we provided each week to them? My guess is a pretty low feeling. ...Feeling as if we were constraining what love means to us, unblossoming its full potential.

So that is how I try to stay connected with head and heart. Measure results. Report the results. Then add on the heart and let a larger, beautiful, fuller meaning expand the concept of giving we are striving to achieve...

Charles, I am looking forward to hearing your insights, and always appreciate your thoughtfulness!

Sincerely,
Pamela

Pamela, I don't know what to say.

I think quality is what brings us into this work, and quantity is what facilitates the world around us understanding.

I am very glad you're here and posting.

It is always challenging to consider how qualitative and quantitative means can be used to articulate the narrative and legitimacy of our efforts. To me, and simply repeating what has already been stated, “context” is key. And, without context, numbers alone are often misleading or, even, arbitrary.

I think that within our world, characterized by a scarcity of resources, we must consider how best to steward our time and money to accomplish as much as possible. With this assumption, shouldn’t we be open and transparent to allowing and encouraging quantitative measurement? Measurement that might allow us to see if we are producing real outcomes (or have outputs as a proxy), to know if we are more efficient this year than last year - and why, and to be aware of other models that our peers are deploying that might be worth reviewing, or even adopting.

I think that quantitative measurement affords a medium for productive discussion and acts as a social object for engaging with like-minded cohorts. If organizations with common purposes are measuring similar things, it would spark dialog – a dialog to understand and compare context so applicability could be understood, learning could be initiated, and improvement could occur.

My gut would be to speak out for more universal measurement so that the story is never lost within the single, rogue, quantitative value.

I too think that numbers within context are useful -- the problem arises when people *other than those directly involved with the initiative* remove the numbers from their context and consider them as though that's all there is to consider.

So one of the contexts that's important is the context of transmission -- who it is that gets to hear the story, how clearly they understand the importance of the non-quantifiable elements, and whether they pass that aspect along when they re-tell the story...

Which makes story skills, and consciousness of the impact of anecdote vs. statistics, both very important...

Ah, yes Charles, avoiding the trap of the Convenient Measure >>>> a nuanced challenge for sure. One way to stay true to ourselves and our missions is to keep the “what are we trying to achieve and how can we measure whether we’re achieving it” question in front of ourselves. As Jim pointed out earlier, “qualitative factors can be measured and analyzed” – we don’t need to give up on measuring the results that matter to us any more than we have to give up on getting a story told with just the right emphasis. Tools from social science as well as business have been developed over decades and we can adapt to our needs. Let’s learn about them and adapt them. More thoughts on this in 'SROI Act II: A Call to Action for Next Generation SROI' on REDF’s website, http://www.redf.org/learn-from-redf/publications. Thanks for the great discussion!

Cynthia Gair
REDF
www.redf.org

Hi Cynthia:

I was over at the REDF site while doing my research for this event -- very happy to see you here, and linking with your site and publications!

I'd agree that "qualitative factors can be measured and analyzed" to some extent -- I just think they tend to trim some of the exceptional outliers. But to me, the whole business of quantity and quality is itself an extraordinary koan, something I'd rather keep pondering on, picking up juicy insights from both nsides of the question as I go...

Thanks again!

I think these are the wrong questions, and the wrong answers. Mr Hipbone may be human. "We" (whichever "we" this is) may be human. But so are "they". Everyone is human, including 'quants', bankers, etc. That's the point of humanity - it's by definition inclusive.

So, being human isn't a difference which makes a difference to anything. The real differences between people, in the contexts being discussed here, are of two kinds: one, between people with different goals; and, two, between people with similar goals but different preferences about how to achieve them. The rest is noise, and it's not going to make any material difference whether people and/or organisations call themselves social entrepreneurs or not, whether they have a warm fuzzy attitude, or embrace a tough guy ideology, whether they see themselves as qualits or quants (which is a silly distinction), and whether or not they believe in being "holistic".

What matters is, what are you trying to do, how well are you doing it, and how can you do better?

Some people will genuinely put the work and the cause first, but they tend to lack self-awareness, a sense of the absurd, and any capacity for irony, so no-one else gets along with them. Other more personable people may be driven by ego - but there is no reason not to let them be, if they get the job done, and don't harm others.

The numbers will always have to stack up if you want to carry on doing the job, and it will have to be a good enough cause to attract enough support from enough people to make it so. But there are no magic numbers; and if the stories sound like fairy tales, they probably are.

Jon Griffith
Centre for Institutional Studies
University of East London

Jon Griffith sent the above to SocialEdge by email, and I posted it here as he suggested we might.

 Greetings Charles,
  I posted in this forum for last Year.In case You missed it I would first like to wish You& Yours a Safe,Healthy,and Sustainably Successful New Year and the same Blessing in the years of this new decade.
These posts resonate and say so much that I wish to comment on ,but not at the moment as I am folding my publicty brochures.I am still trucking,storing,grading adding value and shipping educational resources; currently to schools in the city of Bluefields,islands of Rama Cay,and El Bluff,Costa Caribena ,Nicaragua Meso America.I had a brief visit there last December courtesy Foreign & Commonwealth Office/British Embassy in Costa Rica to educational resources. Posted on my Facebook page and website http://www.grenadarelief.co.uk is going to be up-dated
Iam stillto meet my busines sangel? social venture capitalist? that is an oxymoron for sure.Any introductions to such. Look to see you some day on this hearth.Kind regards.
            Aye David

Hi Jon;

I think you may be conflating two senses of the word "human" -- one refers to all members of homo sapiens, whether they are in fact sapient or not (which I suppose would in turn depend on one's definition of sapience), while the other refers to certain qualities supposedly human, but notoriously *not* found in all members of the human race. That's a difference that makes a difference --

and, btw, how nice ton see a Bateson reference here.

I also think that having a group name for those who engage in business with social goals in the forefront of their thinking is important. I happen to loathe "social entrepreneurs" as a name, but haven't come up with a better one yet -- and in the meanwhile, the existence of a tendency or movement with a name has called forth many resources (university programs for instance) that wouldn't otherwise have been possible.

So that seems to me to be a material difference, too...

Getting past the stumbling blocks of your first two paras, though, I think your third, fourth and fifth bear serious thinking.

Regards...

 Hi Charles,
   I missed out a word in previous post..should be "deliver" educational resources.
 It seems that for me at any rate, who has self-funded this initiative Scottish registered Charity SCO 339331Surplus Educational Supplies Foundation for nearly five years now but has relied on, but not presumed on the goodwill and practical asistance of a number of transport companies here in Scotland and not least one state side sea freight corporation in order to deliver six ocean freight containers of educational resources to Grenadian Government schools,that Quants and Qualits must work together.
I am now at the stage where in order to attract funding from those with deeper pockets than my own I need th expertise of those who can provide the numbers/ On the other hand I will not compromise my integrity /autonomy to extend my reach.
I saw a need and tried to meet it. And Iam still trying to pick up that one starfish off the shore and throw it back in to the Sea. Had I waited to meet those individuals who could provide me with a metric and their own fund$ to validate my rationale I'd still be thinking about it.

"Entrepreneurs do not just manage budgets to create VALUE,they leverage VALUE to create budgets." Ronald Cohen
and " Social Entrepreneurs leverage resources they do not have,while managers and accountants do not ." Ronald Cohen see his book "Turning Risk into Opportunity"
" Entrepreneurs create jobs -not governments."
                                  Peter Cochrane
" The world's a playground and one shouldn't follow the rules at all times."
David Bornstein
   

Always a delight to read you, David, and to know that you're still valiant on the trail of ways to get educational materials to needy places.

Do you keep en eye out for contests or conferences that might offer you contacts for further funding? It seems to me you may need some "quant" help with your high "qualit" project!

This is a great post, thanks for sharing your ideas. I think that K-12 education in the US is a perfect window into this issue. Quants (federal and state education departments) want to be able to document the achievement of students while Qualits (teachers, unions, etc.) would assert (I think) that student achievement is broader and deeper than a set of test scores. Dealing with a human development issue like teaching and learning involves such a complex and interwoven set of factors, some which can be measured, some which can be observed but not necessarily measured, and some which probably should not be measured at all.

As an evaluator, I know that often the best evaluation design is one that integrates quantitative and qualitative data. Without using both, it is often possible to discern what is happening, but completely miss the why of it.

Imposition of an imbalanced model of evaluation, as has been happening in education; in other words, complete reliance on quantitative data for assessing successful teaching and learning, is warping assessment of public education. A child who comes to school hungry, ill and lacking sleep because their guardians were up all night fighting isn't going to score well on a test. Assessing this child's performance based only on quantitative factors would be damning on the teacher, school and curriculum.

Accountability is fine, asking the right questions and designing the right data collection to answer them should not exclude either category of data.

Thanks!

The points you make about education illustrate precisely why I think this conversation is no important.

I'd just like to stress *again* that a quant measurement may be of great value within a qualit context -- the problem is that if is frequently separated from that context and passed on as the sole measure of the worth of a (student or) project.

It gives *aid and comfort to reductionists* who operate entirely on the quant level, thinking all else is a "correction error" -- insignificant, or in any case too time consuming to bother with.

This discussion is so thought-filled and rich. Thank you, Charles, for starting it!

As I am reading, though, more questions arise for me than answers, the most important of which is, “What is the ultimate highest-potential end results we are trying to accomplish as organizations and as social entrepreneurs?” From there, “How can evaluation help us further those ends, and what kind of evaluation will do so?”

In some cases this may mean numbers. In virtually all cases it will mean stories. When we are dealing with changing the lives of people and communities, the most effective answer usually leans more towards “both/and” than "either/or."

Which raises more questions for me:
• How can we ensure our work is aimed at the change we want to create? (In my own experience, lack of social change results not from a failure to measure but failure to aim.)
• How can we engage individuals and communities to develop their own indicators of success (and then engage them in creating that success, rooted in their own strengths and values)?
• How can we create a spirit of learning and exploring and trying new things, to see what will work (and then measure that, to encourage more learning and exploring...)?
• How can we ensure that the measurement approaches we use do no harm? (Many of us are indeed convinced that current measurement systems indeed do harm...)

Lastly, I am reminded of a quote by Chogyam Trungpa, who notes, “The basic problem we seem to be facing is that we are too involved with trying to prove something, which is connected with paranoia and a feeling of poverty. When you are trying to prove or get something, you are not open anymore, you have to check everything, and you have to arrange it “correctly.” It is such a paranoid way to live and it really does not prove anything.”

Again, many thanks for this wonderful and rich conversation!
Hildy

These are great questions Hildy! Thanks for centering back to the results - and how does our evaluation help us further these ends. In my experience social change is messy, often the result of many actions, and both structural and cultural. So - to go back to Charles' point - we need both, and when we bias (especially as funders) to one, we may miss the qualitative.

Hildy's point about communities developing (and owning) their own indicators of success is enormously important - otherwise it is simply filling out the grant evaluation form. I have huge respect for the Women's Funding Network (WFN), which as a network of over 130 foundations globally has focused on how to provide tools to its membership which they can then use with their grantees. WFN (with support of the Kellogg Foundation) created a social change measurement tool called Making the Case (http://www.womensfundingnetwork.org/[…]/about-making-the-case), which is now being used by its members globally. It involves defining intended outcomes in five "shifts"
- Shift in behavoir
- Shift in definition (as in "domestic violence" being a public issue)
- Shift in engagement
- Shift in policy
- Maintaining past gains
As organizations lay out their strategies for acheiving any particular shift, this framework enables an evaluatation that acknowledges the complexity of social change. While this tool was created for non profits, I believe that the macro framework - and how it ties back to a theory of social change can be insructive to for profit social entrepreneurs.

Which leads to a question - for a while I was seeing everyone ask "what's your theory of social change" - to the point where it seemed the "theory" could almost be a fetish in and of itself. Yet constructively laid out, understanding the theory drives measurement. Have others found constructive dialouges about developing / evaluating theories of social change?

Jackie
Criterion Ventures

Hi jackie:

That's fantastic. I've only had a few minutes to look into Making the Case,but I am delighted you've pointed us all in that direction, and hope to get a deeper look -- and make good use of it too -- later this week.

Thanks again, it looks ton be just the kind of practical tool we need to help us get past the qualit/quant dilemma..

Hello again, Hildy:

And thanks for your kind words. To be honest, I prefer questions to answers -- everything is up in the air with a question, and there's room for a breath of spirit to sneak in and rearrange everything. Heidegger once (I believe) distinguished between puzzles (to be solved) and mysteries (to be dwelt within). Questions seem to be the style most fruitful with mysteries, while answers just close the book and leave people feeling satisfied.

Which I think aligns pretty well with Trungpa's remark, I think. I knew him when we were both students of Oxford, and took him to visit some friends of mine in a Benedictine Abbey.

So, anyway -- I love your questions...

Many thanks!

I have been saying for some time now that we need ton get out and about, we need to take the conversations we're having here to other blogs and web-pages, to get into the habit of holding our conversations across the wide, wide, world wide web.

Case in point: While we're here talking about metrics and whether they get in the way of our social entrepreneurial projects, the Harvard Business Review has a piece on its "Conversation" blog titled "Beware of Vanity Metrics" -- and Mario Morino has a post entitled "Social Outcomes: Missing the Forest for the Trees?"

http://blogs.hbr.org/[…]/entrepreneurs_beware_of_vanity_metrics.html
http://www.vppartners.org/[…]/0110_social-outcomes.html

Both of them look to me like limbs of the same conversation we're having here -- but instead of all three limbs getting together and sharing their readers and comments, they're three "separate" conversations.

I'm trying to remedy that with this post. I'd like to encourage you to read both articles, and post comments on the HBR site -- Marino's site doesn't accept comments. Talk about the insights you've gained here, over there, and the insights you've gathered from one or both those sites, back here.

Let's stir up some cross-communication, Let's get the web talking. Let's make sure that SocialEdge conversation reaches the readers of kindred sites, and let's read and do some talking over there, too.

Because when your name becomes familiar there, as well as here, there's a beginning of trust, and further collaboration becomes possible.

And because the web is ideally suited to that!

The Quants have been having a field day with microfinance lately! Based on seemingly just two studies, critics are questioning whether microfinance has made a quantitative improvement in the lives of borrowers, or has had any effect on poverty alleviation on a systemic basis.

After watching from the sidelines as the Quants bloodied the Qualits, the major U.S. based microfinance networks—ACCION, FINCA,the Grameen Foundation, Opportunity International, Unitus and Women’s World Banking—have decided to respond. We have come together—unprecedented in itself—and will be issuing a joint statement on Measuring the Impact of Microfinance: Our Perspective. In sum, we believe that the Quant studies’ insights are fragmentary. To obtain quantifiable data, researchers have had to ask narrow questions over short periods of time – which has significantly limited their ability to see the overall impact or dimensions of microfinance.

As networks representing a wide range of microfinance providers in more than 30 countries across five continents, our work regularly brings us face-to-face with clients. We know for a fact that access to microfinance has measurably improved their lives. Savings accounts facilitate the safe accumulation of assets. Loans allow them to buy tools, or raw materials in bulk, increasing productivity and/or lowering costs. Increased income allows them to pay school fees, to educate their children or to keep them in school. Home loans and mortgages allow them to improve their living conditions. Microinsurance reduces their vulnerability to risk.

Although quantitative research on microfinance’s impact remains underdeveloped, we fully support continued research by qualified academics in how to create inclusive financial systems and inclusive societies – societies where the individual is not shut out of what that society has to offer.

Watch for the release of our joint statement on February 16th.

Many thanks for this, Sarita -- your post is a powerful indicator of the problems with a pure-Quant approach.

Perhaps you can come back and alert us with the URL once your joint statement comes out.

Dear Charles et al.,

I am a would-be social entrepreneur, who is currently enjoying an informative sidetrack in social science. I intend to start getting my hands dirty soon, working on 'concrete' projects. But first I decided to try to come to a better understanding of the philosophical/intellectual background and foundations of social entrepreneurship.

This debate over quants/qualits has been running for some time in the social sciences. Basically it comes down into three groups (within social science that is). Those who want to use numbers to 'explain' everything. Those who want to use interpretation to 'understand' everything. And those qho don't really give a hoot about whether it's interpretation or 'hard' scientific analysis that gets used, as long as the intention is right. (Of course, that is a generalisation, but I'm painting a very large picture with a very large brush here...)

Anyhoo, in the end, numbers can be manipulated to mean just about anything. The intention must be clear when using them. Stories can be extremely informative and inspiring. They can also be too specific and not representative of overall trends.

And no, this post will not clear up any of the issues, but hopefully the obsession with numbers will be broken over the coming years. A balanced, practical mixture of 'hard' numbers, and 'real' and 'humane' considerations would be wonderful...

Rory

Thanks, Rory.

I think the problem social entrepreneurs face could in some ways be described as the problem sociologists face plus compounded interest. A given sociologist can quite acceptably publish a quant paper and leave it at that -- it's an acceptable mode of research, period. A social entrepreneur, on the other hand, might produce a report that is a blend of narrative and numbers -- and then be quoted as though the numbers were all there is.

I see that issue of the later separation out of quantity alone as an additional liability imposed by the "quant problem" on social enterprise.

Hey Charles,

Taking a more general view of this problem, the societal obsession with quantifiable data is (in my opinion) linked to the strong trend towards 'scientism' - or the belief that we humans can measure and learn to control everything in our environs.

The fact is that we cannot, and being (mis)guided by this belief can lead to serious problems. Taking the most salient example, one must look at mainstream macroeconomics. This entire line of thinking has reified the market, and has enabled and been enabled by the disembedding of the economic system from its social roots. Now we end up in a situation where a large proportion of human endeavour and energy goes into trying to feed the economy through GDP-growth-lead development models. These models have been effective up to a point, but are no longer feasible.

Social science (both sociology and economics included) should not be permitted to decouple their analysis from a social bedrock, a rpocess which paves the way for dehumanised conclusions - as has been allowed to happen in the case of most econometric modelling.

It is a major hope of mine that the field of social enterprise can be one of the driving engines in attempts to 're-humanise' analysis and promote a more balanced mix of 'hard' and 'soft' knowledge requirements. To that end, I tip my hat to you and the others out there who work patiently and steadily on that project.

I'd like that too -- I suspect we all would, including those whose emphasis is on building compelling metrics that are faithful to the SE impulse *and* those whose sympathies lie with story.

Here's hoping we can have our necessary quantifications and convey human impact through our narratives too...

What's a donor to do? Give money and read reports with stories about good works. Or give money and insist on numbers that demonstrate what and how much good works have been done? Ask the NPOs how they judge their success. Trust that they want to make change and make it effectively. Meet with them and ask if they are satisfied. Look at their metrics, become investors who are part of the team - use metrics, but don't demand them

As always, I think Patrick O manages to bring my somewhat theoretical issues to a more grounded space (I once had a Lakota shaman friend who said, memorably, that we're already airborne on this planet, but that the spirit brings us "down to earth"), and points out that the donor needs metrics for reassurance just as the recipient needs story to get the life of the project across.

I don't have much to add to that -- just to say I am grateful he said it.

Always a pleasuren -- like so much that goes on, here on the SocialEdge...

I agree. The more we focus on metrics, the less human we become. We focus on metrics to allow us to not personally interact with a problem and yet still try to "understand" it. The problem with that is, we have to interact with the people or the place to really understand a problem.

We can't all go and see or touch the places and things we are putting our money into, but it doesn't mean we need to focus on numbers to know our impact. The internet might be the cause of a lot of de-humanizing problems in our society, but when it comes to monitoring and evaluating where to give money, it can be used to actually make us interact more like humans, if we use it right. We can share videos of the work that we do, interviews with staff and community members, share photos and journal entries and instantly "chat" with the people we would otherwise have had to rely on reported statistics to know about.

So why are we still looking at numbers? When the next generation of donors arrives, the ones who have been social-media-d from birth, they will hopefully not evaluate projects based on numbers which they know can be re-adjusted easily in order to satisfy those of us who have no interactions with that place. They will instead support work that talks to them, that shows them actual failures and successes, through human eyes, not stats. They will invest in people who are honest and speak about the work that they do openly, not through formalized stat-filled reports. And they will hopefully realize that the groups making changes in the world are investing time in people, not just investing times in a certain section of numbers on a census report.

So when they ask you "What's the impact? How sure are you? Have you measured that? What are the numbers?" - show them, tell them, connect them, and entertain them with the details of the TRUTH, which can only be shown through human senses, not numbers. Because the answer from SOMEONE, even if it's not you, should be "Because I saw it, and here is the story."

Hi Daniela:

I'd love to see stories taking primacy over statistics, but it seems to me that the numbers are liable to offer a sort of baseline security for investors for the foreseeable future...

My hope would be that those who value the numbers would read and recognize the insights that you (and others here) are offering, and that those of us on the "story" side will do the same for the insights conveyed by those who know how funding (in particular) has a natural reliance on stats.

*

Part of the issue would seem to be that if you are working to feed hungry mouths, any mouth that's hungry counts -- and mouths can therefore be counted appropriately and helpfully.

If, on the other hand, you are hoping to educate future leaders, then sheer numbers is far less useful as a metric, for reasons that I explained in terms of Trevor Huddleston and his interactions with two of the school children in his care, Desmond Tutu and Hugh Masekela, earlier in this thread.

*

Thanks for your comment, and your work...

During the Tomsk experiment, project sponsor USAID tasked microfinance implementer FINCA with getting metrics to measure the success of the program. FINCA had some difficulty in fulfilling AID's notion of measurement, which was to get before-and-after income statements from program participants. It really isn't very often that I have reason to laugh out loud in my work, so thank God for small favors.

Here's the reality of the matter. Admitting to actual income, to anyone, ever, in Russia is akin playing Russian roulette with a loaded, chambered, safety-off 9mm Glock semi-automatic pistol. It would guarantee that Russian tax police come calling. Here is a photo of Russian tax cops in training:
http://i118.photobucket.com/[…]/_119053_Russian_tax_police_train-1.jpg

Lighter-weight versions or the same theme are available for smaller businesses, but it's still the same theme. If citizens are suspected to have a little extra money, tax cops want some, and they'll get it one way or another. Accounting, such that it is, is with gray books and white books. White books are what are shown to tax police routinely for tax reporting. Gray books tell the real story. AID was pressing FINCA for everyone doing business with Tomsk's microfinance bank to reveal their gray books so as to measure success of the program, which in turn would have killed the program and probably a few people along the way.

Even tax cops know the gray book/white book game. Accountants and bookkeepers are formally taught both gray and white bookkeeping systems during academic training. AID was clueless, at best.

On balance, it's not fair to mention the Russian system without also mentioning the US system. Here is a snapshot of the US system:
http://i118.photobucket.com/[…]/derivative.png

What is that?! That's a derivative.

Note the '=' signs in the mix. Those are arbitrary. Those are thrown in to claim that whatever the first line says is quantitatively the same as whatever the second line says, whatever the second line says is quantitatively the same as whatever the third line says, and whatever the third line says is quantitatively the same as whatever the fourth line says.

Problem: none of those four lines are quantitatively equal. If anyone disagrees, fine. In that case, it is incumbent upon whomever disagrees to prove that all four lines are =, i.e., equal, i.e. quantitatively the same.

Result: the debate would continue for years as each hair is split and examined, whilst I claim from the start that the entire equation is specious nonsense, a series of logical fallacies in the form of non-sequiturs and straw-man arguments.

Meanwhile, back at the ranch:
http://www.pbs.org/wgbh/pag[…]ist&utm_source=proglist

Wherein we learn that derivatives were and are developed and traded in a black box, closed to regulators and outside inspectors, so as to avoid the debate posited above. That process continues practically unabated.

But not to worry. When derivative financial products go bad, there is insurance available to cover the losses. American Insurance Group, AIG, too big to fail, trading in credit default swaps to protect investors from losses on trading derivatives. We've only seen the tip of the iceberg of CDS liabilities, but we'll see much more of the iceberg as an estimated $50 trillion+ in losses surface with US taxpayers -- sometimes referred to as 'people' or 'humans' -- set to be stuck with the tab.

---------------

Here is my preferred depiction of a derivative:
http://i118.photobucket.com/[…]/derivative2.jpg

Correction: AIG is American International Group, not American Insurance Group.

Hey, Terry:

I'm afraid my comprehension of economics is pretty primitive, and i don't have much to say / add.

I just wanted to thank you for posting, and for doing what it takes.

We often talk as if we must choose between quantitative and qualitative approaches to measurement. During this discussion this has been described as a choice between "telling stories" and having "hard data".
I would like to call readers attention to a book by Barry Kibel appropriately titled "Success stories as hard data:An introduction to results mapping" published in 1999 by Kluwer Academic/Plenum Publishers.The forward to this book by Allen Cohen describes Kibel's methodology as a merger of common sense and structured logic that retains the richness of real world success stories without sacrificing a hard nosed focus on quantitative data and measurable outcomes.I have blended Kibel's approach with another methodology described by Robert Brinkerhoff in his book "The success case method:Find out quickly what's working and what's not"[2003,Berrett-Koehler]to create a "success ladder" to quantitate narrative descriptions of succesful outcomes accomplished by a crisis intervention program i have beeb coordinating for the past 6 years.
It is possible to have your cake and eat it too regarding the quantitative/qualitative distinction by adapting the methodologies described in these 2 manuals. The approaches are content free and can be used by any social enterprise that wants to incorporate accountability into its endeavor without sacrificing scientific rigor.
I have also developed a 'Valued outcome scaling " methodology which serves a similar purpose and is described in a rather inaccessible journal[The National Association for Dual Diagnosis Bulletin}. I dont have an electroic version of this article but would be glad to mail a hard copy to readers who are interested.
If anyone has used the approaches described by Kibel and/or Brinkerhoff perhaps he/she would be interested in sharing experiences with me and others?

Just a brief welcome, Al --

Glad you came here and posted. I hope others will chime in -- and if not, to have something more substantive to say myself, tomorrow.

Clearly i need to read up a bit on Kibel and Brinkerhoff!

Since posting my original comment I have taken the opportunity to review previous posts on this topic and have a better sense of the issues raised and the concerns expressed. I also want to thank others for the wealth of information that has been presented,particularly links to other web sites.
One concern had been that cold numbers somehow pose a threat to our humanity.Certainly this is warranted when we encounter phrases like "It doesn't exist if we can't measure it",which some overly zealous quants might make without reflecting on its absurdity and arrogance. measurement is a tool and it can either help or hinder our endeavors depending on what we want to accomplish.
One previous post was concerned with the extent to which measurement reflects the need to meet someone else's goals,as when we have to prove that our project is working by convincing a skeptical audience[e.g.,an external auditor]that our project is working by providing hard data when we know in our hearts that we have encountered many successful applications that we can capture in narrative stories.
The quality improvement model I have ben trained to use is described in detail by my mentor{Dr Donald Wheeler,in his book "Making Sense of Data",published by SPC Press in 2003]which translates the concepts of statistical process control developed for industrial/manufacturing applications into language that is more consistent with human service applications.
Essentially,this approach involves re-iterative answers to the following 3 questions:
1]What are we trying to accomplish? Answering this question requires us to be clear and specific regarding our intended outcomes. We can start with global generalizations,but as we go through several cycles of applying the model we will be forced to become more and more concrete.
2]What methods will we use to accomplish our intended objectives? Answering this question requires us to establish a logical and plausible relationship between our outcomes and the processes we put in place to accomplish them.
3]How will we know if we are successful? Answering this question gets to the crux of the qualitative/quantitative distinction but it puts us in the driver's seat. We want to be able to make a knowledge claim that is supported by the best available evidence and we owe it to ourselves and our customers/collaborators to spell out as clearly as possible what form this evidence will take. One advantage of quantitative data is that it usually results is that lends itself to operational definitions that reduce the likelihood of communication errors between observers. However,as many have noted,some complex phenomena cannot be reduced to events which can be readily counted and objectively measured. Nevertheless, if we are to keep ourselves honest we must be able to spell out the conditions under which we will reject the hypothesis that what we intended to do is working. Failure to do so leads down the path of wishful and potentially delusional thinking. This is the essence of the scientific method ,shorne of fetishistic trappings that make it appear superior to all others. If we can't correct our mistakes based on honest feedback about the results we are accomplishing we are doomed to the insanity that Einstein so poignantly described-doing what we have always been doing and expecting different results!
Sorry for the length of this post but I wanted to give a sense of how this generic model can be applied with either qualitative or quantitative data,as long as we are committed to spelling out the types of evidence that would allow us to confirm or disconfirm our hypothes about what methods will best accomplish our intended results.

Thanks for this, Al.

I wanted to drop in a quick note to all who are reading here, that there's a somewhat parallel conversation about qualitative vs quantitative research in the social sciences, which just led to an interesting post on the Center for a New American Security blog maintained by Abu Muqawama:

http://www.cnas.org/[…]/revenge-nerds.html

I also think the SocialEdge event on The Theory of Change which will start tomorrow, Tues March 9, 2010, will be of interest to all here...

Seems obvious that schools' #1 value is social value and transformative value doesn't it? Yet we make testing metrics the sole form of measuring the worth of the learner and the teacher. Now we even bribe 2nd graders to read! This may not raise test scores but it will raise their moral compass to Goldman Sachs. Of course schools are a value for the economy eventually but first the fire of passionate teaching and learning must be lit!

Having just finished a discussion in a Social Entrepreneurship class on the merits of SROI, I wonder whether much of what is qualatatively measured today will soon be quantified, particularly as investment in green and social enterprises increases. It seems that there is difficulty in monetizing social and environmental impacts, particularly with any real repeatability, precision, and accuracy. However, I strongly believe, that many of those who you would label "quantifiers" who have ended up in financial and statistical fields, may soon populate similar fields within social enterprises and ideally provide direction in developing the metrics necessary to better quantify the social and environmental impacts of different enterprises. Ultimately, my hope is that there is a standardization in different industries to award those for heroic work and silence many organizations who have benefited because of the confusion associated with quantify their impact (lack there of).