Finding out if FORGE works

 

We have been reading about FORGE’s situation in the blogosphere now for a few months, and, like many people, have hoped that FORGE will solve their fiscal problems and have the chance to carry on. But certain questions about what’s happening with FORGE have dogged us – so much so that we made our own offer of pro bono consulting. See Change is a research and evaluation firm, and after reading post after post about whether FORGE will make it, or whether donors will contribute to the organization, our question has been, “Why not simply evaluate FORGE?”

  

No doubt many donors are moved emotionally by FORGE’s model, the plight of the African refugees who are their target population, and perhaps also by Kjerstin and her staff’s initiative and pluck. These donors are probably already giving to FORGE, and will likely continue to, as long as FORGE staff can find them (such as by implementing Curtis’s recommendations about developing a systematic and sustainable fundraising plan).

 

But there are potentially even more donors – individuals and especially foundations – who may be moved emotionally, but who don’t let their emotions alone determine whether or not they write a check. For many of us – though still far too few of us – our pragmatism interacts with our emotions. We want to help a deserving cause, but only if our contribution of time or money will be used wisely and well.

Kjerstin’s experiment in radical transparency was risky in that it may have raised concerns among more pragmatic, even cautious donors about the fiscal and fundraising strategies that FORGE has employed. Though welcome, such honesty may have encouraged some donors to sit on the sidelines while more evidence is gathered. The subsequent consulting support the organization has received may have gone a long way to convince these pragmatists that FORGE’s fiscal house can get stronger. But using money wisely and well in the social sector is only partly about good fundraising and money management. It also relies on the effectiveness of the intervention being used to create social good. Despite the solid face validity of the program model, we still don’t know if FORGE actually works. Does it actually improve well-being for African refugees? Beyond improving well-being, does FORGE actually develop skilled refugee leaders who will use their skills in new situations? Even if FORGE is successful at creating these types of outcomes, what is the likelihood that these outcomes will make a significant difference in African communities over the long term?

 

Our interest as an organization is to provide evidence about effective interventions, so that donors can know how and where to deploy their resources for the greatest social return, and so that non-profit organizations can know how to improve their practices to get closer and closer to their goals. We believe that an essential next step for FORGE is to develop an evaluation strategy, so that they can begin to provide evidence about their effectiveness to the world, and so they can find out about ways to get even better at the work they do. We will work with FORGE to set up just such a strategy in the coming months.

 

FORGE is not a large enough organization, nor are data collection conditions on the ground adequate to support the most rigorous evaluation approaches, such as the use of randomized comparison groups. When we first suggested to Kjerstin that an evaluation would be a useful strategy for FORGE, one of the challenges she mentioned was the “data-poor” environment presented by remote refugee encampments. Indeed, collecting reliable data in such a setting presents significant challenges to systematic research. However, FORGE’s situation is not that different from that of many non-profit organizations, even in the U.S. Getting good data is difficult almost anywhere, and designing a smart study is as much art as science. Most non-profits will never have the resources or logistical support to afford a methodological “Cadillac.” Yet there is still much that we can learn about FORGE and its program model from a wide range of research strategies.

  

There are four areas of work See Change will pursue with FORGE:

 

· Developing a theory of change

· Identifying priorities and methods for conducting a process evaluation

· Identifying a small handful of “best metrics” for an outcome evaluation

· Helping them use their stories as data

 

In the spirit of transparency, we will post regular updates about our work with FORGE staff, and ultimately, our findings. Up next: examining FORGE’s theory of change.