To Standardize or Not to Standardize: Some Core Principles

The question we most often hear first from both investors and social enterprises thinking about measuring their impact is, “What standard impact measures exist in my field?” If a standard does exist, there are obvious benefits to adopting it. A primary benefit is the enhanced potential for collaboration to solve problems. One of the clearest illustrations of this is carbon emissions, which only a few years ago were nothing more than the stuff we’re all exhaling. Now there is a large community concerned with global warming, and because tons of carbon emissions have become a de facto standard measure of progress on this issue, everyone who cares can focus on lowering the number of tons.

Another benefit of adopting a standard measure is that an entity’s performance in terms of a given measure can be compared to that of other entities, making the relative scale of its impact visible. Wal-Mart may report that it reduced its carbon emissions since last year by 1 million tons, and Grupo Ecologico Sierra Gorda that it reduced carbon emissions by 500,000 tons by reforesting thousands of hectares, so at first Wal-Mart seems to have done twice as much as Sierra Gorda… but is this accurate? Standardized measures may facilitate the coordination of effort to solve problems, as well as choices about the best use of resources.  But the first principle for impact metrics is that they should always be viewed in context. 

To frame the context, we consider it a best practice to apply two simple tests of a given measure. The first test is, “Does the measure capture something meaningful?” The second is, “What does it tell us about progress relative to the desired goal?”

Is it meaningful?

For a measure to be generally accepted as meaningful, there must be consensus on the part of those knowledgeable about the subject that the measure does capture something of value. In the case of carbon, while a few people still doubt that global warming exists, the scientific community does not, nor does it doubt that man-made carbon is the key. As such, carbon emissions, and the lack thereof, have acquired value, and a basic, commonly accepted measure is now tons of carbon. By contrast, in many fields debates still rage about what measures really matter. If the goal is helping people break the cycle of poverty, is the essence of this that a person has a job? Or that a person has a job and earns enough income to save money each month, or something else?

In these fields where there is less consensus about how to measure impact, we are seeing a process by which practitioners and investors in the field and the research community: a) focus in on sub-sectors that target more specific contexts, and then b) come to consensus about standard measures of success and impact that speak to that particular context. For example, within the field of poverty eradication and economic development, one sub-sector is community development venture capital (CDVC). CDVC funds invest in geographic regions of the United States where venture capital typically does not flow. These funds generally gauge their success by looking at the combination of the median income level of the geographic region in which portfolio companies are based and the number and quality of jobs generated by portfolio companies as measured by the wages and benefits offered employees.  Whereas another sub-sector that shares economic development goals, such as microfinance, may focus on a very different set of measures to gauge progress out of poverty, such as the building materials used in the homes of borrowers and the growth in the size of loans successfully repaid over time. The process by which consensus on measures is reached varies, and may take many years.

Standardization of impact measures is catalyzed when a compelling reason to come to consensus emerges. Perhaps the most compelling reason of all is when buyers agree to make their purchase decisions based on a certain measurement. Carbon emissions reductions trade by the ton on both regulated and voluntary markets, so groups have a big incentive to measure that they not only replanted forests or increased fuel efficiency, they reduced emissions by X tons. Consensus is hindered when there is not a dominant reason for the parties to agree on a shared measure, and/or when a verifiable measurement seems difficult to take.

Once a meaningful measure has been accepted, the next critical test is whether it is applied in a way that informs us whether what is happening actually matters. Too often “reaching” thousands of people or “reducing carbon by millions of tons compared to last year,” sounds so impressive that folks reward a big number with their support without looking beyond the number to see whether a meaningful result has been achieved. The cardinal sin of impact measurement is to mistake size for value.

We’ll talk more about emerging standards for impact measurement in future posts.