My last post described how Charity Navigator's decision to incorporate outcomes data into its rating system is an important step towards creating a culture in which nonprofits strive for impact and donors reward it. But the task of developing meaningful and appropriate outcome measures is complex. It will require cooperation across the nonprofit sector.
Once they have the incentive, I believe most nonprofits of a certain size can track meaningful data that bears on their impact. The Hewlett Foundation and McKinsey recently partnered in a study on the nonprofit marketplace that confirmed that such data is surprisingly knowable if nonprofits invest modest time and money. The Working Group for Effective Social Investing is also hoping to develop new tools to determine which nonprofits generate the most social value.
The whole nonprofit sector will need to participate in developing outcome measures that are cost-effective for nonprofits to track, and useful to organizations as well as to donors. Ideally, measures should be tailored to different kinds of nonprofits and allow refinement over time. Larger organizations can invest in more complex tracking systems that get closer to actual impact. But small or growing nonprofits might do best tracking interim measures of success while waiting for data on outcomes. As Jacob Harold mentioned in his guest post, when evidence-based research on outcomes is impossible, information from the organization about its goals, strategies, capacity, and progress--combined with the views of stakeholders--can provide good proxies for impact. While not all organizations that measure outcomes necessarily have great impact, donors should have serious doubts about nonprofits that can't provide any information about progress towards accomplishing their goals.
It's also important to make sure that donors understand that numbers don't tell the full story. We don't want donors or nonprofits to prioritize short-term, measurable outcomes at the expense of important long-term--but less measurable--impact. It's easy to count beds filled in a homeless shelter or acres of land purchased for conservation, but harder to measure long-term changes in attitudes or behavior.
Even for two nonprofits doing the same kind of work, outcome measures or benchmarks might differ depending on factors such as the population served, time horizon, political or cultural context, geography, etc. For example, let's take two nonprofits that house homeless individuals: One helps the newly homeless attain transitional housing. Another provides the same service to chronically homeless individuals suffering from mental illness and drug addiction. Based on a measure of how many clients remained housed a year later, the first nonprofit might seem the natural choice for a donation. But its better outcomes would actually result from serving a less challenging set of clients.
Given all this complexity, how exactly could Charity Navigator cooperate with others in building an outcomes-driven culture? Here are several options:
1.Gather better programmatic data from nonprofits, as a short-term substitute for the long-term goal of data on outcomes (DonorEdge is doing this)
2.Gather data from experts (Nonprofit Knowledge Network is working on this)
3.Gather data from other constituents (GreatNonprofits is working on this)
4.Rate nonprofits on how transparently they report outcomes (IntelligentGiving in the UK does this)
Charity Navigator's best option may be to shift its ratings from financial ratios to measures of transparency (#4) while supporting other organizations that gather and aggregate the first three types of information. Charity Navigator's CEO has identified accountability and transparency as one of the three essential components of charity evaluation. While transparency ratings don't measure outcomes directly--and transparency may have its costs--they are a step above Charity Navigator's current financial ratios. Nonprofit Knowledge Network and GreatNonprofits are already developing the systems necessary to gather the opinions of experts (#2) and other constituents (#3). GuideStar will likely be the best aggregator of all this information and outcome data from nonprofits (#1) so that nonprofits are not forced to upload information to multiple sites.
While measures of these sorts are crucial, donors will still need to combine them with other--perhaps less quantifiable--information to identify the best nonprofits. As a sign in Albert Einstein's office read, "Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted." The challenge for Charity Navigator and its partners going forward will be to create demand for impact, while giving donors immediate access to multiple kinds and sources of meaningful information when making charitable decisions.