Evaluating Community Change

December 30, 2016 4:19 PM | Anonymous member (Administrator)


Source: Stanford Social Innovation Review

In 2012, the Greater Cincinnati Foundation (GCF) made a big bet on the ability of collective impact to accelerate systems change. It committed $3.5 million over five years to a number of local “backbone” organizations, with the hope of achieving Community-level social impact on a range of complex problems in the region.

Backbone organizations—typically composed of independent, funded staff dedicated to an initiative—are an important part of the cross-sector, collective impact approach to social change (other elements include a common agenda, mutually reinforcing activities, shared measurement, and continuous communication). Backbone staff help guide the vision and strategy of an initiative, support aligned activities, establish shared measurement practices, build public will, advance policy, and mobilize resources. These activities can all sit within a single organization, or they can have different roles housed in multiple organizations.

GCF’s funding aimed to provide long-term operating support to six different backbones, adding a seventh in 2014, so that they could scale up their programmatic efforts and partnerships, and ultimately improve their ability to drive change in areas like education, workforce development, and environmental sustainability. The foundation also funded a community of practice to support knowledge exchange among the backbones. This year, GCF asked our organization, FSG, to determine if their bet was paying off.  FSG has developed a guide to evaluating collective impact. Here, we share an example of putting this evaluation approach into action. 

Backbone organizations supported by the Greater Cincinnati Foundation included:

Evaluating Community Transformation

Evaluating any aspect of collective impact effort is challenging—change is happening at multiple levels, in non-linear ways, throughout the life of an initiative—and evaluating backbone organizations is no exception. But tracking progress across multiple success factors—such as social, political, and financial capital—can be helpful, and for this project, we ultimately developed a framework to capture community transformation on a variety of levels. The framework is rooted in the belief that innovative and structured collaboration leads to a strengthened civic infrastructure, which translates into systems-level change, and thus accelerates community-level impact.

  • Civic infrastructure. Is there a web of strong, trusting relationships between people and institutions? Backbone organizations play an important role in improving civic infrastructure by building connections, marshalling resources, enabling community engagement, and sharing knowledge.
  • Systems-level change. Insight on this level took shape through seven indicators. We looked for: a culture of learning, dialogue, experimentation, and reflection; formal organizations making changes in their practices; shifting behavior of the target population; increasing funding streams; evolving social and cultural norms; progress on advocacy and public policy goals; and resources and capacity allocated to supporting partners. Backbone agencies engage with their partners on these medium-term outcomes, because they actually influence the attainment of long-term outcomes.
  • Community-level impact. Is the collective impact initiative achieving the long-term, population-level changes it seeks? Backbone agencies help build data collection systems to track impact across health, education, economic growth, and other indicators in the region.

The Greater Cincinnati Foundation’s theory of change to community transformation.

To evaluate the impact of the backbone organizations on each of these levels, we reviewed publicly available information, grant reports, and internal information provided by GCF and the backbone organizations themselves. We also interviewed backbone leaders, stakeholders, and GCF staff and board members, and surveyed organizations and individuals involved in each collective impact effort.

Applying the Framework

To illustrate how this framework helped us see the progress of each backbone across a number of metrics, let’s look at Partnership for a Competitive Workforce (PCW), a workforce development initiative in the Ohio-Kentucky-Indiana tri-state region, and one of the seven backbones GCF supported. PCW’s mission is to meet employer demand by enhancing the skills of its current and future workforce by fostering collaboration between employers, chambers of commerce, workforce boards, educational institutions, and service providers. The initiative has three main objectives:

  1. Connecting businesses with qualified workers
  2. Aligning education with employer needs
  3. Improving work readiness services to help individuals obtain and retain gainful employment

In terms of building a trustworthy civic infrastructure, PCW has successfully brokered relationships among three sectors that traditionally have not worked well together: business, higher education, and workforce investment boards. PCW has acted as a neutral convener, using data (from workforce needs surveys, for example) and mutual interests (developing a strong talent pipeline) to spark conversations and find win-wins. They collected data from service providers and government agencies on workforce development services, awarded credentials, employment status, current income, and demographic information of clients, for example, and used it to facilitate conversations on how to improve providers’ performance.

PCW has also excelled at systems-level change, in two areas: 1) establishing a culture of experimentation and learning, and 2) gaining active participation from partners across sectors. In collaboration with another GCF-supported backbone called Skyward, PCW facilitated the creation of a talent pipeline for the advanced manufacturing industry—the second largest source of private sector employment in Northern Kentucky. Employers were struggling to find skilled workers who had a technology background and could take on roles such as welder, or pharmaceutical or electro-mechanical technician. At the same time, Gateway Community College was having difficulties recruiting students for classes in advanced manufacturing. PCW brought both sides to the table to design a predominantly online training program through which students could obtain an industry credential in less time and at a lower cost, and move into job opportunities quickly.

On the community-level impact front, PCW has served more than 10,000 individuals across five career pathway partnerships since 2008. Of that group, 89 percent of individuals completed training, 78 percent obtained employment, and 67 percent retained employment for more than 12 months. Those who participate in a career pathway program earn up to $7,500 a year more than the previous year.

Community Transformation Across the Backbones

We found a very strong civic infrastructure, a high level of systems change, and positive community-level impact at each of the organizations GCF backed.  A few examples follow:

Civic infrastructure: The Diverse by Design (DBD) initiative facilitated by Agenda 360 and Skyward brings 150 companies together in a community of practice, with 400 volunteers and 5 action teams to strategize around inclusivity, diversity, and culture within their businesses and the region at large. Experienced companies are mentoring others on supplier diversity and the creation of employee affinity groups.

Systems-Level Change

  • A culture of learning, dialogue, experimentation, and reflection.For Green Umbrella, experimentation has led to numerous independent, highly resourced initiatives, such as the Red Bike program (50 stations and more than 100,000 rides in 15 months), Taking Root campaign (planting of 170,000 trees), and the Tri-State Trails Master Plan (1,000-plus miles mapped).
  • Funding streams are increasing. LISC has tapped into this network to advocate for a land bank in Cincinnati, directly resulting in increased investments for demolishment and development in the region. Most notably, over the past 5 years, LISC has aligned funding totaling over $664 million in grants, private investment, and market tax credits, including a $29.5 million HUD Choice Neighborhoods implementation grant in Cincinnati.
  • Progress on advocacy and public policy goals. StrivePartnership and SB6 have helped the Preschool Promise coalition to expand access to high-quality preschool through an annual $15 million levy.

Community-level impact. The StrivePartnership has seen positive, sustained improvement across the vast majority of their student indicators, with 91 percent of its 40 indicators currently trending in the right direction.

Room to Grow

Finally, while we uncovered many successful efforts, we also saw some areas where all the backbones could grow. These included:

  • Incorporating an equity lens when evaluating community-level impact by developing a set of goals, strategies, and metrics that disaggregate progress by race, class, gender, geography and other important factors 
  • Increasing community engagement through all stages of design and implementation (for example, upfront input, co-creation, and sustaining solutions) by embedding the community voice in every aspect of the work
  • Developing community leaders into “system” leaders who can get others to understand the complexity of the change process, and focus on the “health of the whole” versus just the success of their own organization or interest area

The good news is that backbone leaders are already shifting their practices to address racial equity and co-create solutions with communities.

Change requires time and patience; however, most collective impact efforts are probably making progress on at least one level of this evaluation framework. But no matter where your effort stands, it is important to track and articulate these metrics.


Powered by Wild Apricot Membership Software