Douglas E. Luffborough, III is a doctoral student at the University of San Diego. He is also the Executive Director at Turning the Hearts Center.
Performance evaluation tools and designs
The onset of New Public Management (NPM), a term that refers to the restricting of public management using market-oriented strategies to “provide for social needs and increase effectiveness, efficiency, and responsiveness of public services” (Smith, 2011) has played a pivotal role in changing the size and role of the nonprofit sector. NPM is also closely related to the devolution of public programs to community levels of government and society, a move that has led to a more prominent role for nonprofits which are seen to be closer to their communities. This shift has led to a broader movement and conversation around improving the performance and effectiveness of government supported and funded nonprofit organizations (p. 33).
The economic crisis of 2008 through 2010 has had profound effects on government and private funders. This crisis has produced a wave of funding cutbacks, forcing many nonprofit organizations to reduce services, lay off staff, turn away clients, or in some cases have to close their doors altogether. Today, more than ever, nonprofit organizations are forced to do more with less and face increasing demands for accountability, transparency, and improved performance (Smith, 2011, p.35).
Evaluation tools
There are several evaluation tools that nonprofit organizations can use but the first tool highlighted in the literature was the use of logic models. Logic models were developed and required as a result of devolution under the NPM era. Logic models provide a one page road map toward programmatic and organizational success which consists of the following items: problem statement, goals, objectives, activities, outputs, and outcome measures (short and long) (Kaplan, 2001).
The purpose of a logic model is to describe and articulate program theory. There are many potential uses of the logic model tool in explicating program theory for a variety of purposes throughout the life span of programs: for assessing the feasibility of proposed programs and their readiness for evaluation, for program development, for developing performance monitoring systems, and for building knowledge (Savaya & Waysman, 2005). The majority of government funded contracts require logic models as a part of the request for proposal process. Therefore, understanding the theory and practice behind logic models helps nonprofit compete for shrinking federal grant awards.
The next evaluation tool used by nonprofits is the Balanced Scorecard. The Balanced Scorecard was first introduced to the nonprofit sector in 1996, shortly after the founding of the Social Enterprise program at the Harvard Business School. The program conducted a survey and learned that executives and board members of nonprofits consistently rated performance measurement as one of their top three management concerns (Kaplan, 2001, p. 357).
Kaplan (2001) shared that The Balanced Scorecard was initially developed for the private sector to overcome deficiencies in the financial accounting model, “which fails to signal changes in the company’s economic value as an organization makes substantial investments in tangible assets, such as the skills, motivation and capabilities of its employees, customer acquisition and retention, innovative products and services, and information technology” (p. 357).
Since its inception, The Balanced Scorecard has helped organizations to implement new strategies rapidly and effectively, leading in some cases to dramatic performance improvements. The nonprofit literature concurs with the need to articulate a multi-dimensional framework for measuring and managing nonprofit effectiveness and the evidence suggests that The Balanced Scorecard would seem to provide just such a framework (Kaplan, 2001).
The third and final evaluation tool is Efforts-To-Outcomes (ETO). ETO is a case management data collection database used for storing demographic information, logging case notes and tracking qualitative and quantitative performance evaluation measures. The company that created the ETO software is called Social Solutions based in Baltimore, Maryland. Social Solutions is the leading provider of performance management software for human services, connecting efforts to outcomes, people to social services, and service providers and communities to funders (http://www.socialsolutions.com/non-profit-software-demo-video.aspx).
When it comes to developing quality performance evaluation measures the ETO software is a nonprofit software solution designed to help you measure the incremental progress of your participants, understand the effectiveness of your programs, and demonstrate impact to funders and key stakeholders both quickly and efficiently. Performance evaluation measures can be tailored made to match organizational mission and outcomes and the ETO software can track and evaluate the impact. Most commonly used evaluation designs include posttest only and pre-and posttests, however, ETO has the capacity to track more quasi-experimental designs that ensure greater reliability and validity to performance outcomes.
Quality of evaluation design
An evaluation’s quality is related to the design employed. The studies reviewed indicated that many nonprofit organizations have relatively weak evaluation designs in regards to rigor, reliability and validity (Hoefer, 2000, p. 172). Some of the barriers mentioned earlier are possible causes to weak evaluation designs however not all inclusive. In review of the research the post-test-only evaluation design, in which client results are measured only at one point in time after program completion is considered very weak in defending against threats to internal validity (p. 172). Both smaller and newer nonprofit organizations elect to have a posttest evaluation design primarily because their funder required that they had one or because they are one of the easiest to design. Generally, the questions reflect if clients were satisfied with the services implemented and do not necessarily address behavioral changes as a result of the services provided.
Developing a more rigorous approach would be to administer a pre-and post-test evaluation design. Research shows that this is a more popular design that nearly half nonprofit organizations utilize. In implementing this type of design, clients’ results are measured once before and once after the program to determine if there is any change as a result of the services provided (Hoefer, 2000). Well designed pre-and post-test evaluations can be effective in showing positive and/or negative changes in behaviors, attitudes, and choices as a byproduct of services. Nonprofit organizations that conduct this type of evaluation design will be able to better demonstrate overall organizational effectiveness than just having a posttest evaluation design.
A third design, known as time series, helps control for more types of internal validity threats than post-tests and pre-and post-test designs. It is similar to the pre-and post-test design, but clients’ results are measured several times before and after the program. However, one challenge with this type of evaluation design involves follow-up and acquiring information after clients have exited the program as well as a lack of funding available to conduct follow up services (Hoefer, 2000).
The fourth and final evaluation design is the most rigorous provided to clients. The comparison group design controls for more threats to internal validity than any of the previous evaluation designs. This design is also known as a quasi-experimental or semi-quasi-experimental evaluation that focuses on a treatment group (those that receive the proposed intervention) versus a controlled group (those that do not receive the proposed intervention). The biggest challenge with this type of evaluation design often includes length of time for implementation and costs associated with the data collection and analysis process (Hoefer, 2000).
Hoefer (2000) concludes that “Evaluations are being conducted, but the level of methodological quality appears to be rather low” (p. 174-175). Research shows that most nonprofit organizations create performance evaluations not because they want to but because of growing funder expectations regarding accountability. In this case, conducting evaluations becomes a way “to confer legitimacy rather than a way to improve programs” (p. 175).
Conclusion
Chelimsky (1994) states, “Evaluation seems destined to play a major role both in the formulation of new programs and policies and in the assessment of their achievement” (p. 342). From the research it is clear, program evaluation is not yet producing accountability; however, the “glass of evaluation may be considered half full, in that evaluations are being conducted” (Hoefer, 2000, p. 176) but the fidelity as it relates to reliability and validity remains to be seen throughout the nonprofit sector.
References:
Alexander, J. (2000, Spring). Adaptive Strategies of Nonprofit Human Service Organizations in an Era of Devolution and New Public Management. Nonprofit Management & Leadersip. 10(3). Jossey-Bass.
Chelimsky, E. (1994). Making Evaluation Units Effective. In J. S. Wholey, H. P. Hatry, and K.E. Newcomer (eds.), Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass Publishing.
Herman, R., & Renz, D. (1998, Fall). Nonprofit Organizational Effectiveness: Contrasts Between Especially Effective and Less Effective Organizations. Nonprofit Management & Leadership. 9(1). Jossey-Bass.
Hoefer, R. (2000, Winter). Accountability in Action? Program Evaluation in Nonprofit Human Service Agencies. Nonprofit Management & Leadership.11(2). Jossey-Bass.
Kaplan, R. (2001, Spring). Strategic Performance Measurement and Management in Nonprofit Organizations. Nonprofit Management & Leadership. 11(3). Jossey-Bass.
Savaya, R. & Waysman, M. (2005). The Logic Model. Administration in Social Work. 29(85-103). Routledge Taylor and Francis Group.
Scott, W. (1995). Institutions and Organizations. Thousand Oaks, CA. Sage.
Smith, S. (2011). The Nonprofit Sector. In M. Edwards (ed.). The Oxford Handbook of Civil Society. Pages 29-38. New York: Oxford University Press.