Leveraging Language Models like ChatGPT for Evaluation

With algorithms, automation, and AI here to stay, I have been wondering what this partnership with AI might look like for consultants, clients, and colleagues. How might we leverage AI to support our data and evaluation work?

Clearly I am not alone in wondering this. ChatGPT gained over 1 million users during its launch week, and has quickly become the fastest-growing app in history, with over 100 million active within its first two months. Wow.

How Can ChatGPT Support Evaluation?

TL:DR? What I learned is that ChatGPT requires persistence. The current version works best with simple text-based questions. More detailed and nuanced answers require patience, wordsmithing, follow-up questions, and fact-checking. As with any tool, AI can be prone to errors and misinformation, so triangulating sources is essential.

Overall, I found that ChatGPT has the potential to offer foundational guidance to consultants and clients when looking for inspiration around:

Developing evaluation questions
Considering evaluation methods
Identifying data sources
Explaining basic Excel, R, or other code

I view this and other AI as supplementary tools or exercises for consultants, clients, and funders to begin a discussion about what possibilities and opportunities align with a project’s vision, values, goals, and budget, rather than a replacement for conversation, collaboration, lived experience, intentionality, and meaning making.

Ok, let’s see ChatGPT in action. Buckle up and join me on my journey as I asked ChatGPT to write an evaluation plan for a community navigation program. Since I’ve worked in this area, I could easily compare the results against my own practice and experience. Note that in some areas, I have adjusted the formatting from the original output to make it easier to read here.

Writing an Evaluation Plan

I started out by asking ChatGPT to “write a mixed-methods program evaluation plan for a community navigation program.”

The result?

A very high level plan with all the jargon. Due to its basic and brief nature, I consider this information to provide more of a template or outline than an actual plan.

The utility?

This might be helpful for folks who have a more novice understanding of program evaluation and are looking for ideas to begin conversations with a potential consultant or funder. Additional follow-up questions and detail are needed to translate this outline to an actionable and meaningful evaluation strategy.

Introduction

The community navigation program aims to connect community members with various resources and services available in their local area. This evaluation plan utilizes mixed-methods to provide a comprehensive analysis of the program’s effectiveness and areas of improvement.

Evaluation Questions

The following evaluation questions will guide this mixed-methods evaluation:

To what extent has the community navigation program been effective in connecting community members with resources and services?
What are the strengths and weaknesses of the community navigation program?
How satisfied are community members with the program?
What are the barriers and facilitators to accessing resources and services through the community navigation program?

Data Collection Methods

Quantitative data collection:

Surveys: Administer surveys to program participants and non-participants to gather quantitative data on program effectiveness, satisfaction, and barriers to accessing resources and services.
Administrative data: Collect data from program records and administrative sources to assess program utilization rates and demographics of program participants.

Qualitative data collection:

Focus groups: Conduct focus groups with program participants and non-participants to gather qualitative data on program strengths and weaknesses, satisfaction, and barriers to accessing resources and services.
Interviews: Conduct interviews with program staff and key stakeholders to gather their perspectives on program effectiveness, strengths and weaknesses, and recommendations for improvement.

Data Analysis

Quantitative data analysis:

Descriptive statistics: Use descriptive statistics to analyze survey data on program effectiveness, satisfaction, and barriers to accessing resources and services.
Regression analysis: Conduct regression analysis to determine the relationship between program participation and outcomes such as resource utilization and service satisfaction.

Qualitative data analysis:

Thematic analysis: Analyze focus group and interview data using thematic analysis to identify themes related to program effectiveness, strengths and weaknesses, and recommendations for improvement.

Evaluation Plan Outputs

The following outputs will be generated by this mixed-methods program evaluation:

A report summarizing the findings of the evaluation, including key strengths and weaknesses of the community navigation program and recommendations for improvement.
A presentation of the evaluation findings to program staff and key stakeholders.
A summary of the evaluation findings to be shared with program participants and the broader community.

Conclusion

This mixed-methods program evaluation plan provides a comprehensive approach to evaluating the effectiveness of the community navigation program. By using both quantitative and qualitative data collection methods, the evaluation will provide a more in-depth understanding of the program’s strengths and weaknesses and provide valuable recommendations for improvement.

Developing evaluation questions

With the foundation created, I started probing ChatGPT with more information about the project. Next I asked, “What are program evaluation questions for a community-navigation program in a large city?”

The result?

More detailed evaluation questions than in the general plan, separated out by focus area. I was more impressed with these questions and found a high level of overlap with questions I have explored in the past.

The utility?

While still broad, these evaluation questions provide a much stronger starting point for organizations to begin tailoring to their specific needs and context

Here are some potential program evaluation questions for a community-navigation program in a large city:

Program Effectiveness Questions:

To what extent has the community-navigation program increased access to resources and services for community members?
Has the program helped community members overcome barriers to accessing resources and services?
How well has the program been able to connect community members with the resources and services that they need?
What is the impact of the program on community members’ overall well-being and quality of life?
To what extent has the program been successful in achieving its stated goals and objectives?

Program Implementation Questions:

Has the program been implemented as planned? If not, what are the reasons for any deviations?
How well has the program been able to reach its intended target population?
How has the program been received by community members and other stakeholders?
What are the strengths and weaknesses of the program’s implementation?

Program Efficiency and Sustainability Questions:

Is the program being implemented in a cost-effective manner?
How can the program be sustained over time? What are the potential funding sources or partnerships that could help sustain the program?
What are the barriers to program sustainability, and how can they be addressed?

Program Equity Questions:

How has the program impacted different demographic groups in the community?
Is the program equitable in terms of access and outcomes for all community members?
How can the program be adapted to better serve marginalized or underserved populations?

Program Improvement Questions:

What are the areas for improvement in the program, and how can they be addressed?
What are the best practices and lessons learned from similar community-navigation programs in other cities, and how can they be applied to this program?
How can the program be expanded or scaled up to reach more community members or serve additional needs?

These evaluation questions can be tailored to the specific context and goals of the community-navigation program in the large city. By addressing these questions through a comprehensive program evaluation, the program can be improved and sustained over time, ultimately leading to better outcomes for community members.

Exploring evaluation methods

To refine the methodology, I asked, “What are arts-based and creative evaluation methods for a community navigation program?”

The result:

A brief description of five different potential arts-based and participatory evaluation methods. I appreciated the breadth of options and a summary to provide a general sense of each activity.

The utility:

This level of detail was enough to help readers understand potential options without feeling overwhelming. Again, I think this is a great starting point for a conversation between consultants, clients, and funders about which method would be most appropriate for the project budget, timeline, and context.

Arts-based and creative evaluation methods can be a valuable addition to a community navigation program evaluation, as they can provide alternative and unique perspectives and insights into the program’s effectiveness and impact. Here are some potential arts-based and creative evaluation methods for a community navigation program:

Photovoice

Photovoice is a participatory evaluation method that involves community members taking photographs of their experiences and perspectives related to the program. These photographs are then used in group discussions and analysis to explore themes and issues related to the program’s effectiveness and impact.

Storytelling and Narrative Evaluation

Storytelling and narrative evaluation involve gathering personal stories and anecdotes from program participants and using them to explore the program’s impact and effectiveness. This can be done through individual interviews or group discussions, where participants are invited to share their experiences and perspectives related to the program.

Artistic Expression and Performance

Artistic expression and performance can be a powerful tool for exploring the emotional and subjective dimensions of program impact. Participants can be invited to create visual art, poetry, music, or other creative works that represent their experiences and perspectives related to the program. These works can then be exhibited or performed publicly, providing an opportunity for reflection and dialogue about the program.

Collaborative Mapping

Collaborative mapping is a participatory evaluation method that involves community members creating visual maps of their experiences and perspectives related to the program. These maps can be created individually or collaboratively, and can incorporate a range of visual and artistic elements to represent different aspects of the program.

Theater of the Oppressed

Theater of the Oppressed is a participatory evaluation method that uses interactive theater techniques to explore social issues and power dynamics related to the program. Participants are invited to act out different scenarios and experiences related to the program, and are encouraged to experiment with different solutions and strategies for addressing these issues.

By incorporating arts-based and creative evaluation methods into a community navigation program evaluation, a more diverse and holistic understanding of the program’s effectiveness and impact can be gained. These methods can also help to engage community members in the evaluation process and provide a platform for their voices and perspectives to be heard.

Identifying data sources and tools

Of course one key to an evaluation is collecting and analyzing data. I asked ChatGPT, “What are some data sources for community navigation outcomes?”

The result:

A list of different data categories with examples of potential metrics such as enrollment and service utilization. I was actually hoping to learn about specific data sets or data tools when I asked this question, so I ended up asking a few more follow-up questions such as “What data sets are available for community navigation outcomes?” and “What evaluation tools measure community navigation?”

The latter prompt yielded more concrete and measurable tools like the National Association of Community Health Centers; Community Navigation Metrics Tool and the Centers for Disease Control and Prevention’s (CDC) Community Health Worker Assessment and Improvement Matrix (CHW AIM). After reading this, I followed up and asked if the CHW AIM tool was free to use. ChatGPT responded that it was and gave me a summary description of the tool.

The utility:

The original search provided some examples of potential outcomes or types of assessments, but additional follow-up questions were needed to get to more specific tools for the evaluation.

There are several data sources that can be used to measure community navigation outcomes. These data sources can include:

Program Data:

Program data, including enrollment numbers, service utilization, and participant feedback, can provide valuable insights into the effectiveness and impact of community navigation services. This data can be collected through program records, surveys, and focus groups.

Health and Social Service Data:

Health and social service data, including emergency room visits, hospitalizations, and service utilization, can be used to measure the impact of community navigation services on access to and utilization of healthcare and social services.

Community Surveys:

Community surveys can be used to measure community members’ awareness of and satisfaction with community navigation services, as well as their perceptions of the impact of these services on their access to resources and services.

Quality of Life Assessments:

Quality of life assessments can be used to measure the impact of community navigation services on community members’ overall well-being and quality of life. These assessments can include measures of physical and mental health, social connectedness, and access to resources and services.

Financial Data:

Financial data, including program costs and cost savings associated with reduced emergency room visits and hospitalizations, can be used to measure the cost-effectiveness of community navigation services.

By collecting and analyzing data from these various sources, a comprehensive understanding of the impact and effectiveness of community navigation services can be gained. This information can be used to guide program improvements and advocate for the importance of community navigation services in improving health outcomes and reducing healthcare costs.

Exploring basic Excel, R, or other code

Analyzing and visualizing data is another important part of an evaluation to help make sense of and communicate the information.

Others have discussed the benefits and challenges of using ChatGPT for analytic and visualization purposes, including:

Generating example data
Writing R, Excel, and other code
Understanding what certain code means
Identifying the purpose of different data visualizations

Multiple users have experienced challenges with certain analytic functions, noting that ChatGPT may link to outdated statistics packages, provide incorrect code, or perform inaccurate calculations. Like anything, it is important to check recommendations and findings against other sources. But maybe stick to text-based inquiries and stay away from numerical calculations for now, since text is the primary use at the moment.

For more examples and discussion, check out:

So…am I worried that ChatGPT is going to put evaluators out of business?

For now, no.

All the information populated by the AI tool could already be found using Google or another search engine. Plus, most situations that I see in my evaluation work with organizations is not a lack of data or information. Rather, it’s the struggle to interpret and make sense of the information, given competing demands for time and the nuance of each specific context and community.

Evaluation is as much art as science, and this level of complexity requires empathy, creativity, and collaboration. I don’t think we will be able or want to outsource or automate those characteristics in a meaningful way any time soon.

Instead of fearing these tools, perhaps we can view them as supports that help us re-imagine our time, our partnerships, and our work. I hope these tools allow us to use our precious time together to interpret, make meaning, and connect rather than check off agenda items. When used intentionally, ethically, and equitably, I hope that these tools can offer us spaciousness.

What are you imagining and reimagining with the rise of automation and AI in data and evaluation work?

Join the newsletter

Sign up to receive the latest blog posts and offerings about capacity building, program evaluation, and data visualization.