Pancreas Ponderings: What T1D Has Taught Me About Eval

Long before we entered the pre-Covid/during-Covid realm, marked by daily monitoring of case counts, testing, hospitalizations, and death tolls, I had my own pre/during world. Twenty-two years ago the world as I knew it changed as I was rushed to the emergency room and monitored for swelling in my brain. The doctors weren’t sure I would live through the night. With the uttering of one phrase, one diagnosis—Type 1 Diabetes (T1D), my life changed.

Since being diagnosed with T1D, I have had many lessons, successes, and struggles. My daily routine consists of constant monitoring of my blood sugar levels and adjusting my insulin and food intake accordingly. I feel like I should be awarded an honorary degree in predictive analytics based on the calculations needed to anticipate and course-correct every move.

You would think, being an evaluator, that I relish and thrive in this world of numbers, monitoring, and learning. I mean, I have an insulin to carb ratio so I just need to count my carbs and dose my insulin accordingly, and it’s smooth sailing. Right? Wrong. So many things impact blood glucose levels including fiber, glycemic index, hormones, stress, and type and duration of exercise. Often I just feel frustrated and exhausted. How am I, someone who monitors numbers for a living, unable to crack the secret code?

And why, when I have so many facets to myself, do I constantly feel like my success, value, and worth as a human, as someone with T1D, is being judged by one number—my HbA1c? (HbA1c captures a person’s average blood glucose over the past 3 months and is a key indicator for measuring “control” of T1D) 

So I started to ask myself, if I—someone who loves numbers, data, and evaluation—feel this way, how must my evaluation clients feel? 

Here’s what I know:

Numbers are scary. 

When I was a child I had a physician threaten to confiscate my insulin pump if I did not achieve their desired HbA1c level. To this day, over 20 years later, I am still nervous when I go to the doctor. 

Our evaluation clients are nervous too. They fear that diving into the data could result in punitive recourse in terms of loss of funding or programming. They fear that one number could result in judgement at best or worse – closure. Our job as evaluators is to shift the message from evaluation as punitive to evaluation as learning. We need a stronger focus on curiosity, capacity building, and co-creating rather than indicators and benchmarks.

What you ask matters. 

Nearly every visit to my endocrinologist goes like this: doctor downloads my records into their computer, prints them out, reviews them, and then comes into the room. The doctor then proceeds to ask me what I was doing two Thursdays ago at 3pm when my blood glucose level was high. Now I don’t know about you but I usually have a hard enough time telling you my schedule from yesterday, let alone weeks ago. And yet, they ask. Every. Single. Time. I usually make something up.

We see similar approaches in evaluation. Poor survey and interview designs ask people to recall very specific information across unreasonable time intervals. Then we treat those results as fact. Understanding how to ask the right questions is key to gathering good data. See Sheila B. Robinson and Kimberly Firth Leonard’s book, Designing Quality Survey Questions, for guidance.

Data represents people. 

Another fun exercise endocrinologists and diabetes educators like to do is having you write down everything you ate for the past two weeks. I don’t have time for that. I’m sure it’s very informative but I’m not going to prioritize that into my schedule at the moment. It’s just not going to happen.

Here’s the thing. In evaluation, numbers represent people. People have rich and complex lives that often do not fit into a nice neat checkbox or pass a fidelity check. So maybe it’s time we stopped asking them to. We need to fund qualitative, participatory, and developmental evaluation projects that not only measure but value and align with the lived experiences of program staff, clients, and community members. And we need to stop asking people to show up without compensation for their time, feedback, and expertise.

Context is everything. 

As I mentioned earlier, pretty much anything and everything influences blood glucose levels. I can eat the same thing at the same time and take the same amount of insulin three days in a row and get different results every time. This is because my context is changing every day. Perhaps I went for a hard hike one day and gave a presentation another. Maybe it’s performance review time—definitely need extra insulin for those meetings!

Many qualitative and participatory evaluation approaches recognize context as important contributing factors. In public health, we often refer to Bronfenbrenner’s ecological systems theory. And while we say context matters, we rarely intentionally and thoroughly explore the historical and political roots of whatever social issues a program or project is addressing. We must take a more equitable approach to evaluation and understand these underlying systems. See the Equitable Evaluation Initiative’s framework for inspiration. 

We’re going to mess up.

One thing I’ve learned from having T1D, is that there’s no one size fits all model. I have all of the technology and fancy gadgets and I still struggle.

We’re going to struggle during evaluation too. Each client and project is unique, requiring tailored methods and approaches. What worked with one team may not work well with another. While this can seem daunting, it’s also an opportunity to learn, grow, and challenge our assumptions. 

Recently a colleague told me it’s ok if the project isn’t perfect because there will always be some margin of error. He said this to calm my perfectionism. But I wasn’t worried about getting it right to be perfect. I was worried about the implications. I was worried about the people. About what could happen to the clients or the community if my numbers were wrong. I think about this with my health too—what could happen if my numbers are wrong. And that, my friends, feels daunting. That is why data has power. And that is why I strive to be both gentle and vigilant in my health and my evaluation work. 

Unbecoming & reimagining: Leaning into values and fears

This blog post has been a long time coming. I have struggled with voicing my thoughts for a couple of years. I am embarrassed that it took a national crisis and more murders of people of color like Ahmaud Arbery, Breonna Taylor, and George Floyd for me to lean into the fear and vulnerability to write. I am ashamed that I became afraid to use my voice publicly for justice and equity for fear of losing potential clients when people are losing their lives. I recognize that these fears, while real for me, are a privilege. This guilt, shame, and embarrassment are also privileges — white privilege. I have privilege simply because of the color of my skin, and I need to intentionally and consistently use that privilege to enact change.

I’ve known all of these things. I’m a social worker and evaluator. I chose this path because I want to help organizations use data to dismantle oppressive systems and build more equitable and just programs and communities. And still, I’ve become complacent with my public commitment to my values of justice and equity. As the stakes have gotten higher, I have gotten quieter. Not in my private circles, but certainly in my professional and public ones. I fear that my clients may not know my values. That is disheartening. That is shameful. That is unacceptable.

There’s a family story about how my parents didn’t send me to a certain elementary school for first grade because they knew that I would call nonsense on the motivational posters on the walls. You know the ones I’m talking about—respect, integrity, ethics, community. All wonderful concepts that are easier to say than to actually embody. I fear that I’ve become one of those motivational posters.

I have spent my career making calculated decisions to appear nonpartisan on Twitter and my website so that I can continue to operate in the policy evaluation space. I have hidden my values publicly so that clients will continue to hire me. For the last six years, I have been living in fear that speaking out about what I believe in will ruin my evaluation career before I feel like it has launched. That fear is a privilege. That fear stops now.

In doing so, I have erased the parts of me that make me who I am—an assertive, passionate, tenacious leader and advocate for justice. To quiet that voice is complacency. I’ve been complicit in injustice and the perpetuation of our centuries-old racist and inequitable systems. By trying to appear nonpartisan, I have silenced my voice. By not sharing the work of other advocates, especially colleagues of color, I have also silenced the voices of those working to advance equity and justice. By not diversifying the social media accounts, podcasts, authors, and artists I follow, I continue to live in an insular world.

My decision to appear nonpartisan online has made me an actor in perpetuating the problem. The issues that I care about—homelessness, healthcare, racial justice—and the values that I hold should not be viewed as partisan. They are human rights, social justice, and equity issues. If a potential client doesn’t want to hire me because of my commitment to these issues then they are on the wrong side of history and social justice.

Saying I support equity and justice while not actively taking steps to better educate myself and work towards them in every aspect of my life makes me just like those motivational posters.

I can’t go backwards. I can only commit to moving forward and doing better. I am grateful for evaluation colleagues who share their experiences, learnings, and resources for me to do better. I look forward to having more difficult conversations and realizations about how I have actively and passively perpetuated injustice. I also recognize that this is not my (or any other white persons) moment or my movement. This is the time to listen and learn to communities and colleagues of color, and then use those learnings to enact change. 

Intentionally and sustainably (re)committing to racial equity—rather than engaging in optical allyship and performative posting—will look different for everyone. Right now what this commitment looks like for me is:

  • Recognizing my privilege and not taking advantage of it
  • Reading more about the history of oppression and white supremacy
  • Engaging in conversation about racial equity with white friends, family members, and colleagues
  • Diversifying the social media accounts I follow
  • Amplifying the voices of people of color
  • Donating money to support communities of color
  • Diversifying speaking panels, conference programs, leadership teams, and boards of directors
  • Advocating for policy change
  • Challenging racist language used in conversations, meetings, and materials
  • Disaggregating data in my evaluation work and data visualizations

I commit to leaning into and embodying my values of equity, ethics, authenticity, transparency, honesty, and action. I commit to being and doing better.

My next step? Reading How To Be An Antiracist by Ibram X. Kendi. 

What steps are you taking?

Evaluator as Acrobat: The Art of Flexibility

Each day I spend on the trail, walking, running, or hiking teaches me something about myself and the universe. If you know me, you know that I am a Type-A planner. I like researching background details, creating agendas and itineraries, identifying what materials are needed, and orchestrating events so they go as planned. When things don’t go as planned, I can get frustrated. This is something I continuously work on in my personal and professional life. Hiking has consistently challenged this aspect of my personality in the best way possible.

For example, this Fall I set out on a solo hike up my 22nd (of 48) 4000 footer mountains (those with summits at or above 4000 feet) in New Hampshire. I had done extensive research. I knew the weather, the hiking stats, where the closest water source was, and what time I needed to reach different junctions to stay on track. The plan was for me to hike five miles up one mountain to a backcountry campsite where I would meet my friends who were hiking 17 miles across six mountains from the other direction (overachievers!). I had an amazing day on the trail, getting to the summit in record time for me, where I enjoyed 360-degree views of the Pemigewasset Wilderness. It was spectacular!

IMG_4055

When I got down to the campsite, the plan started to unravel. I unpacked my tent and realized it was damp. Then I learned that my friends were too exhausted to make the final steep three-mile push to the campsite. My Hallmark-inspired vision of our trio chatting and cooking dinner on our backpacking stoves while we watched the sunset over the mountains was dissipating. I realized then that 1) I did not want to camp alone in near/sub-freezing temperatures and 2) I still had time to get off the mountain in daylight. I quickly packed up, hiked out, and drove home.

IMG_4043

When I shared this story with friends and family, they unanimously remarked that I must be disappointed. You would think that, as a planner, I would have been. But I wasn’t. Not at all. I was proud of my friends for making the right and safe call to stay where they were. I was proud of myself for making the decision that felt true for me. And I was excited to learn that I was strong enough to hike up 3000 feet of elevation with 30 pounds of gear on my back. Yes, the plan changed. The trail has taught me to always be ready to change course – weather changes quickly, people get injured, and trail conditions can be unexpected. Sometimes the best-laid plans need to change and that is ok. In this case, Plan B ended up being pretty fantastic.

fullsizeoutput_9129.jpeg

This lesson of flexibility has also been advantageous for my consulting career. In evaluation capacity building work, often the best plans and intentions hit a bump in the road. Like people, organizations are dynamic, constantly shifting and evolving.  I’ve witnessed leadership changes, staff changes, financial challenges, disagreements about priorities, changes in funder requirements, and changes in databases. I’ve even had an organization change its mission and key strategies during the time I worked with them.

Each of these factors influences the work and requires a shift to the original plan. Sometimes these are small changes like bringing a new team member into the discussions or revising the logic model. Sometimes these are much larger changes like restructuring all of the data collection and management processes. Rigidity has no place in evaluation capacity building. As a consultant, I must find ways to pivot quickly and strategically when the plan changes. Flexibility is key.

It can be challenging to see the leaves through the trees when a consulting or capacity building project hits a bump in the road or changes course. However, I’ve found that celebrating small successes helps both me and my clients to stay focused and energized as we determine our next steps. I often ask myself questions such as, did we…

  • More clearly articulate the theory of change?
  • Collect a new data point?
  • Collaborate with a new partner?
  • Look at data differently than in the past?
  • Try a new type of data visualization?
  • Meet regularly to discuss and prioritize evaluation?

All of these steps help to build evaluation capacity and evaluative thinking. They aren’t the flashy outcomes or final report, but they do help to build the foundation and facilitate forward movement.

There are many different roads to building capacity and sometimes we have to take a detour from our original plan. If you find yourself amidst a project that’s changing course, remember to celebrate the small successes, expect and anticipate detours, and remember that Plans B, C, or D might turn out even better than Plan A. Onward!

Wearing Many Hats: Where Does the Role of Evaluator End?

Four years ago I uprooted my life and moved halfway across the country to relocate to a state that had never been on my radar. In that time, I’ve established myself in the evaluation community, working both as an external evaluator and as a consultant building the evaluation capacity of organizations. You would think in that amount of time, plus my years of experience prior to this, that I would have a clear answer about my professional identity and how it fits into the broader field. But really, I feel that I find more questions than answers with each passing day.

If you’ve ever tried to explain program evaluation to a non-evaluator, you know what I’m talking about – the glazed over looks you get when mentioning that you’re an evaluator, the people who think you’re a researcher or auditor, and the others who assume you’re a computer programmer. I also received my MSW so I’m fairly confident that part of my family still thinks I’m a therapist, despite the fact that my degree is in policy and evaluation. Add to that the many hats that a program evaluator wears in the 21st century, and it’s really a wonder that we haven’t created a support group. Or maybe we have – shout out to my #EvalTwitter community!

Evaluators wear many different hats, which vary depending on the evaluation approach as well as the stage of the evaluation cycle. Over the years, evaluation theorists have debated the role of the evaluator. Campbell classifies the role of evaluator as methodologist, Scriven says judge, Stake says facilitator, and Wholey says educator (Luo, 2010). Today, evaluators often straddle all of these roles in addition to new roles brought with advancing technology and globalization such as designer and marketer. Other hats include data analyst, project manager, grant writer, strategic planner, coordinator, educator/teacher, coach, and facilitator.

While it is clear that evaluators operate under the American Evaluation Association’s Guiding Principles: systematic inquiry, competence, integrity, respect for people, and common good and equity, the role(s) of an evaluator are not as clearly demarcated. In fact, a commonly cited quote in the field of evaluation is: “evaluation – more than any science – is what people say it is; and people currently are saying it is many different things” (Glass & Ellett, 1980). Even the American Evaluation Association (2004) states that, “based on differences in training, experience, and work settings, the profession of evaluation encompasses diverse perceptions about the primary purpose of evaluation.” If the evaluation field struggles to uniformly define the scope of evaluation roles, it is likely that stakeholders and clients are unclear about what services evaluators offer.  

Two years ago, I led a Birds of a Feather discussion at the American Evaluation Association conference seeking clarity about the many hats an evaluator wears. I was curious to understand:

  • What are the hats evaluators wear?
  • Which of these hats, if any, are in conflict with one another?
  • What are the boundaries of evaluation?
  • How do we communicate these boundaries to our clients?
  • How might globalization change the role(s) of an evaluator?
  • To what degree can and should evaluators play a role in policy change, advocacy, and program development?

Attendees shared that they feel that evaluators can bridge all of the aforementioned roles as long as we acknowledge our biases at the outset of a project and are transparent about our work. The group unanimously shared that open communication is the most important aspect of an evaluation partnership, more so than the specific role(s) being occupied by the evaluator. Attendees also suggested that evaluators develop subjectivity statements at the outset of a project, which according to Given (2008), “is a summary of who researchers [evaluators] are in relation to what and whom they are studying. Researchers [evaluators] develop these from their personal histories, their cultural worldviews, and their professional experiences.”

This all about made my head explode. Have I been overthinking the role(s) of an evaluator and whether and how I fit into the field for no reason? Have I been questioning the ethics of whether we can occupy all of these spaces and still fall under one profession to no avail?

For four years I’ve been pondering – Who am I as an evaluator? Am I even an evaluator? Or am I a social worker with an evaluation and data-driven mindset? Can I be both? Am I a policy analyst who implements data-driven approaches? Am I an advocate that uses data to drive change? How many hats can I wear before my professional identity is so dispersed that it is nonexistent? Does claiming a professional identity even matter?

It seems that, as of yet, there is no consensus about the hats evaluators wear and the boundaries of evaluation. With each colleague I speak with I get another definition or answer, which raises more questions. This reminds me of a quote by Elie Wiesel, “I define myself more by my questions than by my answers, because answers come and go but questions remain.” So for now I will take the attendeesadvice and focus on communication and transparency as I continue to ponder all of these questions.

Interested in pondering these questions with me or being featured in a blog series on this topic? Share a comment or send me a tweet.

Wisdom from the Woods: Everything Looks Scarier in the Darkness

I haven’t written in a while. And by a while I mean half a year. I could attribute that to the fact that work has been busy or that I’ve been spending my free time traveling and exploring. These are, in fact, true statements. But the real reason is that I’ve been afraid. WarningSign

Now here’s the funny part – I’ve climbed nine of New Hampshire’s 48 4000 footer mountains this summer for a total of 21 to date, I’ve climbed trails that are on the Terrifying 25 and on the most dangerous hikes in the world list, and I recently hiked a trail that consisted of more iron rungs than rock. I’ve solo hiked in the Alps and gone paragliding in Switzerland. I’m not scared of most things and I love a good challenge. But here I am. Afraid of my own blog. It sounds laughable and it’s true.

Why the change of heart? Well like most things in my life, it goes back to the trail. A few weeks ago I traveled to Acadia National Park to camp and hike for the weekend. It’s a long drive so I decided to stop and camp along the way. This was my first time ever camping alone so I had a solid plan. I found a nice family campground close to a major town, I booked a site not too far from others, and I planned to sleep in my car instead of my tent since I would be getting there well after dark. I was ready! Except I wasn’t. Everything is scarier in the darkness.

I arrived past sunset to realize the main road in was closed for construction. After navigating some windy, twisty, and dark back roads (thank you GPS!), I located the campground. All the staff had left for the night so I used the map to navigate my way through the confusing maze of gravel campground roads. Finally, I found my site! I set up camp in my car and settled down for the evening. Of course, not before setting off my car alarm by accident (sorry neighbors!). None of the sites around me were booked. In the darkness, even with my headlamp, it looked and felt remote. When I’m with other people I welcome this silence. Alone, I was nervous.

After tossing and turning, I’m embarrassed to say that I started looking up hotels on Yelp. But I was determined to do this. I listen to a lot of the She Explores and Women on the Road podcasts, I just needed to channel my inner adventurer. If you know me, you know that I’m incredibly tenacious. I could do this. I can do hard things. This was not that hard. Everything is scarier in the darkness.

GraceJoyAveI made it through the night unscathed and woke with the sunrise. I surveyed my surroundings. In the light of day, I realized my campsite was fantastic and had a beautiful spot for a tent in the woods. When I walked to the washroom, I laughed upon realizing that just two sites away, the campground backed onto a neighborhood. The campground even had wifi! Then came the icing on the cake. Remember those gravel roads I navigated in the dark? My particular campsite was at the intersection of Grace Lane and Joy Avenue. Well played universe, well played. Everything is scarier in the darkness.

Life is about perspective. It’s about leaning in, taking risks, embracing vulnerability, and failing forward. I know this, I subscribe to this, I preach this. I’ve read the writings of Brene Brown, Elizabeth Gilbert, and Glennon Doyle. So when and where did I lose my path in the woods? I can clearly articulate the change I want to be in this world. But in the darkness of fear, I’ve grappled with how these dreams can come to fruition. Some of the questions I’ve pondered are:

  • Do I have anything meaningful to say?
  • Will what I say negatively impact my future career prospects?
  • Is what I’m saying on brand?
  • Do I even have a brand?
  • What if I ruin my career before I feel like it’s launched?
  • Am I a total imposter?

But here’s the thing. While I haven’t been writing my thoughts, someone else has published on the same topics. My brand is me so I can choose what that looks like. And if I’m being authentic then I should be able to find future career opportunities that embrace me for what I have to offer.

That’s all really easy to say and a lot harder to embody and implement. So here I am. Putting this out there. To remind myself and others that everything is scarier in the darkness, we can do hard things, and what you have to say is important. Onward.

Tripyramids