Evaluator as Acrobat: The Art of Flexibility

Each day I spend on the trail, walking, running, or hiking teaches me something about myself and the universe. If you know me, you know that I am a Type-A planner. I like researching background details, creating agendas and itineraries, identifying what materials are needed, and orchestrating events so they go as planned. When things don’t go as planned, I can get frustrated. This is something I continuously work on in my personal and professional life. Hiking has consistently challenged this aspect of my personality in the best way possible.

For example, this Fall I set out on a solo hike up my 22nd (of 48) 4000 footer mountains (those with summits at or above 4000 feet) in New Hampshire. I had done extensive research. I knew the weather, the hiking stats, where the closest water source was, and what time I needed to reach different junctions to stay on track. The plan was for me to hike five miles up one mountain to a backcountry campsite where I would meet my friends who were hiking 17 miles across six mountains from the other direction (overachievers!). I had an amazing day on the trail, getting to the summit in record time for me, where I enjoyed 360-degree views of the Pemigewasset Wilderness. It was spectacular!

IMG_4055

When I got down to the campsite, the plan started to unravel. I unpacked my tent and realized it was damp. Then I learned that my friends were too exhausted to make the final steep three-mile push to the campsite. My Hallmark-inspired vision of our trio chatting and cooking dinner on our backpacking stoves while we watched the sunset over the mountains was dissipating. I realized then that 1) I did not want to camp alone in near/sub-freezing temperatures and 2) I still had time to get off the mountain in daylight. I quickly packed up, hiked out, and drove home.

IMG_4043

When I shared this story with friends and family, they unanimously remarked that I must be disappointed. You would think that, as a planner, I would have been. But I wasn’t. Not at all. I was proud of my friends for making the right and safe call to stay where they were. I was proud of myself for making the decision that felt true for me. And I was excited to learn that I was strong enough to hike up 3000 feet of elevation with 30 pounds of gear on my back. Yes, the plan changed. The trail has taught me to always be ready to change course – weather changes quickly, people get injured, and trail conditions can be unexpected. Sometimes the best-laid plans need to change and that is ok. In this case, Plan B ended up being pretty fantastic.

fullsizeoutput_9129.jpeg

This lesson of flexibility has also been advantageous for my consulting career. In evaluation capacity building work, often the best plans and intentions hit a bump in the road. Like people, organizations are dynamic, constantly shifting and evolving.  I’ve witnessed leadership changes, staff changes, financial challenges, disagreements about priorities, changes in funder requirements, and changes in databases. I’ve even had an organization change its mission and key strategies during the time I worked with them.

Each of these factors influences the work and requires a shift to the original plan. Sometimes these are small changes like bringing a new team member into the discussions or revising the logic model. Sometimes these are much larger changes like restructuring all of the data collection and management processes. Rigidity has no place in evaluation capacity building. As a consultant, I must find ways to pivot quickly and strategically when the plan changes. Flexibility is key.

It can be challenging to see the leaves through the trees when a consulting or capacity building project hits a bump in the road or changes course. However, I’ve found that celebrating small successes helps both me and my clients to stay focused and energized as we determine our next steps. I often ask myself questions such as, did we…

  • More clearly articulate the theory of change?
  • Collect a new data point?
  • Collaborate with a new partner?
  • Look at data differently than in the past?
  • Try a new type of data visualization?
  • Meet regularly to discuss and prioritize evaluation?

All of these steps help to build evaluation capacity and evaluative thinking. They aren’t the flashy outcomes or final report, but they do help to build the foundation and facilitate forward movement.

There are many different roads to building capacity and sometimes we have to take a detour from our original plan. If you find yourself amidst a project that’s changing course, remember to celebrate the small successes, expect and anticipate detours, and remember that Plans B, C, or D might turn out even better than Plan A. Onward!

Wearing Many Hats: Where Does the Role of Evaluator End?

Four years ago I uprooted my life and moved halfway across the country to relocate to a state that had never been on my radar. In that time, I’ve established myself in the evaluation community, working both as an external evaluator and as a consultant building the evaluation capacity of organizations. You would think in that amount of time, plus my years of experience prior to this, that I would have a clear answer about my professional identity and how it fits into the broader field. But really, I feel that I find more questions than answers with each passing day.

If you’ve ever tried to explain program evaluation to a non-evaluator, you know what I’m talking about – the glazed over looks you get when mentioning that you’re an evaluator, the people who think you’re a researcher or auditor, and the others who assume you’re a computer programmer. I also received my MSW so I’m fairly confident that part of my family still thinks I’m a therapist, despite the fact that my degree is in policy and evaluation. Add to that the many hats that a program evaluator wears in the 21st century, and it’s really a wonder that we haven’t created a support group. Or maybe we have – shout out to my #EvalTwitter community!

Evaluators wear many different hats, which vary depending on the evaluation approach as well as the stage of the evaluation cycle. Over the years, evaluation theorists have debated the role of the evaluator. Campbell classifies the role of evaluator as methodologist, Scriven says judge, Stake says facilitator, and Wholey says educator (Luo, 2010). Today, evaluators often straddle all of these roles in addition to new roles brought with advancing technology and globalization such as designer and marketer. Other hats include data analyst, project manager, grant writer, strategic planner, coordinator, educator/teacher, coach, and facilitator.

While it is clear that evaluators operate under the American Evaluation Association’s Guiding Principles: systematic inquiry, competence, integrity, respect for people, and common good and equity, the role(s) of an evaluator are not as clearly demarcated. In fact, a commonly cited quote in the field of evaluation is: “evaluation – more than any science – is what people say it is; and people currently are saying it is many different things” (Glass & Ellett, 1980). Even the American Evaluation Association (2004) states that, “based on differences in training, experience, and work settings, the profession of evaluation encompasses diverse perceptions about the primary purpose of evaluation.” If the evaluation field struggles to uniformly define the scope of evaluation roles, it is likely that stakeholders and clients are unclear about what services evaluators offer.  

Two years ago, I led a Birds of a Feather discussion at the American Evaluation Association conference seeking clarity about the many hats an evaluator wears. I was curious to understand:

  • What are the hats evaluators wear?
  • Which of these hats, if any, are in conflict with one another?
  • What are the boundaries of evaluation?
  • How do we communicate these boundaries to our clients?
  • How might globalization change the role(s) of an evaluator?
  • To what degree can and should evaluators play a role in policy change, advocacy, and program development?

Attendees shared that they feel that evaluators can bridge all of the aforementioned roles as long as we acknowledge our biases at the outset of a project and are transparent about our work. The group unanimously shared that open communication is the most important aspect of an evaluation partnership, more so than the specific role(s) being occupied by the evaluator. Attendees also suggested that evaluators develop subjectivity statements at the outset of a project, which according to Given (2008), “is a summary of who researchers [evaluators] are in relation to what and whom they are studying. Researchers [evaluators] develop these from their personal histories, their cultural worldviews, and their professional experiences.”

This all about made my head explode. Have I been overthinking the role(s) of an evaluator and whether and how I fit into the field for no reason? Have I been questioning the ethics of whether we can occupy all of these spaces and still fall under one profession to no avail?

For four years I’ve been pondering – Who am I as an evaluator? Am I even an evaluator? Or am I a social worker with an evaluation and data-driven mindset? Can I be both? Am I a policy analyst who implements data-driven approaches? Am I an advocate that uses data to drive change? How many hats can I wear before my professional identity is so dispersed that it is nonexistent? Does claiming a professional identity even matter?

It seems that, as of yet, there is no consensus about the hats evaluators wear and the boundaries of evaluation. With each colleague I speak with I get another definition or answer, which raises more questions. This reminds me of a quote by Elie Wiesel, “I define myself more by my questions than by my answers, because answers come and go but questions remain.” So for now I will take the attendeesadvice and focus on communication and transparency as I continue to ponder all of these questions.

Interested in pondering these questions with me or being featured in a blog series on this topic? Share a comment or send me a tweet.

Wisdom from the Woods: Everything Looks Scarier in the Darkness

I haven’t written in a while. And by a while I mean half a year. I could attribute that to the fact that work has been busy or that I’ve been spending my free time traveling and exploring. These are, in fact, true statements. But the real reason is that I’ve been afraid. WarningSign

Now here’s the funny part – I’ve climbed nine of New Hampshire’s 48 4000 footer mountains this summer for a total of 21 to date, I’ve climbed trails that are on the Terrifying 25 and on the most dangerous hikes in the world list, and I recently hiked a trail that consisted of more iron rungs than rock. I’ve solo hiked in the Alps and gone paragliding in Switzerland. I’m not scared of most things and I love a good challenge. But here I am. Afraid of my own blog. It sounds laughable and it’s true.

Why the change of heart? Well like most things in my life, it goes back to the trail. A few weeks ago I traveled to Acadia National Park to camp and hike for the weekend. It’s a long drive so I decided to stop and camp along the way. This was my first time ever camping alone so I had a solid plan. I found a nice family campground close to a major town, I booked a site not too far from others, and I planned to sleep in my car instead of my tent since I would be getting there well after dark. I was ready! Except I wasn’t. Everything is scarier in the darkness.

I arrived past sunset to realize the main road in was closed for construction. After navigating some windy, twisty, and dark back roads (thank you GPS!), I located the campground. All the staff had left for the night so I used the map to navigate my way through the confusing maze of gravel campground roads. Finally, I found my site! I set up camp in my car and settled down for the evening. Of course, not before setting off my car alarm by accident (sorry neighbors!). None of the sites around me were booked. In the darkness, even with my headlamp, it looked and felt remote. When I’m with other people I welcome this silence. Alone, I was nervous.

After tossing and turning, I’m embarrassed to say that I started looking up hotels on Yelp. But I was determined to do this. I listen to a lot of the She Explores and Women on the Road podcasts, I just needed to channel my inner adventurer. If you know me, you know that I’m incredibly tenacious. I could do this. I can do hard things. This was not that hard. Everything is scarier in the darkness.

GraceJoyAveI made it through the night unscathed and woke with the sunrise. I surveyed my surroundings. In the light of day, I realized my campsite was fantastic and had a beautiful spot for a tent in the woods. When I walked to the washroom, I laughed upon realizing that just two sites away, the campground backed onto a neighborhood. The campground even had wifi! Then came the icing on the cake. Remember those gravel roads I navigated in the dark? My particular campsite was at the intersection of Grace Lane and Joy Avenue. Well played universe, well played. Everything is scarier in the darkness.

Life is about perspective. It’s about leaning in, taking risks, embracing vulnerability, and failing forward. I know this, I subscribe to this, I preach this. I’ve read the writings of Brene Brown, Elizabeth Gilbert, and Glennon Doyle. So when and where did I lose my path in the woods? I can clearly articulate the change I want to be in this world. But in the darkness of fear, I’ve grappled with how these dreams can come to fruition. Some of the questions I’ve pondered are:

  • Do I have anything meaningful to say?
  • Will what I say negatively impact my future career prospects?
  • Is what I’m saying on brand?
  • Do I even have a brand?
  • What if I ruin my career before I feel like it’s launched?
  • Am I a total imposter?

But here’s the thing. While I haven’t been writing my thoughts, someone else has published on the same topics. My brand is me so I can choose what that looks like. And if I’m being authentic then I should be able to find future career opportunities that embrace me for what I have to offer.

That’s all really easy to say and a lot harder to embody and implement. So here I am. Putting this out there. To remind myself and others that everything is scarier in the darkness, we can do hard things, and what you have to say is important. Onward.

Tripyramids

Equal Pay is 41 Years Away: Why Gender Parity Must Be Accelerated

Last week, on March 8, the world celebrated International Women’s Day – “a global day celebrating the social, economic, cultural and political achievements of women. The day also marked a call to action for accelerating gender parity.”

Many people and organizations took to social media to celebrate women. For example, Forbes, BBC, and Boeing shared how women are changing the face of sports, science, and technology. Women in positions of power shared business and life advice. The data visualization community created graphics highlighting women’s progress and disparities. And I shared my story of how local female leaders have positively impacted my life.

But the day was not without drama and backlash. People continue to claim that inequalities like the wage gap do not exist. Some felt threatened and inquired about International Men’s Day (by the way, it’s November 19). Others reportedly did not celebrate the day, citing that it is a communist and socialist holiday.

Clearly, more work must be done and we are far from achieving gender parity.

Women continue to earn less than men. “In 2015, female full-time, year-round workers made only 80 cents for every dollar earned by men” (IWPR, 2018). According to the Institute for Women’s Policy Research,

“If change continues at the same slow pace as it has done for the past fifty years, it will take 41 years—or until 2059—for women to finally reach pay parity. For women of color, the rate of change is even slower: Hispanic women will have to wait until 2233 and Black women will wait until 2124 for equal pay” (IWPR, 2017).

Let me rephrase this. A baby girl born today in 2018 will not experience pay parity until she is 41 years old. If that baby is Hispanic or Black, she would need to live to 215 and 106 years old, respectively, to experience pay parity. That is ridiculous. When are we going to start valuing women as human beings and equal contributors to society?

Malala quote

As if that is not infuriating enough, the wage gap just touches the surface of work needed to achieve gender parity. Across the world, women and girls are experiencing disparities at unacceptable and alarming rates:

  • Women make up half of the population but 70% of the 1 billion poorest people in the world are female (Global Citizen, 2012)
  • One in three women have experienced physical or sexual violence (WHO, 2017)
  • Globally, women make up only 18.9% of the world’s legislators (USAID, 2015)

This is not a global phenomenon solely impacting developing countries. Here in the United States:

  • Over 600 women are sexually assaulted every day (NOW)
  • Employers are not required to offer paid maternal leave, making the US the only country in the developing world not offering this benefit (Washington Post, 2018)
  • Only 27 of Fortune 500 CEOs are women, and only two are women of color (Fortune, 2017)

And one of my personal frustrations – women in state prisons or local jails continue to be deprived of the basic and requisite hygiene products for menstruation. I mean, really? It’s 2018 and we are still dehumanizing women for a bodily function that literally gave the people creating the asinine policies life, once again sending the message to women that their health is not valued or prioritized.

WOMEN'S RIGHTS

So why invest in women?

From a strictly financial perspective, closing the gender and wage gap could add an estimated $12-28 trillion (11-26%) to the global annual GDP by 2025 (McKinsey & Company, 2015). Furthermore, “when 10% more girls go to school, a country’s GDP increases on average by 3%” (USAID, 2015).

From a social justice perspective, investing in women often has a multiplicative impact, also advancing her family and community. Women reinvest more of their incomes back into their families than men (90% vs. 30-40%). Other studies suggest that communities are more inclusive when the percentage of women holding political leadership positions is over 30%.

Well, this all sounds daunting. What can we do to impact change? Here are a few suggestions:

  • Encourage inclusivity
  • Question who is and who is not participating and represented
  • Challenge gender stereotypes
  • Examine the language you are using to describe and communicate with women
  • Support female-owned businesses and hire women for positions of power
  • Pay employees equally for their position and skill set, not their gender
  • Adopt flexible work arrangements
  • Highlight women’s achievements and give credit accordingly
  • Mentor women and girls

How did you celebrate International Women’s Day? What are you doing to accelerate gender parity?

International Women’s Day: Celebrating Local Female Leaders

Today, March 8, marks International Women’s Day (#IWD2018) – “a global day celebrating the social, economic, cultural and political achievements of women. The day also marks a call to action for accelerating gender parity.”

This is an exciting time to identify as a woman. In 2016, a female presidential candidate earned the Democratic Party nomination for the first time and ran a strong campaign that resulted in more popular votes than her opponent. The #MeToo and #TimesUp movements have brought critical attention to and recognition of sexual assault and harassment. And women are launching campaigns and running for political office in record numbers, with support from organizations such as She Should Run and VoteRunLead.

A search of #InternationalWomensDay and #IWD2018 on social media provides insight into incredible female leaders across the world who have or are currently facing adversity, challenging stereotypes, and breaking down barriers. Many are iconic names such Rosa Parks, Malala Yousafzai, and Gloria Steinem. Organizations like Forbes, BBC, and Boeing are highlighting how women are changing the face of sports, science, and technology.

But for every well-known female leader, there are hundreds of women changing lives in their local communities who often remain unrecognized. I met some of these amazing women in El Salvador as part of a study abroad program focused on understanding how the civil war impacted women and children in the country. Through this experience, I had the opportunity to meet with grassroots women’s organizations and female political leaders, and share meals with women living in rural communities. These women changed the trajectory of my life.

Some of the groups we met with included:

  • Pro-Busqúeda: a non-profit organization that investigates cases of and searches for children who were disappeared during the conflict
  • COMADRES: a human rights organization that offers assistance to the victims and families of those imprisoned, murdered, or disappeared during the civil war
  • AMUSAMECO: a weekly group for women in San Salvador that provides social and emotional support while also teaching a profitable skill such as paper quilling and card making

 

AMUSAMECO_cards
Cards made by members of AMUSAMECO

Across the organizations, women shared their experiences with poverty, violence, lack of employment opportunities, and the machismo tradition. The testimonies shared of physical and sexual violence were unfathomable. I learned of abuses I did not know existed and cannot comprehend. Those images will forever remain etched in my brain. I heard about husbands, brothers, and children being disappeared and/or killed. It was excruciatingly heartbreaking and I feel humbled that the women shared their stories.

 

Monument_to_Memory_and_Truth
Monumento a la Memoria y la Verdad (Monument to Memory and Truth)

When I reflect on this experience, I am continually reminded of the resilience, gratitude, and sense of community that these women demonstrated. They truly embody the spirit behind International Women’s Day – to celebrate “the social, economic, cultural and political achievements of women.” Despite having experienced unimaginable violence and loss, the women persisted.

For example, during El Salvador’s civil war (1980-1991), 90,000 people were killed and thousands of children disappeared. Many children were trafficked and sent for adoption in other countries. Pro-Busqúeda has investigated over 800 cases of disappeared children and, as of 2014, had solved 300. The strength and tenacity of the women and families involved with Pro-Busqúeda demonstrate the transformative power of advocacy and activism.

By challenging the status quo, these women have changed the trajectories of their families. They have formed grassroots organizations, learned new trades, leaned on each other for support, and challenged the machismo culture in their homes and the community. One woman shared how she climbed out of her window when her husband forbade her from leaving the home so that she could attend the AMUSAMECO support group. These women are paving the way for future generations with every small shift in cultural and gender norms.

PadreJonCortina_quote
“The fight for justice does not end with the truth, but requires a reparation to the victims.”

Not only are the women changing their lives, they have changed mine as well. The impact this experience had on my life was so profound that it is difficult to put into words. Hearing these women’s stories was one of the key reasons that I pursued social work and public health. This experience taught me the impact that communities have on health and wellbeing. I understood the effect systems and cultural norms can have on advancing or curbing opportunities. I also saw the role resilience and social supports play in overcoming adversity. Moreover, I witnessed the impact of personal stories on shifting norms and policies, which is a testament to the power of lived experience, advocacy, and activism. These women are the strongest and most tenacious change agents I have ever met.

The women involved with Pro-Búsqueda, COMADRES, and AMUSAMECO are just one example of the many women around the world who are challenging the status quo and accelerating gender parity. There are so many others whose stories remain untold. As we celebrate International Women’s Day, I encourage you to recognize the women in your lives and communities that are working to #PressforProgress. We are all stronger when we lift each other up and celebrate each other’s successes.