Thinking Like an Evaluator: 2016 Election Edition

As a public health social worker, I became interested in the field of evaluation because I saw an opportunity to use data to inform programs and policies. Not surprisingly, I geek out over elections, making sure to do my research on each candidate and their policy platform. Twenty-four hours ago, like many of you, I cast my vote in a historic election, knowing that whatever the outcome, we would be entering a new phase for our country.

The people have spoken, the data is in, and, America, we have a problem. In evaluation terms, we ignored the early warning signs. We ignored the indicators on the dashboard. We created a predictive model that failed. We experienced an unanticipated outcome.

Today I’m hearing a lot of different reactions to the results, one of which is that people are planning to leave the country. So many people, in fact, that Canada’s immigration website crashed last night as the results were being processed. When we experience unanticipated or negative outcomes in an evaluation, we don’t run and hide, we explore. We ask ourselves and those involved questions like,

  • What contributed to the results of the election?
  • What are the socio-economic-political factors that influenced the outcome?
  • What can we learn from this experience?
  • What are the anticipated and unanticipated effects of this outcome?
  • What alternative solutions might be able to address this problem?

So, America, it’s time to explore and evaluate not run and hide. Regardless of your political affiliation, this outcome is unacceptable and horrific. Think about it, according to the popular vote, half of those who voted supported a candidate who has continuously and unabashedly demonstrated a disregard for the foundational values of our country.

My first reactions when I explore are that I’m saddened, disheartened, and concerned that the results indicate this phase is not progress but misogyny, sexism, racism, xenophobia, and hate. I’m scared and shaken to my core. I fear for my safety, security, and health while also recognizing the privilege that I have based on the different identities I hold. I worry about my friends and family and those I have yet to meet whose basic human rights are going to be taken away. I fear for future generations who’s wellbeing we’re forecasting with each of our decisions. Have we learned nothing from history?

And I want to stay in the comfort of my bed and hide. But progress and change don’t happen in the comfort zone. This is our moment. We have to get up and show up. We have to lean on each other and figure out how to move forward. We have to be unrelenting advocates for social justice. We have to think like evaluators. It’s time to get curious. Blame and hate have no place in this conversation, they will not bring progress. We have to come together as communities and as a country and practice understanding, kindness, and love. So in the words of Glennon Doyle Melton, “carry on, warriors” – let your light shine in the darkness.

Low Cost, High Impact Dashboards: Part 2

This blog post is a collaboration between Elizabeth Grim and Laura Sundstrom (posted on both of our sites), based on our presentation at the 2016 American Evaluation Association conference in Atlanta, GA. The presentation, entitled “Low Cost, High Impact: How to Create Dashboards on a Budget,” focused on basic dashboard design principles and provided detailed examples of how to create meaningful dashboards using Microsoft Excel and PowerPoint. This is the second part of a pair of blog posts about the presentation. The first part focused on basic dashboard design principles.  

Now that you’ve familiarized yourself with our three-step process for dashboard design (Discuss, Develop, Refine), you’re ready to put your dashboard skills to the test. This post will provide example dashboards that we’ve created, and explain how the dashboard design principles were used. Want to learn more about what questions to ask during each phase of the dashboard design process? Download our Dashboard Design Checklist!

basic-attendance-dashboard-v2

Dashboard 1

This is an example of a basic static dashboard that was created using PowerPoint, but the same design elements are available in Excel if you prefer that software. The dashboard was developed with a nonprofit organization to monitor attendance for a multi-site program. Let’s review some of the dashboard design principles and questions.

Discuss

  • Who is the audience? Organization staff.
  • What is the purpose? To understand if attendance is remaining consistent through the growth of the program and addition of new program sites.

Develop

  • Is the dashboard static or interactive? Static.
  • Which platform was used? PowerPoint was chosen because staff feel more comfortable using and editing in PowerPoint vs. Excel. Additionally, PowerPoint is good for formatting and aligning in a static dashboard.
  • What is the best visual display for each metric? Bullet charts are great for showing progress towards a goal.  Column charts allow us to show comparisons across different groups of participants and to easily display a benchmark.
  • Is color used intentionally? The colors align with the organization’s branding guide. Purple was used for overall program metrics and goals, teal is used for individual program sites.
  • Has the clutter been removed? Yes, grid lines, tick marks, and other distracting elements have been removed.

Refine

  • Who needs to provide feedback? Program staff, program directors, and the executive director all provided feedback to ensure that that dashboard was useful and meaningful at all levels.
  • Can users easily determine what action needs to be taken after reviewing the dashboard? Yes, benchmark lines were added so that users can see areas of success and areas for improvement.

Tip: This beginner dashboard was designed in a way such that it has to be updated manually. You can link an Excel file to the dashboard (or use Excel to create the dashboard in general) so that the data is directly linked to the visuals, rather than having to manually update each graph.

fars-dashboard

Dashboard 2

This is an example of a more advanced interactive dashboard that was created in Excel using data from the Fatality Analysis Reporting System (FARS). The dashboard was developed primarily as a demonstration for the AEA 2016 presentation and is considered to be more advanced because it includes slicers (available in Excel 2010+), Pivot Tables, and Pivot Charts. If you want to learn more about how to create a dashboard with slicers, check out the great work happening at MyOnlineTrainingHub, which served as inspiration and guidance for this example.

Alright, let’s review some of the other dashboard design principles.

Discuss

  • What metrics will be included? Total fatalities and DUI fatalities by month and by state.
  • Are the metrics easy to understand? Yes, but could also (or alternatively) include percentages for comparisons.
  • How often does this information need to be updated? Monthly or annually, depending on the level of analysis.

Develop

  • Which platform will be used? Excel 2016 (have to have Excel 2010+ to use slicers).
  • Will the dashboard be static or interactive? Interactive (in Excel, when you click on the states in the slicer on the left sidebar, the blue lines in the bottom two charts move to represent data from that specific state).
  • Does each color represent only one component? Yes, gray is used for the total and blue is used for the states.

Refine

  • Can users easily determine what action needs to be taken after reviewing the dashboard? Not really, might consider adding some text about potential action steps.
  • Does the dashboard continue to be responsive to the ongoing changing and monitoring needs? Yes, the dashboard was created using Pivot Tables and Pivot Charts so it’s easy to update the dashboard simply by adding new data to the existing table.

Tip: You can also embed your Excel dashboard on a website using Microsoft OneDrive. It’s helpful to test out this method using fake data first while you practice choosing different viewing and sharing options, and to make sure all your functionality works.

So there you have it, you’re now equipped with the knowledge and tools you need to become a dashboard superhero! Want to learn more about what questions to ask during each phase of the dashboard design process? Download our Dashboard Design Checklist!

copy-of-dashboard-one-pager

We want to hear from you! What’s working well during your dashboard design process? What are you struggling with? What questions do you need answered that aren’t included in the checklist?

Low Cost, High Impact Dashboards: Part 1

This blog post is a collaboration between Elizabeth Grim and Laura Sundstrom (posted on both of our sites), based on our presentation at the 2016 American Evaluation Association conference in Atlanta, GA. The presentation, entitled “Low Cost, High Impact: How to Create Dashboards on a Budget,” focused on basic dashboard design principles and provided detailed examples of how to create meaningful dashboards using Microsoft Excel and PowerPoint. This is the first part of a pair of blog posts about the presentation. This first part focuses on basic dashboard design principles.  Watch for the second post providing examples of how to use these principles in action using Excel and PowerPoint.

Dashboards are a great tool for organizations to use to analyze and monitor their objectives and outcomes. While they can seem overwhelming and scary at first, dashboards don’t have to be something that organizations have to dedicate a lot of time and money towards. Here we demystify dashboards and provide examples of how to creating meaningful dashboards using tools organizations already have – Microsoft Office and PowerPoint.

What is a Dashboard?

Stephen Few (2006) defined a dashboard as:

“A visual display of the most important information needed to achieve one or more objectives; consolidated and arranged on a single screen so the information can be monitored at a glance.”

An effective dashboard uses visual displays to clearly and concisely deliver key information to an intended audience. Displaying information on a single page or screen allows users to make comparisons, evaluate, and draw conclusions that are not possible when data is split across multiple pages or screens.   

Dashboards can have different purposes: monitor operations, track progress on outputs and outcomes, strategic planning, and for analytics. Regardless of the specific purpose, a dashboard has to meet these three aims (Smith, 2013).

  1. Increase understanding
  2. Communicate effectively
  3. Inform decision making

Dashboard Design Principles

When we think about creating a dashboard, we think about it as a three step process: Discuss, Develop, Refine.

dashboard-design-steps-image

1. Discuss

Dashboards cannot be created in a vacuum. You need to understand what the dashboard is going to be used for, who needs the information, and what information is most important. The only way to understand this is by talking to your key stakeholders.  

Know your audience: Who is the intended audience of the dashboard? Who needs this information? What type of information do they need?purpose-drive-dashboard

Know your purpose: You have to understand what the audience is going to be using the
dashboard for and how it will be used in order to make a meaningful and usable dashboard.

Identify appropriate metrics: After you know the audience and purpose, you can identify the most appropriate metrics that will meet these needs. You also have to take it one step further and prioritize the metrics. Because you have to fit all of this information onto a single page or screen, you have to be able to have the difficult conversations about prioritization and recognize that you don’t have to say “yes” to every data request.

Just as we cannot be all things to all people, our dashboards cannot serve every purpose and meet the needs of every audience. Your dashboard is only as good as the world you put into it in the beginning.

2. Develop

Next it’s time to put “pen to paper” (or whatever medium you choose), to start to develop your dashboard.  

Identify a platform: There any many different platforms available for dashboard development, with varying price tags and learning curves. Most of our experience is in creating dashboards using Microsoft Excel and PowerPoint. There are many other options out there, such as Google Sheets and Slides, Tableau, and Tableau Public.

Sketch it out:  As you are sketching out the dashboard, here are some important things to remember:

  • einstein-quotesKeep it simple: People have a hard time retaining more than 3-5 pieces of information at a time. Overcomplicating your dashboard with too many metrics will make it hard for users to understand the main message(s). Your dashboard should also be structured in a way that helps users understand the big picture, using flow, relationships, or grouping. 
  • Identify the appropriate chart type for the data: Flashy doesn’t mean better; pretty doesn’t mean effective. You do not have to use every type of chart available to make your dashboard “interesting.” It is more important that your dashboard is meaningful and effective, ensuring that the right chart type is utilized for each metric. If that means that the same chart type is used with every metric, that’s fine!  
  • Eliminate the clutter: It is hard to see the message through the clutter. Only the things that are necessary to the story of the dashboard should be included. Remove “chart junk” – anything on the default chart settings that are decorative or not necessary (like grid lines, tick marks, 3D displays). Use color intentionally to convey meaning – rainbow graphs are out, impactful graphs are in!
  • Consider accessibility concerns: It’s important to ensure that users can easily access the information in your dashboard. Accessibility concerns you will face include:
    • Color blindness: The main forms of color blindness in the U.S. are red and green, and blue and yellow. Try to avoid these color combinations. Check out a tool like Color Oracle to see if your visualizations are readable by users experiencing color blindness.
    • Color Contrast: Some organizations might print your dashboard in black and white, or photocopy your dashboard multiple times. Try viewing your dashboard in black and white and/or print your dashboard in black and white to see if the key messages are able to be interpreted when color is removed.
    • Legibility: Make sure that it is easy for users to read the dashboard. Use appropriate font styles and sizes.

Make a dashboard: Now that you have sketched out the dashboard, taking all of these considerations into account, you are ready to create the prototype. This process may take multiple iterations – your initial prototype will not be your final dashboard.

Document how the metrics were developed and how the visuals were made, what data cleaning occurred, and so on. When you update the dashboard, you won’t have to completely re-do all of your work.

3. Refine

Remember how you aren’t making this dashboard in a vacuum? You need to talk with people at this phase of the dashboard creation too.  dashboard-design-process

Get feedback: Get feedback from potential users on how they interact with the dashboard. Who needs to provide feedback and approve the dashboard? Does the dashboard provide enough context for users? Is it easy for users to understand what actions need to be taken based on what the dashboard is showing? 

Refine: Once you get feedback, the next step is to take their feedback into consideration and refine the dashboard. Remember to prioritize and understand that your dashboard can’t be every thing to every person.

Update regularly: Every dashboard should have an update protocol outlining how often the dashboard will be updated and who is responsible for updating it. How often you update the dashboard will depend on the audience, purpose, and metrics.  

These three design steps (Discuss, Develop, Refine) will help you with any dashboard that you are creating, no matter how complicated or platform used.  

We want to hear your input on how you develop your dashboards. What tricks do you use? What are the important things to include? What do you not need to keep in your dashboard?

And watch out for the second post in this pair about how to apply these principles to making dashboards in Excel and PowerPoint!

More Resources

You can find resources for selecting an appropriate chart type here:

The What, Why, and How of Data Visualization

What is data visualization?

Data visualization is a way of representing information visually, often through the use charts, images, and maps. The goal of data visualization is to inform and communicate information so it should come as no surprise that simplicity is key. A favorite tagline I’ve used in presentations is “pretty doesn’t mean effective.” Similarly, in his book Data at Work, Jorge Camoes describes data visualization as “understated elegance,” noting that “the purpose of data visualization in organizations is not to make beautiful charts; it is to make effective charts.”

Why is data visualization important?

With the advancing technology of the digital age, the rate at which we create and consume data is increasing exponentially. In fact, the 2014 EMC Digital Universe Study found that “the digital universe is doubling in size every two years and will multiply 10-fold between 2013 and 2020 – from 4.4 trillion gigabytes to 44 trillion gigabytes.” To put that into perspective, EMC says that “today, the average household creates enough data to fill 65 iPhones (32gb) per year. In 2020, this will grow to 318 iPhones.” That’s a lot of data!

The days of creating long dense text-filled reports are becoming limited. Data visualization is no longer something novel that can be viewed as an afterthought. Rather, data visualization is a necessity to keep up with and communicate the growing amounts of information being produced. Data visualization is the language of the 21st century; it’s the language of the present and of the future.

Side Note: Data visualization in everyday life

Although the term data visualization can sound a bit intimidating, I believe that we already have many of the tools we need to create great visualizations. We’ve been conditioned from a young age to understand that images represent information. Think back to the games you played as a child like Go Fish, Memory, or Uno. These games taught us how to group things into categories and interpret visual cues. We learned that color, size, and shape are meaningful and are to be used intentionally.

Now think about an experience you may have had more recently as an adult like the last time you went out to eat at a restaurant. Did you notice anything about how the information was organized on the menu? Restaurants usually use section headings to orient the reader to different types of food. They include pictures to make the menu more engaging. Often, restaurants will also highlight or star their specialties so they stand out from the other items. This is data visualization in everyday life!

If you want to learn more about how we perceive information, check out this Khan Academy video about the Gestalt Principles of Perceptual Organization.

How can I create data visualizations?

One common fallacy I hear people say is that they don’t have a budget for data visualization. Creating impactful data visualizations is not dependent on using expensive technology or flashy chart types. The best data visualizations simply tailor the format and message to the specific audience to clearly communicate the information. They strip away the excess visual noise so that the message can be interpreted quickly and easily.

If you’re looking to start creating data visualizations, a great place to begin is Microsoft Office (Word, Excel, PowerPoint). Often simple tweaks to the default settings can produce polished charts and visuals. Additionally, there are a number of free online resources such as Tableau Public, Google Drive, Canva, and Piktochart that can be used to produce presentations, dashboards, charts, social media content, and infographics.

Where else can I learn about data visualization?

Check out my previous blog post on book recommendations for my go-to data visualization reads. I also highly recommend checking out Stephanie Evergreen, Ann Emery, Chris Lysy, and Jorge Camoes’ blogs.

Have other favorite resources for data visualization? Share them in the comments!

Hi, I’m a program evaluator

The most popular question at social events seems to be some variation of “what do you do for a living?” In my experience, the topics I’ve focused on thus far in my career can shut down a dinner party conversation faster than you can ask what’s on the menu. Here’s a spark notes version – homelessness, anxiety, sexual violence against women, substance use, child welfare, and juvenile justice. Apparently these are topics that not everyone feels comfortable discussing at happy hour. I’ve also found that people are not that excited to discuss data or statistics. Sound familiar?

Given these reactions, I’ve contemplated developing a professional happy hour-friendly alter ego. But that would be doing a disservice to our wonderful field of evaluation and my quest to champion the implementation of evaluation and data-driven decision-making. Instead, my response usually sounds something like, “I’m a program evaluator, which means that I help organizations understand their data so they can make their programs even more effective.” My email signature is similar and says that “I make data accessible to improve outcomes and communicate impact.”

3At this point the conversation usually goes one of three ways:

  • The person stares at me blankly and the conversation is steered in another direction
  • The other person thinks that I’m an auditor or accountant
  • The person seemingly loves data and nonprofits as much as I do and asks lots of questions

Here’s the thing – People use data and evaluation in their daily lives. Let’s consider Facebook as an example. Facebook users rate the value of their posts by looking at the number of likes, shares, and comments often on a daily, if not more regular, basis. This is evaluation! People also cook recipes and make adjustments to the ingredients when something doesn’t turn out as desired. This is data-driven decision-making!

So this is what I’m wondering:

  • If people use data and evaluation in their everyday lives, why don’t people know more about program evaluation? We just finished the International Year of Evaluation (2015)!
  • How are we talking about and marketing evaluation as a profession?
  • As an evaluator, how do you define evaluation and how do you answer the question, “what do you do for a living?”

12