By: Phil Arcuria, Research Director
A VP, Dean, & Dept. Chair walked Into a bar to discuss a new potential GCC initiative (where did you think I was going with this?). The initiative will require various types of resources and they want to make sure it “works”. They are discussing potential steps to take to evaluate the efficacy of the initiative. I happened to be sitting at the table next to them and, being a nosy neighbor, offered the following quick tips to help guide their efforts:
Tip 1: Write down the purpose of the initiative and the expected outcome(s). Verbally conveying it is not enough. Writing it down helps actualize it into something “real” that can more easily be refined and shared. One way to articulate the expected outcome is to complete the following sentence, “If the initiative is successful, ________ is expected to happen.” Or, a slight variant, “In order for this initiative to be deemed successful it must_______________.” Further refine the outcomes to capture the properties of SMART goals: Specific, Measurable, Achievable, Realistic, and Time-specific.
Tip 2: Substantiate in writing how you believe the initiative will result in the expected outcome(s). Be specific and detailed. Incorporate in prior research, best practices, how it has worked at other institutions, etc. If no prior research is available, lay out the logical argument on how the initiative will achieve the expected outcome. Pretend you are on an episode of ABC’s Shark Tank and you have two minutes to convince someone to invest the resources required for the initiative (e.g., employee time, funds, etc.). Craft your pitch and read over it. Does it provide a convincing argument of how the initiative will likely result in the expected outcome? If not, further refine it until can stand on its own. This step takes a lot of effort, but if we are not willing to put the effort into substantiating the value of the initiative, should we be asking anyone else to put effort into implementing it?
Tip 3a: Although Realistic is one of the characteristics of a SMART goal it tends to get glossed over. Most of us have a tendency to channel our inner Babe Ruth and swing for the grandiose aspirational outcomes. In higher education, this tends to take two forms. The first is in the unrealistic belief that most initiatives will have a direct effect on increasing persistence and graduation rates. Lots of factors go into whether or not a student persists and/or graduates, ranging from family and work obligations to their level of motivation and academic preparedness. Very few initiatives have the mass needed to directly move these metrics. Instead, focus on outcomes that you believe will be the direct result of the initiative. This also includes ensuring that the outcome and initiative are in alignment. One way to do this is visualize your outcome as a tree and your initiative as a saw. Is your saw proportionate to the size of the tree? If you have a chainsaw to cut down a twig, your outcome is too meek given the initiative. If you have a handsaw to cut down a giant Sequoia, your outcome is too lofty given the initiative. Replace your inner Babe Ruth with your inner Zen and seek balance between the tree and the saw.
Tip 3b: Unrealistic outcomes also come in the form of unachievable performance targets used to quantify them (e.g., [outcome] … will increase 5% over the next year). Take time to talk through what would need to happen for this to occur. For example, if the outcome and performance target is to increase enrollments in course X by Y% over two years, how many more students would need to enroll in course X? Based on the response to Tip 2, is it reasonable that the initiative will result in that? What factors might prevent this from happening (e.g., a drop in overall enrollment) and how likely are they to happen? One way to test how realistic the outcome is to ask yourself, “what percentage of my salary would I be willing to bet that the outcomes is achieved by the specified time frame?” If the percentage is low then you should consider revising the outcome to make it more realistic. It can also be very beneficial to set a range as the performance target rather than a single estimate to account for natural variability from year to year. Also, make sure you distinguish between a percent increase and an increase in percentage points. It may seem like a small nuance but increasing 20% by 10 percent points (20% –> 30%) is a lot different than increasing it by 10 percent (20% –> 22%).
Tip 4: Write down any potential secondary or indirect outcomes of the initiative. There are outcomes that you do not expect the initiative to directly influence but might indirectly influence. In short, success on the initiative will not be determined by these things happening, but they are a nice additional potential outcome. Persistence an graduation rates tend to fall in this category. Do this by expanding the sentence in Tip 1 to, “If the initiative is successful, [direct outcome] is expected to happen, which in turn may lead to [indirect outcome] happening.” Indirect outcomes are like donuts on Friday. They are not essential to measuring the successfulness of the day, but are still worth vigorously pursuing.
Tip 5: Make sure the metric selected to quantify the outcomes directly relates to the outcomes. Sounds straightforward, but it is easy to select the wrong metric given the outcome. My professional leaning is toward quantitative methods. There is something elegant about numbers and their utility. However, it is important to acknowledge that not all things should be evaluated in quantitative terms. Antoine de Saint-Exupéry eloquently reminds us of this in the below excerpt from his classic tale of The Little Prince:
“Grown-ups love figures. When you tell them that you have made a new friend, they never ask you any questions about essential matters. They never say to you, ‘What does his voice sound like? What games does he love best? Does he collect butterflies?’ Instead, they demand: ‘How old is he? How many brothers has he? How much does he weigh? How much money does his father make?’ Only from these figures do they think they have learned anything about him.”
Figures are immensely beneficial and in many cases are the best method for evaluating an expected outcome. But do not let the method drive the need. Go with the approach that best fits the nature of the initiative under evaluation – but do not be deceived into thinking that pursing qualitative outcomes is an easier short cut to evaluating the success of an initiative. In many cases it is a much more difficult path. Ideally, most things should be evaluated from both perspectives.
After enthusiastically conveying these tips, the VP, Dean, and Dept. Chair looked at me with puzzled faces and said, “So is this how research directors spend their Friday nights?” I sheepishly slinked back to my table as I muttered to the group, “the next round is on me.”
For more information, please email us at spa@gccaz.edu.
