UPCEA Updates

These days, students and prospective students are becoming increasingly critical of higher education programs—especially as the cost of higher learning continues to rise. Understandably, students want to make sure they’re getting their money’s worth before enrolling in (and committing to) a program.

In other words, they want to know that the program they’re considering is effective and proven to set students up for success in their respective fields. Meanwhile, it’s not just students demanding this transparency and accountability. Faculty and funders alike are increasingly looking for in-depth program evaluations to assess the impact of higher education programs.

Being able to measure impact of a program effectively can improve accountability in addition to answering important questions held by prospective students, faculty, and funders. So, how do you effectively measure the impact of your higher education programs? There are some actionable steps your institution can take to get started.

Why Measure the Impact of a Program?

First, consider the importance of measuring the impact of higher education programs, including what it actually means to take on this kind of initiative.

Program evaluations in higher education involve a systematic process where data is objectively collected, analyzed, and reviewed to determine the overall success and effectiveness of the program compared to its learning objectives and goals.

Although there are numerous different strategies for measuring the impact of a higher education program, the importance of this practice remains the same regardless of the process used. Program evaluations and assessments help prospective students decide on an educational pathway and give faculty and funders peace of mind while improving accountability and maximizing transparency to stakeholders.

In many ways, program evaluations can also be used to identify weaknesses within a program that can become focus areas for improvement. All of this can help schools improve the quality of their educational programs while securing more funding and resources and attracting or retaining more students.

Defining Your Impact

Where does one even begin when it comes to measuring impact evaluation in education? There is no single way to carry out this type of assessment, but there are some general rules and best practices your organization will want to follow to optimize the process.

First and foremost, your team needs to clearly define the goals and objectives of the higher education program being assessed. By carefully defining those goals and objectives, you’ll be able to come up with a list of key performance indicators (KPIs) that are aligned with those outcomes. KPIs refer to measurable and trackable metrics that you can use to gauge the impact of your program.

When choosing KPIs, it’s crucial that your team selects metrics that are not only easily measurable and trackable but also relevant to your goals and program objectives. We’ll further explore the process of choosing KPIs based on your goals in the next section; however, the most essential thing to remember here is that your program evaluations should always begin with a list of key goals and objectives for each program being assessed.

Key Performance Indicators Based on Your Goals

One of the biggest challenges your team is likely to run into after determining the goals and outcomes of your higher education program is choosing specific KPIs to track based on those goals. This can be difficult because different objectives will require looking at different KPIs, and choosing the wrong KPIs can result in a complete lack of insight when your assessment is complete.

To get a better feel for the kinds of KPIs your team may select based on different goals and objectives, it can be helpful to look at some examples.

Consider, for instance, a micro-credential or certificate program evaluable where your team is looking to determine how successful the program is in preparing professionals for real-life work scenarios. Some of the KPIs you might consider measuring for this type of assessment may include job promotion and program completion metrics. High completion and promotion rates may indicate that your program is adequately preparing students for success and growth in their respective fields, whereas lower rates may help you identify weaknesses and areas for improvement.

In certain cases, program evaluations may be held to assess both the value of the program to learners and to faculty and stakeholders. In these scenarios, KPIs like research impact, faculty performance, and even financial sustainability may be examined more closely.

Other common examples of KPIs often measured in program evaluations include:

• Course completion rate
• Retention rate
• Graduation rate
• Employment rate
• Learner/student satisfaction
• Faculty-learner ratio

Ultimately, in conducting any assessment and evaluation in higher education, your team will be tasked with choosing the appropriate KPIs based on your assessment goals and outcomes. From there, it typically makes the most sense to focus on three or four KPIs to best gauge how well the program aligns with your goals.

Collecting Data for Impact Evaluation

Even once you’ve selected the right KPIs for your program assessment, the work isn’t over. Now, you need to determine the most effective means of collecting the data you need to conduct your impact evaluation. This includes both direct and indirect methods.

Direct measures are those that are easily observable and able to be quantified, whereas indirect measures are more subjective (such as exit surveys).

When selecting methods for data collection, you should also carefully consider the size of the program. If you’re assessing a large program with a lot of students enrolled, then it might actually make more sense to take a smaller sample size and assess data from there.

As you consider the types of data you’ll collect and the processes you’ll follow to gather it, there are a few different types of data that are commonly considered in program evaluations. These include:

• Quantitative data – Surveys, administrative records, tests, financial data
• Qualitative data – Interviews, course evaluations, faculty observations
• Mix-methods approach – Combines quantitative and qualitative for a more comprehensive understanding
• Data sources – Institutional databases, national surveys, program records

No matter which types of data you decide to collect, there are also some key considerations to keep in mind as they relate to ethics in data collection. Make sure that you have a plan in place for keeping all information safe and confidential throughout the entire process. Likewise, you’ll need to ensure that data is being collected transparently and that informed consent standards are being followed. This means that all participants should know if their data is being used and what it is going to be used for.

Analyzing and Interpreting Data

As data is collected, a plan for analyzing and interpreting it must be put into place. This helps ensure all data is properly assessed and compared to the original goals and outcomes of the program. There are many advantages to responsible data collection here, including the ability to use metrics to inform future program decision-making, improve the outcome of the program itself, and enhance overall reporting and communications.

So, what are some of the most tried-and-true methods for analyzing and interpreting data? Consider the following strategies:

• Data visualization – This can include the use of charts, graphs, and other visuals to represent data in a way that is easy to understand.
• Statistical analysis – This involves collecting large amounts of data and assessing it to pinpoint trends and identify other potentially valuable insights.
• Qualitative analysis – This approach looks particularly at the non-measurable interpretations that can be taken from data, such as an understanding of student opinions regarding a program.

There are various ways in which data can be used in program evaluations. Based on your program evaluation, you might use the data for reports, future planning, and showcasing success stories.

 

Start Assessing Your Higher Education Programs With Confidence

No matter the size or scope of your school, implementing program evaluations and assessments is a critical part of ensuring your students and faculty have access to the best work and learning environments possible. While building a plan to measure the impact of a program isn’t always easy, these steps should help your team get started.

Seeking additional help with assessing the impact of your higher education programs? UPCEA is here to assist with custom consulting services that can help you obtain a more objective analysis of your programs along with guidance for further exploration. Our industry experts have years of experience in this realm, offering a wide range of research and consulting options to suit your needs. Plus, we’re happy to create a custom research plan to accommodate your budget.

Learn more about our professional research and consulting services at UPCEA by getting in touch with our team at (202) 659-3130 or emailing [email protected]. We look forward to putting our team to work for you! 

 

Sources

https://upcea.edu/resources/research-benchmarking/ 

https://www.insidehighered.com/news/2022/11/10/how-colleges-measure-and-prove-their-value-key-podcast 

https://claned.com/how-to-measure-learning-impact/