Larry Seawright and I made our presentation this morning at Educause 2008. Our slides are available here.

Together with Stephanie Allen and Whitney Ransom McGowan, Larry and I have been working on an alternative approach to evaluating the effectiveness of teaching & learning technology. Traditionally, evaluation takes the form of comparative-media studies in which one group of students learns via standard methods (control) and others learn with new, experimental methods (test). Over and over (and over) again, these kinds of studies have found differences that are not statistically significant.

The so-called “NSD” (no significant difference) problem is the bane of teaching & learning evaluators the around the world. A growing group of influential scholars has rejected the comparative-media studies approach in favor of design-based research. Borrowing elements of this approach, we have implemented a goal-driven model of instructional design, technology integration, and evaluation at BYU.

Our approach to evaluating the impact of teaching & learning technology (and getting beyond the NSD problem) begins with the end in mind. The first and essential step in this approach is to begin any teaching & learning with technology project with a carefully articulated goal. Without such a goal, there is no clear, shared understanding of what “success” looks like. Hence, evaluation is virtually impossible–if you don’t know what success looks like, i.e. what should be better as the result of a project, what should you evaluate?

Measuring the impact of teaching & learning technology depends on a clear articulation of learning goals, strategies for accomplishing those goals and tactics for implementing those strategies. The goals can then be re-formulated as teaching & learning “problems” and strategies and tactics become “solutions.” Evaluation is then simply the process of measuring the results implemented solutions, as illustrated below:

To facilitate the consistent articulation of teaching & learning goals, we’ve adopted the Sloan-C’s Five Pillars: (1) Student Learning Outcomes, (2) Cost Effectiveness (Scalability), (3) Access, (4) Student Satisfaction, and (5) Faculty Satisfaction. By choosing to explicitly focus on one or more of these goals in every teaching and learning project, we identify what success should look like and, at the same time, establish an evaluation plan for each project.

As the examples in our slides suggest, there are often serendipitous results of teaching & learning technology implementation efforts. For example, a project aimed at improving access might also improve student learning outcomes and student satisfaction. However, by articulating and staying focused on a clear, shared rationale (and funding justification) for projects, we have been able to consistently measure and demonstrate the impact of our teaching & learning technology projects and get beyond the NSD problem.

It all begins by starting with the end in mind.

The Educause Learning Initiative (ELI) has launched a project focusing on the top teaching & learning challenges in higher education. This is looks to be a great initiative. I attended a session on it at Educause today. The project site indicates this is a “new community effort to surface and synthesize trends in higher education.”

Teaching & learning professionals can contribute their ideas via an online survey through the end of the week. Then ELI staffers will identify the top five issues, around which they will facilitate additional discussion and debate. Among other things, they’re hosting a wiki and a social networking site on Ning.

The session today featured some observations from panel members and an audience brainstorming session RE some of the major challenges before us. A sampling of the issues:

1. Promoting authentic / engaged learning
2. Developing “21st Century Literacy” in students AND faculty
3. Getting teaching & learning the priority they deserve (budget, tenure, etc.)
4. Effectively evaluating teaching & learning improvements
5. Encouraging learner ownership of the learning process
6. Closing the teacher-student technology gap

This looks to be a very interesting, engaging initiative. Have something to say about these or related issues? Go participate!

I’ve been thinking “big thoughts” lately, a problem brought on by several recent conversations with David Wiley. I realize I’m repeating something I’ve written before, but the idea is so core to the way I see things that I think it bears repeating–the purpose of institutions of higher education (and all of their associated functions and personnel) is student learning. Learners and the knowledge and skills they acquire are the raison d’etre of colleges & universities. Sure there are folks who might argue that university-based research is just as important, but the number of institutions that could send their students home and still make a case for their continued existence is very small.   So why does this matter to an academic technologist? Because at the end of the day, my purpose is to ensure that our investments in technology promote better, more effective and even more efficient learning. While institutional and instructor efficiency and convenience are laudable goals, however, I’m increasingly of the mind that these goals, by themselves, are not very good justifications for technology expenditures. Unless these efficiencies and conveniences have a direct impact on student learning effectiveness and efficiency, I think we’re missing the mark.  For example, if we relieve some of the administrivia for an instructor in an introductory course, we should ask ourselves what the instructor is doing with the saved time. If he or she simply has more discretionary time, that’s a nice thing, but not necessarily worth significant institutional investment. If, on the other hand, that extra time is dedicated to more one-on-one time mentoring and coaching students, working on mentored research projects with students, or teaching smaller sections of upper-division courses, methinks that is a more justifiable use of institutional teaching & learning improvement resources. Some not-so-random observations that have been bouncing around my noggin related to learners, goals & technologies:

1. We should focus on technologies that support LEARNING activities more than we do on technologies that support TEACHING activities.
2. If we can’t readily explain how we expect a particular technology to improve learning, we should rethink what we’re doing.
3. If a technology yields significant institutional or instructor efficiency, we should ask how learners will benefit from that efficiency, i.e. how will learning be improved?

Perhaps these observations are obvious to most readers, but, again, I believe their important enough that they need to be repeated, again and again, so we don’t forget them. Maybe a teaching & learning with technology mantra is in order: “Teaching & learning technology should always improve learning. Teaching & learning technology should always improve learning. Teaching & learning technology . . .” You get the idea. So, how can we tell if technology has actually improved learning? That’s the subject of a presentation I’m making this Friday at Educause 2008. I’ll post about in a couple of days. 

TTIX 2009 will be held at Utah Valley University in June 2009. The Call for Proposals is now open.

For the second year in a row, the organizers are calling for proposals to be submitted publicly then rated by visitors to the conference website. An innovative use of social technology to promote an interactive discussion and conference around the innovative use of technology in teaching and learning!