Earlier this month, BYU’s sponsoring institution implemented a hiring freeze and other cost-saving measures for 2009 (and perhaps beyond). The current economic “downturn” (insert your own euphemisms here) is impacting higher education coming and going. While the struggling economy and tightened job market are driving more people to seek degrees, the institutions offering those degrees find themselves in situations similar to–or worse than–what we’re facing at BYU.

In a meeting with the Deans the other day, my boss observed that “crisis plus wit equals opportunity.” He was essentially inviting the Deans to think about things they might do creatively and innovatively, in the midst of economic crisis, to address long-standing dilemmas and challenges. I’ve thought quite a bit about this in terms of academic technology the past few days. As I’ve visited with colleagues at other institutions over the years, I’ve participated in many a conversation about institutional leaders not quite catching the vision of distance learning, online learning, hybrid learning, and otherwise technology enhanced learning in higher education. Now that we face growing demand for our services (teaching and learning) at the exact moment that we’re facing resource constraints, we have a golden opportunity to demonstrate the value of our craft.

The Sloan-C Five Pillars of Quality research suggests that online learning ought to yield significant improvements in (1) the quality of student learning outcomes, (2) the efficiency of learning, (3) access to learning, and (4 & 5) student  and faculty satisfaction with the learning process. We have the opportunity before us, perhaps like we will never again in our lifetimes, to demonstrate our ability to deliver on these promises–particularly, the 2nd and 3rd–at our institutions.

My guess is that administrators, deans, chairs and faculty members at colleges and universities around the world are anxious for good ideas and sound proposals about how to manage the problems that confront us in higher education. Let’s not miss this opportunity to meet crisis with wit and do something bold and innovative.

Larry Seawright and I made our presentation this morning at Educause 2008. Our slides are available here.

Together with Stephanie Allen and Whitney Ransom McGowan, Larry and I have been working on an alternative approach to evaluating the effectiveness of teaching & learning technology. Traditionally, evaluation takes the form of comparative-media studies in which one group of students learns via standard methods (control) and others learn with new, experimental methods (test). Over and over (and over) again, these kinds of studies have found differences that are not statistically significant.

The so-called “NSD” (no significant difference) problem is the bane of teaching & learning evaluators the around the world. A growing group of influential scholars has rejected the comparative-media studies approach in favor of design-based research. Borrowing elements of this approach, we have implemented a goal-driven model of instructional design, technology integration, and evaluation at BYU.

Our approach to evaluating the impact of teaching & learning technology (and getting beyond the NSD problem) begins with the end in mind. The first and essential step in this approach is to begin any teaching & learning with technology project with a carefully articulated goal. Without such a goal, there is no clear, shared understanding of what “success” looks like. Hence, evaluation is virtually impossible–if you don’t know what success looks like, i.e. what should be better as the result of a project, what should you evaluate?

Measuring the impact of teaching & learning technology depends on a clear articulation of learning goals, strategies for accomplishing those goals and tactics for implementing those strategies. The goals can then be re-formulated as teaching & learning “problems” and strategies and tactics become “solutions.” Evaluation is then simply the process of measuring the results implemented solutions, as illustrated below:

To facilitate the consistent articulation of teaching & learning goals, we’ve adopted the Sloan-C’s Five Pillars: (1) Student Learning Outcomes, (2) Cost Effectiveness (Scalability), (3) Access, (4) Student Satisfaction, and (5) Faculty Satisfaction. By choosing to explicitly focus on one or more of these goals in every teaching and learning project, we identify what success should look like and, at the same time, establish an evaluation plan for each project.

As the examples in our slides suggest, there are often serendipitous results of teaching & learning technology implementation efforts. For example, a project aimed at improving access might also improve student learning outcomes and student satisfaction. However, by articulating and staying focused on a clear, shared rationale (and funding justification) for projects, we have been able to consistently measure and demonstrate the impact of our teaching & learning technology projects and get beyond the NSD problem.

It all begins by starting with the end in mind.

I’ve been thinking “big thoughts” lately, a problem brought on by several recent conversations with David Wiley. I realize I’m repeating something I’ve written before, but the idea is so core to the way I see things that I think it bears repeating–the purpose of institutions of higher education (and all of their associated functions and personnel) is student learning. Learners and the knowledge and skills they acquire are the raison d’etre of colleges & universities. Sure there are folks who might argue that university-based research is just as important, but the number of institutions that could send their students home and still make a case for their continued existence is very small.   So why does this matter to an academic technologist? Because at the end of the day, my purpose is to ensure that our investments in technology promote better, more effective and even more efficient learning. While institutional and instructor efficiency and convenience are laudable goals, however, I’m increasingly of the mind that these goals, by themselves, are not very good justifications for technology expenditures. Unless these efficiencies and conveniences have a direct impact on student learning effectiveness and efficiency, I think we’re missing the mark.  For example, if we relieve some of the administrivia for an instructor in an introductory course, we should ask ourselves what the instructor is doing with the saved time. If he or she simply has more discretionary time, that’s a nice thing, but not necessarily worth significant institutional investment. If, on the other hand, that extra time is dedicated to more one-on-one time mentoring and coaching students, working on mentored research projects with students, or teaching smaller sections of upper-division courses, methinks that is a more justifiable use of institutional teaching & learning improvement resources. Some not-so-random observations that have been bouncing around my noggin related to learners, goals & technologies:

1. We should focus on technologies that support LEARNING activities more than we do on technologies that support TEACHING activities.
2. If we can’t readily explain how we expect a particular technology to improve learning, we should rethink what we’re doing.
3. If a technology yields significant institutional or instructor efficiency, we should ask how learners will benefit from that efficiency, i.e. how will learning be improved?

Perhaps these observations are obvious to most readers, but, again, I believe their important enough that they need to be repeated, again and again, so we don’t forget them. Maybe a teaching & learning with technology mantra is in order: “Teaching & learning technology should always improve learning. Teaching & learning technology should always improve learning. Teaching & learning technology . . .” You get the idea. So, how can we tell if technology has actually improved learning? That’s the subject of a presentation I’m making this Friday at Educause 2008. I’ll post about in a couple of days. 

The text “answers” service ChaCha invites user to “simply ask your question like you are talking to a smart friend and ChaCha’s advanced technology instantly routes it to the most knowledgeable person on that topic in our Guide community. Your answer is then returned to your phone as a text message within a few minutes.”

I tried a few questions to see how it works. Here are the results:

Q: What is a confidence interval?
A: A group of continuous or discrete adjacent values is used to estimate a statistical parameter (as a mean or variance)

Q: When was the French Revolution?
A: A major transformation of the society and political system of France, the French Revolution lasted from 1789 to 1799.

Q: How does photosynthesis work?
A: Photosynthesis is the process by which green plants convert carbon dioxide to sugar and oxygen using sunlight for energy.

Q: Who wrote the Book of Mormon? (Thought I’d try a difficult one.)
A: No one can say with certainty who wrote it, or which author wrote which portions. There were too many contributors.

The answers are surprisingly cogent. Where do they come from? According to the site, live “ChaCha Guides” are behind the answers:

ChaCha’s Guides are individuals who are part of a vibrant community dedicated to helping people by sharing their knowledge. To become a ChaCha Guide, you must pass a series of tests that verify that you are a good fit with our Guide community. You are then able to go through ChaCha’s Search University and simulation process to become certified as a live ChaCha Guide. This unique approach aims to ensure that only knowledgeable people who have an interest in sharing their knowledge with others are part of ChaCha’s Guide community. ChaCha’s technology is also learning from each answer that is provided by our guides so that we can deliver accurate answers as quickly as possible

It’s unclear to me if the Guides are volunteers or if they are paid. In any case, this service raises interesting questions about the nature of knowledge. When I first heard of ChaCha and did some investigation, I was primarily concerned that students might use such a service to cheat on exams, quizzes and even homework. And I remain concerned that a student might surreptitiously use a cell phone in his or her pocket to “look up” answers on a test.

But this got me thinking about the nature of knowledge and the importance of recall. If a student can (almost) instantaneously get answers to factual questions, how important is it for us to require them to memorize facts? ChaCha will certainly not be the last or most sophisticated tool that provides just-in-time answers to knowledge questions.  As educators and learning technologists, our challenge is to figure out how to make assessment more meaningful and authentic in a world in which rapidly accessing facts is a trivial matter. When anyone can access any bit of knowledge anywhere, anytime, the real premium will increasingly be knowing what to do with that knowledge. Memorization will increasingly give way to analysis, synthesis and the creation of knew knowledge.

Plus cha-cha change . . .

I was intrigued to see that the reauthorization of the Higher Education Act passed by Congress in July and signed into law by Pres. Bush in August includes the establishment of a “National Center for Research in Advanced Information and Digital Technologies.” According to the legislation, the Center’s purpose is to:

“[S]upport a comprehensive research and development program to harness the increasing capacity of advanced information and digital technologies to improve all levels of learning and education, formal and informal, in order to provide Americans with the knowledge and skills needed to compete in the global economy.”

To this end, the Center will award contracts and grants for the following purposes:

(A) to support research to improve education, teaching, and learning that is in the public interest, but that is determined unlikely to be undertaken entirely with private funds;

(B) to support–

(i) precompetitive research, development, and demonstrations;

(ii) assessments of prototypes of innovative digital learning and information technologies, as well as the components and tools needed to create such technologies; and

(iii) pilot testing and evaluation of prototype systems described in clause (ii); and

(C) to encourage the widespread adoption and use of effective, innovative digital approaches to improving education, teaching, and learning.

While the Center has been authorized, but not yet funded via the appropriations process, it appears to have broad political support. Here’s to its future and the research it will foment . . .