The End in Mind

PlugJam (http://plugjam.com) appears to offer a crucial piece of the open learning network puzzle. While it makes intuitive sense to allow seamless integration between campus-based apps and Web 2.0 apps, it’s much easier to think about such integrations than it is to actually pull them off. According the company’s website, “PlugJam is a solution for schools, colleges, and universities looking to bridge the gap between existing campus-based tools and Web 2.0 services, allowing students to use their favorite social networking environment or Web Service to access their campus-based resources.”

This illustration (also from PlugJam’s website) shows how the PlugJam open API facilitates interconnectivity between the campus enterprise and the cloud:

Among other things, PlugJam allows campuses to:

PlugJam has already built ”connectors” for Blackboard, Moodle, Peoplesoft Student, “identity management servers” and “portal servers”.

I can’t wait to see what this tool can do!

(For another take on PlugJam, see “Bringing Student ‘Stuff’ to Campus Enterprise Systems“.)

Larry Seawright and I made our presentation this morning at Educause 2008. Our slides are available here.

Together with Stephanie Allen and Whitney Ransom McGowan, Larry and I have been working on an alternative approach to evaluating the effectiveness of teaching & learning technology. Traditionally, evaluation takes the form of comparative-media studies in which one group of students learns via standard methods (control) and others learn with new, experimental methods (test). Over and over (and over) again, these kinds of studies have found differences that are not statistically significant.

The so-called “NSD” (no significant difference) problem is the bane of teaching & learning evaluators the around the world. A growing group of influential scholars has rejected the comparative-media studies approach in favor of design-based research. Borrowing elements of this approach, we have implemented a goal-driven model of instructional design, technology integration, and evaluation at BYU.

Our approach to evaluating the impact of teaching & learning technology (and getting beyond the NSD problem) begins with the end in mind. The first and essential step in this approach is to begin any teaching & learning with technology project with a carefully articulated goal. Without such a goal, there is no clear, shared understanding of what “success” looks like. Hence, evaluation is virtually impossible–if you don’t know what success looks like, i.e. what should be better as the result of a project, what should you evaluate?

Measuring the impact of teaching & learning technology depends on a clear articulation of learning goals, strategies for accomplishing those goals and tactics for implementing those strategies. The goals can then be re-formulated as teaching & learning “problems” and strategies and tactics become “solutions.” Evaluation is then simply the process of measuring the results implemented solutions, as illustrated below:

To facilitate the consistent articulation of teaching & learning goals, we’ve adopted the Sloan-C’s Five Pillars: (1) Student Learning Outcomes, (2) Cost Effectiveness (Scalability), (3) Access, (4) Student Satisfaction, and (5) Faculty Satisfaction. By choosing to explicitly focus on one or more of these goals in every teaching and learning project, we identify what success should look like and, at the same time, establish an evaluation plan for each project.

As the examples in our slides suggest, there are often serendipitous results of teaching & learning technology implementation efforts. For example, a project aimed at improving access might also improve student learning outcomes and student satisfaction. However, by articulating and staying focused on a clear, shared rationale (and funding justification) for projects, we have been able to consistently measure and demonstrate the impact of our teaching & learning technology projects and get beyond the NSD problem.

It all begins by starting with the end in mind.

The Educause Learning Initiative (ELI) has launched a project focusing on the top teaching & learning challenges in higher education. This is looks to be a great initiative. I attended a session on it at Educause today. The project site indicates this is a “new community effort to surface and synthesize trends in higher education.”

Teaching & learning professionals can contribute their ideas via an online survey through the end of the week. Then ELI staffers will identify the top five issues, around which they will facilitate additional discussion and debate. Among other things, they’re hosting a wiki and a social networking site on Ning.

The session today featured some observations from panel members and an audience brainstorming session RE some of the major challenges before us. A sampling of the issues:

1. Promoting authentic / engaged learning
2. Developing “21st Century Literacy” in students AND faculty
3. Getting teaching & learning the priority they deserve (budget, tenure, etc.)
4. Effectively evaluating teaching & learning improvements
5. Encouraging learner ownership of the learning process
6. Closing the teacher-student technology gap

This looks to be a very interesting, engaging initiative. Have something to say about these or related issues? Go participate!

I’ve been thinking “big thoughts” lately, a problem brought on by several recent conversations with David Wiley. I realize I’m repeating something I’ve written before, but the idea is so core to the way I see things that I think it bears repeating–the purpose of institutions of higher education (and all of their associated functions and personnel) is student learning. Learners and the knowledge and skills they acquire are the raison d’etre of colleges & universities. Sure there are folks who might argue that university-based research is just as important, but the number of institutions that could send their students home and still make a case for their continued existence is very small.   So why does this matter to an academic technologist? Because at the end of the day, my purpose is to ensure that our investments in technology promote better, more effective and even more efficient learning. While institutional and instructor efficiency and convenience are laudable goals, however, I’m increasingly of the mind that these goals, by themselves, are not very good justifications for technology expenditures. Unless these efficiencies and conveniences have a direct impact on student learning effectiveness and efficiency, I think we’re missing the mark.  For example, if we relieve some of the administrivia for an instructor in an introductory course, we should ask ourselves what the instructor is doing with the saved time. If he or she simply has more discretionary time, that’s a nice thing, but not necessarily worth significant institutional investment. If, on the other hand, that extra time is dedicated to more one-on-one time mentoring and coaching students, working on mentored research projects with students, or teaching smaller sections of upper-division courses, methinks that is a more justifiable use of institutional teaching & learning improvement resources. Some not-so-random observations that have been bouncing around my noggin related to learners, goals & technologies:

1. We should focus on technologies that support LEARNING activities more than we do on technologies that support TEACHING activities.
2. If we can’t readily explain how we expect a particular technology to improve learning, we should rethink what we’re doing.
3. If a technology yields significant institutional or instructor efficiency, we should ask how learners will benefit from that efficiency, i.e. how will learning be improved?

Perhaps these observations are obvious to most readers, but, again, I believe their important enough that they need to be repeated, again and again, so we don’t forget them. Maybe a teaching & learning with technology mantra is in order: “Teaching & learning technology should always improve learning. Teaching & learning technology should always improve learning. Teaching & learning technology . . .” You get the idea. So, how can we tell if technology has actually improved learning? That’s the subject of a presentation I’m making this Friday at Educause 2008. I’ll post about in a couple of days. 

TTIX 2009 will be held at Utah Valley University in June 2009. The Call for Proposals is now open.

For the second year in a row, the organizers are calling for proposals to be submitted publicly then rated by visitors to the conference website. An innovative use of social technology to promote an interactive discussion and conference around the innovative use of technology in teaching and learning! 

I attended OpenEd 2008 @ Utah State the last couple of days. Even though I missed the last day of the conference (today) it was one of the best, most inspiring, thought-provoking conferences I’ve attended in a long time.

Here are some highlights /observations:

• Seeing Yale’s OCW demo and being reminded that sometimes quality is more important than quantity. They have seven (yes, 7) courses online that have been viewed by 500,000 people. Not too shabby.
• David Wiley’s declaration: “If my students can Google it, I don’t need to teach it.” The new knowledge economy is much more about what you can do with information than it is what you can memorize. (See my recent post RE ChaCha.)
• Interesting observation by Yoshimi Fukuhara of Keio U that OCW sites are too focused on content and not enough on the learner experience.
• Tusk Project at Tufts U facilitates “personal knowledge management” for students.
• Terry Bays of OCWC suggests it’s critical for institutions to be clear about the goals their pursuing via Open CourseWare, i.e. what benefit(s) will it bring to the institution? Without such clarity, OCW efforts will be difficult to sustain over time. Jacque du Plessis made a similar argument in his presentation on the OCW lifecycle.
• The hike up Logan Canyon. A very nice, refreshing break in the middle of the normal conference grind.
• Finally meeting Brian Lamb after bumping into each other on Twitter and blogs for several months.
• Confirmation from several folks after my presentation that a standalone, CMS-independent gradebook is a critical missing link for the creation of more open, flexible learning networks.
• General mood / ideology / philosophy permeating the conference that learning and learners are much more important that institutional niceties, systems, vendors, etc. etc. etc.

Great conference! Thanks to the organizers, presenters and participants!

I just finished reading The Starfish and the Spider. The authors argue that a growing number of organizations are more like starfish than they are like spiders. Spiders are “centralized” in that they have a head that governs the rest of the body. If you cut off the head, the spider dies. In contrast, starfish do not have a central nervous system. They have no head, no brain. If you cut off a tentacle from a starfish, the starfish will probably grow a new one. In some species, the tentacle might even grow a whole new starfish. 

Exactly how a starfish survives, e.g. how it decides to move from point A to point B, is something of a biological mystery. Without a brain to coordinate the movement of its tentacles, a starfish could just flop around on the ocean floor. But somehow it moves. The legs somehow negotiate with each other to move in the direction that  is best for the organism, say in the direction of something to eat.

Starfish organizations are similar in nature. With little or no centralized coordination, they still manage to pursue and achieve a common (or widely shared) goal.

The authors suggest that starfish organizations have the following characteristics (p. 46-53):

• There’s no one person “in charge”
• There’s no recognizable headquarters
• If you “thump it on the head” it survives
• There’s an amorphous division of roles
• If you “take out a unit” the community survives
• Knowledge and power are distributed
• The organization is flexible
• Units are self-funding
• It’s difficult (or impossible) to count “members” or participants
• Working groups communicate directly with each other (instead of through a central organization)

Alcoholics Anonymous, the music-swapping community, al Qaeda, and craigslist are examples of communities that are more like starfish than they are like spiders. 

Traditional institutions of learning (colleges and universities) are much more like spiders than they are starfish. In contrast to the starfish organizations described in the book, Universities have clearly identifiable leaders (presidents), a headquarters, and are dependent upon the “central administration” to survive. Additionally, there’s a clear division of roles, the organization tends to be more rigid than flexible (think colleges & departments), units get the bulk of their funding from the central organization, and you can count the participants (student FTEs). In some ways, however, Universities can be like starfish–you can (in most cases) take out a unit without severely harming the organization, knowledge and power is distributed, and working groups frequently communicate directly with each other (without coordination from the administration).

In the end, the authors suggest that the most successful, dynamic and vibrant organizations are likely to be hybrids, i.e. part starfish-part spider (sounds like a bad SciFi Channel movie, doesn’t it?). Open Content and Open Educational Resources, and Open CourseWare all have the potential to make institutions of higher education more like starfish. These movements tend to promote broader participation in and ownership of curriculum. There is no “president” of the OER movement. Instead there are “thought leaders.” Instead of a headquarters, some working groups recognized by the community as centers of excellence and best practices. There is an amorphous division of roles. It’s difficult to count the participants. And so on.

But how do we reconcile the fundamentally starfish-like traits of the OER community with the traditional hierarchies of colleges and universities? The ideal hybrid organization is centralized where centralization adds the most value to its members and decentralized where decentralization is best. For example, eBay is centralized when it comes to verifying the identity of buyers and sellers and the security of financial transactions. But it is highly decentralized when it comes to what is bought and sold and at what price in the auctions it hosts. These decisions are made almost exclusively by the members of the eBay community.

As institutions of higher education evolve, they might similarly remain centralized in cases where centralization adds value to students–paying instructors, providing physical learning spaces, certifying student performance, granting credentials and degrees. Accordingly, institutions might become increasingly decentralized when it comes to the creation and delivery of learning materials, the bundling of various learning activities, the provision of teaching and learning tools, etc.

Many faculty and students are already moving in this direction in their teaching and learning activities. Are college and university administrators paying attention?

The text “answers” service ChaCha invites user to “simply ask your question like you are talking to a smart friend and ChaCha’s advanced technology instantly routes it to the most knowledgeable person on that topic in our Guide community. Your answer is then returned to your phone as a text message within a few minutes.”

I tried a few questions to see how it works. Here are the results:

Q: What is a confidence interval?
A: A group of continuous or discrete adjacent values is used to estimate a statistical parameter (as a mean or variance)

Q: When was the French Revolution?
A: A major transformation of the society and political system of France, the French Revolution lasted from 1789 to 1799.

Q: How does photosynthesis work?
A: Photosynthesis is the process by which green plants convert carbon dioxide to sugar and oxygen using sunlight for energy.

Q: Who wrote the Book of Mormon? (Thought I’d try a difficult one.)
A: No one can say with certainty who wrote it, or which author wrote which portions. There were too many contributors.

The answers are surprisingly cogent. Where do they come from? According to the site, live “ChaCha Guides” are behind the answers:

ChaCha’s Guides are individuals who are part of a vibrant community dedicated to helping people by sharing their knowledge. To become a ChaCha Guide, you must pass a series of tests that verify that you are a good fit with our Guide community. You are then able to go through ChaCha’s Search University and simulation process to become certified as a live ChaCha Guide. This unique approach aims to ensure that only knowledgeable people who have an interest in sharing their knowledge with others are part of ChaCha’s Guide community. ChaCha’s technology is also learning from each answer that is provided by our guides so that we can deliver accurate answers as quickly as possible

It’s unclear to me if the Guides are volunteers or if they are paid. In any case, this service raises interesting questions about the nature of knowledge. When I first heard of ChaCha and did some investigation, I was primarily concerned that students might use such a service to cheat on exams, quizzes and even homework. And I remain concerned that a student might surreptitiously use a cell phone in his or her pocket to “look up” answers on a test.

But this got me thinking about the nature of knowledge and the importance of recall. If a student can (almost) instantaneously get answers to factual questions, how important is it for us to require them to memorize facts? ChaCha will certainly not be the last or most sophisticated tool that provides just-in-time answers to knowledge questions.  As educators and learning technologists, our challenge is to figure out how to make assessment more meaningful and authentic in a world in which rapidly accessing facts is a trivial matter. When anyone can access any bit of knowledge anywhere, anytime, the real premium will increasingly be knowing what to do with that knowledge. Memorization will increasingly give way to analysis, synthesis and the creation of knew knowledge.

Plus cha-cha change . . .

I was intrigued to see that the reauthorization of the Higher Education Act passed by Congress in July and signed into law by Pres. Bush in August includes the establishment of a “National Center for Research in Advanced Information and Digital Technologies.” According to the legislation, the Center’s purpose is to:

“[S]upport a comprehensive research and development program to harness the increasing capacity of advanced information and digital technologies to improve all levels of learning and education, formal and informal, in order to provide Americans with the knowledge and skills needed to compete in the global economy.”

To this end, the Center will award contracts and grants for the following purposes:

(A) to support research to improve education, teaching, and learning that is in the public interest, but that is determined unlikely to be undertaken entirely with private funds;

(B) to support–

(i) precompetitive research, development, and demonstrations;

(ii) assessments of prototypes of innovative digital learning and information technologies, as well as the components and tools needed to create such technologies; and

(iii) pilot testing and evaluation of prototype systems described in clause (ii); and

(C) to encourage the widespread adoption and use of effective, innovative digital approaches to improving education, teaching, and learning.

While the Center has been authorized, but not yet funded via the appropriations process, it appears to have broad political support. Here’s to its future and the research it will foment . . .

I started reading Wikinomics this week. In the book, the authors observe that “deep changes in technology, demographics, business, the economy and the world” have ushered in a “new age where people participate” like never before (2008, p. 10). Moreover, they contend that we have already reached a “tipping point where new forms of mass collaboration are changing how goods and services are invented, produced, marketed, and distributed on a global basis.” In The Wisdom of Crowds, Surowiecki explains that large groups of people can be “smart” when they are diverse, individuals in the group are independent from each other, and thought processes are decentralized (2004, p. 42). Another view of so-called “crowdsourcing” suggests that humanity is now capable of “using the kind of collective intelligence once reserved for ants and bees—but now with human IQ driving the mix” (Libert, 2007, p. 1). The result? A “quantum increase in the world’s ability to conceive, create, compute, and connect. We are only beginning to comprehend the consequences.”

The troubling thing to me about all of this is how little mention there is of education in these books. For example, Tapscott and Williams specifically mention education only four times in their 340 page volume on “Wikinomics” (see Index p. 343). The references themselves are also enlightening. The first is a mention of the MIT Open Courseware initiative (p. 22-23). The second references TakingITGlobal’s efforts to reform education by providing a “set of tools and curricular activities that will get students collaborating with other students in other countries” (p. 51). The third refers to the California Department of Education’s Open Source Textbook Project (p. 69). The fourth is merely an additional mention of the California textbook project (p. 301). Note that only one of these references relates to the way students actually learn—the others are about content creation and distribution.

This is additional evidence that technology’s real impact on education is yet to be realized. In a 2007 IRRODL article, David Annand observed: “Much like the Industrial Revolution before it, rapid technological change in the Information Age has to date created significant, fundamental change in virtually all sectors of society except education” (2007, emphasis added).

What are the factors that will bring about a fundamental paradigm shift in learning? For starters, I believe we need to press onward in our efforts to make teaching and learning technology (both tools and content) more modular and interoperable. We also need to do a better job of leveraging the network effect, connecting more learners to more content and more fellow-learners. Finally, none of this will be of any significance if we don’t doggedly stay focused on learning (instead of on making administrative and teaching tasks more efficient).

This is all the subject of an article I’m working on with my BYU colleague Mike Bush. I’ll post a link to it when it’s published.