All are welcome to add to this conversation, taken from Inside Higher Education.

The Assessment Gap

Inside Higher Education, December 11, 2009

http://www.insidehighered.com/news/2009/12/11/assess

PHILADELPHIA — Audience members here at one session of the annual meeting of the Middle States Commission on Higher Education were asked to talk among themselves and then to share their biggest challenges with assessment. Among them: “using assessment results,” “communicating assessment results,” “resistance of faculty,” “herding the cats,” and “too much of an emphasis on process and not enough on purposes.”

These and other answers were variations on a theme: Everyone is doing assessment, but at many colleges, not many educators are using the results. For regional accreditors like Middle States, it’s hard to take much satisfaction in the institutional consensus about having assessment activities in place if they don’t prompt change, given that the argument accreditors and others have made about assessment is that it should spur improvements.

If assessment is only done to satisfy accreditors, after all, it may not be done or taken seriously when there’s not a self-study to be produced.

“We don’t want faculty to do this only because they know that they are coming,” said Stephanie Schull, coordinator for undergraduate assessment at Temple University, with the clear implication that the second “they” was made of up of those in the audience — the deans, provosts and faculty members who serve on the committees that conduct accrediting reviews. And to judge from the presentation by Schull and a colleague here — on how to manage assessment “top-down” and “bottom-up” — many of those leading assessment efforts want their faculty members and administrators not only to do it, but to see the value in it.

The discussion here reflects a growing frustration that assessment efforts are hindered by a lack of follow-through, once data are collected or forms are filed. In October, when assessment experts gathered to mark the 10th anniversary of the National Survey of Student Engagement, which is being used at a growing number of institutions, similar complaints were voiced.

At the Middle States meeting, the Temple officials offered strategies for winning faculty involvement and follow through:

• Recognize differences among departments. Jodi Laufgraben, associate vice provost, noted that many programs at her university have specialized accreditation requirements, and said that, generally, faculty members in those programs are more involved than others in accreditation and assessment requirements. “We sort of have an accreditation culture and a non-accreditation culture,” she said. That means a decentralized approach may be more successful, given differing degrees of acceptance or resistance to assessment. To that end, Temple insists that every college or division do assessment, but doesn’t dictate a single approach, she said.
• Publicize success stories. Temple requires all departments to have periodic program reviews, and some professors entered that process “kicking and screaming,” Laufgraben said. But attitudes changed when some departments were able, based on those reviews, to grow. “Once you have a department saying ‘We have hired two new faculty as a result’ or ‘We changed our graduate program as a result,’ it made a big difference,” she said. Faculty members will respond to seeing assessment make a difference, she suggested.
• Start with the basics. Schull said that assessment experts like those in the room need to have realistic first steps. “Don’t try to get from A to Z. Just have a plan to get to Z,” she said. Many professors view assessment as “very esoteric and very bureaucratic,” she said. They can be approached by encouraging them to “tell stories of what they do to improve,” and they will find in most cases that they are doing assessment (but may not be recording or analyzing it in any way).
• Reward, don’t punish, the identification of flaws. Colleges should link budgeting to evidence that departments are engaged in assessment, Schull said. But they need to “be careful” to do it in a way that doesn’t encourage departments to hide problems. Using assessment to identify and fix a problem is something to be rewarded, she said. “If people think that if they uncover a problem they are going to get dinged, you won’t get any assessment or you’ll get cooked assessment,” she said.

The session at Middle States didn’t feature anyone from Temple’s faculty. Reached later, Arthur Hochner, president of the faculty union (an American Federation of Teachers affiliate), said it was fair to say that Temple’s efforts are decentralized. “There is no general assessment regime,” he said.

But he said that despite being decentralized, and there being many seminars and programs, he wasn’t sure that many faculty members actually felt close to the efforts. “I don’t think it’s as effective as anybody would like,” he said. “It seems to me that a lot of it is driven by administrators who come up with ideas, and try to get faculty to go along with it, and faculty don’t have ownership of the process.”

— Scott Jaschik

Comments on The Assessment Gap

• Assessment is expensive; change is more expensive still
• Posted by Unfunded Mandate on December 11, 2009 at 10:45am EST
• Here’s the quote I love from this piece: “Once you have a department saying ‘We have hired two new faculty as a result’ or ‘We changed our graduate program as a result,’ it made a big difference,” she said. Faculty members will respond to seeing assessment make a difference, she suggested.”

My reading of a lot of assessment movements is that it is easier pay for assessment than it is for the real work of improving education. At my university, there’s no money for new tenure-line faculty, and if we said in our assessment that a committed instructor active in his or her field would help solve a problem this statement would carry zero weight.

Meanwhile, assessment, though cheaper than actually providing resources to colleges and departments, is still quite expensive, though much of it is fairly hidden labor. I am still waiting for my upper administration to say, “we don’t want you to research so much; we want you to work on assessing and revising your graduate program.”

As I’ve written before, and I will continue to write, the problem with outcomes assessment is that it operates in a vacuum, without reference to the resource (time and money) demands on faculty or on universities. And that’s my assessment.

• Posted by MathProf on December 11, 2009 at 12:45pm EST
• We do assessment all the time, and have for centuries. It’s called “exams” or “papers.” We read student work carefully, comment on strengths and weaknesses, and meet with students to help them use what we say to make their work better. We consult with colleagues about the assignments in our classes and about our own exams. We have department meetings where we talk about the individual students and the minors and the majors, how they’re progressing, how our requirements for the major should change in the light of recent scholarship.

Our dean said, “All WASC wants is for you to document what you actually do.” In other words, waste time expanding the paragraph above into a series of time-lines (“met with junior colleague for fifteen minutes, explained why question 3 on his final exam will be misunderstood but that question 5 is an excellent way of probing understanding of topic X” “met with colleagues in a department at a neighboring institution to see how they use new instructional technologies — one hour travel each way and a two-hour meeting” “met with the undergraduate student association to discuss how they can raise money to send students to the next regional meeting”). If this exercise would help improve our program in any substantive way, I’d be all for it. But I think it just takes time away from things that really matter.

Now if you’ll excuse me, I need to go and teach a class, make up an exam, and grade a set of term papers. Three tasks, two of them assessment. That’s what we DO, people.

• correction
• Posted by Assessment Expert on December 11, 2009 at 1:45pm EST
• MAthProf, a correction, if you please…. of your three “tasks” – ‘teach a class, make up an exam, and grade a set of term papers’ only one of those is the assessment of student learning. The creation of an exam is the development of an assessment instrument. Only after the instrument is deployed has assessment taken place.

Assessment, at its core, should be for the benefit of the student. It should be designed to communicate to the student, his/her strengths and weaknesses and next steps. Only when (and if, in many cases) the needs of the student are met, should the assessment be applied to other purposes, i.e. accreditation. Accrediting agencies do serve a valuable purpose, however. They force many (often unwilling) faculty, programs and institutions, to look internally at the learning process. At the core, most accreditation agencies really just want institutions to be able to articulate what they are trying to teach students, verify what students are learning, and look for ways to improve the processes. Why are so many so frightened of that?

The ultimate goal of assessment should not be to create grades or inform accredidators, but to improve the teaching and learning process. Simple, right?

• Ford Falcon designer
• Posted by No Need , retired on December 11, 2009 at 2:15pm EST
• Trust me!

• Wrong, ‘Expert’
• Posted by DFS on December 11, 2009 at 2:15pm EST
• Making up an exam is assessment. The exam is communication to the students of what they are expected to know, and it’s most probably based on what the teacher has already assessed the students should be able to demonstrate knowing, dynamically and immediately based on continual results.

Live in the real world, ‘Expert,’ and let us assess in the way it took for this planet to reach the Moon.

• That’s the rub…how?
• Posted by Chemist , assist. prof/chemistry at Hamline University on December 11, 2009 at 2:45pm EST
• Dear Assessment Expert – How do you improve the teaching and learning process? Assessment is about quantifying what is currently going on in education – at an enormous time cost to those trying to record what they do onto your forms. It doesn’t address the fact that every class and every student is different, learns differently, and teaching needs to be adapted to to the specific needs of the class. Further, teaching methodology is a lot older and more tested than it is given credit for….every new whiz-bang idea that comes out is old news if you look at the historical record…we will get to the end of the assessment movement and see no progress because there will be no big revelation from the data – I predict…anyone remember “New Math”?

• Wall Street Bank Exec
• Posted by No need , retired on December 11, 2009 at 8:00pm EST
• Don’t worry…trust me!

• Some clarifications are in order
• Posted by Barbara Wright , Associate director at WASC on December 11, 2009 at 8:00pm EST
• First — if you haven’t used the results, you haven’t “done” assessment. Assessment is a process with multiple step: defining the learning outcomes you want; collecting evidence that can tell you whether outcomes have been achieved, and at what level of proficiency; interpreting the evidence — collectively, not alone — to make meaning and so you can decide what to do when results are not what you wish; and finally DOING something to improve outcomes.

If you’re just doing step two (collecting data or evidence) and sending off a report, you may be satisfying some accountability requirement, but you’re not doing assessment. You’re doing a pale imitation that consumes time and energy without generating any improvement. Accrediting agencies, believe it or not, really want real assessment.

Second — yes, the “doing something” step takes leadership and resources beyond what faculty members or a department can usually provide. Administrators have a role to play, too. If they don’t hold up their end, there is no reason for faculty or other educators on campus to believe that work on assessment will have any effect, so why do it? Reward systems should also reflect the value of assessment and its potential to improve learning.

Third, assessment can and should help individual students to learn better through clarity about outcomes, high-quality feedback, and the like. Assessment activities should always be educational experiences for students, not a waste of time.

But arguably the most powerful results come at the program level. When faculty can define what learning the major seeks to achieve in its graduates, it can lead to a stronger program in multiple ways: students understand better where they’re going and how the pieces fit together; faculty can gain greater clarity about what their individual courses need to contribute to the major; the program of study becomes more coherent; existing resources can be deployed more effectively; and a more persuasive case for new resources can be made — one based not on assumptions or traditional arguments (like a smaller class size is always better) but on evidence.

Fourth, testing and assigning grades bears some resemblance to the assessment process (see point one above) but it is not identical to it. Testing and grading are highly individual activities for both the faculty member and the student. Grading is also evaluative; it is most often experienced as punitive, whereas assessment (for faculty and students) is a dispassionate look at what the the quality of the learning is and a non-blaming analysis of what to do about it.

Tests may or may not be devised in relation to shared departmental goals. The results are most likely not discussed by the entire faculty, and while instructors may get ideas from a test for how course content or pedagogy might be modified to get better results, there is no built-in expectation, no support system, to ensure that there will be follow-through.

Assessment is a tool for learning: for students, for faculty and programs that can learn how to do better by their students and their disciplines; and for institutions that want to plan and budget for better outcomes. Assessment is an inquiry process: it’s about asking questions and looking for answers, not falling back on traditional assumptions. Aren’t these supposed to be two of the academy’s core values?

• Clarifications Are Redundant by Definition, Barbara Wright.
• Posted by DFS on December 14, 2009 at 10:15pm EST
• I’ll just put this all in the context of what I do — teach math.

“First” is done, using results. Every semester. In fact, everyday. No ‘collectivism’ is necessary; ‘finally’ “DOING” means daily modification! Duh.

Second, administrators cannot, nor should, argue with definitive results, known as Grades. Assuming they have not had us fired for teaching that 2 + 2 = 4.

Third, assessments tell students what they do not know and what they do have to learn. There is no high-quality feedback involved here. Educational experiences must include (1) tidying up what you don’t have a clue about, even though your SAT says you’re a genius, and (2) don’t waste our time in class because your high school said you always should ‘earn’ A’s.

Fourth, and most arguably inarguable, faculty know what their students’ objectives must be, not ’should be.’ There are no “assumptions” or “traditional arguments” about how individual student learning is to proceed — they either perform, or not. Again, Duh.

“Shared departmental goals” — what a hoot! The students must know the material to go on. We don’t want to inflict an ‘idiot’ on our colleagues. We constantly and continually share tests and goals. It is actually done, when results actually matter from course to course, in my field. Again, Duh.

I stand again in defense of Math Prof. We have assessed all along the way, so why don’t the rest of you newly ‘degreed’ interlopers get out of our way?

© Copyright 2009 Inside Higher Ed

Assessment of Student Learning Outcomes

Resource Materials

Overview Materials

A Nice Explanation of Outcomes Assessment as Related to All Assessment:

http://outcomes.lbcc.edu/pdf/InstructProgOutcomes_overview.pdf

Links to Statewide Student Learning Outcomes:

http://valenciacc.edu/slo/

Links to Many Outcomes Assessment Sites, Not Always Applicable:

http://www2.acs.ncsu.edu/UPA/assmt/resource.htm#inst

Select Specific College Sites that May Be Useful

Miami-Dade College:

http://www.mdc.edu/learningoutcomes/default.aspx

Northwest Florida State College:       

http://www.nwfstatecollege.edu/IE/2009/2008-2009-program-review-summary.pdf

Valencia:

http://www.valenciacc.edu/learningcouncil/GenEdOutcomes.asp

Florida Community College Jacksonville:

http://www.fccj.org/campuses/mccs/instruction/liberal_arts/models_assessment.html

http://www.fccj.edu/campuses/mccs/instruction/programdev/assessment/classassess.html

As we long suspected:

“At the most celebrated institutions of higher education in the United States, the teaching quality of the adjuncts is many times better than that of those on the tenure tack.”

Inside Higher Ed didn’t pull any punches in their review of Off-Track Profs: Nontenured Teachers in Higher Education from the MIT Press. The book, just out, reports results on a study funded by the Mellon Foundation that looked at adjuncts teaching at 10 leading research institutions. The authors, university administrators themselves, had seemingly total access to data and personnel.

As important as the finding about ‘quality teaching’ (more on this in a moment), is the study’s analysis of the drivers behind the growth of adjuncts. It isn’t just the cynical need to save money. In fact, the decision to place a course with an adjunct results from many factors. This makes sense, since few university administrators have any sort of cost-cutting philosophy to their leadership. If they did, many aspects of university departments would change before an increased hiring in adjuncts.

As usual, the definition of ‘better teacher’ is based entirely on course evaluations completed by students. Which means the results are worthless. Anti-adjunct (or anti-data) partisans will reject the findings out of hand. And they’d be correct to do so, although the conclusion feels right to us. It would have been interesting to see the authors of the study bounce their data against RateMyProfessors.com data…but that’s another story.

The whole illogical mess is another reflection on the emotional and cultural decision making that drove the financial melt-down. Is the goal to educate students? Or is the goal to bring in research $ and publish obscure texts (the article casually mentions that the course load for “many tenured professors has fallen from four to three a year.” THREE COURSES A YEAR?

Is the goal to sell as many mortgages as possible? Or to make sensible loans that will actually be paid back?

The whole dynamic runs very close to a terrific new piece from Atul Gawande in The New Yorker this week. We’ve blogged about Gawande before; he is taking on the 30,000 foot issues in medicine with an eye for detail and counter intuitive conclusions that are obvious once pointed out. We feel similar work should be (and could be) done in education.

Gawande visits the US county with the highest Medicare costs in the country and asks, “what is different about this place”? The answers are obvious and challenging at the same time. The opening shot, devastating to anyone who isn’t in medicine, is the immediate realization that no one in leadership positions at the hospitals and health care providers he visits knows how much money is being consumed in their facilities. That is, lowering cost is not on their radar screen. The equivalent would be UPS not bothering to know how many packages were moving through its system on a daily basis. The health care executives don’t know because they don’t care; total cost – at the individual patient level – isn’t a concern.

There are some bright spots – health care systems like the Mayo Clinic where physicians are paid salaries, rather than “incentivized to provide services.” These systems spend less money on people – and the result is a healthier population. There is a lot of data behind this conclusion, but basically the more you visit hospitals and doctors, overall, the sicker you get. If your doctor orders three follow up tests, a ‘minor’ preventative surgery, and puts you on drug your outcome is probably going to be poorer than finding more creative ways to deal with the situation.

Unlike healthcare, educators (at the world’s leading institutions!) are basing decisions on course evaluations. Is this identical to deciding on the effecitveness of a hospital by asking a the (surviving) patients how they feel? Just about. The information is useful, but the data cannot be used to drive decision making.

And as in healthcare, the stakes are the same: education costs America huge sums of money, much of it to ‘incentivize’ pseudo-academics to invent ever more elaborate systems of language and criticism that keeps their obscure world afloat. They are no different than the doctors in Texas that enjoy a particularly entrepreneurial culture, and charge the shit out of the Medicare system to spin up their own revenues.

Unlike healthcare, America as a whole does not need to come to some grand conclusion and force change. Education (especially higher education) is an increasingly market-driven business, and efficient providers are wading in…within 10 years the luster of a non-profit, ‘prestigous’ degree while have completely worn away (a generation of graduates with huge bills and no jobs will accelerate this process even more) and 18 year olds will become smart consumers. Universities, as we know them now, are the equivalent of the ice harvesting companies of the early 20th century; they are worried about course evaluations like the ice harvesters designed more effective insulation to move ice long distances. When refrigeration became possible in the home, new companies took over the entire ‘ice’ market.

I just spent two days at a WASC-sponsored conference on “Teaching and Assessing the English Major.”  WASC is our institution’s accrediting agency, the Western Association of Schools and Colleges, so we have to listen to what they say.  A WASC review is a six-year process culminating in an evaluation of “educational effectiveness” which largely derives from an analysis of the program assessment measures in place throughout the campus.  This conference was the first discipline-specific assessment workshop offered by WASC.  Clearly they think there is something lacking in the enthusiasm and expertise of English departments for assessment.

We were supposed to send a team with a specific project to work on.  However, for the most part the presenters presented to us.  When they gave us time to talk at our tables, it was usually for only five minutes, which meant that one person got half way though presenting an issue, and then we were interrupted.

I came to the conference with more enthusiasm for assessment than when I left.  Originally, I saw program assessment as a simple, commonsensical endeavor based on four questions:

• What are we trying to do?
• How are we doing it?
• How do we know that we are doing it?
• How can we improve?

The first question is answered with a series of outcome statements about what the department wants its students to be and do when they complete the program.  The second is about pedagogy and curriculum.  The third is about data gathering and analysis, and the fourth is about applying what was discovered in that analysis to address any gaps or problems that were revealed.   Faculty are generally very focused on the content and effectiveness of the courses that they teach.  They don’t often think about the cumulative effect of the entire program on the student.  An assessment plan that inspires regular conversations about the design of the whole program  and its results will improve the coherence of a department markedly.  It just makes sense.

The WASC representative at the conference stated that WASC is primarily interested in program assessment, not course assessment, and not assessment of individual instructors.  However, much of the material presented at this conference was about course assessment and student self-assessment.  One of the handouts was a chart that every instructor at Alverno College is required to fill out for every course, listing the mission statement of the college, the major outcomes, the course outcomes, and the plan for assessing those outcomes in the course.  This goes far beyond setting some goals, gathering some data, and discussing it at a yearly retreat.  This is putting the plan and the outcomes in the professor’s face on an almost daily basis.

I don’t know for a fact, but they probably put these outcomes on every syllabus too.  Words that are simply duplicated everywhere become invisible.  As scholars of language, we should know that.

I think it is important to define outcomes and have conversations at the program level, where such conversations do not usually occur, and leave instructors the freedom to teach as they will.  If program outcomes become too rigid and ubiquitous, they will become the material of an elegant cage.  Higher education used to be a refuge for brilliant eccentrics.  If we drive them all away we will have an institution of competent drones.

I can imagine a future in which universities create a non-assessment college in order to attract the best students and faculty.

Our department is fairly well along in developing an assessment plan.  We have nine outcomes, with rubrics for four of them, and a capstone course that helps students develop a portfolio of their work over their career in the program.  They collect papers from previous courses, write a new critical paper, and write a reflective essay about how they have grown in the program.  We also do an exit interview.  We have been assessing two outcomes a year, and developing one new rubric per year.  This year the new one is “Research Skills.”

I do think that English departments must change to survive.  People forget that English departments are only about 100 years old.  What can appear, can disappear. The traditional program covers literature, as they say, from Beowulf to Virginia Woolf,  divided into historical periods, sometimes with genre and major author courses.   Faculty specialize in specific periods and authors.  That coverage model is severely threatened now because departments are so understaffed.  Professors are routinely asked to work up courses outside of their specialties.  The coverage is thinner, and the quality less.

However, is coverage of traditional periods, genres, and canonical authors really what English majors need to have?  Most departments also include practical disciplines such as linguistics, and rhetoric and composition.  We train language teachers and writing teachers.  Faculty in all disciplines agree that students should speak and write English well.  In fact, they think that is what English departments do, the primary purpose.  Perhaps they are right.

Our nine current outcomes are heavily weighted toward coverage of period and genre knowledge.  However, when we designed the capstone course, we realized that there would be gaps in coverage, so we focused on interpretive strategies.  The current instructor for the capstone assigns works by Italo Calvino which the students have never read before, and asks them to use what they have learned from the program to interpret these texts and write about them.  I think we are on to something here.  We are assessing the intellectual tool kit that our students have acquired from the program, and their ability to use these tools to analyze new texts.

Focusing on interpretive strategies instead of coverage is necessary in the reduced circumstances we find ourselves in today.  However, I think that these skills are also more marketable for students.  Necessity may have pushed us in a productive direction.  In order to complete this transition, I think we need to revisit our outcomes statements to reflect this new emphasis.

But let’s leave some room for eccentricity and brilliance and some holes in the elegant cage.  We are, after all, the English Department.

A couple of recent articles in Inside Higher Ed caught our eye – one on grades and grade inflation, and the other on the creation of the National Institute for Learning Outcomes Assessment.

It seems obvious to us that grading and assessment are largely the same thing. Barring sampling programs, or initiatives designed to assess program outcomes (aggregating student results rather than considering the success of individuals), grading IS assessment.

It’s just that the typical grade (A-, B+ etc.) is an extraordinarily blunt instrument.

Imagine reading a car review (okay, bad example – who is reading car reviews anymore?) or a film review that is simply a letter grade. Many reviews feature letter grades, but they come after a thousand words of measured criticism. And it is subjective criticism, but we largely accept the skill of a Roger Ebert and take their points seriously. They are assessing the film, and they do it through a narrative response built upon well-established criteria.

Education is even messier than film reviewing, because the letter grades awarded are all over the place. To draw the analogy out a little farther, imagine trying to pick a movie to see from the following:

• 20 films, all rated B+ or higher (with no narrative or other information)
• 20 films, each rated four times by separate reviewers, where the individual grades are all over the map but the averages are still B+ or higher

You wouldn’t know which film to see…and likewise our system of letter grades is useless for assessing knowledge.

Think back to your own college days. What did grades really say about the person? How well they had learned to play the game? How hard they worked? How much talent and mental horsepower they had? Probably all of these. And for this reason grades are a somewhat useful gauge for companies looking to hire college graduates. But they are no measure of learning, even with the amount of effort that goes into creating them.

One of my earliest engineering jobs was an internship at Unisys. I went to lunch with a group of managers who were discussing the internship program. An HR manager brought up the idea of raising the minimum GPA required to a 3.5/4.0 scale. The managers all looked at each other, and to their great credit laughed, and then (to a person) said that if those standards were applied they wouldn’t have earned their initial job at Unisys.

This may be a commentary on Unisys (which didn’t invent the iPod, eBay, Facebook, Google, or the Blackberry), but I don’t think so. It’s the understanding in industry that there is more to learning than a GPA.

Keith Hampton, manager of the LinkedIn Higher Education Management Group, was kind enough to give me the opportunity to spout off on a few topics of interest:

• the state of the outcomes assessment technology field;
• the emergence of customized learning and what’s holding it back;
• the opportunities for data sharing across P-20 and what opportunities we’re missing.

Here’s the link...

Introduction

Peer review is a widely accepted practice, particularly in writing classes, from high school through college and graduate school. The goal of peer review is typically two-fold:

1. To help students get valuable feedback at the draft stage of their work.
2. To help students more deeply understand the goals of the assignment.

Unfortunately, peer review is often used as a busy-work activity, or a process that takes advantage of conscientious students while allowing others to do superficial work. For instance, many teachers will hand out a list of peer review questions in class, and then give students 30 minutes to review two papers written by their colleagues. An open-ended question might be:

• “Did the writer adequately summarize and discuss the topic? Explain.”

Many students will write “Yes” under this question and move on. Without review by the instructor (difficult to do when many instructors have 50 to 150 students), these students can destroy the social contract of a peer review. Other students will spend a lot of time making line edits to the draft – correcting grammar, making minor changes to sentences etc. At the draft stage this is probably inappropriate – the focus should be on ideas and big-picture organization, not embroidery. Plus, some students aren’t qualified to be dictating where the semicolon should go.

Students aren’t alone in having these problems; in 1982, Nancy Sommers published her highly influential piece, “Responding to Student Writing,” in which she commented about how little teachers understand the value of their commenting practices, and that, essentially, they don’t know what their comments do. She raised numerous long-standing points in her evaluation of teachers’ first and second draft comments on papers.

Two of her major findings:

1. Teachers provide paradoxical comments that lead students to focus more on “what teachers commanded them to do than on what they are trying to say” (151).
2. She found “most teachers’ comments are not text-specific and could be interchanged, rubber-stamped, from text to text” (152). One result is that revising, for students, becomes a “guessing game” (153). Sommers concluded by saying, “The challenge we face as teachers is to develop comments which will provide an inherent reason for students to revise” (156).

Most teachers have experienced this last point when a student asks, “what do you want.” The student doesn’t understand the larger goal of the assignment and has learned that achievement comes through figuring out the personal foibles of their current teacher.

These outcomes are unfortunate, because peer review (and written feedback from teachers) can be one of the most powerful learning experiences for students.

From our perspective, peer review should:

1. Help the students improve their work through the drafting process.
2. Deepen the understanding of the assignment and its goals for both authors and reviewers (and teachers!).
3. Allow instructors to assign more authentic work without requiring they read and grade piles of papers – so they do more coaching than grading.
4. Give students opportunities to create authentic work – that is, peer reviews written (or spoken) to a real audience: the author. Peer review, in the words of Drexel University’s Dr. Scott Warnock, “can be some of the most important writing students ever do – because they have a real audience for their work.”

Peer Review and the LMS

Before considering pedagogy, the first challenge to peer review is largely one of workflow. How to effectively set up teams of reviewers? How to manage student work? How to enforce deadlines and reward students who are conscientious?

Learning Management Systems like Blackboard, Blackboard Vista/CE6, and Moodle can streamline the peer review process.

Through much trial and error and philosophizing, along with combining best practices from many educators, from middle-school through graduate school, we propose a process to minimize the load on the teacher and maximize the benefits of peer review. This process leverages the best of the Learning Management Systems to automate much of the process, leaving the teacher to focus on value-added activities far more useful than busywork.

PEER REVIEW PROCESS

1)    DEVELOP THE ASSIGNMENT

Students need to do something. Without getting into curriculum design debates, we will assume that ‘something’ is fairly clearly defined. Let’s also assume that there are defined deadlines for work. Here’s a suggested timeline for a typical 4 to 5 page paper:
Drafts: Tuesday, 9/22  by 11:59pm
Peer reviews: Friday, 9/26 by 11:59pm
Final Drafts: Tuesday, 9/29 by 11:59pm

One of the immediately valuable aspects to peer review is its ability to combat procrastination. Too often we assign projects, give a deadline four or five weeks out, and expect students to perform. No manager would assign a major project to a new employee and then check back four weeks later (at least no effective manager). They would require status reports, check in informally from time to time etc. So by requiring some sort of draft a week before the final deadline, teachers can help students get started.

To help gain students’ attention, we highly recommend that the quality of peer reviews be worth 20% or more of the final Assignment grade (based largely on effort, not skill, since this can be difficult work for students).

We also recommend that students be required to include a cover letter (in the same document), addressed to their reviewers, that details their goals with the assignment and specifies the top two or three areas in which they would like feedback. This helps students reflect on their work and also give the peer reviewers a ‘heads up’ concerning specific issues.

2)    DESIGN A RUBRIC OR SET OF CRITERIA

This is the stage where things can begin to get challenging for teachers new to the idea. Too often the criteria are in our heads, or we don’t share them with students until well after the students have begun working on their projects. Luckily, the 80/20 rule applies to student work: 80% of the projects we assign, from middle school through graduate school, are covered by 20% of the potential criteria we might think of. So there are lots of models to borrow from.
The criterion listed below helps students decide how original an argument might be (useful from high school through college), breaks the issue down and uses language students can probably understand.

It could be easily modified for a particular audience – made either more sophisticated or simplified to highlight the differences between performance levels. You can find the entire rubric in Appendix A.

This rubric might be exactly the same for students and teachers, or teachers might use a slightly different version (with more direct language). The rubric should be posted along with the Assignment. Ideally, students would help design and refine the rubric.

It is important to use open-ended questions when designing peer reviews; this example gives just the declarative ‘observations’ that help the student distinguish between levels of performance. To see how open ended questions can help structure the review process, see Appendix C: Maximizing Peer Review with Waypoint Outcomes.

1. (Excellent) Wow, this is a highly insightful argument – you go far beyond our classroom discussion and readings.
2. (Good) Okay, you`ve definitely introduced an interesting (not predictable) argument..but it doesn`t seem thought through enough or lacks necessary evidence.
3. (Fair) This is bordering on a summary of information rather than an argument.
4. (Poor) There is hardly any insight here – it`s mostly summary of various points of view or conventional wisdom (that is, predictable).
5. (Unacceptable) ZERO argument. The author never presents fresh ideas or interpretation.

There are a number of pedagogical choices implied in the above sample criterion: whether to include specific performance descriptors (Good etc.), how sophisticated or informal to make the language etc. Each educator should make these decisions to suit their needs.

This rubric or criteria set should be made available to students in advance, to guide the development of their draft and to structure their response to the student.

3a)    COLLECT THE DRAFTS – VIA DISCUSSION

Many instructors will try to assign specific students to specific peer review groups. Certainly, if a teacher knows their students very well and can specify teams, then all the better.

However, creating and managing groups manually can create a pile of work for the instructor. What happens when a student emails to say they are sick, and they can’t post a draft? Does the instructor need to edit the peer review groups and move students around?

The trick is to reward the students who complete their work on time and not reward (notice the term isn’t necessarily punish) those who do not. Whether collected in a face-to-face class, or enforced with deadlines in the LMS, a certain percentage of students will not submit a draft on time, endangering the entire process or requiring instructor intervention to rebalance the work. In our experience, assigning exact ‘peer review partners’ is inefficient and unnecessary to achieve the stated goals of a peer review process.

Whether randomly created or designed, the Groups feature of most CMSs can help structure the who-reviews-whom question.
There are benefits to different strategies, but here’s an approach using Groups and Discussion Boards. Let’s assume that we want students to peer review two of their fellow students’ papers:

1. Create Groups and place six students in each (so a class/section with 24 students would have four Groups). This way, every student is guaranteed to get two reviews, even if one or two of their fellow students neglects to post a draft or neglects their work.
2. Create a Discussion Topic for each Group – so only the students in the Group have access to the discussion. Make sure the Discussion Topic will be unavailable after the deadline.
3. Require students to post an attachment and the plain text of their draft. This way, the instructor can quickly review drafts without opening attachments, while students can open fully formatted versions.
4. Students review the two drafts posted before (chronologically) their own. This way the process isn’t destroyed by students who do not post on time, and again we reward those who adhere to deadlines. Students who post first review the students who post last.
5. Put some teeth into the process (see Appendix B for sample instructions intended for students). We don’t recommend punishment, but rather the requirement that students actively do something – visit the writing center or similar.

Students must post a draft by the deadline to receive peer reviews. If they do not post their own work by the deadline, they still must review two other students’ work.

Obviously the particular solution will need to suit the teacher’s sensibilities, but the idea is to give the students who might miss the deadline more work to do. If they know this in advance, they are more likely to seek the easiest path – following instructions.

Peer reviewers can simply post a response to the author’s original discussion post, using the rubric (criteria) as a guide.

3b)    COLLECT THE DRAFTS – VIA ASSIGNMENT

Discussion Topics can be convenient for grouping students and segmenting the reviews; most LMS platforms will tell the instructor how many unread messages are posted to a. So in our example, there should be 6 posts the morning after the deadline. But often students will post twice (they forget to attach, post a newer version etc.). If an instructor has only one class, it may be possible to click into each discussion topic and see who has posted work. But this quickly becomes busy-work with multiple sections.

An alternative is to use the Assignment drop-box, which in most LMS platforms more clearly shows late submissions (or no submissions). The drawback is losing the small group approach, because students will be able to see ALL submissions. Again depending on instructor sensibility and philosophy, using the Assignment drop-box can be a much more efficient approach. Groups of students need to be documented along with the assignment specifications, so that students can search all submissions for the students they are assigned to review.

If the Assignment feature is used, it may be possible to allow students to type or upload comments, depending on the LMS. Or a separate Discussion topic can be setup for responses. The same guiding principle, of grouping students into sub-groups and requiring they review submissions made chronologically just before theirs can elegantly handle the inevitability of non-compliance with instructions.

4)    MANAGE THE PEER REVIEWS

A similar problem awaits the teacher trying to manage and review the comments students write for one another. One solution is simply to wait for the submission of final drafts, and direct students to include the peer reviews they received. Again, inevitably, some students will forget to attach their peer reviews, defeating the instructor who wants to evaluate the quality of peer reviews.

A much less desirable solution is for the teacher to click through potentially hundreds of pages in the LMS reading posts or comments.
Workarounds can be used: students might need to post their reviews to a separate Discussion Topic, so the instructor can more easily review compliance and quality.

5)    COLLECT FINAL DRAFTS

Increasingly instructors are collecting work electronically – for environmental reasons, convenience (no lugging piles of papers around), and to grade electronically. Whatever the process for collecting work, we suggest students be required to submit:

1. A cover letter, addressed to the instructor that:
   1. details their goals for the assignment
   2. discusses their progress from first draft to final
   3. reflects on feedback they have received earlier in their academic careers
   4. responds to the comments they received from the peer review process
2. The final draft, complete with a works cited page if required
3. First draft
4. Peer review(s)

The cover letter requirement helps students learn different rhetorical strategies (switching from a formal academic voice to a more business-like letter-writing voice), and can be a terrific leading indicator for instructors when it comes time to grade. Hastily written cover letters are often indicative of hastily written drafts. Good cover letters can help students maximize credit for the assignment because the instructor may better understand the subtleties of the work

MAXIMIZING PEER REVIEW

All suggestions to this point have assumed a generic Learning Management System with the typical features of Assignment Drop-boxes and Discussion Topics.
Some LMSs have simple evaluation tools, like ‘grade forms’ in Blackboard Vista/CE6, the Outcomes module in Moodle 1.9+, and the peer review feature in Blackboard 8.0+. These features have varying degrees of utility, and none of them are grounded in the latest composition research – or even the common sense to assume that there is more to evaluating complex work than simple Likert Scales.

Waypoint Outcomes, developed by Subjective Metrics and used by over 40 institutions in the US, Canada, and Europe, has focused on peer review from its earliest days. Whatever the approach taken by an instructor, Waypoint can structure students’ response, help the instructor coach rather than click, and vastly improve the peer review experience for all parties.

Waypoint Outcomes is a tool for building, sharing, and using sophisticated rubrics to create exceptional feedback. Waypoint is tightly integrated with Blackboard, Blackboard Vista/CE6, and Moodle.

Using Waypoint, and instructor develops a set of criteria, creates a peer review Project to grant access to the rubric to students, then manages the peer review process. Students access Waypoint via the LMS, click their way through a detailed rubric, annotating and explaining their choices as they go, then save their work and email the evaluation to the author of the work.

For more on Waypoint, visit http://www.gowaypoint.com

BIBLIOGRAPHY

Sommers, Nancy (May 1982). “Responding to Student Writing.” College Composition and Communication. 33.2: 148-156.

Conan, Neil. “Procrastination Nation.” NPR: Talk of the Nation 12/15/2005 13 Jun 2008 <http://www.npr.org/templates/story/story.php?storyId=5053416>.

Like a lot of bloggers and teachers interested in technology and education, I’m a geek. It makes perfect sense to me to do just about everything I can with technology. I’m very fast with a computer, and while my physical desk might be a pile of (seemingly) disorganized papers, my computer is always immaculately organized.

Okay, my inbox gets a bit crazed, but that’s why Xobni invented Xobni.

So I’ve been collecting student work in electronic format for a long time – mostly because it seems so inelegant to walk around with 80 student papers (and a pain when commuting on my bike) when I could have each students’ work stored neatly in a folder on my hard drive. I always have access to my comments and student work, can collect the work in the first place via Blackboard, and don’t have to worry about misplacing student work etc. etc.

I remember the moment that I realized teachers might have another reason for collecting work electronically. I was talking to a technology guru at Carleton College, and asking about workflow and teacher adoption of their elearning platform, Moodle. I have this conversation a lot, and it usually highlights one of the the dirty secrets in higher education: low utilization rate for elearning tools. There are budgets, entire staffs dedicated to technology, but few faculty use tools like Moodle or Blackboard to any great extent. Maybe they post a few files, an announcement or two… I have visited (prestigious) schools where full-time “instructional technologists” will actually scan a professor’s hardcopy syllabus and place the PDF into the appropriate Blackboard course. And that single document is the only resource in the course. For an entire term.

Of course there are the power users – geeks like me, and large introductory classes that make use of online testing (and automatic grading).

But the technology guru at Carleton surprised me. She told me that most faculty collect work electronically and review it on their computers. I was surprised, and asked why. She told me that the faculty had changed their workflow for environmental reasons. They changed their ways to save paper.

I was not surprised, then, to discover that Carleton College was rated in the top ten of over 300 colleges and universities ranked in the 2009 College Sustainability Report Card, published by the nonprofit Sustainable Endowment Institute. The report card covers many areas of an institution’s operations, and the endowment (what the school invests in etc.) and commitment to sustainability (evidenced by an ‘office of sustainability’ and a full-time person directing said office) are major components.

A teacher’s actions, however, are far more visible to students than complex and long-term investments. What struck me the most about the news coverage of the report card (the media loves a ranked list!) had to do with high school students’ attitudes: “Sixty-three percent of 10,300 college applicants recently polled by the Princeton Review said that a college’s commitment to the environment could affect their decision.” Since the report card  didn’t touch on elearning or academic technology, I take the Carleton faculty’s commitment to be a cultural expression of the school’s larger commitment and an indicator that they earned their ‘A-.’

The folks at Waypoint Outcomes just posted a white paper that discusses best practices in designing peer review, particularly in blended (hybrid) or purely online teaching modalities.

The approach described is LMS-agnostic, and only mentions Waypoint via Appendices to highlight how Waypoint improves upon many of the issues/opportunities highlighted.

>> Download the PDF version of the white paper

We are very interested in feedback – what strategies have you found to be effective?

Read an excerpt:

Introduction

Peer review is a widely accepted practice, particularly in writing classes, from high school through college and graduate school. The goal of peer review is typically two-fold:

1)      To help students get valuable feedback at the draft stage of their work.

2)      To help students more deeply understand the goals of the assignment.

Unfortunately, peer review is often used as a busy-work activity, or a process that takes advantage of conscientious students while allowing others to do superficial work. For instance, many teachers will hand out a list of peer review questions in class, and then give students 30 minutes to review two papers written by their colleagues. An open-ended question might be:

• “Did the writer adequately summarize and discuss the topic? Explain.”

Many students will write “Yes” under this question and move on. Without review by the instructor (difficult to do when many instructors have 50 to 150 students), these students can destroy the social contract of a peer review. Other students will spend a lot of time making line edits to the draft – correcting grammar, making minor changes to sentences etc. At the draft stage this is probably inappropriate – the focus should be on ideas and big-picture organization, not embroidery. Plus, some students aren’t qualified to be dictating where the semicolon should go.

“Peer review can be some of the most important writing students ever do – because they have a real audience for their work.”

Dr. Scott Warnock
Drexel University

Students aren’t alone in having these problems; in 1982, Nancy Sommers published her highly influential piece, “Responding to Student Writing,” in which she commented about how little teachers understand the value of their commenting practices, and that, essentially, they don’t know what their comments do. She raised numerous long-standing points in her evaluation of teachers’ first and second draft comments on papers. Two of her major findings:

1)      Teachers provide paradoxical comments that lead students to focus more on “what teachers commanded them to do than on what they are trying to say” (151).

2)      She found “most teachers’ comments are not text-specific and could be interchanged, rubber-stamped, from text to text” (152). One result is that revising, for students, becomes a “guessing game” (153). Sommers concluded by saying, “The challenge we face as teachers is to develop comments which will provide an inherent reason for students to revise” (156).

Most teachers have experienced this last point when a student asks, “what do you want.” The student doesn’t understand the larger goal of the assignment and has learned that achievement comes through figuring out the personal foibles of their current teacher.

These outcomes are unfortunate, because peer review (and written feedback from teachers) can be one of the most powerful learning experiences for students.

From our perspective, peer review should:

1)      Help the students improve their work through the drafting process.

2)      Deepen the understanding of the assignment and its goals for both authors and reviewers (and teachers!).

3)      Allow instructors to assign more authentic work without requiring they read and grade piles of papers – so they do more coaching than grading.

4)      Give students opportunities to create authentic work – that is, peer reviews written (or spoken) to a real audience: the author. Peer review, in the words of Drexel University’s Dr. Scott Warnock, “can be some of the most important writing students ever do – because they have a real audience for their work.”

Read the rest in the PDF

]]> http://authenticassessment.wordpress.com/2008/07/10/assessing-critical-thinking/ Fri, 11 Jul 2008 02:02:19 +0000 11trees http://authenticassessment.wordpress.com/2008/07/10/assessing-critical-thinking/

Many of our users know about the Waypoint Public Library – a shared library of both Assignments and Elements created by our clients. Each month we’ll highlight a unique approach to assessment and feedback and make it easy for you to copy and utilize it.

As a first installment we thought we’d start with a double-shot of critical thinking, a crucial skill difficult to assess and of interest to educators from middle school through graduate school. These Assignments don’t formally address “critical thinking” as a skill, but seek to differentiate summarizing facts from making original connections while synthesizing information.

There are two versions of this Waypoint Assignment: one intended for peer review (pdf), and the other for an instructor (pdf) to use. Specific references (to writing handbooks etc.) have been removed. You can easily copy these Assignments from the Public Library and edit them to suit. The two Assignments are:

• Critical Thinking/Writing (SubMet Library) 7/10/08
• Critical Thinking/Writing Peer Review (SubMet Library) 5/30/08

They both make use of Checklists, but you’ll notice that the first few Observations in the instructor versions have traditional ‘rubric’ choices. So the detailed Observations could be easily dropped and the Checklist Element converted to a Performance Element.

>> Read more about copying an Assignment from the Public LIbrary
>> See the detailed version of the instructor Assignment
>> See the detailed version of the peer review Assignment

I had the good fortune to be invited to attend Dr. Ken Bain’s Best Teachers Summer Institute this week.

The conference, now in its twelth (or so) year, brings teachers from around the world together to share best practices based on Dr. Bain’s work. He published an award winning book in 2004, called What the Best College Teachers Do that draws universal lessons from the work of master teachers.

Since first meeting Dr. Bain at a conference last February, I have adjusted my own teaching with impressive results. Much of what you will find in his book feels like common sense, but you feel like you are saving ten or twenty years of trial and error on your own part, and standing on the shoulders of the great teachers working around you (who may not be readily available as mentors in your own institution!).

A brief example I took from this week’s conference: assign readings based on questions that come up in class, rather than assigning readings to generate class discussion.

I increasingly use readings available through my university’s online databases rather than anthologies, and I frequently use very contemporary readings. For instance, I used five articles from the New Yorker magazine as part of my recent persuasive writing course in-class essay final. We had read Zadie Smith’s fantastic novel White Teeth, and I asked students to pick one of the following articles and then make a connection between the article and the novel. Since we had been working on these kinds of abstract connections all term, that was the only direction I gave. The articles:

o Our Own Devices: Does technology drive history?

· From the May 12 New Yorker by Jill Lepore

· Explores the impact of individual inventors (Edison etc.)

o Pixel Perfect: Pascal Dangin’s virtual reality?

· From the May 12 New Yorker by Lauren Collins

· Discusses the guy who retouches fashion photographs

o The Last Bite: Is the world’s food system collapsing?

· From the May 19 New Yorker by Bee Wilson

· Discusses market forces changing the role of food in our society

o After Empire: Chinua Achebe and the great African novel

· From the May 26 New Yorker by Ruth Franklin

· Discusses Achebe’s career against the backdrop of colonialism and post-colonialism in Africa. (You don’t need to know his writing to really enjoy this).

o Hungry Minds: Tales from a Chelsea Soup Kitchen (not available online)

· From the May 26 New Yorker by Ian Frazier

· Uplifting history of a soup kitchen in NYC

So students were able to enjoy some choice in what they read and I was guaranteed not to have to read the same essay 85 times, especially since the novel is so multifaceted. Between picking various scenes/characters from the novel, an article from the above list, and a particular issue from those long articles, I receive a wonderfully diverse set of essays.

Some of the connections were truly impressive. Here’s a long introductory paragraph linking the immigrant experience of Samad Iqbal, one of the novel’s central characters, with (of all things) the worldwide food crisis:

My parents came to America as immigrants. They were surprised at first to see the vast difference in cultures between American and India, but they were able to realize that change is imminent. They have to get used to things around here. In White Teeth, Samad and Alsana are immigrants to England. The difference between them and my parents is that they are reluctant to change. They do not let people help them or change their ideals. They also end up breaking their family because of their stubbornness. Similarly, in a New Yorker article, the same stubbornness can result in a major food crisis in the coming decade. Unless people decide to change their ways we may end up in a famine.

Another, linking Irie’s body-image issues in the novel to the “Pixel Perfect” article about photo-retouching.

What is the social beauty norm? Since 1857, artists have been retouching “those portraits for which the camera may be said to have laid out the foundation.” This allows us to assume that vanity over one’s appearance – to the point of altering a picture – is not a recent occurrence.

Irie Jones, a character in White Teeth by Zadie Smith, exemplifies a typical self-conscious, gullible teenager. She dislikes the shape of her curvy body and her wild, curly hair. Chapter 11 begins with her looking ata sign that says, “Lose Weight to Earn Money.” Knowing that losing her curves would take a long time, she decides to instead take matters into her own hands and change another facet of her appearance – her hair.

(then later in the essay – this one takes a non-5 paragraph approach)

The reason Millat doesn’t sleep with Irie until close to the end of the novel is because he is searching for flaws in the pretty girls. Altered pictures, such as the ones by Dangin, motivate us to find fault in perfect forms. When we see regular people on the streets, we see their “blips,” for instance, the boy with the slightly crooked nose, or the girl with frizzy hair – and we move on. Yet when we see a perfectly modified picture, our human curiousity drives us to search for her flaws, which may take a while to find, and we keep looking.

Millat has this natural curiousity; he spends time with pretty girls to find their “blips.” Irie, to him, is a girl he knows from childhood, and he already knows all of her imperfections, inside and out. He has a drive to see what else is out there, even if truly deep down he knows he likes Irie and thinks she is beautiful with all her curves and curls.”

These students (both essays developed the ideas much further – but to me they had a clearly original idea from the beginning) connected deeply with both the issues/themes in the novel and an idea from the article they chose. So this was a successful use of assigned readings; but I still predicted what they might want to read rather than letting our class discussion take us (in an investigative manner) to the readings.

There was a moment at the conference when the idea of outcomes assessment, and the gulf between what we actually DO as teachers and our real goals for our students, became very clear.

A journalism professor commented on the idea of giving feedback, and allowing students to experiment, before any formal grading. He did a lot of drafts with students, and while many students put considerable effort into these early versions, a number of students only became motivated when there was a real deadeline. Another set of students had trouble with the idea of a deadline – they seemed content to hand work in late, which as the professor said, could cost hundreds of thousands of dollars in the real world of journalism. “Meeting deadlines is an outcome,” he said.

Sometimes it is only when we try to explain a complex idea to others that we truly understand it ourselves.

Deadlines are important in many more disciplines than journalism. I taught in an interdisciplinary engineering program for nine years. Students wrote proposals, and we talked to them about industry, and how even the slightest deviation from the expected format/content (never mind being late) could result in a proposal being discarded unread.

But did we assess this crucial skill (or value)? Only in the sense of deducting points from late submissions. And even then the system was incoherent; some professors deducted points for late submissions, others didn’t.

I only had a brief chance to talk to the journalism professor. I wasn’t able to find out if his program articulated this crucial outcome in their academic goals, or if they stayed with higher-level abstractions. He had experimented with many approaches, none of them satisfactory. He felt the issue of deadlines and the quality of revision defied the rubrics approach, and I know what he means. If “timeliness” is a criteria on a rubric, and you get a Pulitizer-winning article two days late, how can the document even qualify for reading and grading?

This issue leads to the more general one of procrastination in students of all levels, backgrounds, and disciplines. We do very little in higher education to stack the deck in favor of our students, preferring to assign term papers in week 2 of the term, then collect 20 page documents in week 15 (many of them written in the 24 hours preceeding the deadline). That isn’t how industry works – what boss gives a new employee 13 weeks of unsupervised labor on a major project?

As for the challenge of designing an assessment for the outcome of “meets deadlines,” it seemed to me there were several opportunities.

One would be to get students comfortable with the challenge of completing work in a rush. Perhaps, at the beginning of class (the first class of the term?) the students must interview their neighbor and write a 250 word profile due at the end of class. Because it is a journalism class, the expectation would be “press ready.” With a few of these under their belts, enforced by a strict deadline, students would grow more confident in their abilities to write under stress.

As for revising in a journalism class, maybe some real-world tension could invigorate things. If 25 students are enrolled, perhaps the professor (the editor) tells the students up front that 10 of the 25 drafts will actually have to run as-is in the next edition of the magazine/newspaper. So students would be afraid that their rather shaky draft would be published…potentially motivating them to write better drafts.

This might be overly dramatic for some, but by the time students are in an advanced journalism course they should have editing skills, writing skills, and time-management skills. The expectation is fair, and would probably invigorate students rather than the standard “writing for the teacher” response to draft review.

Schools put a lot of effort into surveys: alumni surveys, course surveys, faculty surveys.

An article in yesterday’s Chronicle summarizes work done at Cornell University to study the effectiveness of surveys of student engagement. Here’s the main take-away:

Their paper examines response rates of Cornell’s class of 2006 as the students progress through the university. In the fall of 2002, the authors say, 96 percent of first-time, full-time freshmen responded to the Cooperative Institutional Research Program Freshman Survey, a paper-and-pencil questionnaire administered by the Higher Education Research Institute at the University of California at Los Angeles.

But in similar surveys, given online in the students’ freshman, sophomore, and junior years, the response rates were 50, 41, and 30 percent, respectively. A final survey of graduating seniors collected data from 38 percent of them.

Those who completed the follow-up surveys were predominantly women, the Cornell researchers say, and they had higher grade-point averages than those who did not respond.

Surveys are easy: a relatively small number of people (often one) can administer online surveys to thousands of students, then collect the data. Other forms of assessment are much more time consuming and require culture change, the mustering of resources etc. So while the community of survey specialists worry about ’survey fatigue,’ whether students are completing surveys after 9pm (when they could be ‘partying’), and other questions familiar to most marketing executives, our institutions are increasingly dependent on this single data source for major decision making.

The statisticians argue that something is better than nothing, and that they can control for all kinds of oddities. According to the article, 30% is an acceptable response rate. What is stunning to us about the report is that it is considered news. Every institution has this kind of data: the ability to bounce student attributes against survey data (and other data). But, in general, student evaluations are taken in a very literal way (as anecdotal evidence, without context) and are used to make major decisions about tenure and curriculum redesign.

Do results like Cornell’s invalidate the process? Not at all – but they should cause changes to the survey process (the goal of outcomes assessment: to improve processes through rigorous analysis). Similarly, the vast amount of data available in a school’s back-end database (Banner, Datatel) should be put to much greater use. How much variability in grade inflation occur in particular courses? Which courses/programs receive the best course evaluations? This data leads to questions that would help improve outcomes while addressing some of the incoherence students experience. Tying these various data streams together would help build a complete picture. So if a teacher gets slammed for being too ‘hard’ on course evaluations? Perhaps they are grading significantly harder than their colleagues (which doesn’t necessarily mean they should be the ones to change!). This sort of inquiry is second-nature to most academics; we just don’t apply our analytical and research skills to our most important undertaking: teaching.

We were pleasantly surprised to see Waypoint (web-based software for creating and using interactive rubrics…find out more here) featured in Bill Vilburg’s LMSPodcast series.

Bill is the Director of Instructional Advancement at the University of Miami, and does in-depth interviews on issues concerning Learning Management Systems. He has ambitiously set out to interview the all of the presenters at this year’s Blackboard World Conference in Las Vegas.

Last week he interviewed Dr. Rosemary Skeele, from Seton Hall University and Dan Driscoll, from Drexel.

All the interviews that Bill does are in-depth and wonderfully paced. The most exciting aspect to the interviews is how little time is spent talking about Waypoint. The interviews are all about the challenges of designing effective peer reviews, leveraging Blackboard and Blackboard Vista, and developing data that is used to improve curricula. Waypoint is just the mechanism.

Peer review, in particular, is an under-utilized tool in education. When done right (just listen to Dan Driscoll’s process) it is a fantastic way for teachers to coach more, grade less, and radically alter students’ relationship with writing. With the release of Blackboard Version 8, there is a window of attention on the subject because v.8 has a rudimentary Likert Scale commenting tool built into it. Since Waypoint was designed from day one with peer review in mind – peer review of any artifact or product – and is based on sound composition and pedagogical theory, we look forward to an increased dialogue on the subject.

You can find the podcasts here:

LMS 43 Dan Driscoll, Drexel University

Dan Driscoll uses the Waypoint add-on system to create a peer review system in his first-year composition courses at Drexel. He discusses how he sets up the rubrics and then has the students fill them out. The process of applying the rubric to the papers gves students as much or more value than the feedback given back to the original author. Dan will be presenting “Course-Embedded Assessment and the Peer Review Process” at BbWorld’08, July 15-17.

>> Play the Podcast

LMS 42 Rosemary Skeele, Seton Hall

Rosemary Skeele describes how Seton Hall is using the Waypoint addon for Blackboard to help assess learning, primiarily for accreditation purposes. Waypoint allows you to integrate rubrics into Blackboard and in the process opens new possibilities. Rosemary will be presenting “Blackboard and Waypoint: Perfect Together” at BbWorld’08, July 15-17.

>> Play the Podcast

We’re just finishing up three day’s at NC State’s Assessment Symposium. 500 educators from around the USA have come together to talk about student learning, “closing the loop,” and accreditation.

Many of the sessions are focused not just on data-gathering, but on teaching and learning. A number of attendees have talked about the change they’ve seen since even last year: a focus on bringing assessment into the process of teaching (!). That is, avoiding the mad dash to develop data just for accreditation that often results in two databases of student learning outcomes. One presenter said that on her campus administrators referred to the “shadow database,” which reminded me of a business owner keeping two sets of books – one for the IRS and one for the real world.

We gave a 60 minute presentation on best practices in course-embedded assessment. We must have had at least 50 people in attendance…not to learn about Waypoint as much as to gain insight into how schools execute.

I spoke in three general areas:

1. Getting faculty help with the challenges of formal assessment
2. “Closing the loop” – using data to inform changes in curricula
3. Using a sampling approach to gather data quickly and efficiently for benchmarking purposes

Getting faculty help with the challenges of formal assessment:

We increasingly talk to senior administrators about the need to look at authentic assessment and course-embedded assessment as more than a challenge in software training. This work is not about clicking the right buttons in Blackboard or Waypoint. Too often a roll-out plan is based on a two hour “hands-on” training session, where faculty watch an outside consultant or instructional technologist walk through basic functionality. This approach only works with the people who don’t need it, the types who can kick the tires, click around, and figure software out.

Constructing knowledge is much more powerful than having it pushed at you. So the two best-practices that I highlighted, one from a private university and the other from a state university, are essential to any real culture change.

The first approach is about instructional technologists working with faculty – in faculty offices, on faculty schedules, to develop assessments (Waypoint or otherwise). The instructional technologists, more facile with web-based tools perhaps, can help with the work while also bringing a structured methodology to rubric and assessment design. And after a personal meeting or two, the faculty member has something they can use. After a training session, faculty usually have a headache.

The institution in question has built this collaborative work around a writing across the curriculum initiative, and for the first time a technology and teaching initiative has caught on in all four major academic divisions at the same time, which is a first.

The other “helping faculty” approach is an institution’s commitment to a three day course to teach and implement rubrics. Three days. Faculty have to write a short proposal to be considered, and space is limited. The department head has to sign off on the application, which puts the process on the radar for annual reviews (smart move). There is even a $200 stipend to be used for professional development activities.

All participants have been given Arter and McTighe’s book, Scoring Rubrics in the Classroom. The goal is to develop an understanding of rubric design, and then to build detailed rubrics - both on paper and then into Waypoint. The seminar is being taught by faculty at the school, not outsiders.

For me, these are two concrete and attainable ways that many institutions could take a longer-range approach to supporting faculty in their development of rubrics, outcomes data, and most importantly, the improvement of their own teaching.

Once I get back to the office, I’ll post the second and third areas that we discussed: “closing the loop” and gathering data quickly via sampling.

The University of Pennsylvania’s student newspaper, The Daily Pennsylvanian, recently included a student-written piece on the lack of feedback in higher education. The article is called “Fighting for Feedback,” and it is a show stopper.

If we want to talk about accountability and outcomes assessment, what greater measure is there than students receiving feedback from their professors? David Kanter’s article begins,

The first time I got a paper back from a professor here at Penn, I was a little confused.

Other than a few perfunctory, illegible comments found scribbled in the margins, insightful, constructive criticism was nowhere in sight. I thought (incorrectly, I suppose) that I would receive extensive feedback on each assignment. I soon learned that unmarked papers and vague comments were the norm.

This is a message we’ve heard over and over again. It isn’t a Penn thing. It isn’t an Ivy League thing, or a private school thing. It’s just the old paradigm of pushing information at students rather than helping them discover knowledge for themselves. Some of us are more talented pushers than others, and can make that mode of education work through charisma and talent. But the majority of experiences we’ve all had in our educational careers are closer to what David describes than a true dialogue over issues and ideas.

I see the surprise in my students’ eyes when I hand them back their first feedback on a major assignment – a half-page or so of neatly typed and categorized comments, often with an annotated document attached. The surprise is created not by my straightforward feedback, but by the 12 years of hit and miss comments they’ve received previously.

If institutions are worried about being accountable to stakeholders – like parents, alumni, state and federal governments – isn’t David’s description an indictment of the service a student receives in exchange for $45,000 per year?

We are proud to be working with our client institutions to help move assessment and feedback into a new era. It doesn’t have to happen through software. Intelligently designed rubrics coupled with authentic assignments that don’t feel like make-work to students can make a world of difference, even if they only happen once a year or once a term. Dr. Greenhalgh’s army of TAs at the Wharton School are making a difference and doing what no single professor could do. Other schools have writing fellows, Writing Intensive Tutors, formal programs to educate faculty about writing across the curriculum, and a serious approach to data-gathering.

It’s a party that’s getting bigger, and if students stand up and “fight for feedback” universities will pay attention and put resources behind these initiatives.

A couple of terrific articles recently that have serious implications (and lots to teach us) as educators.

The first, from Campus Technology, argues that higher education has co-opted the ePortfolio from its intended role as a reflective and creative student project to become a tool for accreditation reporting. Since our focus with Waypoint has always been on the assessment engine, and not the attempt to build yet another portfolio solution, we are in total agreement.

The second article, from The New Yorker magazine, makes a devastating case against the medical establishment (you’d think we would have run out of reasons to bash medicine) and its hubris. Atul Gawande makes a compelling comparison between contemporary medical doctors and the test pilots of The Right Stuff fame. As a teacher, the comparison hit home. Could simple checklists help our students complete the tasks we assign more creatively and more competently?

Here are the citations:

1. Trent Batson, “The ePortfolio Hijacked,“ Campus Technology, 12/12/2007
2. Atul Gawande, “Annals of Medicine: The Checklist,” The New Yorker, 12/10/2007

Dr. Helen Barrett, who has been writing about ePortfolios for years in a variety of media, summarized the issue with the use of ePortfolios eloquently in her blog:

Somehow, we need to get back on track with the metaphor of “ePortfolio as Story” and not only “ePortfolio as Test” or we will lose a powerful tool for reflection and lifelong learning. The challenge we have is accommodating the strong pressures for institutions to produce tangible evidence of achievement for external audiences (accreditation and government agencies), so that faculty and students can also focus on the internal audiences (small, private, personal) to realize growth over time. I am concerned about the “opportunity cost” (the value of the benefits forgone) in the current focus on accountability portfolios. How can we find a balance?

In our experience, institutions often prioritize obtaining an “ePortfolio” solution as a way to accomplish a multitude of goals. One is accreditation reporting, although ePortfolio implementations in all but teacher education programs rarely produce the data necessary for accreditation. Another goal is staying with current pedagogical trends and integrating technology into the curriculum. Another major goal is curriculum redesign – using evidence of student learning to adjust pedagogy. A final, and admirable goal, is to help students develop more advanced critical thinking and reflective skills. This last goal was the prime mover for portfolios in the first place. However, a program does not need a massive database server and sophisticated software to manage a selection of documents and reflective writing! Google Docs, blogs, and simple web-pages can suffice. And often with technology, the least complicated option is by far the best. The result is less training (of students and faculty), less upkeep, and a greater focus on the essence of a portfolio initiative: student creativity and reflection on learning.

Drexel University’s LeBow College of Business provides a great example of integrating portfolios into the curriculum with the original (non-hijacked) intentions of portfolios in education. LeBow uses a dedicated portfolio software product, integrated with the schools Learning Management System (Blackboard Vista in this case), that does not do any assessment. Students have considerable flexibility when building their portfolio along with required artifacts. LeBow, with Drexel’s strong focus on cooperative education, even includes students’ reflections on their work experiences. You can read more about LeBow’s My LIFEfolio here (5mb PDF file).

LeBow must also generate rich data on student learning for AACSB accreditation. In fact, AACSB has some of the most stringent requirements for data-gathering outside of teacher preparation. To gather the minimum of 3 years of data necessary, LeBow samples documents from the curriculum, strips away unique student identifiers, and then uses a combination of business professors, English professors, and outside assessors to evaluate the documents against a detailed rubric. This analysis and data gathering is facilitated by Waypoint. This winter, LeBow will use Waypoint inside of courses to assess presentation skills. So assessment for improved curriculum, data-gathering for accreditation, and ePortfolios for learning are complimenting one another rather than working at cross-purposes.

No matter how badly educators misuse assessment, accreditation, and portfolio tools, no one’s life will be lost. But Atul Gawande, in a New Yorker article from a few week’s ago, documents a devastating fact of modern medicine: simple checklists can save lives in I.C.U.s and other places, but the simple procedural change is ignored. He writes,

This is the reality of intensive care: at any point, we are as apt to harm as we are to heal. Line infections are so common that they are considered a routine complication. I.C.U.s put five million lines into patients each year, and national statistics show that, after ten days, four per cent of those lines become infected. Line infections occur in eighty thousand people a year in the United States, and are fatal between five and twenty-eight per cent of the time, depending on how sick one is at the start. Those who survive line infections spend on average a week longer in intensive care.

The article is a fantastic read, and I highly recommend it. Although it seems counterintuitive given the complexity of medicine, a John Hopkins Hospital doctor named Peter Pronovost has proven the power of checklists to dramatically decrease infections and complications. The particular point that caught my imagination came with a comparison between modern doctors and the famed U.S. test pilots of the 1950s:

Test pilots strapped themselves into machines of barely controlled power and complexity, and a quarter of them were killed on the job. The pilots had to have focus, daring, wits, and an ability to improvise—the right stuff. But as knowledge of how to control the risks of flying accumulated—as checklists and flight simulators became more prevalent and sophisticated—the danger diminished, values of safety and conscientiousness prevailed, and the rock-star status of the test pilots was gone.

I couldn’t help think of the good teachers I have known, both professionally and as a student. I’m sure many were methodical in their own ways, and maybe even used checklists (or similar). But something they all have in common is charisma – or rock-star status. Granted, the rock-star status was in the world of teaching history or economics, but they were compelling because of their personalities, their stories, their ability to relate to students, and to demand excellence. They would have done quite well without Blackboard, Waypoint, even computers. But they are the minority.

The other connection that I immediately made had to do with students’ achieving basic competency against specific requirements. For instance, including a compelling title to an essay, or making sure they include a counterargument. I have taken to requiring a cover letter to all work submitted in my classes, and I ask students to reflect on their previous work, peer review feedback, and my previous feedback along with explaining their approach to a particular assignment (one reason I like these cover letters is that they are often a few hundred words long, and provide an easy way to get more writing into the assignment!). In my mind, this cover letter works a bit like a checklist: students need to write about a variety of issues that are important to the assignment. The trouble is, this is to “teachery” for many students. They simply don’t make the connection. Plus, they’re completing this letter a few minutes before they submit the final draft. So it doesn’t help many of them to run through requirements in a letter when they have little time to correct issues they identify.

Checklists are direct. If I simply embedded a checklist in the curriculum (using Waypoint in conjunction with a Learning Management System, of course) a few days in advance of an assignment’s due date, then required students complete the checklist as part of their homework, students would internalize the assignment requirements in advance of the deadline. This would help students with issues of procrastination, reinforce the goals of the assignment, and level the playing field. Most beautifully, just as a checklist designed for a specific procedure in an I.C.U. has applicability to thousands of locations, checklists for major assignments and the high school and college levels could be applied in hundreds of courses.

Doctors and test pilots still need tremendous skill and training, but some of the most avoidable complications to medicine and flying can be avoided by clearly stating a procedure and requiring follow-through. Students who procrastinate, can’t think critically, don’t write proper transitions, and ignore feedback on previous assignments might be best served by a more rigorous but simple approach to documenting expectations via checklists embedded in the curriculum.

I’ll mock up something that gets at this issue and report back on my success (or not) with students sometime in January. In the meantime, we welcome your input concerning the application of ePortfolios in learning and how clearly we document our expectations of students in a way they can digest and use.

From middle schools through world-class MBA programs, the more we talk and listen, the more the plug and chug and crank approach to learning clearly doesn’t work.

That line is a famous quote from a professor I had years ago. He still teaches Physics, and is supremely dedicated to his job and his students. A great entertainer, I remember enjoying his lectures for his good-natured teasing, funny stories, and huge energy. But he was teaching engineers basic physics and the math that goes with it. We were not an overly talented bunch, and he just hammered through the problems. At the crucial moment, when the incline plane or pulley system had been sufficiently analyzed and the only step remaining was the calculations, he would command “PLUG AND CHUG AND CRANK!” Generations of students have loved him for that, have imitated him endlessly with great affection. I learned physics – at least enough to be a successful engineer as judged by my GPA (magna cum laude:)

But did I understand anything? Not really. Labs were performed by rote memorization, with a TA laying out the steps to get the students out as quickly as possible. Exams were all about studying ‘back exams.’ If a professor was lazy and didn’t change the problems from year to year then that was their luck out and we breezed through the exam.

There were three seminal moments for me as an engineer: the moment I realized that I might really fail and have to switch majors, the Valentine’s Day massacre when I scored the second-highest grade on a math exam covering non-Cartesian coordinate systems, and the statistics course I took as a junior with Dr. Banu Onaral.

The first two moments are subjects for another conversation, but after a term or two of surviving, then a few years of thriving because of my abilities to plug and chug and crank, I finally hit a professor who wanted more and had designed assessments to actually measure the ‘more.’

The class was called Non-Deterministic System – statistics for engineers, I suppose to prepare us for the insane probabilities involved in the physics of semiconductors. But the class was very theoretical. Basically, the problems posed a statistical problem and you had to solve for the probability of a given event. The events were simple but abstract, and revolved around packs of cards and dice, flung into the air with specific parameters.

My roommate at the time was brilliant at these problems. He would glance over the description, avoid the distractions and intended confusions, and pronounce “7/32.” He was always right. I struggled – with the math, with the abstractions, but mostly with the ambiguity of the questions.

Dr. Onaral didn’t give math exams. She gave writing exams. You had to write out your interpretation of the problem and the logic you used to solve it. So all term I don’t think I got anything “right,” at least in the intended manner, but I got excellent grades and an ‘A’ for the course. The exams would come back with notes written in the margins of the problems – little personalized questions and exclamations. And I ‘got’ that the professor had taken the time to engage with my point of view and to look beyond the expected answer. Just that single class, amidst a sea of plugging and chugging, was hugely encouraging to me. That summer, when I interned for a steel mill and was sent to North Carolina to work alongside displaced furniture makers for twice their hourly pay, I was well prepared for the ambiguities of it all. And a more confident engineer.

So as we’ve talked to business professors who run MBA programs where every exam is a written exam, middle-school teachers who have ended the bell system, and therefore don’t just drop everything after 40 minutes to switch from ‘English’ to ‘History,’ and English professors who have rethought how first-year English should be taught, we’ve been humbled and overjoyed to find like-minded educators. And they’re happy to find us: to streamline the response process, to create exceptional feedback for their students, and to develop rich data (based on the criteria they value) for accreditation and internal discussions. There is always a place for plug and chug and crank – at some points in life you just have to take things on faith – and my physics professor proved that he could make up for the mechanical nature of his approach through charm and sheer entertainment. But for thirty or forty thousand bucks a year (private education in the US) and the demands of a 21st century information economy, students and parents should demand more.

At one of Grant Wiggins workshops over the summer I saw him present this quote, and it is one I always use in presentations and training:

“At both ends of the school career, we deemphasize one-shot, uniform testing in favor of a careful assessment, from different perspectives, of the student’s own projects. We focus more on the student’s ability to extend or play with ideas than on the correctness of answers to generic questions. Each piece of work, be it a drawing or a dissertation, is examined – often through dialogue – for what it reveals about the learner’s habits of mind and ability to create meaning, not his or her ‘knowledge’ of ‘facts.’

At the beginning and end of formal education, we understand that intellectual accomplishment is best judged through a ‘subjective’ but rigorous interaction of mind and mind.”

Just one or two experiences built on this model can be extremely memorable and inspiring for students – and for schools that care about their students’ life after graduation, care about retention rates, and care about alumni involvement, it should be a no-brainer to consciously work more authentic assessment into the curriculum.

We spoke at the Lehigh Valley Technology in the Classroom Symposium today.

Higher education institutions from the Lehigh Valley (in and around Allentown, PA – about an hour north of Philadelphia) gathered to share approaches to common challenges, immerse themselves in some of the latest and greatest things going on in ed tech, and relax their way through a very hot PA summer day.

It was a calm and pleasant way to spend some time getting to know some new schools, talk to the Tablet PC wizzes from Gateway, and talk about everybody’s favorite subject, outcomes.

We got the coveted just-before-lunch time slot, just after a very compelling Beth Ritter-Guth described her amazing uses of Second Life to immerse students in the literary worlds of Beowulf, Edgar Allen Poe, and Dante. Tough act to follow. Now if some foundation would just throw a couple of million bucks into making a totally immersing, photo-realistic Yoknapatawpha County.

Here’s my list of resources, which several people asked for:

• NC State University Internet Resources for Higher Education Outcomes Assessment
• Dr. Helen C. Barrett’s uber-site on eportfolios (just about everything you could think to ask)
  • Particular discussion of the need to split portfolio functionality from assessment functionality

High level outcomes mapping tools

• WEAVEonline
• Bb Outcome$
• Trakdat

Tools for developing actual data (!) on student achievement

• NSSE: National Survey of Student Engagement
• Collegiate Learning Assessment
• Waypoint Outcomes
• The IDEA Center and other surveying tools

TK20, TaskStream, Chalk&Wire, LiveText are standalone eportfolio solutions with an assessment engine built in. They were developed, primarily, for teacher education programs (accrediting purposes) and have a unique business model – the student pays. So for $89 for 5 years, or similar, a student purchases their own space on the company’s servers to host their portfolio, ostensibly so that the student can have easy access to their portfolio after they graduate (which is particularly important to recently graduated teachers).

Dr. Helen Barrett, seemingly the worlds most prolific and expert commentator on all things eportfolio, writes about this family of tools with Dr. Judy Wilkerson, and expresses concern that assessment and evidence don’t necessarily play well together – that trying to pack both into the same tool is a “conflicting paradigm.” I’m in agreement on this one. Plus, from a basic workflow perspective, having assessment tied to the portfolio makes it awkward to just trot out a rubric to respond to a simple in-class writing assignment or similar. Read more here.

We’ve had a crazy month – between NECC 2007 in Atlanta (end of June) and Bb World 2007 in Boston (mid-July), I’ve been doing a lot more traveling than normal. I’ve got a couple of blog entries drafted, so stay tuned. One entry is on student-centered rubric design and the other compares a brilliant recent article by James Surowiecki, writing in The New Yorker about fuel economy and ice hockey, to the use of elearning tools and outcomes assessment in higher ed.