Tag Archives: assessment

Retrieval Practice and Ed Tech

From July 18,2014 NYT, Sunday Review

From July 18,2014 NYT, Sunday Review

Last month, the NYTimes ran an education article entitled How Tests Make Us Smarter, by Henry Roedigger [July 18, 2014]. The basic idea put forth in the article was that frequent testing begets frequent retrieval of the tested information; the more often we attempt to retrieve information as we learn it, the more sophisticated mental structures we create around it, the better it sticks.

Roedigger’s piece sent me off on a hunt for the original research cited in the article.  That led me to this article authored by Jeffrey Karpicke and Janell Blunt (Purdue University):

Retrieval Practice Produces More Learning than Elaborative Studying with Concept Mapping, [January 2011]

Their research points to the conclusion that testing provides retrieval practice and, as such, works as a memory modifier – it strengths the information in the learner’s brain, making it more likely to be recalled later. In their article, Karpicke and Blunt cite the researcher, Robert Bjork, a cognitive psychologist at UCLA who has, hands-down, the best lab name I’ve ever heard, “The Learning and Forgetting Lab”. Bjork is well-known in education circles for coining the phrase “desirable difficulties”. His cog psych research points to the conclusion that roughening up the learning path (“desirable difficulties”) leads to greater retention and comprehension. The practice conditions that produce desirable difficulties are spacing (distributing practice over time), interleaving (for example, mixing in equal parts tutorials and practice problems), and testing (retrieval).  Bjork talks about the “study-study-study-TEST” model versus the (preferred) “study-test-test-test” model. I’ve blogged about Bjork’s work previously.  Here is a good explanation of his research, When and Why Introducing Difficulties and Errors Can Enhance Instruction, Courtney Clark and Robert Bjork (UCLA)

The key take-home in both Bjork’s and Karpicke’s work is the importance of using testing as a learning tool instead of just as an assessment tool. Frequent, low-stakes, formative testing, companioned with timely feedback, allows learners to strengthen their retrieval and rehearse the correct information.

With this research base in mind, the next natural step is to address the practical question of how best to provide regular retrieval practice? How to encourage practice that enhances learning in the classroom and doesn’t add to the feeling of being tested to death. To my mind, that’s where educational technology comes in. With tablets or laptops in the hands of each student in a wired classroom, educators can construct lesson plans around frequent, low-stake performances that provide regular feedback.  For instance, reflective blog posts, short student presentations, practice problems, and peer-to-peer teaching in small groups.

The interesting thing about this way of looking at educational technology, is that it doesn’t drive the change – it becomes a facilitating tool.  First you want to improve teaching/learning, next you examine what the educational literature tells us about what works, and then you look for the tools to help drive that change.


Filed under Reflections on Teaching

Reflection On – and In – Action

I’m working on a new project, with the Forum Corporation, about workplace learning.  It’s been a really interesting journey that’s thrown me into some fascinating reading and put me in touch with some very interesting experts – Chris Briggs, a learning expert from Indiana University School of Informatics,  George Siemens, the learning theorist and co-founder of “connectivism”,  and the Harvard neuroscientist,  Srini Pillay.

One element of this research is the concept of reflection.  We know that learning must consist of action—mental, physical, or both—in conjunction with an opportunity to reflect on the action and its outcome. People learn by doing and thinking about what they have done.  Reflection generates lessons for future action.  It allows people to examine experiences, to find meaning in them, and to generate new insight and knowledge.

But when do we reflect?   If we do at all, that is.  In our busy, workaday worlds, we can’t always make time for reflection.  But brain research (and our experience) tells us that reflection is an essential part of learning – without reflection we are unlikely to experience behavior change, and it is highly unlikely that the learning will be retained.

When we do reflect, most often it is reflection on action – that is, we set aside some time, after the fact, and we think back on what we have done in order to improve or change our future actions. Equally important, yet not often discussed, is reflection in action –that is, reflecting in the midst of our actions, without interrupting them, and reshaping what we are doing as a result.  As an example, think of the way a skilled pianist reflects while she is playing – How could it be better?  What mistakes am I making?  What refinements might improve my performance here?

Just last night, I had the practical opportunity to fully absorb that notion.  I’ve been asked to do some “test cooking” for a friend who is publishing a cookbook.  She has a number of recipes that she wants others (less proficient others, like me) to test for her and report back.  She is interested in information like – were the instructions easy to follow, were the ingredients easy to obtain, how long did it take to make the dish, how did it taste, and do you have suggestions for improving it?

Last night I prepared my second test dish for her – linguini with kale and pickled lemon (I know it sounds gross, but it’s really tasty). As I was preparing the dish, I realized that I was reflecting in action.  I took in the ingredient list critically – asking myself if it made logical sense.  I carefully measured amounts and time, making note of them.  I tasted as I went along, marking subtleties.  I looked for inconsistencies in the directions.

All of this attention and reflection happened while I prepared the meal.  It was most definitely reflection in action.  I was spurred on by the certain knowledge that my friend would ask me questions about the experience when I was done.  And the impact of that knowledge, for me, was a much more vivid, memorable, and intense cooking experience.  I also know that the next time I make linguine with kale, I will not need a recipe.  It’s burned into my longterm memory.

The resulting dish is pictured above – and I can tell you that I enjoyed it more than most meals I create.  I suspect that pleasure had quite a bit to do with how carefully I reflected on the making of it – while I was making it.


Filed under Reflections on Teaching

Blooming with Students

Bloom's Taxonomy Butterfly

One of the sharathon session at this year’s Biology Leadership Conference focused on a novel use of Bloom’s Taxonomy.  Mary Pat Wenderoth (University of Washington) gave a great talk on the way she uses Bloom’s with students in her freshman majors’ biology course.  For those unfamiliar with Bloom’s Taxonomy, it’s a useful – and widely recognized – system for classifying learning objectives according to cognitive domains.  It was first presented in 1956 through the publication of a handbook by Benjamin Bloom (an educational psychologist) and since then, has undergone many revisions and alterations. Today, educators most often apply the taxonomy to asseessment items, in order to make certain that their students are functioning in higher order cognitive domains (that is, they possess a deeper synthesis versus a rote mastery of facts). Here’s a grid, summarizing the original Bloom levels with definitions, sample verbs and behaviors.

Wenderoth uses these Bloom’s cognitive domains:  evaluation, synthesis, analysis, application, comprehension, and knowledge – with analysis, synthesis, and evaluation being the recognized, higher order domains.

In her talk, Mary Pat described the start of her investigation into the use of Bloom’s at UW when they set out to create a “Blooming Biology Tool”.  They assembled a team of three biology faculty and designed a rubric, based on Bloom’s Taxonomy.  Then they rated 700 (gasp!) exam questions taken from 1st year medical school tests, the MCAT, the biology GRE, undergraduate biology courses, and the AP Biology exam, according to their rubric. This gave them a solid, working knowledge of the way assessment items work (and don’t work) and some really good ideas for incorporating the use of Bloom’s in their courses.

First day of biology class at University of Washington

Wenderoth began introducing Bloom’s Taxonomy on the first day of her introductory biology class. Her idea was to make clear to the students that not all questions are created equal and encourage the students to monitor their own learning. On that first day, they come up with a mnemonic to remember the taxonomy – they use that all year.  They “Bloom” questions in class (before they answer them) and they “Bloom” last year’s exam.  As the semester progresses, the students get increasingly comfortable with the scheme and begin to recognize the type of questions that give them the most trouble.

When Wenderoth creates her exams, she indicates the taxonomic level of each question, right on the exam (a good double check for her, to make sure that her exam is in alignment with her plan).  Then, when she hands back the students’ scores, they are also given their “Bloom scores” (Wenderoth uses an online grading tool to calculate this).  So students know exactly how they scored on, say, “Knowledge” questions vs. “Analysis” questions.

As a result, Wenderoth reports students come to her office saying things like, “I need help with “Analysis-level” questions.”  And that, explains, Wenderoth, is a much better place to start with a student.

From there, Wenderoth and her colleagues have started to develop learning activities for students, based on specific Bloom’s areas. So, for instance, if you are having trouble with comprehension questions, she recommends activities where you describe a biological process in your own words, provide examples of a process, write a sentence using a key term, or discuss the process with peers.

I loved this whole idea.  What a great way to make learning more transparent!  It takes the mystery out of learning for students and encourages them to think about their learning in a productive way.  It also helps to give students insight into the way their instructor thinks – how the course is organized – how the exams are created and graded.  Mary Pat wrapped up her talk with a few quotes from her students…this was one of my favorites:

“I remember initially thinking, “Why are we wasting valuable class time on Bloom’s taxonomy?” I felt that Bloom’s taxonomy was a burden, but I now use Bloom’s taxonomy unconsciously to attack many problems. It is a method used to help organize my thoughts before I act.”

p.s. Here’s a link to a pdf of the bloomsposterv4 butterfly poster shown at the top of this post.


Filed under Reflections on Teaching

Learning Obliquely

Last week Kate Haney turned me onto a really interesting article on the web site for Intelligent Life (a quarterly journal published by the Economist) about professionalism that I’ve been pondering ever since (thanks, Kate!).  The article starts out talking about professionalism in sports (cricket, specifically) and the way that our understanding of the terms “amateur” and “professional” has shifted with time.  While “amateur” might have been used at one time to describe someone who embarks on something (a hobby, sport) for the love of it – not to be paid or to be perfect – but because they’re having fun with it.  But these days, when we use the term “amateur”, it is usually in a pejorative sense.  As in sloppy, slap-dash, and without care.  Professionalism, the author suggests, is in ascendancy.  We are all encouraged to be professional.  To have professional wins.  To take a professional attitude.  To rise above – and act professionally.  But what does that get us, really?

The author goes on to talk about professionalism in education and how it can have the unfortunate consequence of hemming teachers in, limiting spontaneity, and short-changing students… Teachers being asked to prepare detailed lesson plans for every minute of their day, curriculum guidelines, mission statements, covenants, and – of course, the hand maiden of professionalism — Assessment.

But it’s when he connects these thoughts to a forthcoming book by the economist John Kay, entitled Obliquity, that my mind really started to race.  In his book, apparently, Kay puts forth the notion that complex goals (like investment banking, and, say, teaching young minds) are best gone at obliquely. That is, as a by-product of something else, rather than an end in and of itself.  What an interesting notion to consider.

And consider it I did, at the grocery store, that same week. I was going about my business when I spotted a young woman with a toddler who was clearly putting his first language connections together.  The boy had a bag of M&M candies clutched in his hand.  As they approached the cashier, the boy stopped suddenly in his tracks, visibly gob-smacked. He stood there; staring at the cashier who, as it turned out, was wearing a smock decorated with the M&M cartoon characters.  The boy’s mother was oblivious to the amazing moment this boy was undergoing, since she was busy putting her groceries on the conveyer belt..  I watched carefully as he stepped a little closer to check his data. Nodding his head, he said in a clear, loud voice, “SAME!”

A little oblique learning going on there.


Filed under Reflections on Teaching

New Literacies: Inside and Outside School

“We are living in the middle of the largest increase in expressive capability in the history of the human race.”   Clay Shirky

I can’t help but notice the startling contrast between the world inside school these days and the world outside of school.  Outside of schools, students are talking about music they’re producing, online communities they are part of, conversations they’re having online with people at a distance, and sharing, sharing, sharing.

creative commons

Thomas Favre-Bulle, Creative Commons

Inside school?  Well, not so much. Inside school looks pretty much the same way it has for a very, very long time.

The National School Board Association recently published results of three surveys regarding social networking and I was not surprised to learn, in a recent THE Journal article, that 52% of all districts interviewed prohibited any use of social networking sites in school. But here’s the kicker – “almost 60% of students who use social networking talk about education topics online and, surprisingly, more than 50% talk specifically about schoolwork.”

There’s a new literacy out there and, you know what? …it’s happening with or without our schools. But just think for a minute how much better it might be if our teachers, administrators, and school resources were supporting and guiding that literacy journey that our students are taking without us. Just think what teachers could contribute.  Yes, students are pretty facile with all of this new media, but there’s still so much they need to know. For example, students need to know how to determine the veracity of a web site (and find out who owns the domain), how to safely navigate an online social network, how to make good judgments about what they post online, how to edit a wikipedia entry, how to navigate the sea of information available to them, how to use of the creative commons, how to curate their own work, how to connect with experts and peers  – how to embed, share, mash-up, remix, and animate.

The federal No Child Left Behind Act and standardized state curricula and assessments don’t reflect these  literacies. There’s no doubt about the fact that NCLB and the way student achievement is measured has had a strong influence on daily life in the classroom. By aligning our schools along the narrow band of what can be evaluated in a high stakes exam, we fail to assign value to the new knowledge and skills that our  youth need to become effective participants in a global, networked environment. It’s time for our schools to teach and foster responsible student mastery of new literacy forms.  That doesn’t mean throwing away the old ones – couldn’t we augment them with the new literacies?  We have a responsibility to teach students to critically understand and responsibly use these new forms of media  – and in order to do that, we need to understand them ourselves.

Web 2.0 is an ideal platform for this kind of participatory learning.  These tools help us to reach out to others, join in the conversation, creatively express ourselves, and find our teachers. As Chris Dede says, we want students to not just be problem solvers, but problem finders – out there, working it, finding fresh areas for investigation.

Leave a comment

Filed under Teaching with Technology, Uncategorized

Clever Use of VoiceThread

Exam review using VoiceThread.

Exam review using VoiceThread.

My friend, Tod Duncan (UC Denver) just sent this VoiceThread link to me.  It will take you to a Voicethread that he created to review the results of a recent exam given in his introductory biology course.  There’s so much to love about this!

First off, I appreciate the tone of he takes in the recording.  A friendly, casual, companionable, let’s-you-and-I-just-talk-this-through sort of tone.  That’s bound to put the students at ease. I really like the way he subtley reinforces good test-taking strategies, like thinking through the way to eliminate impossible or unlikely choices in a multiple choice exam.

It also strikes me that reviewing an exam this way would be extremely efficient.  Rather than go over the test individually with students during office hours, one by one by one, students can link to this VoiceThread and listen to it.  And they can listen as many times as they need to.  He could also use this with future students, as a test preparation tool.  It’s unlikely Tod will use the same exam questions next time around, but hearing their instructor’s analysis of past assessment items will help them prepare for new ones.  Tod just posted this so, right now, there are no student comments embedded, but using the comment feature in VoiceThread, students could post further questions or requests for clarification to Tod or to their fellow students and get a conversation started.  Brilliant, just brilliant.


Filed under Teaching with Technology