ERRR #010. Catherine Scott on Meta-memory and More

Listen to all past episodes of the ERRR podcast here.

Catherine taught art and technics in high schools and primary school before training as a school psychologist. She went on to complete a phd in psychology at Macquarie University and has taught psychology and research methods to a variety of students at the under graduate and graduate levels, including those studying teaching, clinical psychology, nursing, school counselling, physiotherapy, speech and occupational therapy. She has taught in several Australian universities and overseas. Catherine was state president of the Victorian branch of the Australian College of Educators and served as chair of the ACE national council and member of the ACE Board. She is a registered psychologist.

Catherine nominated two articles to form the basis for this ERRR discussion. The first is entitled Meta-memory and Successful Learning, and the second is the chapter on memory from Catherine’s excellent book, Learn to Teach, Teach to Learn. Catherine is a generalist and, as such, the discussion in this podcast is very wide ranging touching on numerous topics from memory, to mindfulness, to such questions as what is meant by ‘questioning for understanding’ as well as ‘what exactly is ‘Depth’ in learning?’

Links mentioned during the interview

TOT019: The most amazing school in the world! + other Twitter takeaways

Teacher Ollie’s Takeaways is a weekly-ish post (and sometimes a podcast!) bringing together some of the fascinating things that Ollie read throughout the week-ish. Find all past Teacher Ollie’s Takeaways here

What are the maths wars and what do I need to know about them?

Turns out the What Works Clearinghouse has some serious methodological issues

Department wide interleaving. Someone did it!!! Here’s what happened : )

The Dyslexic brain: What happens in there?

How should I start the year with my new class? (With video!)

The most amazing school on Earth! Has to be read to be believed!…?!?

People still believe in learning styles! This may be why…

Might wanna read this if you’re keen to improve teaching and learning in your department

‘Performance Assessments’ of teachers… what the evidence says.

Embed from Getty Images

A recent article on Education Minister Simon Birmingham’s website entitled ‘Lifting the Bar on Quality Teacher Education‘ states that  ‘The Turnbull Government is lifting the bar on teaching quality with the rollout of performance assessments for new teachers graduating in 2018.’ and that ‘Minister Birmingham said all new teachers would need to pass a teacher performance assessment before they could graduate.’

This prompted me to share a correspondence on teacher observations and assessments that I had recently with Heather HIll, Jerome T. Murphy Professor in Education at the Harvard Graduate School of Education. I share it here in the hope that Heather’s response can be considered by those groups and individuals who are making decisions regarding this proposed teacher performance assessment.

I asked Heather (with minor edits):

Hi Heather.

Ollie Lovell here, head of senior maths at a Melbourne School (Australia). Trying to build a healthy culture of teachers visiting each other’s classes.

Dylan William quotes you on the Craig Barton podcast (an excellent episode by the way!)  and says ‘Heather Hill’s work at Harvard suggested that a teacher would need to be observed teaching 5 different classes, with every observation made by 6 independent observers to  really be able to reliably judge a teacher.’

But searching through your work I wasn’t able to find this statistic referenced at all?

Am I just looking in the incorrect places?

In some ways I’d love for this statistic to be true. It would  help me to introduce more classroom observations in a non-threatening way. But in another way, I’m doubtful, because There are things that we can observe (feedback, spaced repetition, good modelling of problem solving and metacognition, etc) that can point to whether a teacher is or isn’t on the right track.

Further to this point, I’ve seen people quoting this paper (1) recently as suggesting that “We can’t tell good teaching when we see it” (see tweet here) but in this paper they used ‘thin slicing’, only showing assessors less than 5 minutes of footage from teachers’ classrooms. My response to this would have been ‘Teaching takes place in time, but learning takes place over time (John Mason).’ It makes sense that it’s impossible to tell the quality of a teacher from such a small clip, but that doesn’t necessarily mean that accurate assessments can’t happen in less than 30 observations. I feel that, personally, by observing a teacher in 2 to 3 lessons I can get a pretty good feel for what they’re like, as well as identify some of their key strengths and areas for improvement.

If you’ve got time, I’d love to know where this original stat came from and your take on assessing teachers through observations more generally.

Hope this finds you well Heather and I hope to hear back. 

All the best.

Ollie.

Ref 1: Strong, M., Gargani, J., & Hacifazlioğlu, Ö. (2011). Do we know a successful teacher when we see one? Experiments in the identification of effective teachers. Journal of Teacher Education, 62(4), 367–382.

Heather replied (shared here with Heather’s permission).

 

Hi Ollie.

Thanks for your question about how many observations are necessary. It really depends upon the purpose for use.

1. If the use is teacher professional development. I wouldn’t worry too much about score reliability if the observations are used for informal/growth purposes. It’s much more valuable to have teachers and observers actually processing the instruction they are seeing, and then talking about it, than to be spending their time worrying about the “right” score for a lesson.

That principle is actually the basis for our own coaching program, which we built around our observation instrument (the MQI):

http://mqicoaching.cepr.harvard.edu

The goal is to have teachers learn the MQI (though any instrument would do), then analyze their own instruction vis-a-vis the MQI, and plan for improvement by using the upper MQI score points as targets. So for instance, if a teacher concludes that she is a “low” for student engagement, she then plans with her coach how to become a “mid” on this item. The coach serves as a therapist of sorts, giving teachers tools, cheering her on, and making sure she stays on course rather than telling the teacher exactly what to do. During this process, we’re not actually too concerned that either the teacher (or even coach) scores correctly; we do want folks to be noticing what we notice, however, about instruction. A granular distinction, but one that makes coaching much easier.

2. If the use is for formal evaluation. Here, score reliability matters much more, especially if there’s going to be consequential decisions made based on teacher scores. You don’t want to be wrong about promoting a teacher or selecting a coach based on excellent classroom instruction. For my own instrument, it originally looked like we needed 4 observations each scored by 2 raters (see a paper I wrote with Matt Kraft and Charalambos Charalambous in Educational Researcher) to get reliable scores. However, my colleague Andrew Ho and colleagues came up with the 6 observations/5 observer estimates from the Measures of Effective Teaching data:

http://k12education.gatesfoundation.org/wp-content/uploads/2015/12/MET_Reliability-of-Classroom-Observations_Research-Paper.pdf

And looking at our own reliabliity data from recent uses of the MQI, I tend to believe his estimate more than our own. I’d also add that better score reliability can probably be achieved if a “community of practice” is doing the scoring — folks who have taken the instrument and adapted it slightly to their own ideas and needs. It’s a bet that I have, but not one that I’ve tested (other than informally).

The actual MQI instrument itself and its training is here:

http://isites.harvard.edu/icb/icb.do?keyword=mqi_training

We’re always happy to answer questions, either about the instrument, scoring, or the coaching.

Best,

Heather

I hope that Professor Hill’s thoughtful take on assessing teachers through observations can be taken into consideration as decisions are made regarding this new policy.

Edit: Since posting this I’ve had this document brought to my attention which, helpfully, provides some background to this current policy push. It also provides some guidance as to what such performance assessments could look like. 

 

ERRR #009. Andrew Martin, Load Reduction Instruction, Motivation and Engagement

Listen to all past episodes of the ERRR podcast here.

Our guest this month is Professor Andrew Martin. Andrew, is Scientia Professor, Professor of Educational Psychology, and Co-Chair of the Educational Psychology Research Group in the School of Education at the University of New South Wales. Andrew specialises in motivation, engagement, achievement, and quantitative research methods. He is also Honorary Research Fellow in the Department of Education at the University of Oxford, Honorary Professor in the School of Education and Social Work at the University of Sydney, Fellow of the American Psychological Association, and the list goes on.

Although the bulk of his research focuses on motivation, engagement, and achievement, Andrew is also published in important cognate areas such as Aboriginal/Indigenous education, gifted and talented, academic resilience and academic buoyancy, adaptability, personal bests, pedagogy, parenting, and teacher-student relationships.

Andrew’s research also bridges other disciplines through assessing motivation and engagement in sport, music, and work. Based on sole and first authorships, Andrew placed 1st in the most recent International Rankings of the Most Published Educational Psychologists. He has written over 250 peer reviewed journal articles, chapters, and papers in published conference proceedings, 3 books for parents and teachers (published in 5 languages), and more!

Andrew has been listed in The Bulletin magazine’s ‘SMART 100 Australians’ and was one of only three academics judged to be in the Top 10 in the field of Education in Australia. His PhD was judged the Most Outstanding Doctoral Dissertation in Educational Psychology by Division 15 of the APA and also judged the Most Outstanding PhD in Education in Australia by the Australian Association for Research in Education.

Andrew really knows how to make an impact and it’s an inspiring story that we hear, in this episode, about how he went from a less-than academically focussed teenage boy, to the current heights of his academic career.

The paper that Andrew nominated for this episode of the ERRR was entitled ‘Using Load Reduction Instruction (LRI) to boost motivation and engagement’. The central concept of this paper, Load Reduction Instruction, based upon Cognitive Load Theory, offers a basis for critiquing different instructional approaches and helps us to move beyond simplistic comparisons of ‘traditional’ vs. ‘progressive’, or ’didactic’ vs. ‘diologic’ teaching techniques. The LRI approach enables both direct and guided discovery approaches to be considered effective, and provides a structured framework for determining ‘under what conditions each approach is most suitable. It’s a cracker of an episode and covers some of the topics that have most powerfully shaped my own teaching journey to date.

Links mentioned during the interview

  • Exciting as yet unpublished report linking LRI with Motivation and Engagement. It’s… as yet unpublished! But I believe the title (once out) will be: The Load Reduction Instruction Scale (LRIS): Examining Psychometric Properties among High School Students
  • Robert Slavin and the What Works Clearing House (See also ‘Evidence for ESSA‘)

TOT018: Teaching Critical Thinking, Rewards and Motivation + More twitter takeaways!

Teacher Ollie’s Takeaways is a weekly-ish post (and sometimes a podcast!) bringing together some of the fascinating things that Ollie read throughout the week-ish! Find all past posts of Teacher Ollie’s Takeaways here

A detailed look at the literature on the use of rewards and their relationship to intrinsic motivation

How, to use, commas!,

Aboriginal Astronomy

How can we teach Critical Thinking Skills?

TOT017: Teacher expectations and cultural narratives…+ More Twitter takeaways

Teacher Ollie’s Takeaways is a weekly-ish post (and sometimes a podcast!) bringing together some of the fascinating things that Ollie read throughout the week-ish! Find all past posts of Teacher Ollie’s Takeaways here

Is it worth writing learning intentions on the board? Has data been collected on this???

No data…

Following this Shirley replied, but I can’t for the life of me find the reply! It said that what’s important isn’t that learning intentions are written on the board for students, but that they’re clear in the heads of teachers. Also, writing or not writing learning intentions on the board isn’t something that’s been empirically tested in an RCT. Nor will it probably ever be…

What lies at the heart of the Prog vs. Trad debate?

Link: http://www.rupertwegerif.name/blog/defining-dialogic-education

Reading Analogue time, a quick strategy

Quick clip: Bjork speaking on desirable difficulties

Discussing Race in the Classroom. Some Prompts

Career Aspirations of young people, and how they change

How the narratives we have surrounding students impacts their education

Link to the article.

TOT016: Cognitive Science for Teaching Summary…+ more Twitter takeaways

Teacher Ollie’s Takeaways is a weekly-ish post (and sometimes a podcast!) bringing together some of the fascinating things that Ollie read throughout the week-ish! Find all past posts of Teacher Ollie’s Takeaways here

Building Community how-to

Cognitive Science for Teaching. A fantastic summary!

Desirable Difficulty and the Bjorks’ take on cog sci for learning

TOT015: Drinking alcohol aids learning + more twitter takeaways

Teacher Ollie’s Takeaways is a weekly-ish post (and sometimes a podcast!) bringing together some of the fascinating things that Ollie read throughout the week-ish! Find all past posts of Teacher Ollie’s Takeaways here

Presentation advice from the greats!

Students need saving, and teachers are the ones to do it…not

Drinking alcohol enhances learning…wha???

Google doc addon that emails parents

An analysis of meta-analysis

Task switching costs… An activity to demonstrate in the classroom

Great post on spacing maths content

Present new material in small steps with student practice after each step: How’s it look?

Embed from Getty Images

The second recommendation in Rosenshine’s ‘Principles of Instruction is “Present new material in small steps with student practice after each step”. The basis for this recommendation is the fact that working memory is limited and, for learning to occur it’s important to avoid overloading working memory. But that isn’t the focus of this post. In this post I just wanted to share what ‘new material in small steps with student practice after each step’ can look like in the classroom.

As a rule of thumb, the longer a teacher talks for the more likely they are delivering sufficient information to overload their students’ working memory. As I reflected upon this point, prompted by Craig Barton’s in-depth interview with Kris Boulton recently, I found myself thinking, ‘I wonder how long I talk for?’ It was time to collect some data.

Next lesson I split my notebook into three columns ‘explain’, ‘student work’, and ‘check solution’ (I always teach my maths lessons in an ‘I do’ then ‘You do’ format, then go over the solutions as a class), then I got to recording! First class I got distracted and fell off the timing bandwagon (first half of the page) but second class I remembered to stay on task and that whole class (90 mins) is recorded in the image below (red box).

To set the scene, I wanted students to be able to answer the exam question presented by the end of the lesson. This required them to be able to go from a transition diagram and an initial state matrix to the result after multiple periods with or without the addition of extra units each period, as well as determining the result of such transitions ‘in the long run’, and working backwards in such a relation. I split this up into the following sub-steps for the purposes of instruction.

  • Constructing a transition matrix from a transition diagram.
  • Applying a transition diagram to interpret a transition
  • Applying a transition matrix to interpret change after one transition
  • Understanding transition matrices as recurrent relations (And results after multiple periods with a formula)
  • ‘In the long term’: Steady state solutions to Transition matrices
  • Results after multiple periods (using brute force, that means with a calculator)
  • Transition matrix modelling when the total number of units changes.
  • Working backwards in matrix multiplications

20813172_10214166499896038_1281376121_o

The astute observer will note that the total time adds up to about 60 mins. The additional time was taken up with approx. 20 mins of revising previous content and 10 mins talking about an upcoming assessment and doing a ‘brain break’.

Below is the lesson as I presented it, with the timing for each segment added in italics (images weren’t in the original as students had all questions in front of them. I added them for readers here)

I found it really valuable to look at the timing of my lessons in this level of detail. I’d love to know if it’s prompted any similar reflections for you.

References:

Rosenshine, B. (2012). Principles of Instruction: Research-Based Strategies That All Teachers Should Know. American Educator, 36(1), 12.