Stay on Target

Last updated on 2024-02-19 | Edit this page

Overview

Questions

  • How can you measure learners’ progress towards your lesson objectives?
  • Why is it important to identify misconceptions as early as possible?
  • Why should we create assessments before we have written the explanatory content of our lesson?

Objectives

After completing this episode, participants should be able to…

  • Describe the importance of regular assessment while a lesson is being taught.
  • Design assessments to identify the misconceptions learners might have during your lesson.
An overview of the iterative process of lesson design and development, adapted from Nicholl's five phases, with step 2, 'Design assessments for these outcomes' highlighted.
In the next two episodes, we will design assessments to measure learners’ attainment of the objectives we defined previously.

As we have seen previously, defining objectives for a lesson (or a teaching episode) can help to focus your content on the most important learning outcomes and outline the scope of your lesson project. The goal of the remaining steps of lesson development is to ensure that what learners learn from following your lesson matches its defined objectives as closely as possible. To do so, you need to develop assessments to monitor progression towards your learning outcomes.

Assessments

In order to measure progress and evaluate if and what learning occurred - we use various types of assessments:

  • summative assessments - used to verify whether learners achieved the stated learning objectives after instruction.
  • formative assessments - used to detect changes in learner performance during instruction, to provide feedback and insight into the learners’ developing mental models of the topic taught and to identify any old or developing misconceptions.

Summative assessments sum up what learning has been achieved after training (e.g. via exams). They give valuable data about learning attainment by individuals and entire cohorts but are not used to guide further progress. They may not be as suitable for short courses, but may be necessary for those that give marks/grades or certificates of completion.

Formative assessments are applied throughout a course and with several different purposes: they provide a way to move new information from working memory to long-term memory; they can inform instructors’ decisions about how to modify instruction to better promote learning; they also inform learners about changes they may need to make to improve their learning. Ideally, they should be used often (e.g. after every 15-20 minutes of teaching), providing opportunities to instructors to change pace and refocus learners’ attention. For short courses, formative assessments are usually more valuable and easier to implement in practice than summative assessments - they need not be complex or time-consuming, just informative enough about learning for both instructors and learners.

The most effective way to test learner understanding is to do such assessments in class - they engage all learners and allow instructors to check learners’ confidence with the content and its delivery, can help you deal with any potential misunderstandings as soon as they arise, and maximise the value of workshop for everyone. Such formative assessments also help with metacognition - the awareness a learner has that they are succeeding in learning something new.

Any instructional tool that generates feedback and is used in a formative way to check for learners’ understanding can be described as “formative assessment”. For example,

  • reflection at the end of a session to help process learning - e.g. asking learners to write down things they learned, things they want to know more about and any questions they still have
  • concept maps and diagrams - asking learners to reflect by drawing/labeling a concept map/diagram or writing down a list of new concepts and skills they’ve learned and (optionally) how they relate to one another or connect with previous knowledge
  • checking in - gauging learners’ satisfaction and understanding using agreed signals (e.g. raising different coloured post-it/sticky notes or Zoom reactions to indicate that the pace is too fast/slow, that they completed/have not completed an exercise).
  • think, pair, share - learners think about an answer to a question, pair up with a classmate to discuss their answer, and then share out the consensus they came to with the class.

Many other formative assessment tools can be found in Briggs’ list of “21 ways to check for student understanding” or Edutopia’s “56 Examples of Formative Assessment”.

Exercises are one important type of formative assessment. We will now have a look into exercises that perform misconception checks and ask students diagnostic questions; we will cover a few other types of exercises that help with retaining new knowledge in one of the later episodes.

Detecting Misconceptions

Detecting and correcting misconceptions and fixing learners’ incorrect/broken mental models is as important as presenting your learners with new knowledge and correct information. Why is it important to identify misconceptions as early as possible? When mental models are broken, learning can occur slower than you might expect1. The longer a prior incorrect model is in use, and the more extensively it has to be “unlearned”, the more it can actively interfere with the incorporation of the new correct knowledge (since it will contradict the misconceptions already present in the mental model).

Example Misconceptions

Everyday Life Example

An example misconception from everyday life (borrowed from The Carpentries Instructor Training) is, knowing that a bigger object (more volume) pushes out more water out of a basin than a smaller object (less volume), assuming that the similar principle will apply for objects of different weights. Surprisingly, heavier objects of the same volume as lighter ones will not push out more water. Four sets of words inside rectangles, with labeled arrows connecting them. "Ball" is at the left, and "Water", at right. "Big Ball" and "Small Ball" are stacked vertically between them. Arrows from "Ball" are labeled with "Heavy Ball" and "Light Ball", and arrows to "Water" are labeled as "Pushes out MORE" and "Pushes out "LESS. There is a red "X" over the arrows labeled "Pushes out MORE" and "Pushes out LESS"

Coding Example

Another misconception, sometimes encountered by people who are learning programming languages such as R and Python having used spreadsheets, is that variables in programs calculated by referencing other variables in formulas will retain the connection and be updated automatically when the referenced variables change. Variables in R or Python programs can use other variables in calculations (e.g. a = b + 3), but they do not behave like cells in spreadsheets and updating the value of variable b after the previous assignment will not change the value of a.

Misconceptions often arise around topics of copyright and licencing of work/content (e.g. data, software, written material). Copyright allows a creator to state that they own the work they have created. This declaration is optional - even if the creator does not explicitly assert it, copyright of the work exists from the moment of creation. A licence is a legal document which sets down the terms under which the creator is releasing what they have created for others to use, modify, extend or exploit. If this information is not provided, some will not reuse the work at all while others will assume they are free to do whatever they want with the work (as there is nothing to say that they cannot). This former is the correct assumption, even if this was not the intention of the copyright owner.

Exercise: misconceptions (5 minutes)

What are the common misconceptions learners can have about the topic of your lesson? How might you identify that misconception in your learners while they follow your lesson? Share your answer in the collaborative notes document.

Hint: Try thinking about related or common tools the learners might know and how applying that prior knowledge might lead to a misconception with the topic you are teaching.

Multiple Choice Questions (MCQs)

Multiple choice question (MCQ) exercises are types of a formative assessment that can help you target anticipated misconceptions. When designed carefully, each incorrect answer in a MCQ will have diagnostic power and provide valuable insight into how a mental model is broken. For example, suppose we are teaching children multi-digit addition. An example of a well-designed MCQ (borrowed from The Carpentries Instructor Training) in this case could be:

MCQ: What is 27 + 15?

  1. 42
  2. 32
  3. 312
  4. 33

The correct answer is 42, but each of the other answers provides a valuable insight:

  1. they do not understand the concept of a carry and are throwing it away completely
  2. they understand the concept of a carry and know that they cannot just discard the carried ‘1’, but do not understand that it is actually a ten and needs to be added into the next column - they are treating each column of numbers as unconnected to its neighbours.
  3. they understand that they need to carry ‘1’ but are adding it to the wrong column.

Their diagnostic power means that each of the wrong choices helps an instructor figure out precisely what misconceptions learners had adopted and in which ways their mental models are broken. As a result the instructor may decide to review a particular concept or change the pace of instruction. At the same time, learners get feedback about what they have misunderstood and what they need to focus their study efforts on - we call this guided practice.

Choosing Plausible Distractors

When using a multiple choice question for formative assessment, the incorrect answers you provide as options are at least as important as the correct answer because they offer the most useful insight into the mental model your audience is building. But choosing incorrect answers can be difficult, especially early in a lesson where you are more likely to encounter misconceptions that learners have arrived with as opposed to those they have picked up while following the lesson.

Prior Knowledge and Plausible Distractors

When identifying plausible distractors, it is essential to consider the prior knowledge you expect learners to have of your lesson topic.

The plausible distractors in the MCQ example above are chosen on the assumption that learners have been taught addition by “carrying” numbers: the incorrect answers are designed to diagnose misconceptions associated with that method.

However, as pointed out by Maneesha Sane, if a learner has been taught to add numbers by another method, e.g. first rounding them up or down to values that are more easy to remember and combine (e.g. 27 to 30), they might have misconceptions that cause them to arrive at an answer that is incorrect but not represented in the MCQ options.

Example MCQs

Coding MCQ

Thinking back of the misconceptions of how values are assigned, referenced and updated in programming languages, here is an example MCQ that can probe learners for such misconceptions.

MCQ: Look at the following 3 assignment statements in Python.

PYTHON 

n = 300
m = n
n = -100

What is the result at the end of the above assignments?

  1. n = 300 and m = 300
  2. n = -100 and m = -100
  3. n = 200 and m = 300
  4. n = -100 and m = 300

The correct answer is d., while plausible distractors identify different misconceptions:

  1. they understand the value held by n is now also held by m but do not understand the value of n has been updated since the initial assignment (e.g. treating it as a constant).
  2. they do not understand n and m are separate variables/containers for values (which can hold the same values) but rather assume that m is referencing variable n instead of copying its value at the time of assignment (e.g. behaving as a reference to another cell in a spreadsheet).
  3. they think that the reassignment for n actually subtracts 100 from the original value.

An example MCQ to check on people’s misconceptions around licencing and reusing other people’s work could be designed as follows.

MCQ: Which of the following statements are true and which are false?

  1. I don’t need permission because I am only using the copyrighted work in educational or non-profit purposes
  2. I should always know the licence of any code, data, libraries, pictures or other work that you reuse or redistribute
  3. Since I’m planning to give credit to the authors who created the work I reuse, I do not have to worry about or need permission
  4. Material I obtain from the Internet is publicly accessible so no explicit permission is required
  5. The work I want to use does not have a copyright notice on it, so it’s not protected by copyright and I’m free to use it

The correct answers are as follows:

  1. False - you always need an explicit permission from the creator to use their work.
  2. True - you should make sure that you have the permission for all the work that you are reusing, modifying or sharing.
  3. False - if you give credit to a work’s owner, that only means you are not plagiarising other people’s work and claiming it as your own, however that does not mean that you have the permission to use it.
  4. False - publicly accessible work is not the same the work in the public domain. The owner explicitly must put their work in the public domain but attaching the appropriate licence to it, before you can freely reuse it.
  5. False - the use of copyright notice is optional as copyright exists implicitly from the moment the work is created.

To help identify plausible distractors, you can think about problems or questions from previous training events and what people struggled with, think about your own misconceptions in the past, or ask colleagues about their experiences. It is important to consider expert awareness gap, the phenomenon where experts in a topic forget what it is like not to have a good mental model of it. Observing how others learn your topic and asking colleagues to review lesson content as you design it can help mitigate this.

You should aim to create all your assessments before you have written the explanatory content of your lesson (recall Nicholl’s backward design). These assessments will guide your lesson design process by knowing exactly which knowledge you’d expect from your learners at any point in the lesson.

Exercise: designing a diagnostic exercise (20 minutes)

Create a multiple choice question (MCQ) that could be used in your lesson, to detect the misconception you identified above. As well as the correct answer, include 1-3 answer options that are not obviously incorrect (plausible distractors) and have diagnostic power i.e. each incorrect answer helps you pinpoint the exact misconception carried by the learner. Write down what misconception is being tested for each incorrect answer.

Share your MCQ in the collaborative notes document.

Exercise: reviewing formative assessments (10 minutes)

(this exercise will only work if participants have sufficient knowledge of their partner’s topic)

The Trainers will group you into pairs.

Review the MCQ designed by your partner. When providing feedback, try to answer the following questions:

  • Is the question clear and easy to understand? Could the wording be improved in some way?
  • Are the incorrect answers to the MCQ plausible distractors?
  • Do the incorrect answers provide diagnostic power, to help an Instructor identify the misconception the learner has?
  • Are there any incorrect answers missing i.e. are there other misconceptions that could be detected with this MCQ?

Share your feedback in the collaborative notes document.

Key Points

  • The goal of lesson development is to ensure that the attained curriculum matches the intended curriculum as closely as possible.
  • Assessments are a way to determine whether the objectives you defined for the lesson have been reached.
  • Formative assessment happens during teaching and provides feedback both to an instructor and a learner - about progress and whether learning of new concepts occurred but also about any misunderstandings and misconceptions which can hinder further learning.
  • It is important to detect misconceptions as early as possible and formative assessments (such as multiple choice questions) can help us with this.

  1. See chapter 1, How Does Students’ Prior Knowledge Affect Their Learning?, of Ambrose et al. 2010.↩︎