Building Skill with Practice

Last updated on 2026-06-29 | Edit this page

Estimated time: 35 minutes

Overview

Questions

  • What are the implications of genAI tool use for guided practice among novice learners?
  • How good are current genAI tools at producing solutions to the exercises in Carpentries lessons?

Objectives

  • Explore the capabilities of genAI tools in the context of Carpentries lessons.
  • Discuss how genAI tool use could impact skill acquisition for novice learners.
  • Discuss how widespread use of genAI tools for code generation might affect the relative importance of the skills we teach.

Guided practice is a central principle in Carpentries Instructor Training. We provide learners with guided practice during a workshop and equip them with sufficient expertise to continue guiding their own practice after they leave. Novices need this initial guided instruction because they do not have sufficient expertise to know what they do not know or to articulate what they want to do.

GenAI chatbots may be unhelpful to novices who want to learn a new topic because:

  • They may produce responses that do not account for the learner’s lack of expertise.
  • They often over-cater to the prompt they were given: providing additional information, taking additional steps, and suggesting next steps that may be jumping too far ahead.
  • They can reliably generate correct snippets of code, including solutions to most if not all of the exercises in our lessons.

However, novices are unable to assess the quality of a generated solution. Are there other ways that they could be encouraged to use genAI tools, to better support their learning?

Challenge

How Good is GenAI at Solving Carpentries Exercises?

Choose one of the suggested lessons in the spoiler below. Try passing the text of some of the exercises from your chosen lesson to duck.ai or another genAI chatbot of your choice.

  • How many exercises from the lesson was the chatbot able to solve?
  • Did you find any exercises that the chatbot could not solve?

When reporting your results, include details of any custom settings you used on the chatbot. How might those have affected the output you received? Similarly, did you need to provide any additional information to the chatbot? E.g. upload example data, paste extra content from elsewhere in the episode, etc.

General notes:

  • At the time of writing, some chatbots do not support uploading of csv files (e.g. duck.ai). For lessons with csv datasets, you could try describing the dataset, pasting in the first several lines of data, or using a chatbot that supports csv uploads (e.g. Claude.ai).
  • Some lessons don’t work well for this exercise because they require more specialized set-up than there is time available for this exercise. However, many exercises in these lessons can still be completed by genAI tools once the setup is accomplished. If you have extra time, feel free to test them, and think about whether an additional steps you needed to take could realistically be expected of learners in a workshop.

The following lessons work well for this exercise:

The following lessons don’t work as well for this exercise:

The above exercise has demonstrated that chatbots will often provide correct responses to the types of challenges we pose to learners during our workshops. We might also wonder how consistent these responses are - in other words, if all the learners in your workshop enter the same prompt, will they reliably receive the same answer? Does that depend on whether everyone is using the same genAI tool?

Challenge

How Consistent are GenAI Answers?

Copy the following prompt into your genAI chatbot of choice.

MARKDOWN 

In bash how would you write a loop that echoes all 10 numbers from 0 to 9?

Share your results in the Etherpad, along with information about the tool and model you used.

  • How many different responses were provided?
  • How many of the responses were correct?
  • Were the correct responses all useful for novices?

For comparison, the solution provided in the Software Carpentry Unix Shell lesson is:

BASH 

$ for loop_variable in 0 1 2 3 4 5 6 7 8 9
> do
>     echo $loop_variable
> done

Some responses obtained might be:

BASH 

$ for i in {0..9}; do
>    echo "$i"
> done

The two examples below are quite similar to one another, but differ in whether quotation marks are placed around the variable.

BASH 

$ for ((i=0; i<10; i++)); do
>    echo "$i"
> done

BASH 

$ for ((i=0; i<10; i++)); do
>  echo $i
> done

The example below will only work if the learner has initiated the bash program (or zsh or similar) in their terminal.

BASH 

$ i=0
> while [ $i -lt 10 ]; do
>  echo $i
>  ((i++))
> done

BASH 

$ for i in $(seq 0 9); do
>  echo $i
> done

The example below assumes the learner is writing a shell script, rather than working interactively within the terminal.

BASH 

#!/bin/bash

for i in {0..9}; do
	echo "$i"
done

We’ve demonstrated that chatbots are able to generate correct solutions for many Carpentries-style exercises, but that the responses generated are not consistent. Not only are they not consistent, but each example above introduces new pieces of syntax that haven’t been covered in the lesson yet. We will discuss more the challenges this can cause with cognitive load later in this training.

Discussion

What Does This Change?

How does the ability of genAI tools to generate source code influence the relative importance of the following different skills to a novice learning to program?

  • Writing a syntactically valid for loop.
  • Tracing the order of execution.
  • Choosing descriptive names for variables.
  • Debugging, e.g., interpreting error messages.

Referring back to your answers, what is one approach you could take as an Instructor during a workshop to (de)emphasise the importance of one or more of the skills listed above?

Discussion

How Much Expertise Do You Need?

Prompt your chatbot with a coding task. You can choose an exercise from a lesson (Carpentries or otherwise), or come up with your own task. Consider the output produced: How much knowledge of programming concepts do you think you would need to have to be able to understand or validate what the output is doing?

On this chart (FIXME: add link to tldraw chart), mark your result based on your judgement of how complex your coding task was and how much expertise would be needed to understand the response.

Key Points
  • Novices using chatbots in a workshop are less likely to benefit from guided practice.
  • Current chatbot tools can produce functioning solutions to many exercises in Carpentries lessons.