Building Skill with Practice

Overview

Questions

• What are the implications of genAI tool use for guided practice among novice learners?
• How good are current genAI tools at producing solutions to the exercises in Carpentries lessons?

Objectives

• Explore the capabilities of genAI tools in the context of Carpentries lessons.
• Discuss how genAI tool use could impact skill acquisition for novice learners.
• Discuss how widespread use of genAI tools for code generation might affect the relative importance of the skills we teach.

Guided practice is a central principle in Carpentries Instructor Training. We provide learners with guided practice during a workshop and equip them with sufficient expertise to continue guiding their own practice after they leave. Novices need this initial guided instruction because they do not have sufficient expertise to know what they do not know or to articulate what they want to do.

GenAI chatbots may be unhelpful to novices who want to learn a new topic because:

• They may produce responses that do not account for the learner’s lack of expertise.
• They often over-cater to the prompt they were given: providing additional information, taking additional steps, and suggesting next steps that may be jumping too far ahead.
• They can reliably generate correct snippets of code, including solutions to most if not all of the exercises in our lessons.

However, novices are unable to assess the quality of a generated solution. Are there other ways that they could be encouraged to use genAI tools, to better support their learning?

Challenge

How Good is GenAI at Solving Carpentries Exercises?

Choose one of the suggested lessons in the spoiler below. Try passing the text of some of the exercises from your chosen lesson to duck.ai or another genAI chatbot of your choice.

• How many exercises from the lesson was the chatbot able to solve?
• Did you find any exercises that the chatbot could not solve?

When reporting your results, include details of any custom settings you used on the chatbot. How might those have affected the output you received? Similarly, did you need to provide any additional information to the chatbot? E.g. upload example data, paste extra content from elsewhere in the episode, etc.

Suggested Lessons

General notes:

• At the time of writing, some chatbots do not support uploading of csv files (e.g. duck.ai). For lessons with csv datasets, you could try describing the dataset, pasting in the first several lines of data, or using a chatbot that supports csv uploads (e.g. Claude.ai).
• Some lessons don’t work well for this exercise because they require more specialized set-up than there is time available for this exercise. However, many exercises in these lessons can still be completed by genAI tools once the setup is accomplished. If you have extra time, feel free to test them, and think about whether an additional steps you needed to take could realistically be expected of learners in a workshop.

The following lessons work well for this exercise:

• Data Carpentry:
  • Data Analysis and Visualization in R for Ecologists
  • Data Analysis and Visualization in Python for Ecologists
  • Análysis y visualización de datos usando Python
  • Project Organization and Management for Genomics
  • Introduction to R for Geospatial Data
• Library Carpentry:
  • Introduction to Regular Expressions
  • The Unix Shell
• Software Carpentry:
  • The Unix Shell
  • Version Control with Git
  • Programming with Python
  • Plotting and Programming in Python - after the first episode.
  • Programming with R
  • R for Reproducible Scientific Analysis
• HPC Carpentry:
  • HPC Workflow Management with Snakemake

The following lessons don’t work as well for this exercise:

• Data Carpentry:
  • Foundations of Astronomical Data Science
  • Data Organization in Spreadsheets for Ecologists
  • Data Organization in Spreadsheets for Social Scientists]
  • Data Cleaning with OpenRefine for Ecologists
  • Data Cleaning with OpenRefine for Social Scientists
  • Introduction to the Command Line for Genomics
  • Data Wrangling and Processing for Genomics
  • Introduction to Cloud Computing for Genomics
  • Introduction to Geospatial Concepts
  • Introduction to Geospatial Raster and Vector Data with R
• Library Carpentry:
  • Tidy data for librarians
  • OpenRefine
  • Introduction to Git
• HPC Carpentry:
  • Introduction to High-Performance Computing

The above exercise has demonstrated that chatbots will often provide correct responses to the types of challenges we pose to learners during our workshops. Previous experience has also shown us that the tools are quite consistent in the solutions they produce to these relatively small and introductory exercises, but some variation in responses can be assumed.

Discussion

What Does This Change?

How does the ability of genAI tools to generate source code influence the relative importance of the following different skills to a novice learning to program?

• Writing a syntactically valid for loop.
• Tracing the order of execution.
• Choosing descriptive names for variables.
• Debugging, e.g., interpreting error messages.

Referring back to your answers, what is one approach you could take as an Instructor during a workshop to (de)emphasise the importance of one or more of the skills listed above?

We’ve demonstrated that chatbots are able to generate technically correct solutions for many Carpentries-style exercises. However, many of the code solutions generated introduce new pieces of syntax that haven’t been covered in the lesson yet, and may be accompanied by explanatory text that invokes concepts and terminology that learners have not yet encountered. We will discuss this challenge in the next episodes.

Key Points

• Novices using chatbots in a workshop are less likely to benefit from guided practice.
• Current chatbot tools can produce functioning solutions to many exercises in Carpentries lessons.