Expertise and Instruction

Overview

Questions

• How does a user’s expertise affect their use of genAI tools?
• What can Instructors do to discourage learners from relying on chatbots to solve exercises during a workshop?

Objectives

• Explore the influence of expertise on the use of genAI tools and the quality and interpretation of their output.
• Recognise the importance of a user’s mental model in guiding and taking responsibility for output generated by an AI model.

Many (research) software engineers have concluded that genAI tools are now sufficiently effective and reliable for code generation that they no longer need to do much programming by hand. These experts report that they have found ways of working with genAI tools that leverage their expertise.

Despite widely publicised claims that anybody can now code using genAI, novices are unlikely to achieve equivalently good results. Novices lack a mental model for conceptualising and articulating what they want to achieve, how it could be achieved, how it could go wrong, etc.

The simplified model of skill acquisition presented in Instructor Training.

Of course, these novices may not expect to achieve the same results as an expert. They may be motivated to use genAI tools because they feel that it will help them produce better results, faster than they could do on their own.

Conceptual understanding and knowledge of relevant terminology are essential for learners to be able to articulate what they want to achieve, and to process the responses they receive. This is comparable to how the use of relevant terminology in search terms is likely to produce more relevant internet search results. But the model of skill acquisition tells us that the barrier is more than a limited vocabulary: one characteristic of a novice is that they do not know what they do not know. This means that a novice cannot articulate what they want to do, because they may not yet know what it is.

To keep improving the results they receive, and take responsibility for AI-generated outputs, learners must build their own expertise.

Challenge

Copying/pasting Exercises

One challenge that Instructors now report encountering in workshops, is that some learners will choose to copy and paste the text of an exercise into a chatbot, and the code it produces to “solve” the exercise. What are some ways that you might notice that this is happening in a workshop? Suggest one strategy that an Instructor could employ to discourage or prevent learners from doing this.

Show me the solution

Indications of an AI-generated solution include:

• Use of functions, syntax, language features, etc that have not yet been introduced in the workshop.
• Complex code (more on this later).
• An abundance of comments explaining the lines of code.
• Inability for the learner to explain how the solution works.

Strategies suggested by Trainers include:

• Explain the importance of exercises for learning (transfer from working to long-term memory, metacognition)
• Celebrate the successful, manual completion of an exercise.
• Ask learners to explain their solutions.
• Follow up an exercise by asking learners to make small modifications to change the behaviour of their solutions.

(Note to Trainers: if you receive more suggestions from participants while delivering this bonus module, please open a pull request to add them to the curriculum.)

Key Points

• A user needs a working mental model of a domain in order to articulate what they want to achieve and to evaluate the results they get from an AI tool.
• Instructors should encourage learners to develop their own expertise and avoid giving up their agency to genAI tools.