The Carpentries Instructor Training Assessment Report

1 Executive Summary and Key Findings

• The Carpentries’ Instructor Training Program:

  • trains people to teach The Carpentries curricula through an active training that includes fundamentals of cognitive psychology, mechanics of a positive learning environment, and content specific to The Carpentries community.

• The purposes of this report are to:

  • provide an update on the successes, challenges, and growth of the Instructor Training program to the Executive Council, Trainers Leadership, Trainers, and broader Carpentries community; and

  • help the Instructor training team, Core Team, Executive Council, and Trainers Leadership identify trends and opportunities that can drive continuous program improvement.

• This report relies on two primary sources of data:

  • pre and post Instructor Training Surveys, and
  • programmatic data from The Carpentries AMY database.

Instructor Training Outcomes

• Trainees complete surveys before and after attending Instructor Training about their experience during the training and their skills, knowledge, and motivation.

• Trainees overwhelmingly report that Instructor Training is welcoming and worth their time. They also provide strong ratings of Trainers’ communication skills, considerateness, and enthusiasm. This positive feedback is a testament to the outstanding efforts of Carpentries Instructor Trainers.

• Trainees reported increased skills and knowledge after Instructor Training, suggesting the training effectively meets our goal of increasing familiarity with evidence-based teaching practices.

• A majority of trainees (on average 74% across topic areas) report an increase in motivation to learn about teaching practices.

• We ask trainees how likely they are to recommend Instructor Training to a friend or colleague. Over the last five years, average recommendation scores have been between 8.43 and 9 (out of 10). These scores suggest that not only are trainees satisfied with their experience, but they are likely to recommend it to others.

• Most trainees hear about Instructor Training through word-of-mouth (81%), making recommendation scores particularly important.

Trainee Demographics

• Trainees provide demographic information before attending Instructor Training which helps us better understand who we are teaching.

• A majority of trainees (62%) are located in the United States, followed by the United Kingdom, Canada, and Australia. Other countries in the top ten include European countries as well as South Africa and Argentina.

• The highest percentage of trainees come from the fields of Computer Science (19%), Bioinformatics/Genomics (16%), and Library Science (12%) which likely speaks to the origins of the Software, Data Carpentry, and Library Carpentry communities.

• The highest percentage of our trainees are graduate students (26%) followed closely by research staff (24%). We also have a sizeable portion of faculty (16%) and postdoctoral researchers (13%). We have smaller numbers of trainees that come from government and industry.

• Because of regulations, we only collect ethnic identity information from US-based trainees. Among US trainees from 2018 to 2020, the greatest number of trainees identify as White (66%), followed by Asian (17%) and Hispanic/Latino(a) (8%).

Outcomes Related to The Carpentries Community

• While one goal of The Carpentries Instructor Training Program is to develop the individual Instructor, another goal is to build The Carpentries community. The following findings are associated with the Carpentries community.

• After attending an Instructor Training event, trainees can choose to complete three checkout exercises: 1) attending a community discussion, 2) completing a teaching demonstration, and 3) contributing to a Carpentries lesson. After these steps, trainees are awarded a badge and are considered certified Carpentries Instructors, qualified to teach any Software, Data, or Library Carpentry workshops.

• Over the past three years (2017 - 2020), the total number of trainees who become certified Instructors has increased from 464 to 625 annually.

• The percentage of trainees who complete certification is generally between 50-60%. This number declined slightly, from 57% to 54% between 2017 and 2019, rising to 55% in 2020.

• In 2019 and 2020, 28% of trainees who attended Instructor Training never began checkout, while 15% completed some but not all steps. Of those steps, trainees participated in a community discussion most often. Far fewer trainees completed a teaching demonstration or shared a lesson contribution.

• After certification, Instructors typically teach their first workshop within six months.

Growth of Instructor Training Efforts

• The Carpentries received a grant from the Chan Zuckerberg Initiative to support growth in our Instructor and Trainer Training Efforts between 2019 and 2022. This section reports on metrics related to that growth.

• In 2019, we certified 10 new Instructor Trainers (“Trainers”) to teach Instructor Training and host teaching demonstrations. In 2020, we trained two cohorts of new Trainers and certified our highest number of Trainers in one year to date, 29. In July 2021, we certified an additional 14 new Trainers.

• In 2019, 1,168 trainees attended Instructor Training. In 2020, the number of trainees declined to 895. As of August, we have trained 717 in 2021.

Conclusion

• Trainees rate their Instructor Training experience as highly positive. They find the training welcoming, learn valuable skills, and leave with increased motivation to learn.

• Areas of potential growth include adding support to ensure that trainees who begin the checkout process have the support they need to complete it, and leveraging the growth in our Instructor Trainer community to expand our training audience. These and other goals must address continuing global unpredictability, supporting the ambitions of our community-members amid the strain of changing workplace and family landscapes.

• The success of The Carpentries Instructor Training program hinges on the strength of our outstanding Trainer community. Trainers create welcoming and inclusive spaces for our trainees to learn. They are knowledgeable, enthusiastic, and kind. Our Trainers contribute thousands of hours to maintaining our curriculum, teaching Instructor Training, hosting teaching demonstrations, and continuing to improve their own teaching skills.

2 Introduction

2.1 Carpentries Instructor Training Program

Carpentries workshops teach researchers the skills and perspectives to work more effectively and reproducibly with data and software. Our workshops use open, collaboratively-developed Data Carpentry, Library Carpentry, and Software Carpentry lessons with materials on topics from data organisation to version control and are taught by volunteer Instructors trained through our Instructor Training program.

The Carpentries’ Instructor Training Program is firmly grounded in educational research. We train people how to teach The Carpentries curricula through an active training that includes fundamentals of cognitive psychology, mechanics of a positive learning environment, and content specific to The Carpentries. Instructor Training is typically a two full-day or four half-day experience. As with our other lessons, our Instructor Training curriculum is open, collaboratively developed, and regularly updated.

The Carpentries Instructor Training has the following goals:

• Introduce trainees to evidence-based best practices of teaching.
• Teach how to create a positive environment for learners at The Carpentries workshops.
• Provide opportunities for trainees to practice and build their teaching skills.
• Help trainees become integrated into The Carpentries community.
• Prepare trainees to use these teaching skills in teaching The Carpentries workshops.

When people learn about Instructor Training, they sometimes anticipate that we will teach content that is not included. Examples of things we do not teach are:

• How to program in R or Python, use git or any other topics taught in Carpentries workshops.
• How to create lessons from scratch (although some principles we teach do apply).

After attending an Instructor Training event, trainees can choose to complete three checkout exercises: 1. Attending a community discussion. The purpose of this step is to introduce trainees to The Carpentries global community as a resource that they can use in the future when they prepare to teach and reflect on a workshop. 2. Completing a teaching demonstration. The purpose of this step is to support trainees in preparing to teach a full workshop and to provide an opportunity to give and receive feedback on a prepared lesson. This also includes feedback from a Carpentries Instructor Trainer and serves as a checkpoint where trainees may be invited to repeat a demonstration in order to ensure that they are fully prepared to implement Carpentries teaching practices. 3. Contributing to a Carpentries lesson or repository. The purpose of this step is to ensure that trainees are prepared to contribute to collaborative maintenance of the lessons they choose to teach in the future. After these steps, trainees are awarded a Carpentries Instructor badge and are considered certified Carpentries Instructors, qualified to teach any Carpentries workshops.

The Carpentries website has more details about how to become an Instructor and join The Carpentries community or become an Organisational Member to grow a local community of Instructors within your organisation.

2.2 Purposes of this Report

The purposes of this report are to:

• provide an update on the growth, successes, and challenges of the Instructor Training program to the Executive Council, Trainers Leadership Group, Trainers, and broader Carpentries community.
• help the Instructor training team, Core Team, Executive Council, and Trainers Leadership Group identify trends and opportunities that can drive continuous program improvement.

3 Methods

The Instructor Training program’s assessment efforts focus on two primary sources of data:

3.1 Pre and Post Instructor Training Surveys

For the Instructor Training outcomes section of this report, we analyse trainees’ perceptions of the Instructor Training and their Instructor Trainers. We also look at how their perception of their skills and knowledge changes from before to after the training.

3.2 AMY Database

For the Community Outcomes Section of the report, we analyse current and historical programmatic Instructor Training information. The Carpentries began systematically recording data from workshops in 2012. These data are stored in our AMY database. Included are the number of and geographic reach of Instructor Training events, the number of trainees and their certification rates, and details on the training of new Instructor Trainers.

4 Instructor Training Outcomes

As mentioned previously, the stated goals of The Carpentries Instructor Training program are to:

• Introduce trainees to evidence-based best practices of teaching.
• Teach how to create a positive environment for learners at The Carpentries workshops.
• Provide opportunities for trainees to practice and build their teaching skills.
• Help trainees become integrated into The Carpentries community.
• Prepare trainees to use these teaching skills in teaching The Carpentries workshops.

To determine whether our Instructor Training program is meeting these goals, we ask all trainees to complete a survey about their skills and experience with Instructor Training curriculum content both before and after the training. We also ask demographic questions as well as their perceptions of the training and their Instructor Trainers. Finally, we ask how likely they are to recommend this Instructor Training to a friend or colleague.

4.1 What Skills Did Trainees Learn? Changes in Perceived Skills Before and After an Instructor Training.

First, let’s look at if trainees believe their skills changed from before to after the Instructor Training.

We asked trainees to describe their ability to do the following tasks using a five-point Likert scale ranging from “Extremely able” to “Not at all able”:

• Teach using live coding
• Create a Carpentries workshop website
• Use a concept map for lesson planning
• Write a multiple choice question with diagnostic power
• Write effective learning objectives

Note that the number of responses only reflects those who completed both pre-and post-surveys for graphs comparing Before and After Instructor Training.

4.1.1 Ability to Teach using Live Coding

Before Instructor Training, a substantial number of trainees fit into each ability level, with the highest number reporting being moderately able to teach using live coding. After Instructor Training, there was a shift. Fewer trainees reported being neutral, slightly able, or not at all able, and more trainees reported being extremely able or moderately able.

Figure 4.1: Ability to Teach Using Live Coding

4.1.2 Ability to Create a Carpentries Workshop Website

Before Instructor Training, the highest number of trainees report that they are not at all able to create a Carpentries workshop website. After Instructor Training, the highest number of trainees say that they are moderately able.

Figure 4.2: Ability to Create a Carpentries Workshop Website

4.1.3 Ability to Use a Concept Map for Lesson Planning

Before Instructor Training, the highest number of trainees reported being not at all able to use a concept map for lesson planning. After Instructor Training, the highest number of trainees reported being moderately able.

Figure 4.3: Ability to Use a Concept Map for Lesson Planning

4.1.4 Ability to Write Multiple Choice Questions with Diagnostic Power

Before Instructor Training, trainees were almost evenly split from moderately able to not at all able to write a multiple choice question with diagnostic power. After Instructor Training, trainees overwhelming reported being moderately able.

Figure 4.4: Ability to Write Multiple Choice Questions with Diagnostic Power

4.1.5 Ability to Write Effective Learning Objectives

Before Instructor Training, trainees were divided on their ability to write effective learning objectives. However, after Instructor Training, there is an apparent increase in the number of trainees reporting being moderately able or extremely able to write effective learning objectives.

Figure 4.5: Ability to Write Effective Learning Objectives

4.2 What Knowledge Did Trainees Acquire? Changes in Perceived Knowledge Before and After an Instructor Training.

Next, we will examine whether trainees believe their knowledge changed from before to after the Instructor Training.

Trainees rated their knowledge about the following topics on a three-point scale from “Extensive knowledge of the topic” to “Little or no knowledge of the topic”:

• Formative vs. summative assessment
• Giving effective feedback
• What makes someone a novice or expert
• How human memory works
• Cognitive load
• What The Carpentries is and what it does
• Strategies to motivate learners

4.2.1 Knowledge of Formative vs. Summative Assessment

Before Instructor Training, trainees primarily reported that they had little to no knowledge of formative vs. summative assessment. However, following the training, the highest number of trainees reported having some knowledge of formative vs. summative assessment.

Figure 4.6: Knowledge about Formative vs. Summative Assessment

4.2.2 Knowledge of How to Give Effective Feedback

Before Instructor Training, trainees were most likely to report that they had some knowledge of how to give effective feedback. After Instructor Training, the number of trainees reporting extensive knowledge increased while the number reporting little to no knowledge dropped to almost zero.

Figure 4.7: Knowledge of Giving Effective Feedback

4.2.3 Knowledge of What Makes Someone a Novice or Expert

Before Instructor Training, trainees were most likely to report that they had some knowledge of what makes someone a novice or expert. After Instructor Training, the number of trainees reporting extensive knowledge increased substantially while the number reporting little to no knowledge dropped to almost zero.

Figure 4.8: Knowledge of What Makes Someone a Novice or Expert

4.2.4 Knowledge of How Human Memory Works

Before Instructor Training, trainees were most likely to report that they had some knowledge of how human memory works. After Instructor Training, trainees reporting extensive knowledge increased while the number reporting little to no knowledge decreased substantially.

Figure 4.9: Knowledge of How Human Memory Works

4.2.5 Knowledge of Cognitive Load

Before Instructor Training, most trainees were split between reporting some knowledge or little to no knowledge of cognitive load. After Instructor Training, there was an increase in trainees reporting both extensive knowledge and some knowledge.

Figure 4.10: Knowledge of Cognitive Load

4.2.6 Knowledge of What The Carpentries is and What it Does

Before Instructor Training, many trainees reported having some knowledge of what The Carpentries is and what it does. After Instructor Training, trainees reporting some knowledge and little to no knowledge decreased while trainees reporting extensive knowledge increased substantially.

Figure 4.11: Knowledge of What The Carpentries is and What it Does

4.2.7 Knowledge of Strategies to Motivate Learners

Before Instructor Training, the highest number of trainees reported having some knowledge of strategies to motivate learners. After Instructor Training, those reporting some knowledge remained consistent while those with little or no knowledge dropped, and those reporting extensive knowledge increased substantially.

Figure 4.12: Knowledge of Strategies to Motivate Learners

4.3 Summary of Skills and Knowledge

We ask trainees to report on the perceptions of their skills and knowledge before and after Instructor Training in twelve teaching-related areas covered during the training.

Trainees report higher levels of ability or knowledge after Instructor Training for every skill and knowledge type. These results demonstrate that our Instructor Training program effectively meets its goals of introducing trainees to evidence-based teaching practices and providing skills they can use to improve their teaching.

While there is an increase in ability or knowledge across all areas, there is some variability. For example, we see a greater increase in trainees’ ability to teach using live coding (something we cover and practice in Instructor Training) than their ability to create a Carpentries workshop website (an exercise that many Trainers tend to skip). Similarly, we see a less clear increase in trainees’ ability to use a concept map for lesson planning. While this is an activity we cover extensively in Instructor Training, we consistently hear that some trainees struggle with this topic. These reflections suggest that perhaps there are opportunities in our curriculum to strengthen how we teach building workshop websites and concept maps.

4.4 Did Trainees Report a Change in Their Motivation to Learn After Instructor Training?

After Instructor Training, we asked trainees to describe how their motivation to learn about the following topics has changed:

• Formative vs. summative assessment
• Giving effective feedback
• What makes someone a novice or expert
• How human memory works
• Cognitive load
• What The Carpentries is and what it does
• Strategies to motivate learners

The majority of trainees reported increased motivation to continue learning across all seven topic areas. Percentages of trainees reporting increased motivation range from 63% for “What makes someone a novice or an expert” to 87% for “Strategies to motivate learners,” an average of 75% across topic areas.

Figure 4.13: Change in Trainee Motivation to Learn More by Topic

4.5 How Was Instructor Training? Trainees’ Perceptions of the Instructor Training Experience.

After Instructor Training, trainees rated their level of agreement with the following statements on a 5-point Likert scale ranging from “Strongly Agree” to “Strongly Disagree”:

• The amount of information covered was reasonable for the allotted time.
• The overall atmosphere was welcoming.
• The material presented matched the description of the training event.
• This training was worth my time.

4.5.1 The amount of information covered was reasonable for the allotted time.

The majority of trainees (86%) either agreed or strongly agreed that the amount of information covered in Instructor Training was reasonable for the training time.

Figure 4.14: The amount of information covered was reasonable for allotted time

4.5.2 The overall atmosphere was welcoming.

The overwhelming majority of trainees (98%) either agreed or strongly agreed that the atmosphere during Instructor Training was welcoming.

Figure 4.15: The overall atmosphere was welcoming

4.5.3 The material presented matched the description of the training event.

The majority of trainees (94%) either agreed or strongly agreed that the description of Instructor Training matched their experience.

Figure 4.16: The material presented matched the description of the training event

4.5.4 This training was worth my time.

The majority of trainees (92%) either agreed or strongly agreed that Instructor training was worth their time.

Figure 4.17: The training was worth my time

4.5.5 I would recommend this training to a friend or colleague.

After Instructor Training, trainees indicated how likely they are to recommend Instructor Training to a friend or colleague on a scale of 1 to 10. Over the last five years, average recommendation scores have been between 8.43 and 9. These consistently high scores suggest that trainees are so satisfied with their experience they suggest others take it as well.

Figure 4.18: Likelihood of Recommendation by Year

4.5.6 Trainees Perceptions of the Pace of Instructor Training

Trainees indicated if they felt the pace of Instructor Training was too fast, too slow, or just right. The majority of trainees thought the speed was just right (56%). An additional 38% were split between thinking it was slightly fast or slightly slow.

These responses echo a common finding when asking about the pace of teaching. Most people will say the speed is just right, but invariably some trainees will think it was too fast or too slow. Because everyone has different experience levels and needs, the pace will not be perfect for everyone. However, we can try to make it just right for the majority.

Figure 4.19: Instructor Trainees’ Perception of Pace

4.5.7 Trainees Perceptions of the Balance of Instructor Training

Trainees indicated how they felt about the balance between lecture and hands-on activities during Instructor Training. The majority of trainees (69%) thought the balance between lecture and hands-on work was good. If trainees thought that balance was off, they were more likely to think there was too much lecture (24%) rather than too much hands-on work (7%).

Figure 4.20: Instructor Trainees’ Perceptions of Balance

4.5.8 Summary of Trainees’ Perceptions of the Instructor Training Experience

Trainees reported positive perceptions of Instructor Training. They reported that it was welcoming, was taught at a comfortable pace, and had a good balance of lecture and hands on materials. Most trainees thought Instructor Training was worth their time and were very likely to recommend it to others. One potential area of improvement is the amount of information we cover in Instructor Training. While 86% of trainees either agreed or strongly agreed that the amount of material covered was reasonable, in contrast to other questions in this section, the majority (1,440 vs. 787) agreed rather than strongly agreeing. The idea that we may be trying to cover too much material is not new. We discuss this often in the Trainer community and have been working to address it in ongoing curriculum updates.

4.6 What Were Trainees’ Perceptions of Instructor Trainers?

After Instructor Training, trainees were asked to select how they felt the Instructor Trainer(s) performed based on the following statements:

• Instructor Trainer(s) gave clear answers to your questions.
• Instructor Trainer(s) were good communicators.
• Instructor Trainer(s) were considerate.
• Instructor Trainer(s) were enthusiastic.

4.6.1 Instructor Trainer(s) gave clear answers to your questions.

The majority of trainees (97%) thought Trainers gave clear answers to their questions All of the time or Often.

Figure 4.21: Instructor Trainers gave clear answers to your questions

4.6.2 Instructor Trainer(s) were good communicators.

The majority of trainees (97%) thought Trainers were good communicators All of the time or Often.

Figure 4.22: Instructor Trainers were good communicators

4.6.3 Instructor Trainer(s) were considerate.

The majority of trainees (99%) thought Trainers were considerate All of the time or Often.

Figure 4.23: Instructor Trainers were considerate

4.6.4 Instructor Trainer(s) were enthusiastic.

The majority of trainees (98%) thought Trainers were enthusiastic All of the time or Often.

Figure 4.24: Instructor Trainers were enthusiastic

4.6.5 Summary of Trainees’ Perceptions of Instructor Trainers

We are fortunate to work with such outstanding Instructor Trainers. Our Trainers are rated highly across all categories. Trainees see them as good communicators who answer questions clearly, and are considerate and enthusiastic.

4.7 How did you find out about the Carpentries Instructor Training?

Before Instructor Training, we ask trainees how they found out about The Carpentries Instructor Training. The greatest number of trainees found out through a friend or colleague (55%) followed by an advisor or supervisor (26%). Smaller numbers of trainees learned about Instructor Training through Carpentries communications and social media.

Figure 4.25: How Trainees Found Out About The Carpentries Instructor Training

4.8 Trainee Demographics

4.8.1 Country

Before Instructor Training, we ask trainees in what country their training takes place. A limitation of these data is that many trainings occur online, but it is unclear how someone taking the training online should answer this question. We assume that those online trainees answered with the country they were in at the time.

That limitation aside, this first graph shows the top ten countries trainees reported. Most trainees were in the United States, followed by the UK, Canada, and Australia. However, this list also has representation from European countries (e.g., Germany, Netherlands, Sweden) and South Africa, and Argentina.

Figure 4.26: Top 10 Countries Trainees Come From

Below is a map showing all countries trainees have reported. The colors represent the number of trainees who reported that country from 2018 to 2020. Light grey indicates countries not reported.

Figure 4.27: Instructor Trainees by Country

4.8.2 Instructor Trainees by Ethnic Identity

Trainees located in the United States were asked about their ethnic identity.

In the US from 2018 to 2020, the greatest number of trainees identify as white (66%), followed by Asian (17%) and Hispanic/Latino(a) (8%).

Figure 4.28: Instructor Trainees by Ethnic Identity

4.8.3 Instructor Trainees by Professional Profiles

The greatest percentage of trainees, as reported on the Instructor Training pre-surveys, come from the following domains: Computer Science (19%), Bioinformatics/Genomics (16%), and Library Science (12%). This distribution may reflect the origins of the Software, Data Carpentry, and Library Carpentry communities.

Figure 4.29: Instructor Trainees by Domain

The greatest percentage of our trainees are graduate students (26%) followed closely by research staff (24%). We also have a sizeable portion of faculty (16%) and postdoctoral researchers (13%). We have smaller numbers of trainees that come from government and industry.

Figure 4.30: Instructor Trainees by Position

4.9 Why are you Attending The Carpentries Instructor Training?

According to our trainees, the most common reason for attending Instructor Training is to improve their general teaching skills followed closely by being able to teach Carpentries workshops.

Figure 4.31: Instructor Trainee Motivation for Attending Instructor Training

5 Instructor Certification and Teaching: Outcomes, Challenges, and Opportunities

While one goal of The Carpentries Instructor Training Program is to develop the individual Instructor, another goal is related to building The Carpentries community. Each certified Carpentries Instructor becomes a member of a thriving community of practice.

5.1 Instructor Certification By Year

One goal of our Instructor Training program is to certify Instructors who will go on to teach Data Carpentry, Library Carpentry, and Software Carpentry workshops. However, a secondary goal is to share our pedagogical approach with those who want to improve their teaching skills. We are happy to train those who do not intend to certify and teach with The Carpentries. Therefore, not all individuals trained are expected to certify as Instructors. To become certified, one must complete a three-step checkout process following training.

The Carpentries certified 630 instructors in 2019, the most in one year to date. This was a 10.9% increase from the year before. In 2020, we certified 494 instructors. This decrease reflects a reduction in the number of people trained in 2020, likely due to the pandemic.

Figure 5.1: Instructors Certified by Year

5.2 Checkout and Certification

One potential opportunity for improvement is in our Instructor Training program is related to checkout. Specifically, we could work to reduce the number of individuals who participate in the training, but do not complete the instructor certification process. As mentioned previously, the goals of our Instructor Training program are twofold: 1) Certify new instructors who will go on to teach our workshops, but 2) To provide training in our pedagogy to those interested in improving their teaching practices regardless of whether they plan to teach for The Carpentries. With that in mind, our goal is not to have an instructor certification rate of 100%. However, if there are trainees who intend to certify but do not complete the checkout process, this may be an opportunity.

Our overall certification rate, since 2013, is 51.3% and has ranged from 44% in 2014 to 57% in 2015. Between 2017 - 2019, we saw a steady decline in the certification rate. In 2020, the certification rate increased slightly (from 54% in 2019 to 55% in 2020).

Figure 5.2: Percentage of Trainees Who Certify as Instructors

5.2.1 The Checkout Process

After completing Instructor Training, there are three steps to “checkout”, that is, become a certified Instructor. Those three steps are:

1. Homework: Contribute to The Carpentries lessons or glossary;
2. Discussion: Take part in an online community discussion session; and
3. Demo: Teach a short demonstration lesson.

At the time trainees take their post-workshop survey, the vast majority say they understand the checkout process.

Figure 5.3: Instructor Trainees’ Understanding of What Needs to be Done to Complete Instructor Training

However, between 2019 and 2020, only 57% of trainees completed all three checkout steps, 28% of trainees completed no checkout steps, and 15% completed just one or two of the steps. We do not yet measure how many trainees intend to checkout. However, we assume that trainees who started the checkout process did have this intention. In seeking to reduce barriers to checkout and increase the number of certified Instructors, this 15% of trainees who begin but do not complete checkout may be a good place to start.

Figure 5.4: Percentage of Trainees Who Start Checkout

If we are to address why some trainees begin but do not complete checkout, it is helpful to understand what steps they complete and where they may get stuck.

Of the 282 trainees that did not complete checkout between 2019 and 2020, 240 (or 85%) attended a community discussion, while only 83 (29%) and 64 (23%) trainees completed the teaching demonstration and lesson contribution, respectively.

These results suggest that focusing our efforts on the teaching demonstration and lesson contribution steps could increase the checkout rate and, in turn, the number of certified Carpentries Instructors. Possible next steps to help more trainees complete the teaching demonstration and lesson contribution include:

• looking for opportunities to clarify instructions or expectations,
• providing additional support to trainees through these steps, and
• considering whether any changes need to be made to the formats.

Updates have already been proposed to the instructions housed in the Instructor Training Curriculum that may begin to address some of these challenges. The Instructor Training Team will be working with the Trainers Leadership Group, the Maintainers community, and other relevant stakeholders towards further addressing next steps in the coming months.

Figure 5.5: Number of Trainees Who Complete Each Checkout Step

5.2.2 Time from Training to Certified Instructor

Another checkout factor to consider is the time it takes a trainee to complete certification. In 2020, trainees took a median of 78 days from finishing training to completing certification or just under three months.

Figure 5.6: Median Time from Instructor Training to Certification

5.3 Instructor Teaching Activity

Once Instructors are certified, we are interested in how active they are. To start, it is helpful to look at how long it takes for a new Instructor to teach their first workshop.

The graph below shows the median time from completing Instructor Training to teaching your first workshop. Note that someone can teach a Carpentries workshop without being certified as long as they teach with a certified Instructor. In this analysis, we are excluding those who taught before they were certified. The median time from completing Instructor Training to teaching your first workshop has ranged from 185 days in 2015 to 82 days in 2017. In 2020, the average number of days was 182, or just under six months.

Figure 5.7: Median Time from Instructor Training to First Time Teaching

Once Instructors begin teaching, it is helpful to understand how often they teach.

The following graph includes all Instructors, certified and not certified, who have taught a Carpentries workshop. The greatest number of Instructors have taught one workshop. However, the number of Instructors who have taught between 2 and 5 workshops is almost as high. The vast majority of Instructors fall into these two categories. However, we also have over 300 Instructors who have taught 6 or more workshops. Note that the graph below does not include certified Instructors who have never taught a workshop.

Figure 5.8: Number of Total Workshops Instructors Have Taught

5.4 Summary of Instructor Certification and Teaching

From 2017 - 2019, we certified between 464 and 625 new Instructors annually. This number was consistently increasing until 2020, where challenges related to the pandemic affected our efforts and outcomes. The certification rate was 54% in 2019 rising slightly to 55% in 2020.

We have determined that 28% of trainees never begin checkout, while 15% complete some but not all of the steps. Of those steps, trainees attend a community discussion most often followed by a substantial drop in those who complete the teaching demonstration or lesson contribution steps. These data reveal an opportunity to increase support for the teaching demonstration and lesson contribution steps, increasing our certification rate.

After certification, we find that Instructors typically teach their first workshop within 6 months. Most Instructors teach between 1 and 5 workshops, but we have a small but active group of over 300 Instructors that have taught more than 6.

6 Growth of Instructor Training Efforts

The Carpentries received a grant from the Chan Zuckerberg Initiative in which we committed to growing our Instructor and Trainer Training Efforts by 2022.

Our primary metric to reflect efforts is the number of Instructors and Trainers trained. This number is more valuable than the number of trainings run because trainings have varying numbers of trainees and do not reflect efforts directed towards increased efficiency. Instructors trained is also more useful than Instructors certified because certifications rely on downstream factors (e.g., local community support for checkout completion) in addition to efforts toward expanding access to training.

6.1 The Impact of the Global COVID-19 Pandemic

Since 2020, the world has been fighting a global COVID-19 pandemic. The pandemic has created a great deal of uncertainty for individuals and organizations around the globe. Throughout this report, the effects of this uncertainty are reflected in a decrease in our data. As parts of the world begin to open up, we look forward to being able to train more people again.

6.2 Number of Trained Instructor Trainers

Until 2016, Software Carpentry founder and former Executive Director Greg Wilson ran all Instructor Training events himself online. Knowing the limitations of having only one Instructor Trainer, in 2016, The Carpentries began training additional Instructor Trainers (aka “Trainers” in Carpentries contexts).

Training additional Instructor Trainers expanded our reach, allowing us to run several Instructor Training events a month across different time zones. It also allowed us to build capacity at member organizations that have onsite Instructor Trainers. These Trainers run events for their institution, building a community of trained and certified instructors there. By bringing on new Trainers in many parts of the world, we create a global Trainer community. This international community has expanded our geographic reach, allowing us to reach communities with which we may not otherwise connect.

The Carpentries Trainer community comprises experienced Instructors and others with advanced educational training or related expertise who inspire and prepare the next wave of Instructors in our community. They work as a team to maintain and teach the Instructor Training curriculum to prepare prospective Instructors to join The Carpentries community.

As of August 2021, The Carpentries has 98 certified Trainers. Unlike Instructors, certified Instructor Trainers are asked to select “inactive” status if they are unable to continue participating in their role.

In 2019, we certified 10 new Trainers. In 2020, we trained two cohorts of new Trainers and certified our highest number of Trainers in one year to date, 29. In July 2021, we certified an additional 14 new Trainers.

6.3 Number of Trained Instructors

As of August 2021, The Carpentries has 3,335 certified Instructors. Note that once an Instructor is certified, they remain certified indefinitely. Thus, this number does not reflect the current number of active Instructors.

The Carpentries trained 1,168 people in Instructor Training in 2019. This was a 12.5% increase from 2018 and a 42.6% increase from 2017.

Before 2020, the number of people we trained increased year on year. In 2020, despite the pandemic, we still taught 895 people. This decrease of 23% resulted in a number of trainees on par with our efforts in 2017.

Figure 6.1: People Trained in Instructor Training by Year

6.4 Number of Instructor Trainings

In 2019, The Carpentries ran 67 Instructor Trainings. This was similar to the 70 trainings we ran in 2018, but a 36.7% increase from 2017. In 2020, we were able to host 49 Instructor Trainings. This 26% decrease also reflected a return to levels similar to 2017.

Figure 6.2: Instructor Trainings by Year

6.5 Pipelines for Trainees: Member vs. Open

Most of our trainees are trained through organisational agreements with affiliated institutions or groups, also called Memberships. Others are accepted through our Open Application program. The Open Application program is a limited scholarship program where people not trained through a Membership can attend Instructor Training free of charge. In 2020, 716 individuals were trained through a Membership while 178 were trained through the Open Application program.

Figure 6.3: Number of People Trained Through Membership Versus Open Applications

Before 2018, we could not separate trainees by whether they were trained as part of a Membership Agreement or through our Open Training program.

From 2018 to 2019, we saw an 51.7% increase in those being trained under a membership agreement and a 25% increase in those being trained through Open Training.

7 Conclusion

7.1 Summary

The purposes of this report are to:

• provide an update on the growth, successes, and challenges of the Instructor Training program to the Executive Council, Trainers Leadership Group, Trainers, and broader Carpentries community.
• help the Instructor training team, Core Team, Executive Council, and Trainers Leadership Group identify trends and opportunities that can drive continuous program improvement.

The report is comprised of three main components: 1) Instructor Training Outcomes, 2) Instructor Certification and Teaching, and 3) Growth of Instructor Training Efforts.

7.1.1 Instructor Training Outcomes

Results from our pre and post-Instructor Training surveys demonstrate that trainees learn new skills and knowledge, have more motivation to learn and feel the training was worth their time. They rate our Trainers as strong communicators, considerate, and enthusiastic. Trainees also report they are very likely to recommend Instructor Training to others. This point is significant as the majority of trainees find about Instructor Training through word-of-mouth.

7.1.2 Instructor Certification and Teaching

Drawing on data in our AMY database, we identified that from 2017 - 2019, we certified between 464 and 625 new Instructors annually. This number was consistently increasing until 2020, where challenges related to the pandemic affected our efforts and outcomes. The certification rate was 54% in 2019, rising slightly to 55% in 2020. We have determined that 28% of trainees never begin checkout, but 15% complete some but not all of the steps. Of those steps, most attend a community discussion followed by a substantial drop in those who complete the teaching demonstration or lesson contribution steps. These data reveal an opportunity to increase support for the teaching demonstration and lesson contribution steps, increasing our certification rate.

After certification, Instructors typically teach their first workshop within 6 months. Most Instructors teach between 1 and 5 workshops, but we have over 300 Instructors that have taught more than 6.

7.1.3 Growth of Instructor Training Efforts

The Carpentries received a grant from the Chan Zuckerberg Initiative in which we committed to growing our Instructor and Trainer Training Efforts by 2022.

In 2020, we trained two cohorts of new Trainers and certified our highest number of Trainers in one year to date, 29. In July 2021, we certified a cohort of 14 new Trainers.

Instructor Training has declined during the pandemic, showing a 23% drop in the number of people trained in 2020 compared with 2019 (895 vs. 1,168). It has remained somewhat lower than average in 2021. However, we still trained almost 900 people in 2020. Recruitment and retention will continue to present a challenge in an unpredictable pandemic landscape. We look forward to finding new ways to support the ambitions of our community-members amid the challenges they face.

7.2 Next Steps

Work on this report will be ongoing as new questions emerge and our ability to answer them advances. This preliminary report serves as a building block to future assessment work. Areas of interest for continued development include, but are certainly not limited to:

• deeper exploration of our applicant, trainee, Instructor, and Trainer demographics as we continue to focus on equity, inclusion, and accessibility. The limited data currently available to explore these topics suggest that we need to re-examine our assessment efforts in this area.
• deeper analysis of trainee outcomes broken down by paths to training, e.g. through membership vs sponsored admission for individuals or groups.
• more comprehensive understanding of Instructor and Trainer teaching activity, including teaching frequency and retention.
• impact of curriculum updates, policy changes, and other qualitative shifts on survey responses.
• long-term assessment of Instructor Training to evaluate perspectives of Instructors as they begin to teach.

7.3 Final Thoughts

The success of The Carpentries Instructor Training program hinges on the strength of our outstanding Trainer community. Our Trainers create welcoming and inclusive spaces for our trainees to learn. They are knowledgeable, enthusiastic, and kind. Our Trainers donate thousands of hours to maintaining our curriculum, teaching Instructor Training, hosting teaching demonstrations, and continuing to improve their own teaching skills. Thank you to each of you for all you do.