The Carpentries: Programmatic Assessment Report

Publication Date: August 3, 2018

Introduction: The Carpentries

Authors: Maneesha Sane, Erin Becker

What is The Carpentries?

Software Carpentry (SWC) and Data Carpentry (DC) are two programs of The Carpentries (a fiscally sponsored project of Community Initiatives). We teach essential computing and data skills. We exist because the skills needed to do computational, data-intensive research are not part of basic research training in most disciplines.

About Software Carpentry

Software Carpentry enables researchers to create purpose-built tools, whether it be a Unix shell script to automate repetitive tasks, or software code in programming languages such as Python, R, or MATLAB. These enable researchers to build programs that can be read, re-used, and validated, greatly enhancing the sharing and reproducibility of their research.

About Data Carpentry

Data Carpentry learners are taught to work with data more effectively. Workshops focus on the data lifecycle, covering data organization, cleaning and management through to data analysis and visualization. Lessons are domain-specific, with coverage in ecology, genomics, and social sciences. Future releases include lessons for working with geospatial data and additional lessons in genomics and social sciences.

What The Carpentries offers

  • A suite of open source, collaboratively-built, community-developed lessons
  • Workshops based on a learn-by-doing, ‘code with me’ approach
  • Instructor training in evidence-based pedagogical training methods
  • An active global community subscribing to an inclusive code of conduct
  • A supportive learning culture
  • Ongoing development opportunities via our peer mentoring program
  • Community-led discussions of open source software, open science, and teaching methodology

The Carpentries began systematically recording data for our workshops in 2012. We use this data to investigate how The Carpentries has grown over the years including number and geographic reach of our workshops, and learners at these workshops. We also look at our Instructor Training program, including number and geographic reach of instructor training events, number of trainees and their completion rates, and onboarding of new Instructor Trainers.

Data are collected by a team of Workshop Administrators. In Africa, Australia, Canada, New Zealand, and the United Kingdom, Workshop Administrators are affiliated with our member institutions and provide in-kind staff time. A full-time Carpentries staff member is the Workshop Administrator for the rest the world.

Introduction: This Report

Section 1: Carpentries Workshops

This section revisits data and projections made in an earlier report, noting specific instances of unexpected growth or reach to new communities.

Section 2: Instructor Training Applications

For this report, we analyzed the applications to our Instructor Training program. The goal was to compare the applicant pool with the badged instructor pool and to be able to look at application and completion rates across certain characteristics. These findings are explored further below.

As the source data includes individual application data, we can not share the source data publicly. If you are interested in learning more about how this data was analyzed, please contact us at team@carpentries.org.

Section 1: Carpentries Workshops

January 1, 2012 through June 30, 2018

We revisited some of the data explored in an earlier report. Looking at our expected growth, we see that projections for the number of workshops in 2018 have not changed significantly from the first quarter to the second quarter. Our first quarter projections showed 137 Data Carpentry and 299 Software Carpentry workshops, for a total of 436 workshops. Our revised projections show 122 Data Carpentry and 304 Software Carpentry workshops, for a total of 426 workshops. This shows us running about 41 more Data Carpentry and 47 more Software Carpentry workshops in 2018 than we did in 2017.

We did see geographic growth. In June 2018, The Carpentries saw its first workshop in Pakistan. By the second quarter of 2018, we also saw Ethiopia join the ranks of countries hosting 10 or more workshops. This growth happened in only two years, representing the remarkable growth of The Carpentries community across Africa.

Figure 1: Workshops by Carpentry by Year

This bar chart shows the number of Data Carpentry (DC) and Software Carpentry (SWC) workshops each year. Data for 2018 is a projection. The proportion of workshops in the first two quarters of 2017 relative to the full year was applied to actual first two quarters data from 2018 to calculate this projection. Source data can be found in Table 1 in the Appendix.

Figure 2: First workshop by year

This map notes the year each country held its first workshop. Source data can be found in Table 2 in the appendix.

Figure 3: Countries hosting 10 or more workshops

This chart notes all countries hosting 10 or more workshops since 2012. Figures for 2018 are a projection. Source data can be found in Table 3 in the appendix.

Section 2: Carpentries Instructor Training

July 1, 2016 through June 30, 2018

The Carpentries runs instructor training events as a benefit to member sites who want to build a pool of onsite instructors, increasing their capacity to run self organized, onsite workshops for their community. When possible, The Carpentries also offers seats in these events to those applying from an open pool, as a way to increase our reach and connect with individuals who may not otherwise have access to our resources.

This two-day class has the following overall goals:

  • Introduction to evidence-based best-practices of teaching.
  • Teaching how to create a positive environment for learners at your workshops.
  • Provide opportunities for practicing and building teaching skills.
  • Help integrate trainees into The Carpentries community.
  • Prepare trainees to use these teaching skills in teaching Carpentry workshops.

Because we have only two days, some things are beyond the scope of this class. This training does not cover:

  • How to program in R or Python, use git, or any of the other topics taught in Carpentry workshops.
  • How to create new lessons from scratch

For this report, we looked at all applications that came in through our open application process (those that did not specifically identify themselves with a member site), from July 1, 2016 through June 30, 2018. In looking at completion rates, we only looked at people who attended events before March 31, 2018. Because all trainees have three months to complete the checkout process, those who attended later would not be expected to complete certification before June 30, 2018. The analyses below do not include applications from member sites, as these trainees are selected by the member site, and do not go through our review process.

This review included

  • Open training applications (n=892)
  • Attendees at open training events (n=454)
  • Badged instructors from open training events (n=257)

This represents an overall completion rate of 56.6%. In this report, we compare this overall completion rate with completion rates by previous training in teaching, previous experience in teaching, previous involvement with The Carpentries, areas of expertise, programming language experience, by occupation, and by country.

We did not look at how many people attended events in relation to the overall application rate for several reasons. First, not all applicants have had their applications reviewed to be invited to a training event. Then, of those who have been invited, not all have had the opportunity to attend (typically due to scheduling constraints). Lastly, of those who have not yet been invited, they may be invited to a future event or asked to re-apply, so are not considered to be rejected from the application process.

Open Applications by Previous Training in Teaching

We looked at our applications by our applicants' previous training in teaching. Most people have had no previous training in teaching. This demonstrates the importance and singularity of work we do - while it is a needed skill, people are not getting this type of training anywhere else. Just like The Carpentries is motivated by seeing scientists and researchers who are not being taught programming skills, we see people who may teach or want to teach are not being taught teaching skills.

At the same time, people with at least some previous training in teaching tend to certify at higher rates, possibly indicating that coming in with some teaching background motivates people to continue through the Carpentries certification process.

Figure 4: Open Applications by Previous Training in Teaching

This bar chart shows the number of applications received by people by their previous training in teaching between July 1, 2016 and June 30, 2018. Source data can be found in Table 4 in the appendix.

Figure 5: Completion Rates by Previous Training in Teaching

This chart shows the percent of applicants who attended instructor training events before March 31, 2018 and completed their Instructor certification by June 30, 2018, by their previous training in teaching. The overall average of 56.6% is represented by the red line. Source data can be found in Table 4 in the appendix.

Open Applications by Previous Experience in Teaching

In addition to looking at applicants' previous training in teaching, we looked at their previous experience in teaching. The majority of applications come from people who have experience teaching a full course. This demonstrates that while they have this experience, they may not have had training in how to teach, and want to become better at it. Completion rates are steady around 50% for most groups. Those whose only teaching experience was a few hours or as at teaching assistant did not certify at all. These also represent the smallest number of applications and may not be a representative sample.

Figure 6: Open Applications by Previous Experience in Teaching

This bar chart shows the number of applications received by people by their previous experience teaching between July 1, 2016 and June 30, 2018. Source data can be found in Table 5 in the appendix.

Figure 7: Completion Rates by Previous Experience in Teaching

This chart shows the percent of applicants who attended instructor training events before March 31, 2018 and completed their Instructor certification by June 30, 2018, by their previous experience in teaching. The overall average of 56.6% is represented by the red line. Source data can be found in Table 5 in the appendix.

Open Applications by Previous Involvement with The Carpentries

We looked at our applications by previous involvement with The Carpentries. Applicants could self identify that they had experience as a helper, instructor, host, organizer, or contributor. Many applicants had multiple roles. We can not directly compare between groups as many people held more than one role. However, we can look at some characteristics of each group.

The majority of our applications came from people who had no previous involvement with us. This is impetus for us to understand how they learned about The Carpentries and to better understand their motivations for going through our instructor training events.

The next highest group of applications is from learners. However, they complete the certification process at lowest levels. This may be because they still feel they are at the learner/novice level and don't feel they know enough to teach. This may be an indication that while they value what they learned, they may not feel ready to teach it to others. Our pedagogical model values instructors who are only slightly ahead of their learners, because these instructors are often better able to relate to the learners and less affected by the expert blind spot. However, this may show that they don't feel confident or motivated enough to be in an instructor role.

People who have had previous experience with The Carpentries (helpers, hosts, instructors, contributors) all badge at levels higher than overall average. These are people who have had deeper experiences with The Carpentries, have more buy in and motivation, and are already coming in knowing about how we work. This may indicate that people who have seen our teaching practices, and are familiar with our curriculum are motivated to complete the certification process to make them official Carpentries instructors.

Figure 8: Open Applications by Previous Involvement with The Carpentries

This bar chart shows the number of applications received by people by previous involvement with The Carpentries between July 1, 2016 and June 30, 2018. Applicants could choose multiple roles, so the total will be more than the actual number of applications. Source data can be found in Table 6 in the appendix.

Figure 9: Completion Rate by Previous Involvement with The Carpentries

This chart shows the percent of applicants who attended instructor training events before March 31, 2018 and completed their Instructor certification by June 30, 2018, by their previous involvement with The Carpentries. The overall average of 56.6% is represented by the red line. Source data can be found in Table 6 in the appendix.

Open Applications by Areas of Expertise

We looked at our applications by area of expertise. Applicants could self identify their areas of expertise. Many applicants identify with multiple domains. We can not directly compare between groups. However, we can look at some characteristics of each group.

The majority of applications came from people with a background in the life sciences, computer science, high performance computing, and/or mathematics and statistics. These groups were all near or slightly above the average completion rate.

Through a recent grant from the Alfred P. Sloan Foundation we are expanding our reach to new curricula including Social Sciences, Chemistry, and Economics. The data here show the importance of developing strategies for recruiting new instructors and onboarding existing instructors to develop, teach, and maintain these lessons.

Figure 10: Open Applications by Area of Expertise

This bar chart shows the number of applications received by people by their areas of expertise between July 1, 2016 and June 30, 2018. Applicants could choose multiple domains, so the total will be more than the actual number of applications. Source data can be found in Table 7 in the appendix.

Figure 11: Completion Rates by Area of Expertise

This chart shows the percent of applicants who attended instructor training events before March 31, 2018 and completed their Instructor certification by June 30, 2018, by their areas of expertise. The overall average of 56.6% is represented by the red line. Source data can be found in Table 7 in the appendix.

Open Applications by Programming Language Usage

The majority of our applications come from people who use a programming language frequently -- daily or several times a week. Certification rates are also lower among those who use a programming language infrequenty. In presenting our Instructor Training program to member sites and to the broader community, we emphasize that our training is about pedagogy, cognitive psychology, and how to teach. We note that techinical skills are not taught. However, the aim of our Instructor Training program is to prepare people to teach our curriculum, which includes programming languages like R and Python, as well as other computational programming skills like version control in Git. We may be seeing a subset of our applicant pool who are looking to improve their teaching skills, but do not badge because they do not feel prepared to teach the tools we teach in our workshops. To this end, we have implemented a mentoring program where experienced instructors can support new instructors in being more prepared to teach, and would like to see more new instructors take advantage of this.

Figure 12: Open Applications by Programming Language Usage

This bar chart shows the number of applications received by people by frequency of programming language usage between July 1, 2016 and June 30, 2018. Source data can be found in Table 8 in the appendix.

Figure 13: Completion Rates by Programming Language Usage

This chart shows the percent of applicants who attended instructor training events before March 31, 2018 and completed their Instructor certification by June 30, 2018, by their frequency of programming language usage. The overall average of 56.6% is represented by the red line. Source data can be found in Table 8 in the appendix.

Open Applications by Occupation

We looked at the occupation or career stage of our applicants. We found the majority of applications to be from graduate students. They also completed at lower rates than our overall average. Teaching faculty also completed at lower than average rates. This may be because as active teachers, they are interested in developing their teaching skills but not necessarily in teaching specifically with The Carpentries. Completion rates are lowest for those who did not share their occupation. These may be people who are transitioning career stages and are not ready to commit to the certification process and teaching; rather they may see this as an opportunity for something to do while in transition.

We see high completion rates for researchers and librarians, perhaps demonstrating these are the people using the technology skills we teach regularly and are in a position to share these tools with others. While undergraduates also have high completion rates, this may be skewed by the fact that they represent a small percentage of our overall applications.

Figure 14: Open Applications by Occupation

This bar chart shows the number of applications received by people from each occupation between July 1, 2016 and June 30, 2018. Source data can be found in Table 9 in the appendix.

Figure 15: Completion Rates by Occupation

This chart shows the percent of applicants who attended instructor training events before March 31, 2018 and completed their Instructor certification by June 30, 2018, by occupation. The overall average of 56.6% is represented by the red line. Source data can be found in Table 9 in the appendix.

Open Applications by Country

We looked at the number of applications that came in by country. The United States accounts for half of all applications (50.7%). Outside of the US, most applications come from Australia, Canada, South Africa, and the United Kingdom.

We also looked at the completion rate by country. Countries with 100% completion rates are those with only one trainee in that country. Countries with 0% completion rates have 1-5 trainees, and are often in places where we don't yet have a strong presence. Trainees may not feel support or motivation from a larger community. If we are bringing on new instructors it is incumbent on us develop and implement methods to support them even though they may be geographically isolated from the larger Carpentries community.

We see both high applcation rates and high completion rates in Australia and United Kingdom. It is worthwhile to investigate what is happening with Instructors and Trainers in those regions to to learn how we can apply these successes to other communities.

Figure 16: Applications by Country (bar chart)

This bar chart shows the number of applications received from each country. The United States, with 453 applications, is not included in the chart to avoid skewing the remaining data. Source data can be found in Table 10 in the appendix.

Figure 17: Completion Rates by Country (bar chart)

This bar chart shows the percent completion rates for each country. Not all countries with applications are represented here as some may have applied but not yet attended an event. Source data can be found in Table 10 in the appendix.

Figure 18: Applications by country (map)

This map shows the number of applications received from each country between July 1, 2016 and June 30, 2018. Countries in darker shades had higher application counts. Source data can be found in Table 10 in the appendix.