The Carpentries: Programmatic Assessment Report¶
Publication Date: August 3, 2018
Introduction: The Carpentries¶
Authors: Maneesha Sane, Erin Becker¶
What is The Carpentries?¶
Software Carpentry (SWC) and Data Carpentry (DC) are two programs of The Carpentries (a fiscally sponsored project of Community Initiatives). We teach essential computing and data skills. We exist because the skills needed to do computational, data-intensive research are not part of basic research training in most disciplines.
About Software Carpentry¶
Software Carpentry enables researchers to create purpose-built tools, whether it be a Unix shell script to automate repetitive tasks, or software code in programming languages such as Python, R, or MATLAB. These enable researchers to build programs that can be read, re-used, and validated, greatly enhancing the sharing and reproducibility of their research.
About Data Carpentry¶
Data Carpentry learners are taught to work with data more effectively. Workshops focus on the data lifecycle, covering data organization, cleaning and management through to data analysis and visualization. Lessons are domain-specific, with coverage in ecology, genomics, and social sciences. Future releases include lessons for working with geospatial data and additional lessons in genomics and social sciences.
What The Carpentries offers¶
- A suite of open source, collaboratively-built, community-developed lessons
- Workshops based on a learn-by-doing, ‘code with me’ approach
- Instructor training in evidence-based pedagogical training methods
- An active global community subscribing to an inclusive code of conduct
- A supportive learning culture
- Ongoing development opportunities via our peer mentoring program
- Community-led discussions of open source software, open science, and teaching methodology
The Carpentries began systematically recording data for our workshops in 2012. We use this data to investigate how The Carpentries has grown over the years including number and geographic reach of our workshops, and learners at these workshops. We also look at our Instructor Training program, including number and geographic reach of instructor training events, number of trainees and their completion rates, and onboarding of new Instructor Trainers.
Data are collected by a team of Workshop Administrators. In Africa, Australia, Canada, New Zealand, and the United Kingdom, Workshop Administrators are affiliated with our member institutions and provide in-kind staff time. A full-time Carpentries staff member is the Workshop Administrator for the rest the world.
Introduction: This Report¶
Section 1: Carpentries Workshops¶
This section revisits data and projections made in an earlier report, noting specific instances of unexpected growth or reach to new communities.
Section 2: Instructor Training Applications¶
For this report, we analyzed the applications to our Instructor Training program. The goal was to compare the applicant pool with the badged instructor pool and to be able to look at application and completion rates across certain characteristics. These findings are explored further below.
As the source data includes individual application data, we can not share the source data publicly. If you are interested in learning more about how this data was analyzed, please contact us at team@carpentries.org.
Section 1: Carpentries Workshops¶
January 1, 2012 through June 30, 2018
We revisited some of the data explored in an earlier report. Looking at our expected growth, we see that projections for the number of workshops in 2018 have not changed significantly from the first quarter to the second quarter. Our first quarter projections showed 137 Data Carpentry and 299 Software Carpentry workshops, for a total of 436 workshops. Our revised projections show 122 Data Carpentry and 304 Software Carpentry workshops, for a total of 426 workshops. This shows us running about 41 more Data Carpentry and 47 more Software Carpentry workshops in 2018 than we did in 2017.
We did see geographic growth. In June 2018, The Carpentries saw its first workshop in Pakistan. By the second quarter of 2018, we also saw Ethiopia join the ranks of countries hosting 10 or more workshops. This growth happened in only two years, representing the remarkable growth of The Carpentries community across Africa.
Figure 1: Workshops by Carpentry by Year¶
This bar chart shows the number of Data Carpentry (DC) and Software Carpentry (SWC) workshops each year. Data for 2018 is a projection. The proportion of workshops in the first two quarters of 2017 relative to the full year was applied to actual first two quarters data from 2018 to calculate this projection. Source data can be found in Table 1 in the Appendix.
Figure 2: First workshop by year¶
This map notes the year each country held its first workshop. Source data can be found in Table 2 in the appendix.
Figure 3: Countries hosting 10 or more workshops¶
This chart notes all countries hosting 10 or more workshops since 2012. Figures for 2018 are a projection. Source data can be found in Table 3 in the appendix.
Section 2: Carpentries Instructor Training¶
July 1, 2016 through June 30, 2018
The Carpentries runs instructor training events as a benefit to member sites who want to build a pool of onsite instructors, increasing their capacity to run self organized, onsite workshops for their community. When possible, The Carpentries also offers seats in these events to those applying from an open pool, as a way to increase our reach and connect with individuals who may not otherwise have access to our resources.
This two-day class has the following overall goals:
- Introduction to evidence-based best-practices of teaching.
- Teaching how to create a positive environment for learners at your workshops.
- Provide opportunities for practicing and building teaching skills.
- Help integrate trainees into The Carpentries community.
- Prepare trainees to use these teaching skills in teaching Carpentry workshops.
Because we have only two days, some things are beyond the scope of this class. This training does not cover:
- How to program in R or Python, use git, or any of the other topics taught in Carpentry workshops.
- How to create new lessons from scratch
For this report, we looked at all applications that came in through our open application process (those that did not specifically identify themselves with a member site), from July 1, 2016 through June 30, 2018. In looking at completion rates, we only looked at people who attended events before March 31, 2018. Because all trainees have three months to complete the checkout process, those who attended later would not be expected to complete certification before June 30, 2018. The analyses below do not include applications from member sites, as these trainees are selected by the member site, and do not go through our review process.
This review included
- Open training applications (n=892)
- Attendees at open training events (n=454)
- Badged instructors from open training events (n=257)
This represents an overall completion rate of 56.6%. In this report, we compare this overall completion rate with completion rates by previous training in teaching, previous experience in teaching, previous involvement with The Carpentries, areas of expertise, programming language experience, by occupation, and by country.
We did not look at how many people attended events in relation to the overall application rate for several reasons. First, not all applicants have had their applications reviewed to be invited to a training event. Then, of those who have been invited, not all have had the opportunity to attend (typically due to scheduling constraints). Lastly, of those who have not yet been invited, they may be invited to a future event or asked to re-apply, so are not considered to be rejected from the application process.
Open Applications by Previous Training in Teaching¶
We looked at our applications by our applicants' previous training in teaching. Most people have had no previous training in teaching. This demonstrates the importance and singularity of work we do - while it is a needed skill, people are not getting this type of training anywhere else. Just like The Carpentries is motivated by seeing scientists and researchers who are not being taught programming skills, we see people who may teach or want to teach are not being taught teaching skills.
At the same time, people with at least some previous training in teaching tend to certify at higher rates, possibly indicating that coming in with some teaching background motivates people to continue through the Carpentries certification process.
Figure 4: Open Applications by Previous Training in Teaching¶
This bar chart shows the number of applications received by people by their previous training in teaching between July 1, 2016 and June 30, 2018. Source data can be found in Table 4 in the appendix.
Figure 5: Completion Rates by Previous Training in Teaching¶
This chart shows the percent of applicants who attended instructor training events before March 31, 2018 and completed their Instructor certification by June 30, 2018, by their previous training in teaching. The overall average of 56.6% is represented by the red line. Source data can be found in Table 4 in the appendix.
Open Applications by Previous Experience in Teaching¶
In addition to looking at applicants' previous training in teaching, we looked at their previous experience in teaching. The majority of applications come from people who have experience teaching a full course. This demonstrates that while they have this experience, they may not have had training in how to teach, and want to become better at it. Completion rates are steady around 50% for most groups. Those whose only teaching experience was a few hours or as at teaching assistant did not certify at all. These also represent the smallest number of applications and may not be a representative sample.
Figure 6: Open Applications by Previous Experience in Teaching¶
This bar chart shows the number of applications received by people by their previous experience teaching between July 1, 2016 and June 30, 2018. Source data can be found in Table 5 in the appendix.
Figure 7: Completion Rates by Previous Experience in Teaching¶
This chart shows the percent of applicants who attended instructor training events before March 31, 2018 and completed their Instructor certification by June 30, 2018, by their previous experience in teaching. The overall average of 56.6% is represented by the red line. Source data can be found in Table 5 in the appendix.
Open Applications by Previous Involvement with The Carpentries¶
We looked at our applications by previous involvement with The Carpentries. Applicants could self identify that they had experience as a helper, instructor, host, organizer, or contributor. Many applicants had multiple roles. We can not directly compare between groups as many people held more than one role. However, we can look at some characteristics of each group.
The majority of our applications came from people who had no previous involvement with us. This is impetus for us to understand how they learned about The Carpentries and to better understand their motivations for going through our instructor training events.
The next highest group of applications is from learners. However, they complete the certification process at lowest levels. This may be because they still feel they are at the learner/novice level and don't feel they know enough to teach. This may be an indication that while they value what they learned, they may not feel ready to teach it to others. Our pedagogical model values instructors who are only slightly ahead of their learners, because these instructors are often better able to relate to the learners and less affected by the expert blind spot. However, this may show that they don't feel confident or motivated enough to be in an instructor role.
People who have had previous experience with The Carpentries (helpers, hosts, instructors, contributors) all badge at levels higher than overall average. These are people who have had deeper experiences with The Carpentries, have more buy in and motivation, and are already coming in knowing about how we work. This may indicate that people who have seen our teaching practices, and are familiar with our curriculum are motivated to complete the certification process to make them official Carpentries instructors.
Figure 8: Open Applications by Previous Involvement with The Carpentries¶
This bar chart shows the number of applications received by people by previous involvement with The Carpentries between July 1, 2016 and June 30, 2018. Applicants could choose multiple roles, so the total will be more than the actual number of applications. Source data can be found in Table 6 in the appendix.
Figure 9: Completion Rate by Previous Involvement with The Carpentries¶
This chart shows the percent of applicants who attended instructor training events before March 31, 2018 and completed their Instructor certification by June 30, 2018, by their previous involvement with The Carpentries. The overall average of 56.6% is represented by the red line. Source data can be found in Table 6 in the appendix.
Open Applications by Areas of Expertise¶
We looked at our applications by area of expertise. Applicants could self identify their areas of expertise. Many applicants identify with multiple domains. We can not directly compare between groups. However, we can look at some characteristics of each group.
The majority of applications came from people with a background in the life sciences, computer science, high performance computing, and/or mathematics and statistics. These groups were all near or slightly above the average completion rate.
Through a recent grant from the Alfred P. Sloan Foundation we are expanding our reach to new curricula including Social Sciences, Chemistry, and Economics. The data here show the importance of developing strategies for recruiting new instructors and onboarding existing instructors to develop, teach, and maintain these lessons.
Figure 10: Open Applications by Area of Expertise¶
This bar chart shows the number of applications received by people by their areas of expertise between July 1, 2016 and June 30, 2018. Applicants could choose multiple domains, so the total will be more than the actual number of applications. Source data can be found in Table 7 in the appendix.
Figure 11: Completion Rates by Area of Expertise¶
This chart shows the percent of applicants who attended instructor training events before March 31, 2018 and completed their Instructor certification by June 30, 2018, by their areas of expertise. The overall average of 56.6% is represented by the red line. Source data can be found in Table 7 in the appendix.
Open Applications by Programming Language Usage¶
The majority of our applications come from people who use a programming language frequently -- daily or several times a week. Certification rates are also lower among those who use a programming language infrequenty. In presenting our Instructor Training program to member sites and to the broader community, we emphasize that our training is about pedagogy, cognitive psychology, and how to teach. We note that techinical skills are not taught. However, the aim of our Instructor Training program is to prepare people to teach our curriculum, which includes programming languages like R and Python, as well as other computational programming skills like version control in Git. We may be seeing a subset of our applicant pool who are looking to improve their teaching skills, but do not badge because they do not feel prepared to teach the tools we teach in our workshops. To this end, we have implemented a mentoring program where experienced instructors can support new instructors in being more prepared to teach, and would like to see more new instructors take advantage of this.
Figure 12: Open Applications by Programming Language Usage¶
This bar chart shows the number of applications received by people by frequency of programming language usage between July 1, 2016 and June 30, 2018. Source data can be found in Table 8 in the appendix.
Figure 13: Completion Rates by Programming Language Usage¶
This chart shows the percent of applicants who attended instructor training events before March 31, 2018 and completed their Instructor certification by June 30, 2018, by their frequency of programming language usage. The overall average of 56.6% is represented by the red line. Source data can be found in Table 8 in the appendix.
Open Applications by Occupation¶
We looked at the occupation or career stage of our applicants. We found the majority of applications to be from graduate students. They also completed at lower rates than our overall average. Teaching faculty also completed at lower than average rates. This may be because as active teachers, they are interested in developing their teaching skills but not necessarily in teaching specifically with The Carpentries. Completion rates are lowest for those who did not share their occupation. These may be people who are transitioning career stages and are not ready to commit to the certification process and teaching; rather they may see this as an opportunity for something to do while in transition.
We see high completion rates for researchers and librarians, perhaps demonstrating these are the people using the technology skills we teach regularly and are in a position to share these tools with others. While undergraduates also have high completion rates, this may be skewed by the fact that they represent a small percentage of our overall applications.
Figure 14: Open Applications by Occupation¶
This bar chart shows the number of applications received by people from each occupation between July 1, 2016 and June 30, 2018. Source data can be found in Table 9 in the appendix.
Figure 15: Completion Rates by Occupation¶
This chart shows the percent of applicants who attended instructor training events before March 31, 2018 and completed their Instructor certification by June 30, 2018, by occupation. The overall average of 56.6% is represented by the red line. Source data can be found in Table 9 in the appendix.
Open Applications by Country¶
We looked at the number of applications that came in by country. The United States accounts for half of all applications (50.7%). Outside of the US, most applications come from Australia, Canada, South Africa, and the United Kingdom.
We also looked at the completion rate by country. Countries with 100% completion rates are those with only one trainee in that country. Countries with 0% completion rates have 1-5 trainees, and are often in places where we don't yet have a strong presence. Trainees may not feel support or motivation from a larger community. If we are bringing on new instructors it is incumbent on us develop and implement methods to support them even though they may be geographically isolated from the larger Carpentries community.
We see both high applcation rates and high completion rates in Australia and United Kingdom. It is worthwhile to investigate what is happening with Instructors and Trainers in those regions to to learn how we can apply these successes to other communities.
Figure 16: Applications by Country (bar chart)¶
This bar chart shows the number of applications received from each country. The United States, with 453 applications, is not included in the chart to avoid skewing the remaining data. Source data can be found in Table 10 in the appendix.
Figure 17: Completion Rates by Country (bar chart)¶
This bar chart shows the percent completion rates for each country. Not all countries with applications are represented here as some may have applied but not yet attended an event. Source data can be found in Table 10 in the appendix.
Figure 18: Applications by country (map)¶
This map shows the number of applications received from each country between July 1, 2016 and June 30, 2018. Countries in darker shades had higher application counts. Source data can be found in Table 10 in the appendix.
Figure 19: Completion rates by country (map)¶
This map shows the percent of applicants who attended instructor training events before March 31, 2018 and completed their Instructor certification by June 30, 2018. The global average is 56.6%. Countries in darker shades had higher rates of completion. Source data can be found in Table 10 in the appendix.
Open Applications by Month¶
Lastly, we looked at the frequency of open applications. This was important to help get a sense of overall demand for the training we offer.
July 2016 is likely an anomoly as that is when we introduced this application process and may show a surge in applications because of people who were waiting on new process to apply. Excluding this outlier, we saw an average of 33 applications per month, or approximately one a day. This demonstrates a strong demand for our instructor training from people who may not be affiliated with a member site.
Figure 20: Open Applications by Month¶
The bar chart below shows the number of applications received each month from July 2016 through June 2018. The red line shows the overall average of 33. Source data can be found in Table 11 in the appendix.
Summary¶
This data demonstrates the strong and ongoing demand for The Carpentries Instructor Training program.
We see that most of our applicants have little to no training in teaching, yet most are also the primary instructor for a full course. Software Carpentry and Data Carpentry workshops are motivated by recognizing that universities are not teaching computational programming skills, even though scientists and researchers need them. In the same light, our Instructor Training events may be motivated by people recognizing the need for a skill they otherwise do not have access to.
Our overall completion rate is just over half (56.6%), which means almost half of our trainees may be looking at this not as a means to teach with The Carpentries but also as a way to become better teachers themselves. This training can be seen not only as a pathway into The Carpentries community but also as a general professional development opportunity, especially for teaching faculty. This suggests that The Carpentries should explore our own motivations for offering instructor training. Our original reason for offering instructor training is to grow our community of instructors who can run Carpentries workshops at sites around the world, increasing the impact of Software and Data Carpentry workshops. Another reason may also be to fill a need that teaching faculty and other professionals have to improve their own skills. This would mean that badge rate is not necessarily a measure of success, and that we should look to other measures to demonstrate our impact.
Looking at those who do complete the badging process, we see that previous experience is a strong indicator of certification. Those with previous experience as helpers, instructors, or contributors badge at higher rates than the average. Community is an important value of The Carpentries, and we see the impact of this when people who already know our community members and are familiar with our curricula and teaching practices are more likely to become badged instructors.
We also see that demand is constant, with about one application coming in each day. Knowing that not all applicants will be accepted or will enroll, this still means that The Carpentries needs to examine our capacity to offer instructor training to the open applicant pool, especially in relation to the obligations to member sites.
Appendix¶
Table 1: Workshops by Carpentry By Year¶
This table shows the number of Data Carpentry (DC) and Software Carpentry (SWC) workshops each year. Data for 2018 is a projection. The proportion of workshops in the first two quarters of 2017 relative to the full year was applied to actual first two quarters data from 2018.
Table 2: List of countries that held their first workshop each year¶
This table the first year each country held its first workshop. Countries are sorted alphabetically within each year, not by when in the year the workshop occured.
Table 3: Total workshops by country by year¶
This table lists the number of workshop each country has run each year.
Table 4: Open Applications by Previous Training in Teaching¶
This table lists the number of applications, number attending, number of trainees completing badging, and percent completion rate their previous training in teaching.
Table 5: Open Applications by Previous Experience in Teaching¶
This table lists the number of applications, number attending, number of trainees completing badging, and percent completion rate their previous experience in teaching.
Table 6: Open Applications by Previous Involvement with The Carpentries¶
This table lists the number of applications, number attending, number of trainees completing badging, and percent completion rate by previous involvement with The Carpentries. Applicants could choose multiple roles so the totals will add up to more than the total number applications.
Table 7: Open Applications by Areas of Expertise¶
This table lists the number of applications, number attending, number of trainees completing badging, and percent completion rate by areas of expertise. Applicants could choose multiple domains so the totals will add up to more than the total number applications.
Table 8: Open Applications by Programming Language Usage¶
This table lists the number of applications, number attending, number of trainees completing badging, and percent completion rate by their frequency of programming language usage.
Table 9: Applications by Occupation¶
This table lists the number of applications, number attending, number of trainees completing badging, and percent completion rate by occupation.
Table 10: Applications by Country¶
This table lists the number of applications, number attending, number of trainees completing badging, and percent completion rate by country.
Table 11: Open Applications by month¶
This table shows the number of applications we received each month for our open training seats.