Analysis of Software Carpentry’s Post-Workshop Surveys

Contributors: Kari L. Jordan, Ben Marwick, Naupaka Zimmerman, Erin Becker, and Jonah Duckles
July 2017

For nearly 20 years, Software Carpentry has developed material and trained instructors to teach computing skills to researchers in science, medicine, engineering, and other disciplines. This report is an analysis of the post-workshop survey responses collected for Software Carpentry’s workshops from March 2015 through July 2017. In this two year period, nearly 4,000 responses were collected.

A PDF of the survey questions, the data used in this analysis, and full R code are located on the assessment repo on GitHub. Special thank you to Ben Marwick, Naupaka Zimmerman, Erin Becker, and Jonah Duckles. These individuals made valuable contributions to the code that was used to create the figures in this report.

Community members are invited to contribute code to this analysis. Feel free to use the data and tell us about your findings.

Respondent Demographics

A host of initiatives have been developed and implemented globally to address gender disparities in computing. Software Carpentry’s volunteer instructors have hosted hundreds of workshops since 1998, and the post-workshop survey data shows parity in attendance of males compared to females.

Gender n %
Female 1575 48.5
Male 1597 49.2
Other 10 0.3
Prefer not to say 64 2.0

A breakdown of Software Carpentry’s learners by status is provided below.

42% of Software Carpentry’s post-workshop survey respondents are Graduate Students.

A breakdown of respondents by research domain/field of work or study is provided below. Respondents were asked to check all that apply. The majority of Software Carpentry learners work in Life Sciences.

Research Domain n %
Life Sciences (Genetics, genomics, bioinformatics ) 927 24.9
Life Science - Organismal/systems (ecology, botany, zoology, microbiology, neuroscience) 894 24.0
Planetary sciences (geology, climatology, oceanography, etc.) 245 6.6
Mathematics/statistics 225 6.0
Physics 217 5.8
Civil, mechanical, chemical, or nuclear engineering 167 4.5
Medicine and/or Pharmacy 161 4.3
Chemistry 149 4.0
Social sciences 149 4.0
Library and information science 121 3.2
Economics/business 98 2.6
Humanities 98 2.6
Psychology 88 2.4
Education 79 2.1
High performance computing 79 2.1
Space sciences 33 0.9

Respondent Perception of Workshop Content and Atmosphere

Software Carpentry has developed an interactive instructional approach that includes direct instruction (i.e. explicit teaching and demonstrations), indirect instruction (i.e. problem solving and discovery), and experiential learning. Respondents have mixed feelings about the pace of the workshop they attended, as outlined below.

Pace n %
Just right 1317 38.5
Slightly fast 1057 30.9
Slightly slow 730 21.3
Too fast 157 4.6
Too slow 164 4.8

Respondents were asked to indicate their perception of the balance of lecture to hands-on work in the workshop. A breakdown of their responses is provided below.

Balance: Lecture to Hands-On Work n %
Too much lecture 59 1.7
Slightly too much lecture 354 10.3
Balanced (lecture/hands-on) 2773 81.0
Slightly too much hands-on 204 6.0
Too much hands-on 35 1.0

81% of respondents felt the workshop they attended was well balanced between lecture and hands-on learning.

Learners were asked to rate their level of agreement on a scale of 1 (Strongly Disagree) to 5 (Strongly Agree) for the following statements regarding the atmosphere and content of the workshop they attended:

  • Atmosphere: The overall atmosphere of the workshop was welcoming.
  • Material: The material presented matched the workshop description.
  • Recommend: I would recommend this workshop to a friend/colleague.
  • Skills: I learned skills that I will be able to use in my research/work.
  • Worth: The workshop was worth my time.
  • Information: The amount of information covered at the workshop was reasonable for allotted time.

The following Likert chart is an analysis of learner responses to the statements above.

The data strongly suggests that Software Carpentry provides a welcoming environment for its learners where the material not only matches the workshop description, but is worth the time learners spend learning it. Learners acquire skills they are able to apply to their research and/or job function in the time allotted over the two-day period. Lastly, learners feel impressed to recommend the workshop to a friend or colleague.

Respondent Perception of Workshop Instructors and Helpers

A strength of Software Carpentry’s ecosystem is its instructors and helpers. Learners who responded to Software Carpentry’s post-workshop survey were asked to rate how they felt instructors and helpers worked as a team based on the following criteria:

  • Considerate: Instructors/Helpers were considerate.
  • Enthusiastic: Instructors/Helpers were enthusiastic.
  • Communicators: Instructors/Helpers were good communicators.
  • Clear.Answers: Instructors/Helpers gave clear answers to your questions.

The two Likert plots below provide an analysis of respondent answers.

Software Carpentry’s workshop instructors and helpers create an environment where learners are able to receive clear answers to their questions. Additionally, instructors and helpers communicate clearly with learners and are both enthusiastic about the material being taught and considerate of Software Carpentry learners.

Additionally, 94.1% of respondents felt there were enough helpers in the workshop they attended.

Respondent Self-Reported Knowledge of Tools Covered

Software Carpentry seeks to know more about learners’ experience with the tools covered in their workshop before attending, and after attending the workshop. A series of questions were developed around learners’ prior knowledge and perception of the tools they learned. Workshops are self-organized, and each workshop includes the following core topics:

  • Automating tasks using the Unix shell;
  • Structured programming in Python, R, or MATLAB; and
  • Version control using Git or Mercurial.

Learners were asked to indicate their level of knowledge of the Unix shell, R, Python, Git, and SQL prior to attending a Software Carpentry workshop. Not all workshops cover SQL.

Firstly, a breakdown of how much of the information presented was new to the respondents is provided below.

25% of respondents said about half of the material they learned in a workshop was new to them, and 41% of respondents reported that most of the information they learned in the workshop they attended was new to them, while 15% reported that all of the information they learned was new.

Taking a look at each tool more closely, the Likert plot below provides a breakdown of respondents’ self-reported knowledge of the tools covered before attending a workshop. From the figure we see that more than half of respondents had little or no knowledge of the tools covered in their workshop.

An interesting result is the percentage of learners who had little or no knowledge of Git and SQL (78% and 79% respectively). However, 36%, 39%, and 33% of respondents already had some knowledge of R, the Unix shell, and/or Python, respectively.

Individuals who are new to computer programming tend to be intimidated for lack of familiarity with syntax and terms. As Software Carpentry learners have varying knowledge levels of the tools covered pre-workshop, we are interested in understanding learners self-reported feeling of intimidation with these tools. The Likert plot below provides a breakdown by tool (Git, SQL, etc.).

Taking a closer look at the responses by percentage in the table below, we see that at least 44.5% of respondents felt at least one of the tools covered in the workshop they attended was either slightly or very intimidating.

Perception Git Python R SQL Shell
Not at all intimidating 9.2 11.9 15.6 11.5 14.9
Not very intimidating 10.5 14.2 15.4 10.0 15.3
Neither intimidating nor unintimidating 23.5 24.9 24.4 25.1 21.3
Slightly intimidating to me 33.7 26.1 24.1 27.5 28.5
Very intimidating to me 23.1 22.9 20.4 26.0 20.0

Self-Reported Knowledge Increase

Software Carpentry is not only interested in creating an atmosphere where learning programming becomes less intimidating, but we want for learners to leave with increased knowledge of the tools that were covered in their workshop. The Likert plot below provides a breakdown of respondents’ self-reported knowledge increase.

One thing to note is that SQL is not covered in all workshops (60% of respondents said this was not covered). This explains the large portion of respondents that had no increase in their knowledge of SQL.

51% of respondents said their knowledge of Git increased a great deal. This is great news, as 78% of respondents reported having little or no knowledge of Git before attending a Software Carpentry workshop!

Self-Reported Knowledge Increase: Pre/Post Comparison

Now let’s take another look at learners’ prior knowledge with the tools covered in the workshop they attended, compared to after the workshop. The grid below provides the breakdown by tool

Motivation to Learn Post-Workshop

Another goal of Software Carpentry is for learners to leave the workshop motivated to continue their learning. From the figure below, we see that learners are more motivated to continue learning and improving upon the skills that were covered in their workshop.

Respondent Ability to Perform Computing Tasks

Motivation is important, but being confident in your ability to complete specific computing tasks is an equally important goal of Software Carpentry. The grid below shows respondents’ self-reported ability to complete tasks including:

  • Using pipes to connect shell commands
  • Writing a ‘for loop’ to automate tasks
  • Initializing a repository with git
  • Writing a function
  • Importing a library or package in R or Python
  • Writing a unit test in Python or R
  • Writing an SQL query

It also provides their self-reported level of confidence in being able to complete the tasks above after completing the workshop.

Let’s take a closer look with a table. The data shows an increase in confidence for nearly all of the computing tasks outlined above.

Skill Ability Pre-Workshop n %
Pipes Maybe 337 10.7
Pipes No 1942 61.8
Pipes Yes 865 27.5
Loops Maybe 439 13.9
Loops No 1211 38.5
Loops Yes 1499 47.6
Git-Repo Maybe 241 7.7
Git-Repo No 2274 72.2
Git-Repo Yes 633 20.1
Function Maybe 465 14.8
Function No 1024 32.6
Function Yes 1652 52.6
Import-Library Maybe 277 8.9
Import-Library No 1207 38.9
Import-Library Yes 1616 52.1
Unit-Test Maybe 416 14.5
Unit-Test No 1994 69.5
Unit-Test Yes 461 16.1
SQL-Query Maybe 225 9.2
SQL-Query No 1783 72.8
SQL-Query Yes 441 18.0
Skill Confidence Post-Workshop n %
Pipes Confidence increased a bit 979 31.8
Pipes Confidence increased greatly 944 30.7
Pipes Confidence increased slightly 629 20.4
Pipes N/A - Not covered at workshop 74 2.4
Pipes No change in confidence 453 14.7
Loops Confidence increased a bit 895 29.0
Loops Confidence increased greatly 726 23.5
Loops Confidence increased slightly 585 19.0
Loops N/A - Not covered at workshop 85 2.8
Loops No change in confidence 794 25.7
Git-Repo Confidence increased a bit 829 26.9
Git-Repo Confidence increased greatly 1430 46.4
Git-Repo Confidence increased slightly 455 14.8
Git-Repo N/A - Not covered at workshop 59 1.9
Git-Repo No change in confidence 307 10.0
Function Confidence increased a bit 922 30.0
Function Confidence increased greatly 801 26.0
Function Confidence increased slightly 546 17.8
Function N/A - Not covered at workshop 73 2.4
Function No change in confidence 734 23.9
Import-Library Confidence increased a bit 638 21.1
Import-Library Confidence increased greatly 866 28.6
Import-Library Confidence increased slightly 428 14.1
Import-Library N/A - Not covered at workshop 153 5.1
Import-Library No change in confidence 942 31.1
Unit-Test Confidence increased a bit 538 19.1
Unit-Test Confidence increased greatly 367 13.1
Unit-Test Confidence increased slightly 456 16.2
Unit-Test N/A - Not covered at workshop 812 28.9
Unit-Test No change in confidence 637 22.7
SQL-Query Confidence increased a bit 173 7.1
SQL-Query Confidence increased greatly 176 7.2
SQL-Query Confidence increased slightly 162 6.6
SQL-Query N/A - Not covered at workshop 1671 68.1
SQL-Query No change in confidence 271 11.0

Summary

Software Carpentry workshops improve learner skill, ability, and confidence in using computing tools like Python, Git, and the Unix shell. Additionally, respondents are satisfied with the caliber of workshop instructors and helpers. To close out this report, I offer a list of interesting questions that could be answered with this data, and encourage community members to get involved by using the data in this analysis to answer these questions:

  • Are there differences in gender for intimidation, confidence, and/or motivation to use the tools post-workshop?
  • Is there a relationship between respondent confidence and their motivation to use the tools post-workshop?
  • Is there a relationship between respondent knowledge increase and their confidence and/or motivation post-workshop?
  • Is there a relationship between respondent intimidation pre-workshop and their confidence and/or motivation to use the tools post-workshop?

What other questions can be answered from this data? Additionally, here are a few other questions that we can discuss as a community:

  • Why is SQL the least covered workshop topic?
  • Has your confidence and/or motivation to use the tools we teach changed as a result of being a member of Software Carpentry’s community? How? Why?