3 Testing The Workbench
We are still assembling the documentation for this part of the site. If you would like to contribute, please feel free to open an issue.
3.1 Introduction
The first stage of your testing journey is to become convinced that testing has enough benefits to justify the work. For some of us, this is easy to accept. Others must learn the hard way.
— Wickham and Bryan, Testing Basics, R Packages second edition
If you use software that lacks automated tests, you are the tests.
— Jenny Bryan source tweet (2018-09-22 01:13 UTC)
Every single package that runs code in the lesson infrastructure is tested before it ever reaches any lesson. This is important because we want to give the lesson authors and maintainers as much freedom as they need to write a lesson while maintaining predictability and integrity. We also want to give our community confidence that this system works.
Whenever a new feature or bug fix is added to The Workbench, it is imperative that a test is associated and verified before it gets sent into production.
Tests can be run locally and via continuous integration. This page introduces some of the testing strategies used in The Workbench and the caveats that come with these strategies.
3.2 Unit Testing
The tests under test/testthat/ are run in alphabetical order using the {testthat} package (see https://r-pkgs.org/testing-basics.html) via devtools::test() or devtools::check().
3.2.1 Conditionally Skipped Tests
The tests often need special conditions in order to run and sometimes those conditions are not possible. One of the most common conditions to skip is if the testing happens on CRAN. They are very hawkish about how long test suites can run and it’s often difficult to detect the state of a CRAN machine, so it’s better to skip long-running tests or those with complex environmental dependencies on CRAN (which does not yet apply to {sandpaper}).
3.3 Continous Integration
All the unit tests are run in continuous integration for every push and pull request that occurs. They also run every week. This provisions the current releases of the R package dependencies along with development versions of critical dependencies such as {renv}.
In continous integration, we run on with the following conditions to make sure it works not only on GitHub, but also on local user machines:
- test coverage (no package structure) with released versions on Ubuntu Linux (though reporting is stalled)
- For each platform (Ubuntu Linux, macOS, and Windows)
R CMD check, which checks the structure of the package and documentation- all run on these versions of R: current, devel, and two previous R versions
Because of occasional provisioning failures on macOS and Windows, we require only that Ubuntu Linux latest version passes check for merging pull requests.
3.4 Lesson Integration Testing
Unit tests are great for testing all functionality using known and stable inputs, it is important to test using known inputs that are in a constant state of flux: live lessons. This is where The Workbench Integration Test comes in. It will run a weekly test on a defined set of lessons using the current development versions of varnish and sandpaper. These tests are useful for three purposes:
- ongoing integrity for the workbench lessons lessons that use different features of The Workbench such as R code execution
- real-world effects of new sandpaper and varnish versions (including those in pull requests)
- inspecting changes in HTML and markdown output
These lessons that we use are
- Instructor Training
- Lesson with the most content, contributors, activity, and used features. This particular lesson is a bit of a stress test for the infrastructure.
- R for SocialScientists
- This is one of the first R-based lessons to be transitioned and it uses the tidyverse as a dependency.
- Workbench Documentation
- The workbench documentation. If this doesn’t work, nothing will.
- Raster and Vector Geospatial Data with R
- This lesson uses R packages that rely on a geospatial software stack, which can be complex. Failures here likely mean that there are problems with provisioning external C libraries.
- BioConductor RNAseq
- Lesson using BioConductor R packages by people at BioConductor. If this does not work, then there likely is a provisioning problem between BioConductor and {renv}.
3.4.1 Testing sandpaper and varnish pull requests
To test a pull request version, you can head over the the main workflow and use the button that says “Run Workflow”. When you want to test a varnish or sandpaper pull request, you can use the [REPO]#[PR] syntax (e.g. carpentries/sandpaper#429 to run sandpaper pull request 429) in the entry fields for varnish and sandpaper version. If you don’t have a pull request to work from, you can use the [REPO]@[REF] syntax (e.g. carpentries/sandpaper@test-this-thing to run sandpaper test-this-thing branch).
3.4.2 Inspecting changes
The output for all the tests are stored in branches that are named respective for their test repositories. For example, datacarpentry/r-socialsci/markdown and datacarpentry/r-socialsci/site contain the markdown and HTML outputs for the R for Social Scientists lesson. By inspecting the diffs from the commits, you can see how the output has changed from run to run, which is useful if you are confirming that a feature will be automatically deployed.
3.5 Ad-hoc Testing
There are times when you cannot automate your way through testing and you just have to suck it up and get your (virtual) hands dirty. You find yourself in this kind of situation if you are testing out GitHub workflows, GitHub actions, or if you are implementing a new feature and you need to see that it works reliably and safely. It’s this point in time when you use ad-hoc testing on a brand new lesson repository that you give yourself permission to mess around in and will delete when finished.
3.5.1 GitHub Workflows
A GitHub Workflow is a YAML document that lives inside the .github/workflows folder of a repository. This sets up the environment needed to build a lesson. When you debug these, ask yourself if you really want to update the GitHub Actions instead. Because these are copied to each lesson, they need to be updated in each lesson (which is accomplished through the automated pull request workflow). If you’ve determined that the workflow needs to be modified, you can modify them inside your test lesson until you get the desired results. Once that is done, copy them over to sandpaper/inst/workflows and add a NEWS item that’s under the heading ## CONTINUOUS INTEGRATION describing your change.
3.5.2 GitHub Actions
A GitHub Action is a single step in a GitHub Workflow and can be written in nearly any language.
The GitHub actions that are in carpentries/actions are written in BASH, node JavaScript, and R and cobbled together with YAML. When developing a new feature, work on a branch and then, in your shiny new test lesson, replace the @main in the GitHub Workflows to your branch name. This way, you can know immediately if the fix or feature worked without having to interrupt someone’s flow.
3.5.3 New config items or settings
If you implement a new config item (e.g. a lang: tag), a temporary lesson is a great way to test it. To do so, you can use the sandpaper: or varnish: keys in your lesson config to specify the version of sandpaper or varnish you want to test.