7  Testing {sandpaper}

7.1 Introduction

{sandpaper} is the largest package and the main user and deployment interface for The Workbench. The tests are all designed to work within a lesson context, which means a couple of things to be aware of.

  1. Some tests will take a long time to run. IO procedures are often one of the most time-consuming steps in computing and {sandpaper} does a lot of it.
  2. Most tests will be dependent on the previous tests in a file. Because the tests work on a folder on disk, we need to provision different scenarios of lesson state. It is far simpler to do this by side-effect rather than copying, setting up, and tearing down the state of the lesson for each and every test.

7.2 Test Setup

There is nothing that you as the contributor/developer need to do to set up for running these tests beyond what you have already done in your development setup. This section describes conceptually how {testthat} and {sandpaper} setup the testing environment after you run devtools::test().

In short, this is the process:

  1. {sandpaper} is loaded
  2. Helper test files are loaded
  3. The setup script is loaded, which provisions the test lesson
  4. Each test file matchingtests/testthat/test-*.R is run, resetting the test lesson at the top of each file.

7.2.1 Test Helpers

Sandpaper has a few test helpers that handle some of the more tedious side-effects of testing.

helper-child.R
Sets up an episode that contains a child document for testing.
helper-hash.R
Expectation expect_hashed() that an episode file MD5 sum (expected) matches the MD5 sum (actual) we recorded in the site/built/md5sum.txt database.
helper-processing.R
Provides an output string that demonstrates a screen output of a file being processed by {knitr}. This is used with expect_output().
helper-snap-transform.R
A function that is passed to the transform parameter of expect_snapshot() and will mask the temporary directory path so that the snapshot does not continuously invalidate on every run.
helper-translate.R
Defines three expectations to test translations: expect_set_translated(), expect_title_translated() and expect_h1_translated(). All are documented in the helper file and in test-utils-translate.R

7.2.2 Setup Script

The first script to run is tests/testthat/setup.R, where a test lesson and a local git remote is created and stored in a temporary location for the duration of the test suite and a reset function is exposed for the tests.

7.3 Conditionally Skipped Tests

Each link below will open a code search for the different types of skipped tests in {sandpaper}

skip_on_os
These are often tests that are skipped on Windows, usually for the reason that {renv} and filepaths behave slightly differently on Windows in a continuous integration setting.
skip_if
Tests that are skipped under various conditions. For example, if Git is not installed on a system, we should not test any of the CI functions because they rely on Git.
skip("<for reasons>")
This pattern is more rare. It’s really useful in a situation where you are refactoring and know that a lot of tests will fail. If you sprinkle in these skips, you can focus on testing the core functionality of the refactor and then address the side-effects. Regarding the skips that remain: during testing, sometimes we encounter a ghost in the machine and we cannot set up the right conditions to run the test properly or the test was created with an earlier model of the package that we haven’t been able to shake. In these cases, instead of deleting the test or commenting out code, we add the skip() function and write a message of why we skipped it so if we need to come back to it later, we can.

7.4 Continuous Integration

7.4.1 Package Cache

Running tests on Continuous Integration is tricky in part because we need to set up a {renv} package cache to work on Mac, Windows, and Linux systems. In practise, we have to set up a specific RENV_PATHS_ROOT folder for each system.

7.4.1.1 Windows

For Windows, the setup is even more complex because there are weird caveats in how pandoc and {renv} work on the CI version of Windows.

7.4.2 Dependencies

At the moment, we test the current versions of dependencies when we are running tests in the test-coverage.yaml file. For the R-CMD-check.yaml file, however, we test the development version of {renv}. The reason why we do this is because in March 2023, {renv} 0.17.0 was released and subsequently broke bioconductor-based R Markdown lessons and new R Markdown lessons that needed to be bootstrapped (see sandpaper#406).

This can lead to a situation where the tests will pass on the test coverage check, but fail for R CMD check, which is diagnostic because it tells us that there is an upstream issue in {renv} that we can address before it becomes a problem after a CRAN release.