Appendix B — Examples and Flight Rules

aka: what to do when something’s not working

This part of the developers manual is a living document and serves as a window into the mind of a Workbench Maintainer as they address issues and new feature requests.

The following sections of this appendix will not distinguish between bugs and features, because often the process for fixing bugs is not sufficiently different than that of adding a new feature (though there are exceptions, which I will highlight below). Rather, a more logical division is to distinguish between if the issue is within The Workbench Packages, In an Upstream dependency, with GitHub Actions, and finally, issues that are more structural in nature and require more extensive user experience (UX) testing.

B.1 Within The Workbench R Packages

Bugs or features in this category are entirely within our control and are theoretically the easiest/most quick implementations. Items in this category can be split into either single package items which can be fixed with a single pull request or cross-package items, which require coordination of pull requests to achieve.

B.1.1 Single Package

In this section, we outline issues that are addressed within a single package. Note that this does not indicate that these issues are straightforward to address.

B.1.1.1 Single-Function Issues

If you are here, you have determined that the bug or feature that you are working on will affect a single function or data pathway. These issues are often the most straightforward to address. Below, I’ve documented narratives for these issues.

Markdown file for 404 page created with read-only permissions
s479 Situation

The user is attempting to build a lesson, but they are unable to because an error appears during the “Create 404 page” step:

── Creating 404 page ───────────────────────────────────────────────────────────────────────────────────────────────────
Error in file(file, ifelse(append, "a", "w")) : 
  cannot open the connection
In addition: Warning messages:
[snip]
7: In file(file, ifelse(append, "a", "w")) :
  cannot open file '/tmp/RtmpmaiZ5B/file73cdc48b90540.md': Permission denied
s479 Diagnosis

This permissions issue was a problem with copying a read-only file without adjusting the subsequent permissions. It stemmed from build_404() calling render_html() with the presence of a links.md document at the top of the lesson. Packages in the user’s R library are installed by the systems administrator and the user does not have permissions to write or append any file in that folder. When build_404() runs, it passes a template markdown file to render_html(). If there is an option called sandpaper.links set to a filepath, then this function will copy the input file to a temporary file and append the links to that temporary file before rendering it to HTML. Because the input file was read-only, copying the file to the temporary directory retained its permissions and we were unable to append the links.

s479 Solution

Pull Request: carpentries/sandpaper#482

The solution was to add a single line adding user write permissions before the links file was appended: fs::file_chmod(tmpin, "u+w").

s479 Alternative Solution

This additionally could have been avoided by temporarily unsetting the sandpaper.links option for the build_404() function before it calls render_html(). This would prevent it entering the loop where it appends the links, making the process slightly faster. The only downside is that we would need to do that for all of the other templated pages (though I believe that might be the only one).

s479 Narrative

  • Issue: carpentries/sandpaper#479

  • First impression: this is running on a version of Linux that Zhian does not know anything about. My thoughts are that it will be a difficult fix.

  • I suspect the bug might be due to the {fs} package and ask for the user if they could test out the following code snippet:

    tmp <- fs::file_temp(ext = ".md")
cat("test\n", file = tmp, append = TRUE)
readLines(tmp)

  • I looked again did not recognise the code snippet where the issue seemed to be (file(file, ifelse(append, 'a', 'w'))) - so this seems to be a problem with a non-Workbench package.

  • initially went looking for this code in the fs package, but GitHub search showed that this code doesn’t appear there either.

  • next guess is the cat function, which is used to add link references to the end of files. I searched the R code base and found the snippet in the cat()function

  • that seemed to be where that code snippet was coming from, but the problem really originated a few lines above the call to cat: when the template for the 404 page (which is saved where the package was installed) is copied—file_copy() copies a file and all of its permissions—so the copy is read-only for non-admin users.

  • I opened a PR to test for the error, then applied a fix to prevent it from being thrown again.

  • Asked the reporter to install the patch on their system and report back on whether it worked.

  • They reported back that it did—and pointed out a typo!

  • After merging PR, create a new release (see process in The Release Workflow)

B.1.1.2 Multi-Function Issues

B.1.1.3 Test Failures With No User Impact

B.1.2 Aross Packages

Some features require manipulation across packages. Most often, this will mean a feature that is needed between {sandpaper} (user interface) and {varnish} (templates). To a rarer extent would you need features bewteen {sandpaper} and {pegboard}. You would almost never need a feature between {pegboard} and {varnish}.

B.1.2.1 Order of Operations

Features across packages require a bit of extra care because you are dealing with two moving targets. Remember that it’s not impossible to change features spanning packages in a backwards-compatible manner. If you are worried that something will break, I will remind you that not only do we have unit tests, but we also have The Workbench integration tests that allows us to test development versions of The Workbench across a variety of lessons. On top of that, you have the ability to set the versions of The Workbench used to build any given lesson. With these three tools, you should be able to address any issue that arises from a multi-package feature/issue.

This section talks about the order of operations in testing and release, which can be boiled down to these general rules:

  1. Always test {sandpaper} against the development version of the other package
  2. Release the dependent package first. Unless you have made breaking changes, {varnish} and {pegboard} should be released before {sandpaper}.

With that, here are scattered details of the order of operations for developing features for two packages.

Make sure to use the Remotes field in the {sandpaper} DESCRIPTION file to ensure that you are testing against the correct version of whatever other package you are using. That is, if you are testing feature-a in {sandpaper} and there is a companion feature-a branch in {varnish}, you want to specify Remotes: carpentries/varnish@feature-a in the DESCRIPTION file for {sandpaper}. This will not only allow the unit tests to work against the correct version of the dependent package, but it will also allow the integration tests and user tests to work.

In general, when you are creating a feature across two packages you will want to ensure the following:

B.1.2.2 Examples

Adding DOI badge for peer reviewed lessons

Replace {pkg} with the first letter of the package the issue occurs in and {num} with the issue number

s535v97 Situation
s535v97 Diagnosis

In carpentries/workbench#67 and carpentries/varnish#40, Toby Hodges had pointed out that while we had badges for lessons in pre-alpha, alpha, and beta phases of development, we did not yet have a badge for a published lesson with a DOI.

In the former infrastructure, we had indicated these statuses by adding a banner to the top of the lesson. This banner would take up extra room at the top of the lesson, so we changed it to a badge since we had room next to the logo. In this case, Toby was asking if we could add a doi key to the config.yaml that would allow the badge to be displayed on the website.

s535v97 Solution

Toby had opened two pull requests on 2023-10-30:

  • carpentries/sandpaper#535 where he modified the inst/templates/pkgdown-yaml-template.txt and create_pkgdown_yaml() to include a doi element.
  • carpentries/varnish#97 where he added HTML to the header that would insert a stylized DOI badge

On 2023-11-10, Zhian Kamvar added a catch for full URL of a DOI vs the raw DOI itself in {sandpaper} and determined that the contribution to {varnish} needed a couple of extra tweaks before being merged.

{varnish} was merged at 11:43AM and {sandpaper} was merged at 11:47AM

s535v97 Alternative Solution

One of the tools that Toby had used was passing data through create_pkgdown_yaml(). It could have been done without that function and just passed through initialize_metadata() or something similar. For details on how data flows from {sandpaper} to {varnish}, read Data Flow from Source to Website in the {sandpaper} documentation.

It’s important to understand why Zhian used create_pkgdown_yaml() in the first place: to get {pkgdown} to recognise the site/ folder as a package so he could hijack {pkgdown}’s machinery to provision a website. When he first started, he did not realize that he could pass any number of data elements directly to the {varnish} templates (as is done through the global variable caches), so everything was written to site/_pkgdown.yaml. This was cumbersome because it involves the IO for the pkgdown template file, the config file, and the resulting pkgdown yaml. It was much easier to load the config file via initialize_metadata() and use that to carry the configuration items.

B.2 Upstream R Packages

B.2.1 renv

B.2.1.1 Background

The {renv} package is a key player for allowing {sandpaper} to provision and maintain the R packages required to build R-based lessons. The motivation and strategy for how it works can be found in the Building Lessons With A Package Cache article in the {sandpaper} docuementation.

It’s worth diving into carpentries/sandpaper#21 to see the discussion and thoughts around the origin of the design for using this feature. It was implemented during a three week period between 2021-08-24 and 2021-09-16, as detailed in the pull request carpentries/sandpaper#158.

B.2.1.2 In practices

We have to consider {renv} in practice from the standpoint of both local computers and on GitHub, which can behave very differently and require different tools to address their tasks.

There are three tools and packages that use {renv}:

  1. {sandpaper} is designed to provide a way to manage dependencies in a lesson
  2. {vise} was originally intended as a project to split out {sandpaper} code that used {renv} to simplify the testing. At the moment, it provides utilities for automatically provisioning C libraries on Ubunutu Linux and running the equivalent of sandpaper::update_cache() in a GitHub Actions context.
  3. carpentries/actions these contain R code within YAML files 😱 that will call {renv} and {vise}.

B.2.1.3 Debugging Tips

B.2.1.3.1 Setting up a reproducible environment

Browse the {renv} issues opened by @zkamvar. In nearly all of these issues, I provide a reproducible example. They generally follow the pattern of:

  1. create a temporary file
  2. make it a directory and move there
  3. add any files that are needed before the renv project is set up (if specific to problem)
  4. set up a {renv} project with renv::init()
  5. demonstrate problem
tmp <- tempfile()
dir.create(tmp)
setwd(tmp)
writeLines("library(R6)", "test.R")
renv::init(profile = "test")
# demonstrate problem here
B.2.1.3.2 Choosing a minimal example

Presenting a failing CI run with a Workbench lesson is reproducible, but it’s not minimal. Presenting the same with a smaller lesson is still not minimal. Often times, the issue involves detecting and installing a new package. In this case, you want to choose a package that has few to no dependencies and is not listed as a dependency for knitr. One example that I use often is the {cowsay} package. It has a total of three dependencies and is not depended on by anything. If I need a quick package with zero dependencies, I will reach for {R6} or {aweek}. Both of these packages are under 100 Kilobytes, are pure R code, do not have any dependencies, and do not require compillation.

B.2.1.3.3 When you just can’t reproduce it locally

There have been times when {renv} seems to fail only on GitHub Actions. This was the case for {renv} version 0.17.0, as I reported in rstudio/renv#1161.

The important thing in these situations is to stay calm and try to narrow down as much as possible the exact conditions that will create the problem. Once you have those, you can open an issue on the {renv} issue tracker. If you are at this point, it’s important to not expect this to be resolved quickly, because it is likely out of your control. The best you can do is to try the debugging techniques that Kevin provides and report back on the issue thread.

Once the issue is resolved: Thank Kevin for his help. This is a very important point. Maintainers often only hear from their users if somehting is going wrong, so it’s important to let them know that they are appreciated.

B.3 GitHub Actions

There is a category of issues that fail explicitly with GitHub actions only. As with the other issues, it is important not to panic when you encounter these, but instead, take stock of what happens when these happen. In general, here are the steps you should take to diagnosing problems:

  1. Prepare your inbox to filter messages from GitHub. A good way is to check that the subject line matches “Run failed 01 Build and Deploy” or “Run failed 03 Update Cache”.
  2. Find and read the error message from the log. You can find this by looking for the red X and expanding that section. It will normally scroll down to the error.
  3. Read the context of the error message. Most errors happen because of problems upstream, so it’s important to see what was happening when that error happened
  4. Restart the action. GitHub has a button near the top right of the action. If it’s ephemeral (e.g. networking error) then it will run fine on the next run.
  5. Test the build locally
  6. Test it on the carpentries/sandpaper-docs repository
  7. Test it on a brand new repository created via the templates (https://bit.ly/new-lesson-md or https://bit.ly/new-lesson-rmd)

Below I will attempt to outline common problem areas and their potential solutions. Note: this is not an exhaustive list, but you have to remember that in these situations, you are effectively debugging someone else’s computer, so patience is absolutely required because this is more difficult than plucking a single grain of rice with metal chopsticks.

B.3.1 Networking Failures

These are often the easiest problems to solve. You will encounter an error that says somthing like “network timed out” and it fails when either checking out the repository or installing pandoc. In these situations, you should check https://status.github.com to see if this is a problem with GitHub, and then you should restart the build. Most of the time, the build will work on the second try.

B.3.2 Upstream System Dependency Issues

SSL error

bioc-intro109 Situation

On 2023-07-20, @lgatto, who maintains the carpentries-incubator/bioc-intro repository reported an error in their builds during the “Setup Package Cache” of the sandpaper-main.yaml workflow:

Register Repositories
Using github PAT from envvar GITHUB_PAT
Error: Error: Failed to install 'vise' from GitHub:
  SSL peer certificate or SSH remote key was not OK: [api.github.com] SSL: no alternative certificate subject name matches target host name 'api.github.com'
Execution halted
Error: Process completed with exit code 1.

This error was causing the carpentries/vise package to not be installed and the {renv} package cache could not be provisioned for lessons. Because it involved the {renv} package cache, this was a problem that was limited to R-based lessons.

bioc-intro109 Diagnosis

The cause of the problem was identified by Gabor Csardi as a bug specific to the development version of curl on Ubuntu: https://bugs.launchpad.net/ubuntu/+source/curl/+bug/2028170.

The key error message here was SSL: no alternative certificate subject name matches target host name 'api.github.com'. The curl program was checking the validity of the SSL certificate for github.com, but it was ignoring the rules for *.github.com, so when it found api.github.com, it saw that as an invalid site that should not be trusted.

Even though the default version of curl on GitHub’s runners is not the dev version, when we provision the workbench in the “Setup Lesson Engine” step, we query the system dependencies from the R-universe:

$ curl https://carpentries.r-universe.dev/stats/sysdeps 2> /dev/null \
  | jq -r '.headers[0] | select(. != null)'
libcurl4-openssl-dev
libfontconfig-dev
libfreetype-dev
libfribidi-dev
libharfbuzz-dev
libicu-dev
libgit2-dev
libjpeg-turbo8-dev
libpng-dev
libtiff-dev
libxml2-dev
libxslt1-dev
libssl-dev

This installed the development version of curl into our system and introduced the bug during the “Setup Package Cache” step.

bioc-intro109 Solution

The solution was to wait for a fix.

bioc-intro109 Alternative Solution

An alternative solution that we tested was to switch the download method for R packages to “wget”, but this was a non-starter because this caused the installation times for R packages and the workbench to rise because they would need to be compiled (posit PPM did not serve the binary packages over wget).

bioc-intro109 Narrative

B.3.3 Workflow Mis-Configuration

Sometimes the workflow files themselves are mis-configured. In these cases, the fix will involve updating the files in sandpaper’s inst/workflows/ directory. If the pr-comment.yaml or update-workflows.yaml files are not affected, then a pull request will be created automatically to all lessons within a week after you submit the patch, but if these files are affected, then you will have to manually submit the patch.

files with spaces in names cause pr-comment.yaml workflow to fail

s399 Situation
s399 Diagnosis
s399 Solution
s399 Alternative Solution
s399 Narrative

B.3.4 Permissions Changes

B.3.4.1 carpentries-bot

B.3.5 Actions

B.4 Structural Features