6 Rendering Lesson Content

6.1 Introduction

The Workbench is designed on the following principles for developing lesson materials for teaching:

Lessons are comprised primarily of episodes, with additional special pages that aggregate content including references, glossaries, key points and images
Pages within lessons have links back to their source content so people can edit the pages they are viewing
Lessons are writtin in Markdown or RMarkdown
Lessons can be rendered and previewed locally
Lessons can be tested, built and deployed online for broad consumption

The initial Workbench design phase (2020–2021) focussed on functionality that would enhance the above features of lessons:

A web template that was tested for its accessibility
A separate instructor and learner view
An internal package cache for R Markdown-based lessons

These features are implemented in a largely modular fashion, allowing separation of the various pieces of functionality and easing development and testing.

6.2 Lesson Build Internals

The main workhorse of the Workbench is the {sandpaper} package, specifically the sandpaper::build_lesson() function.

The sandpaper preview and deploy functions use build_lesson() to underpin their processes.

In summary, these three functions are:

venue	function	purpose
local	`sandpaper::build_lesson()`	render content for offline use
local	`sandpaper::serve()`	dynamically render and preview content
remote	`sandpaper:::ci_deploy()`	render content and deploy to branches

All of these methods will also call sandpaper::validate_lesson() (which also sets up global metadata and menu variables) and the two-step internal functions sandpaper:::build_markdown() and sandpaper:::build_site(). We’ll look in detail at the process for each below.

6.3 Preflight Checks

A number of pre-flight steps happen before any source files are built. These check for pandoc, validate the lesson, and configure global elements. The site is then built and combined with the global variables and templates.

So, before a lesson can be built, we need to confirm the following:

We have access to the tools needed to build a lesson (e.g. pandoc). This is achieved via the sandpaper::check_pandoc()
We are inside a lesson that can be built with the Workbench

6.3.1 `validate_lesson()`

The lesson validator peforms lesson validation through the methods in the pegboard::Lesson R6 class.

Validation first loads the lesson, via the sandpaper::this_lesson() function, which loads and caches the pegboard::Lesson object. It also caches elements that are mostly duplicated across episodes with small tweaks for each episode:

metadata in JSON-LD format
sidebar
extras menu for learner and instructor views
translations of menu elements defined in {varnish}

6.4 `build_markdown()`

Markdown generation for the lesson is controlled by the internal function sandpaper:::build_markdown().

6.4.1 Generating Markdown

For Markdown lessons, source content md files are copied to the site/built folder for processing.

For R Markdown lessons, the rmd files need to have content rendered to markdown first so that it can be further processed for rendering. This is carried out with the {knitr} R package in a separate R process.

Rendering performance

Because R Markdown files can take some time to render, we use MD5 sums of the episode contents (stored in the site/built/md5sum.txt file) to skip any files that have not changed.

sequenceDiagram
    autonumber
    participant episodes/episode.Rmd
    box rgb(255, 214, 216) The Workbench
    participant {sandpaper}
    end
    box rgb(230, 234, 240) Document Engine
    participant {renv}
    participant {knitr}
    end
    box rgb(255, 231, 168) Generated Files
    participant site/built/md5sum.txt
    participant site/built/episode.md
    end

    site/built/md5sum.txt -->> {sandpaper}: READ file cache
    {sandpaper} -->> {knitr}: RUN conversion
    episodes/episode.Rmd -->> {knitr}: PROCESS changed file(s)
    {knitr} -->> site/built/episode.md: WRITE Markdown
    {sandpaper} -->> site/built/md5sum.txt: WRITE file cache

Package Cache and Reproducibility

One package that is missing from the above diagram is {renv}.

When episodes are rendered from R Markdown to Markdown, the build environment is cached by the {renv} package.

Please see the package cache documentation for more information.

For either source file format, markdown has been written and the state of the cache is updated so if we re-run build_markdown(), then it will show that no changes have occured.

We use this function in the pull request workflows to demonstrate the changes in markdown source files, which is useful when package versions change, causing the output to potentially change.

After this step, the internal function sandpaper:::build_site() is run where the markdown files are converted to HTML with pandoc and stored in an R object. This R object is then manipulated and then written to an HTML file with the {varnish} website templates applied.

6.5 `build_site()`

The following sections will discuss the HTML generation (the following section), manipulation (the section after that), and applying the template (the final section) separately because, while these processes are each run via the internal sandpaper:::build_site() function, they are functionally separate.

6.5.1 Generating HTML

Each markdown file is processed into HTML via pandoc and returned to R as text. This is done via the internal function sandpaper:::render_html().

sequenceDiagram
    autonumber
    box rgb(255, 214, 216) The Workbench
    participant {sandpaper}
    end
    box rgb(230, 234, 240) Document Engine
    participant pandoc
    end
    box rgb(255, 231, 168) Generated Files
    participant site/built/episode.md
    end

    {sandpaper} -->> pandoc: LOAD pandoc with lua filters
    site/built/episode.md -->> pandoc: READ markdown
    pandoc -->> {sandpaper}: RENDER HTML as text

From here, the HTML exists as the internal body content of a website without a header, footer, or any styling. It is nearly ready for insertion into a website template. The next section details the flow we use to tweak the HTML content.

6.5.2 Processing HTML

The HTML needs to be tweaked because the output from pandoc still needs some modification. We tweak the content by first converting the HTML into an Abstract Syntax Tree (AST). This allows us to programmatically manipulate tags in the HTML without resorting to using regular expressions.

Using a language called lua, we perform the first stage of HTML modification. Then, we update links, images, headings, structure that we could not fix using lua filters.

Internationalisation

We also apply translations to some of the menu elements that are not templated in {varnish}.

More information about translated lesson elements can be found in the TODO documentation.

We then use the information from the episode to complete the global menu variable with links to the second level headings in the episode.

sequenceDiagram
    autonumber
    box rgb(255, 214, 216) The Workbench
    participant {sandpaper}
    end
    box rgb(230, 241, 255) R Object
    participant HTML(AST)
    end
    box rgb(230, 234, 240) Helper Package
    participant {xml2}
    end

    {sandpaper} -->> {xml2}: READ HTML
    {xml2} -->> HTML(AST): PARSE HTML
    activate {sandpaper}
    note right of HTML(AST): sandpaper:::fix_nodes()
    {xml2} -->> HTML(AST): update structure
    HTML(AST) -->> {sandpaper}: extract menu items
    note right of {sandpaper}: generate learner and instructor versions
    deactivate {sandpaper}

Working With XML

Much of the modifications listed here are carried out on XML formatted documents.

(HTML is indeed a descendent of XML!)

Working with XML data is performed in R by the {xml2} package.

There are helpful handbooks and guides at the following links:

6.5.3 Applying The Website Template

Now that we have an HTML AST that has been modified successfully, we are ready to write this to HTML. This process is achieved by passing the AST and associated lesson metadata to {pkgdown} where it performs a little more manipulation, applies the {varnish} template, and writes it to disk as the final rendered output. A web browser can now be used to view the lesson.

sequenceDiagram
    autonumber
    box rgb(255, 214, 216) The Workbench
    participant {sandpaper}
    participant {varnish}
    end
    box rgb(230, 241, 255) R Object
    participant HTML(AST)
    end
    box rgb(230, 234, 240) Helper Package
    participant {pkgdown}
    end
    box rgb(255, 231, 168) Generated Files
    participant site/docs/episode.html
    end

    activate {sandpaper}
    {sandpaper} -->> {pkgdown}: Set global menu variables
    HTML(AST) -->> {pkgdown}: Hand off HTML to pkgdown
    deactivate {sandpaper}
    activate {pkgdown}
    {varnish} -->> {pkgdown}: Load template
    {pkgdown} -->> site/docs/episode.html: WRITE website
    deactivate {pkgdown}

6.1 Introduction

6.2 Lesson Build Internals

6.3 Preflight Checks

6.3.1 validate_lesson()

6.4 build_markdown()

6.4.1 Generating Markdown

6.5 build_site()

6.5.1 Generating HTML

6.5.2 Processing HTML

6.5.3 Applying The Website Template

6.3.1 `validate_lesson()`

6.4 `build_markdown()`

6.5 `build_site()`