Fragment: Revealing Otherwise Hidden Answers In Jupyter Notebooks

Some notes culled from an internal feedback forum regarding how to provide answers to questions set in Jupyter notebooks. To keep things simple, this does not extend to providing any automated testing (eg software code tests, computer marked assessments etc); just the provision of worked answers to questions. The sort of thing that might appear in the back of a text book referenced from a question set in the body of the book.

The issue at hand is: how do you provide friction that stops the student just looking at the answer without trying to answer the question themselves? And how much friction should you provide?

In our notebook using data management and analysis course, some of the original notebooks took a naive approach of presenting answers to questions in a separate notebook. This add lots of friction and a quite unpleasant and irritiating interaction, involving opening the answer notebook in a new tab, waiting for the kernel start, and then potentially managing screen real estate and multiple windows to compare the original question and student’s own answer with the answer. Other technical issues include state managament, if for answer the desired answer was not in and of itself reproducible, but instead required some “boilerplate” setting up of the kernel state to get it into a situation where the answer code could be meaningfully run.

One of the advantages of having answer code in a separate notebook is that a ‘Run All’ action in the parent notebook wonlt run the answer code.

Another approach taken in the original notebooks was to use an exercise extension that provided a button that could be clicked to reveal a hidden anwser in the original notebook. This required an extension to be enabled in the environment in which the original notebook was authored, so that the question and answer cells could be selected and then marked up with exercise cell metadata via a toolbar button, and on the student machine, so that the exercise button could be actively be rendered and the answer cell initially hidden by the notebook UI when the notebook was loaded in the browser. (I’m not sure if it would also be possible to explicitly code a button using HTML in a markdown cell that could hide/reveal another classed or id‘d HTML element; I suspect any javascript would be sanitised out of the markdown?)

Most recently, all the notebooks containing exercises in ourr databases course now use a model where we use a collapsible heading to hide the answer. Clicking the collapsed header indicator in the notebook sidebar reveals the answer, which is typically copmposed of several cells, both markdown and code cells. The answer is typically provided in a narrative form, often in the style of a “tutor at your side”, providing not just an explained walkthrough of the answer but also identifying likely possible incorrect answers and misunderstandings, why they might at first appear to have been “reasonable” answers, and why they are ultimately incorrect.

This answer reveal mechanic is very low friction, and degrades badly: if the collapse headings extension is note enabled, the answer is rendered as just any other notebook cell.

When using these approaches, care needs to be taken when considering what the Run All action might do when the answer code cells are reached: does the answer code run? Does it create an error? Is the code ignored and code execution continued in cells after the hidden answer cell.

In production workflows where a notebook may be executed to generated a particular output document, such as a PDF with all cells run, how are answer cells to be treated? Should they be displayed? SHould they be displayed and answer cells run? This might depend on audience: for example, a PDF output for a tutor may contain answers and answer outputs, a student PDF may actually be two or three documents: one containing the unanswered text, one containing the answer code but not executed, one cotnaining the answer code and executed code outputs.

Plagiarising myself from a forthcoming post, as well as adding friction, I think wonder if what we’re actually trying to achieve by the way the answer is accessed or aotherwise revealed might include various other psychological and motivational factors that affect learning. For example, setting up the anticipation that pays-off as a positive reward for a student who gets the answer right, or a slightly negative payoff that makes a student chastise themselves for clicking right to the answer. At the same time, we also need to accommodate students who may have struggled with a question and is reluctantly appealing to the answer as a request for help. There are also factors relating to the teaching, insofar as what we expect the student to do before they reveal the answer, what expectation students may have (or that we may set up) regarding what the “answer” might provide, how we expect the student to engage with the answer, and what we mght expect them to do after reading the answer.

So what other approaches might be possible?

If the answer can be contained in just the contents of a sinlge code cell, the answer can be placed in an external file, for example,, in the same directory or an inconvenient to access solutions directory such as ../../solutions. This effectively hides the solution but allows a student to reder it by running the following in a code cell:

%load ../../solutions/

The content of the file will be loaded into the code cell.

One advantage of this approach is that a student has to take a positive action to render the answer code into the cell. You can make this easier or harder depending on how much friction you want to add. Eg low friction: just comment out the magic in the code cell, so all a student has to do is uncomment and run it. Higher friction: make them work out what the file name is and write it themselves, then run the code cell.

I’m not sure I’m convinced having things strewn across multiple files is the best way of adding the right amount and right sort of friction to the answer lookup problem though. It also causes maintenance issues, unless you are generating separate student activity notebook and student hint or answer notebooks from a single master notebook that contains activities and hints/answers in the same document.

Other approaches I’ve seen to hiding and revealing answers include creating a package that contains the answers, and then running a call onto that package to display the answer. For example, something like:

from course_activities import answers

An alternative to rendering answers from a python package would be to take more of a “macro” approach and define a magic which maybe has a slightly different feeling to it. For example:

%hint week2_section1_activity3

In this approach, you could explore different sorts of psychological friction, using nudges:

%oh_go_on_then_give_me_a_hint_for week2_section1_activity3


%ffs_why_is_this_so_hard week2_section1_activity3

If the magic incantation is memorable, and students know the pattern for creating phrases like week2_section1_activity3 eg from the notebook URL pattern, then they can invoke the hint themself by saying it. You could even have aliases for revealing the hint, which the student could use in different situations to remind themselves of their state of mind when they were doing the activity.

%I_did_it_but_what_did_you_suggest_for week2_section1_activity3

With a little bit of thought, we could perhaps come up with a recipe that combines approaches to provide an experience that does degrade gracefully and that perhaps also allows us to play with different sorts of friction.

For example, suppose we extend the Collapsible Heading notebook extension so that if we tag a cell as an answer-header cell and a code cell below put a #%load line. In an unextended notebook, the student has to uncomment the load magic and run the cell to reveal the answer; in an extended notebook, the extension colour codes the answer cell and collapses the answer; when the student clicks to expand the answer, the extension looks for #%load lines in the collapsed area, uncomments and executes them to load in the answer, expands the answer cells automatically.

We could go further and provide an extension that lets an author write an answer in eg a markdown cell and then “answerify” it by posting it as metadata in preceding Answer reveal cell. In an unextended student notebook, the answer would be hidden in metadata. To reveal it, a student could inspect the cell metadata. Clunky for them and answer wouldn’t be rendered, but doable. In extended notebook clicking reveal would cause the extension to extract the answer from metadata and then render it in the notebook.

In terms of adding friction, we could perhaps add a delay between clicking a reveal answer button and displaying an answer, although if the delay is too long, a student may pre-emptively come to click the answer button before they actually want to tanswer the button so they don’t have to wait. The use of audible signal might also provide psychological nudges that subconsciously influence the student’s decision about when to reveal an answer.

There are lots of things to explore I think but they all come with a different set of opportunity costs and use case implications. But a better understanding of what we’re actually trying to achieve would be a start… And that’s presumably been covered in the interactive learning design literature? And if it hasn’t, then why not? (Am I overthinking all this again?!)

PS It’s also worth noting something I haven’t covered: notebooks that include interactive self-test questions, or self-test questions where code cells are somehow cross-referenced with code tests. (The nteract testbook recipe looks like it could be interesting in this respcect, eg where a user might be able to run tests from one notebook against another notebook. The otter-grader might also be worth looking at in this regard.

Quiz Night Video Chat Server

Our turn to host quiz night this week, so rather than use Zoom, I thought I’d give Jitsi a spin.

Installation onto a Digital Ocean box is easy enough using the Jitsi server marketplace droplet, althugh it does require you to set up a domain name so the https routing works, which in turn seems to be required to grant Jitsi access via your browser to the camera and microphone.

As per guidance here, to add a domain to a Digital Ocean account simply means adding the domain name (eg to your account and then from your domain control panel, adding the Digital Ocean nameservers (,,

It’s then trivial to create and map a subdomain to a Digital Ocean droplet:

The installation process requires creating the droplet from the marketplace, assigning the domain name, and then from the terminal running some set-up scripts:

  • install server: ./
  • add LetsEncrypt https support: ./

Out of the can, anyone can gain access, although you can add a password to restrict access to a video room once you’ve created it.

To local access down when it comes to creating new meeting rooms, you can add simple auth to created user accounts. The Digital Ocean tutorial How To Install Jitsi Meet on Ubuntu 18.04 “Step 5 — Locking Conference Creation” provides step by step instructions. (It would be nice if a simple script were provided to automate that a little…)

The site is react powered, and I couldn’t spot an easy way to customise the landing page. The text all seems to be provided via language pack files (eg /usr/share/jitsi-meet/lang/main-enGB.json) but I couldn’t see how to actually put any changes into effect? (Again, another simple script in the marketplace droplet to help automate or walk you through that would be useful…)

Deconstructing the TM351 Virtual Computing Environment via VS Code

For 2020J, which is to say, the 2020 October presentation, of our TM351 Data Management and Analysis course, we’ve deprecated the original VirtualBox packaged virtual machine and moved to a monolithic Docker container that packages all the required software applications and services (a Jupyer notebook server, postgres and mongoDB database servers, and OpenRefine).

As with the VM, the container is headless and exposes applications over http via browser based user interfaces. We also rebranded away from “TM351 VM” to “TM351 VCE”, where VCE stands for Virtual Computing Environment.

Once Docker is installed, the environment is installed and launched purely from the command line using a docker run command. Students early in to the forums have suggested moving to docker compose, which simplifies the command line command significantly, but also at the cost of having to supply a docker-compose.yaml . With OU workflows, it can take weeks, if not months, to get files onto the VLE for the first time, and days to weeks to post updates (along with a host of news announcements and internal strife about the possibility of tutors/ALs and students having different versions of the file). As we need to support cross-platfrom operation, and as the startup command specifies file paths for volume mounts, we’d need different docker-compose files (I think?) because file paths on Mac/Linux hosts, versus Windows hosts, use a different file path syntax (forward vs back slashes as path delimiters. [If anyone can tell me how to write a docker-compose.yaml files with arbitrary paths on the volume mounts, please let me know via the comments…]

Something else that has cropped up early in the forums is mention of VS Code, which presents a way to personalise the way in which the course materials are used.

By default, the course materials we provide for practical activities are all based on Jupyter notebooks, delivered via the Jupyter notebook server in the VCE (or via an OU hosted notebook server we are also exploring this year). The activities are essentially inlined using notebook code cells within a notebook that presents a linear teaching text narrative.

Students access the notebooks via their web browser, wherever the notebook server is situated. For students running the Docker VCE, notebook files (and OpenRefine project files) exist in a directory on the student’s own computer that is then mounted into the container; make changes to the notebooks in the container and those changes are saved in the notebooks mounted from host. Delete the container, and the notebooks are still on your desktop. For students using the online hosted notebook server, there is no way of synchronising files back to the student desktop, as far as I am aware; there was an opportunity to explore how we might allow students to use something like private Github repositories to persist their files in a space they control, but to my knowledge that has not been explored (a missed opportunity, to my mind…).

Using the VS Code Python extension, students installing VS Code on their own computer can connect to the Jupyter server running in the containerised VCE and (I don’t know if the permissions allow this on the hosted server).

The following tm351vce.code-workspace file describes the required settings:

"folders": [
"path": "."
"settings": {
"python.dataScience.jupyterServerURI": "http://localhost:35180/?token=letmein"

The VSCode Python extension renders notebooks, so students can open local copies of files from their own desktop and execute code cells against the containerised kernel. If permissions on the hosted Jupyter service allow remote/external connections, this would provide a workaround for synching notebooks files: students would work with notebook files saved on their own computer but executed against the hosted server kernel.

Queries can be run against the database servers via the code cells in the normal way (we use some magic to support this for the postgres database).

If we make some minor tweaks to the config files for the PostgreSQL and MongoDB database servers, we can use the VS Code PostgreSQL extension and MongoDB extension to run queries from VS Code directly against the databases.

For example, the postgres database:


and the mongo database:


Note that this is now outside the narrative context of the notebooks, although it strikes me that we could generate .sql and .json text files from notebooks that show code literally and comment out the narrative text (the markdown text in the notebooks).

However, we wouldn’t be able to work directly with the data returned from the database via Python/pandas dataframes, as we do in the notebook case. (Note also that in the notebooks we use a Python API for querying the mongo database, rather than directly issuing Javascript based queries.)

At this point you might ask why we would want to deconstruct / decompose the original structured notebook+notebook UI environment and allow students to use VS Code to access the computational environment, not least when we are in the process of updating the notebooks and the notebook environment to use extensions that add additional style and features to the user environment. Several reasons come to my mind that are motivated by finding ways in which we can essentially lose control, as educators, of the user interface whilst still being reasonably confident that the computational environment will continue to perform as we intend (this stance will probably make many of my colleagues shudder; I call it supporting personalisation…):

  • we want students to take ownership of their computational environment; this includes being able to access it from their own clients that may be better suited to their needs, eg in terms of familiarity, accessibility, productivity, etc;
  • a lot of our students are already working in software development and already have toolchains they are working with. Whilst we see benefits of using the notebook UI from a teaching and learning perspective, the fact remains that students can also complete the activities in other user environments. We should not hinder them from using their own environments — the code should still continue to run in the same way — as long as we explain how the experience may not be the same as the one we are providing, and also noting that some of the graphics / extensions we use in the notebooks may not work in the same way, or may not even work at all, in the VS Code environment.

If students encounter issues when using their own environment, rather than the one we provide, we can’t offer support. If the personalised learning environment is not as supportive for teaching and learning as the environment we provide, it is the student’s choice to use it. As with the Jupyter environment, the VS Code environment sits at the centre of a wide ecosystem of third party extesions. If we can make our materials available in that environment, particulalry for students already familiar with that environment, they may be able to help us by identifying and demonstrating new ways, perhaps even more effective ways, of using the VS Code tooling to support their learning than the enviorment we provide. (One example might be the better support VS Code has for code linting and debugging, which are things we don’t teach, and that our chosen environment perhaps even prevents students who know how to use such tools from making use of them. Of course, you could argue we are doing students a service by grounding them back in the basics where they have to do their own linting and print() statement debugging… Another might be the Live Share/collaboration service that lets two or more users work collaboratively in the same notebook, which might be useful for personal tutorial sessions etc.)

From my perspective, I believe that, over time, we should try to create materials that continue to work effectively to support both teaching and learning in environments that students may already be working in, and not just the user interface environments we provide, not least becuase we potentially increase the number of ways in which students can see how they might make use of those tools / environments.

PS I do note that there may be licensing related issues with VS Code and the VS Code extensions store, which are not as open as they could be; VSCodium perhaps provides a way around that.

Rally Review Charts Recap

At the start of the year, I was planning on spending free time tinkering with rally data visualisations and trying to pitch some notebook originated articles to Racecar Engineering. The start of lockdown brought some balance, and time away from scrren and keyboard in the garden, but then pointless made up organisational deadlines kicked in and my 12 hour plus days in the front of the screen kicked back in. As Autumn sets in, I’m going to try to cut back working hours, if not screen hours, by spending my mornings playing with rally data.

The code I have in place at the moment is shonky as anything and in desperate need of restarting from scratch. Most of the charts are derived from mutliple separate steps, so I’m thinking about pipeline approaches, perhaps based around simple web services (so a pipeline step is actually a service call). Thiw will give me an opportunity to spend some time seeing how production systems actually build pipeline and microsservice architectures to see if there is anything I can pinch.

One class of charts, shamelessly stolen from @WRCStan of @PushingPace /, are pace maps that use distance along the axis and a function of time on the y-axis.

One flavour of pace map uses a line chart to show the cumulative gap between a selected driver and other drivers. The following chart, for example, shows the progress of Thierry Neuville on WRC Rally Italia Sardegna, 2020. On the y-axis is the accumulated gap in seconds, at the end of each stage, to the identified drivers. Dani Sordo was ahead for the whole of the rally, building a good lead over the first four stages, then holding pace, then losing pace in the back end of the rally. Neuville trailed Seb Ogier over the first five stages, then after borrowing from Sordo’s set-up made a come back and a good battle with Ogier from SS6 right until the end.

Off the pace chart.

The different widths of the stage identify the stage length; the graph is thus a transposed distance-time graph, with the gradient showing the difference in pace in seconds per kilometer.

The other type of pace chart uses lines to create some sort of stylised onion skin histogram. In this map, the vertical dimension is seconds per km gained / loat relative to each driver on the stage, with stage distance again on the x-axis. The area is thus and indicator of the total time gained / lost on the stage, which marks this out as a histogram.

In the example below, bars are filled relative to a specific driver. In this case, we’re plotting pace deltas relative to Thierry Neuville, and highlighting Neuville’s pace gap to Dani Sordo.

Pace mapper.

The other sort of chart I’ve been producing is more of a chartable:

Rally review chart.

This view combines tabular chart, with in-cell bar charts and various embedded charts. The intention was to combine glanceable visual +/- deltas with actual numbers attached as well as graphics that could depict trends and spikes. The chart also allows you to select which driver to rebase the other rows to: this allows you to generate a report that tells the rally story from the perspective of a specified driver.

My thinking is that to create the Rally Review chart, I should perhaps have a range of services that each create one component, along with a tool that lets me construct the finished table from component columns in the desired order.

Some columns may contain a graphical summary over values contained in a set of neighbouring columns, in which case it would make sense for the service to itself be a combination of services: one to generate rebased data, others to generate and return views over the data. (Rebasing data may be expensive computationally, so if we can do it once, rather than repeatedly, that makes sense.)

In terms of trasnformations, then, there are are least two sorts of transformation service required:

  • pace transformations, that take time quantities on a stage and make pace out them by taking the stage distance into account;
  • driver time rebasing transformations, that rebase times relative to a specified driver.

Rummaging through other old graphics (I must have the code somewhere bit not sure where; this will almost certainly need redoing to cope with the data I currently have in the form I have it…), I also turn up some things I’d like to revisit.

First up, stage split charts that are inspired by seasonal subseries plots:

Stage split subseries

These plots show the accumulated delta on a stage relative to a specified driver. Positions on the stage at each split are shown by overplotted labels. If we made the x-axis a split distance dimension, the gradient would show pace difference. As it is, the subseries just indicate trend over splits:

Another old chart I used to quite like is based on a variant of quantised slope chart or bump charts (a bit like postion charts.

This is fine for demonstrating changes in position for a particular loop but gets cluttered if there are more than three or four stages. The colour indicates whether a driver gained or lost time realtive to the overall leader (I think!). The number represents the number of consecutive stage wins in the loop by that driver. The bold font on the right indicates the driver improved overall position over the course of the loop. The rows at the bottom are labelled with position numbers if they rank outside the top 10 (this should really be if they rank lower than the number of entruies in the WRC class).

One thing I haven’t tried, but probably should, is a slope graph comparing times for each driver where there are two passes of the same stage.

I did have a go in the past at more general position / bump charts too but these are perhaps a little too cluttered to be useful:

Again, the oberprintined numbers on the first position row indicate the number ofconsecutive stage wins for that driver; the labels on the lower rows are out of top 10 position labels.

What may be more useful would be adding the gap to leader of diff to car ahead, either on stage or overall, with colour indicating either that quantity, heatmap style, or for the overall case, whether that gap / difference increased or decreased on the stage.