Whilst at the nbgrader workshop in Edinburgh a couple of weeks ago, a couple of things jumped out at me. Firstly, Jess Hamrick’s design requirement that students should be able to complete ngrader assignments without need to install the nbgrader package or any of its extensions. This is completely reasonable — we want to make it as easy as possible for students to complete an assignment which means minimising the chances of anything going wrong. Secondly, a comment that the ability to lock cells in an assignment notebook is available as an undocumented feature in nbgrader.
The ability to lock cells is an interesting one when it comes to using notebooks for assessment (it also has interesting pedagogical implications when delivering teaching materials in a notebook form). One thing that might make sense in some situations would be to prevent students, by default, from editing any cells not associated with a submission. (That is, the only editable cells should be cells that are “markable” code or free text cells.) This would provide more of a form driven “exam script” style presentation. Furthermore, if we are autograding a notebook, we want to make sure that students don’t mess up the autograding; neither do we want students to put their answers in cells we can’t detect as requiring grading.
I was aware of a freeze Jupyter notebook extension that allows cells to take on various states. Code cells, for example, can be read-only (executable, but the code cannot be changed) or frozen (code be neither altered nor executed), and markdown cells can be read-only (viewable as markdown but non-editable) or frozen (neither viewable as markdown nor editable). But could this also be achieved without an extension, as per the design requirement?
Interestingly, it seems that recent notebook releases do support various cell presentation controls that can be invoked using particular metadata elements.
In particular, the deletable and editable metadata elements seem to be recognised by the Jupyter notebook server (deletable example, editable example); they take effect when set to false.
From the nbformat docs, the deletable metadata element will prevent the deletion of the cell if set to False, but the editable element doesn’t appear to be documented?
To see the effect of setting these metadata elements, you can enable the Edit Metadata controls from the notebook View - Cell Toolbar menu:
edit the required field:
and then try to edit or delete the cell:
Within nbgrader itself, the nbgrader lockcells preprocessor sets cell.metadata['deletable'] = False and in some cases also sets cell.metadata['editable'] = False. [I didn’t spot in the nbgrader` docs a clear description of what’s set to what and how?]
Using native notebook controls, then, we should be able to make certain cells undeletable and uneditable without the need to install any extensions. Students can still add new cells into the notebook but there will be no nbgrader metadata associated with the cell; as such, these cells would presumably be ignored by the autograder and not highlighted for manual grading. (There is a recently added nbgrader task style that seems to allow submissions over multiple cells but I haven’t had a chance to explore this yet.) It’s maybe also worth noting that if all cells in a notebook are tagged with some metadata as part of the assignment creation step, even if just a cell ID, then nbgrader would be able to detect any cells newly added to the notebook by a student.
Within an assignment notebook, certain code and metadata cells can be designated as Read Only. Originally, the semantics of this was such that the source of those cells was recorded into a database when the assignment was created; during autograding nbgrader checked whether the source of the student’s version of the cell has changed and if it had, the changed content would be replaced by the version in the database to undo any student changes. With native notebook, metadata driven support of non-editable cells, cells are now read-only unless the student changes the metadata field.
Making all assessment notebook cells undeletable and making all cells other than cells where we require students to provide an answer uneditable gives us a certain amount of control over the notebook document. But can we take this further? Providing locked down environments where only “assessable” cells are editable made me start to wonder about whether it would be possible to put an nbgrader assignment into a Jupyter Book like environment, with ThebeLab enabled code cells (and free text answer / markdown cells?) that students could use to run and test their code and then somehow export their answers.
Rather than letting students edit an assignment notebook freely, we would provide them with a truly fixed environment that could execute only clearly identified (user editable) code cells against a known/predefined environment. Ideally, we’d then also provide a means to save the code the student had created in the code cells.
Note: in other automated code assessment tools, the ability to save code may not be strictly necessary. For example, the Moodle CodeRunner question type provides automated assessment “inline”: students write code in a web form, execute it within a remote computational environment, and then receive an automated grading against predefined tests. nbgrader doesn’t operate in this way, but it might be interesting if it could…
As a first attempt at providing a “fixed guidance, editable answer” view, I had a play with nbpreview, a single page standalone web app that lets you upload a notebook and then preview it using notebook.js, a Javascript notebook previewer. In particular, I made a minor tweak to notebook.js to simplify the way in which code cells were rendered, and a tweak to nbpreview that provided support for executing the code against a MyBinder container launched using ThebeLab. (See also this related issue thread.)
The resulting demo, which you can find here, allows you to upload a notebook .ipynb file and preview it in a normal way.
Clicking the Activate button will make the code cells editable and runnable, via ThebeLab, against a specified MyBinder launched environment.
The Download button is a proof of concept that will export the contents of all the code cells in a raw form. My intention for this, in the nbgrader context, is to be able to download a JSON object file that associates the nbgrader cell ID. The notebook.js package really needs some further revision so that the nbgrader cell ID metadata is passed as an ID attribute of each code block into the previewed web page; further modification to identify “assessable” markdown cell content is also required, along with a tweak to the ThebeLab.js package so that assessable and identifiable free text / markdown cells are converted into editable text area form elements when the Activate button is clicked. User modified free text cells should also be exportable / downloadable.
As to how downloaded cell content might be returned to nbgrader, I can think of a several possibilities.
One way would be to create a notebook template document that can be repopulated with content from the answer download.
A similar effect could also be achieved by simply rewriting the contents of each assessable cell in the assessment notebook with the contents of the associated (similarly ID’d) element in the downloaded JSON file. Submission could then proceed using the current nbgrader submission approach.
Another approach might be to modify the autograder so that answer cells from student submissions (either pulled from the downloaded JSON answers file, or extracted from notebook answer cells) are parsed into an “answers” table in the database; autograding could then be run against templated code cells and content pulled from the database’s “answers” table.
An advantage of this approach is that an “assessment form” UI such as the one above might then allow answers to be uploaded directly into the database rather than downloaded as an answer object file.
The database mediated approach might then also support a CodeRunner style mode of operation in which assessments containing only automatically graded elements can be autograded directly in response to a student hitting a “Grade me now” button on the assessment page web app.
OUseful.Info, the blog... 