Quick First Look At Moodle CodeRunner

One of the tools we have to support programming activities in our Moodle VLE is a CodeRunner backed interactive question type.

From the blurb, CodeRunner is “a free open-source question-type plug-in for Moodle that can run program code submitted by students in answer to a wide range of programming questions in many different languages”.

Of interest to me on the courses I’m involved with are support for Python3, SQL (or at least, the dialect supported by SQLite), and R, which looks like it has hacky support via a command line call to R from a Python3 question type…

The question set up includes an execution environment selection, but I’m not sure how easy it is to define bespoke ones (e.g. a Python 3 environment with particular packages preinstalled):

The R support looks like it’s not offered natively, but seems to be hacked together via a system call from the Python environment:

(I guess that means we could also hack a way to running code against an arbitrary Jupyter kernel?)

A slot is provided for a valid example answer, but it doesn’t look like there’s a way I can interactively edit and test that code (a simple terminal onto the underlying execution environment would be really handy. (The jupytergraffiti Jupyter notebook extension has some interesting ideas about inline terminals and workflows around a similar sort of use case, in which the contents of a code cell are saved into a Python file that is then executed from a terminal.)

I can save the code and have it validated automatically, but that’s not really interactive. (Also, I’m not sure about the semantics of ‘Validate on save’? Does that mean it runs the code against the defined tests?)

The test definitions also look like they don’t let me interactively test them? It’s also not clear where I’m expected to pick up the Expected output from. I’m guessing this is used as an exact match string, so, erm, I really need it to be right? That cell really should be automatically populated by running the test case against a correct answer? Human hands should have nothing to do with it…

I’ve not got far enough into yet to know if I can call on arbitrary packages that need installing on top of the base environment (I suspect I can’t?).

It looks like there is an opportunity to provide files that are available at run time…

so I guess if I upload a Python package zip file there I might be able to install from it?

Hmm… nearly…. so it looks like we could hack a way round package requirements by tweaking settings… but are things like time limits defined globally? Or at least, at a level above the question type level? (What if I have a question for which I know any legitimate answer will take a longish time to execute?)

I also haven’t figured out how to properly inspect the test environment:

Co-opting a question preview as an error displaying terminal seems really ineffective? (There has to be a better way, surely?)

Developing sample answers / tests in my own environment means I need to make sure my environment is exactly the same as the one used by CodeRunner and then copying exact match expected answers is fraught with danger?

Keeping student environments (where students might try out sample code before submitting it) and test environments in sync could also be a real issue. Our TM351 VM environment has maybe 30 or 40 custom packages installed, in various known versions, (although there’s nothing to stop students updating them), which may change year on year. Trying to use the default CodeRunner Python3 environment is not likely to work for anything other than trivial questions that don’t use the packages that form the core part of the teaching.

One of the things it would be useful for us to test is chart generation, but it’s not obvious that that would work within the CodeRunner context. (We could get students to generate a py/matplotlib or R/ggplot2 chart (that is, chart object) and then try to introspect on that as part of the tests, I suppose, but the defined tests would be at a state level students are never exposed to? Or we could maybe try to take a hash of an image file against a correct answer image file hash?

Code execution itself seems to be handled via something called Jobe (Job Engine), “a server that supports running of small compile-and-run jobs in a variety of programming languages … developed as a remote sandbox for use by CodeRunner”. This looks to largely be a solo project, although I note some commits from the OU, which is good to see. One of the things I’ve been repeatedly told is we can’t trust solo code projects for our own production use cases (too risky / too unsustainable), so, erm….

(At this point, I should declare I lobbied early on for us to look at using CodeRunner, I think round about the time we were first looking at Jupyter notebooks (or IPython notebooks, as they were then). By the time it arrived, our course needs had moved on as we developed the course’s computational environment, and if anything, nbgrader was looking a better bet: CodeRunner didn’t seem flexible enough on the back end, or as an environment for naturally (i.e. quickly and easily) developing and testing questions. I think there are various other test regimes out there that demonstrated for use in an academic quiz/test/automated assessment context, and I need to do a quick overview of them all to see what they offer…)

It does look like you’re largely stuck with the provided environments though, which is less than useful, particularly when considered across a range of courses at different levels, each with a different focus, each with its own environmental context needs.

Once again, I wonder if anyone has looked at using a Jupyter powered backend, rather than Jobe, which would give access to all the Jupyter kernels, as well as custom environments? This could make CodeRunner a bit more useable for us, and would allow bespoke course/presentation code environments to be more easily defined.