Fragment – TM351 Notebooks Jupyter Books In the VM and Via An Electron App

Another fragment because I keep trying to tidy the code so it’s ready to share and then keep getting distracted…

Previously, I’ve already posted crude proof of concept demos transforming OU-XML to markdown and rendering it via Jupyter Book and wrapping a Jupyter Book in an electron app shell.

I’ve managed to get a Jupyter Book generated from TM351 notebooks working under a local server inside the TM351 VM executing code against the notebook server in the VM, and also managed to connect the electron app version to the notebook server inside the VM. (The TM351 VM already runs a webserver, so I could just define the Jupyter Book to live down a particular directory path when I built it, and then pop it into a corresponding subdirectory in the root directory served by the webserver.)

The following not very informative demo shows me running a query against the database inside the VM twice, adding a row with a particular PK (which works) then trying again (which correctly fails because that PK is already in use).

The magic we use inside the VM for previewing a schema diagram of the current db also seems to work.

The VM solution currently requires two sorts of trust for executing code against the VM kernel from the electron app to work correctly: firstly, that the server is on a specified port (we default to a mapping 35180 when distributing the VM and this port is baked into the electron app book pages); secondly, that no-one attacks the VM — the notebook server is running pretty wide open:

jupyter notebook --port=8888 --ip=0.0.0.0 --no-browser --notebook-dir=/vagrant/notebooks --allow-root --NotebookApp.allow_origin='*' --NotebookApp.allow_remote_access=True  --NotebookApp.token=notverysecret

One of the nice things about the Jupyter Book view is that it bakes in search. I’ve had a couple of aborted looks at notebook search before (eg Initial Sketch – Searching Jupyter Notebooks Using lunr and More Thoughts On Jupyter Notebook Search as well as via nbgallery) and the Jupyter Book solution uses lunr which was one of the things I’d previously used to sketch a VM notebook search engine.

At the moment, the search part works but the links are broken – a template needs tweaking somewhere,  I think, or in the short term we could workaround with a hacky fix by post-processing the generated pages and fixing the broken URLs with a string replace.

One of the problems with lunr is is that the index is baked into a JSON file. datasette is now able to serve feeds off the current state of a SQLite3 database, even if the database is updated by another process, which makes me wonder: is one way of getting a “live” notebook search into the VM would to monitor the notebook directory, update a SQLite database with current notebook contents whenever the a notebook is added or updated (we could archive deleted notebooks in the database and just mark them as deleted without actually deleting them) and then use datasette to generate and serve a current JSON index to lunr whenever a page is refreshed?

So what does this electron and Jupyter Book stuff mean anyway?

1. we can wrap TM351 notebooks in a Jupyter Book style UI and serve it inside the VM against the notebook server running in the VM. This provides an alternative UI and some interesting pedagogical differences. For example, in the traditional notebook UI, students are free to run, edit, add and delete cells, whereas in the book UI they can only run and edit them. Also, in the book, if the page is refreshed after any edits, the original code content will be re-presented and the edited version lost;
2. we can wrap the TM351 notebooks in a Jupyter Book style UI and wrap that in an electron container. A student can then view the book UI on the desktop and if the VM is available on the expected port, execute code against it. If the VM is not available, the students can still read the notebooks, if not actually execute the code. (One thing we could do is publish a version of the book with all the code cells run and showing their outputs…)
3. the electron app could also be built to tell ThebeLab (which handles the remote code execution) to launch a kernel somewhere else, such as on MyBinder or on an OU managed on-demand Binder kernel server. (I guess we could also add electron menu options that might allow us to select a ThebeLab execution environment?) This latter approach could be interesting: it means we could provide compute backend support to students *only* when they need it in quite a natural way. The notebook content will always be available and readable, even offline, although in the offline case code would not be (remotely) executable.
4. distributing the notebooks *as an app* means we can distribute content we donlt want to share openly and in public whilst still freeloading on the possibility of executing code in the appified Jupyter Book on a MyBinder compute backend server;
5. on my to do list (still fettlin’ the code) is to provide a combined Jupyter Book view that includes content from the VLE as well as the notebooks. With both sets of content in the same Jupyter book, we could then readily run a search over VLE and notebook content at the same time (though forum search would still be missing). At the moment, search is an issue becuase: 1) VLE search sucks and it only works over the VLE hosted content, which *does not* include the notebooks; 2) we don’t offer any notebook search tools (other than hints of things students could do themseleves to search notebooks).

I’m not sure I’m going to get everything as “safe” to leave as I’d hoped to before a holiday away from’t compootah, so I just hope its stable enough for me to remember how to finish off all the various unfinished bits in a week or two!