TJ Fragment: Sharing Desktop Apps Via Jupyter-Server-Proxy Et Al.

It’s been some time since I last had a play with remote desktops, so here’s a placeholder / round up of a couple of related Jupyter server proxy extensions that seem to fit the bill.

For those at the back who aren’t keeping up, jupyter-server-proxy applications are incredibly useful: they extent the Jupyter server to proxy other services running in the same environment. So if you have a Jupyter server running on example.url/nbserver/, and another application that publishes a web UI in the same environment, you can publish that application, using jupyter-server-proxy, via example.url/myapplication. As an example, for out TM351 Data Management and Analysis course, we proxy OpenRefine using jupyter-server-proxy (example [still missing docs].).

Applications that are published using a jupyter-server-proxy wrapper are typically applications that publish an HTML UI. So what do you do if the application you want to share is a desktop application? One way is to to share the desktop via a browser (HTML) interface. Two popular ways of doing this are:

  • novnc: an “open source VNC client – it’s is both a VNC client JavaScript library as well as an application built on top of that library”;
  • xpra: “an open-source multi-platform persistent remote display server and client for forwarding applications and desktop screens”.

Both of these applications allow you to share (Linux) desktop applications via a web browser, and both of them are available as jupyter-server-proxy extensions (subject to the correct operating system packages also being installed).

As far as novnc goes, jupyterhub/jupyter-remote-desktop-proxy will “run a Linux desktop on the Jupyter single-user server, and proxy it to your browser using VNC via Jupyter”. A TightVNC server is bundled with the application as a fallback if no other VNC server is available. One popular application used wrapped by several people using jupyter-remote-desktop-proxy is QGIS; for example, giswqs/jupyter-qgis. I used it to demonstrate how we could make a legacy Windows desktop application available via a browser by running using Wine on a Linux desktop and then sharing it via the jupyter-remote-desktop-proxy.

For xpra, the not very active (but maybe it’s stable enough?!) FZJ-JSC/jupyter-xprahtml5-proxy seems to allow you to “integrate Xpra in your Jupyter environment for an fast, feature-rich and easy to use remote desktop in the browser”. However, no MyBinder demo is provided and I haven’t had a chance yet to give this a go. (I have tried XPRA in other contexts, though, such as here: Running Legacy Windows Desktop Applications Under Wine Directly in the Browser Via XPRA Containers.)

Another way of sharing desktops is to use the Microsoft Remote Desktop Protocol (aka RDP). Again, I’ve used that in various demos (eg This is What I Keep Trying to Say…) but not via a jupyter-server-proxy. I’m not sure if there is a jupyter-server-proxy example out there for publishing a proxied RDP port?

Just in passing, I also note this recipe for a Docker compose configuration that uses a bespoke container to act as a desktop sharing bridge: Viewing Dockerised Desktops via an X11 Bridge, novnc and RDP, Sort of…. I’m not sure how that might fit into in Jupyter set up? Could a Jupyter server container be composed with a bridge container, and then proxy the bridge services?

Finally, another way to share stuff is to to use WebRTC. The maartenbreddels/ipywebrtc extension can “expose the WebRTC and MediaStream API in a Jupyter notebook/JupyterLab environment” allowing you to create a MediaStream out of an ipywidget, a video/image/audio file, or a webcam and use it as the bases for a movie, image snapshot or audio recording. I keep thinking this might be really useful for recording screencast scenes or other teaching related assets, but I haven’t fully grocked the full use of it. (Something like Jupyter Graffiti also falls into this class, which can be used to record a “tour” or walkthrough of a notebook that can also be interrupted by live interaction or the user going off-piste. The jupyterlab-contrib/jupyterlab-tour extension also provides an example of a traditional UI tour for JupyterLab, although I’m not sure how easy it is to script/create your own tours. Such a thing might be useful for guiding a user around a custom JupyterLab workspace layout, for example. [To my mind, workspaces are the most useful and least talked about feature of the JupyerLab UI….] More generally, shepherd.js looks interesting as a generic website tour supporting Javascript package.) What I’m not sure about is the extent to which I could share, or proxy access to, of a WebRTC MediaStream that could be accessed live by a remote user.

Another way of sharing the content of a live notebook is to use the new realtime collaboration features in JupyterLab (see the official announcement/background post: How we made Jupyter Notebooks collaborative with Yjs). (A handy spin-off of this is that it now provides a hacky workaround way of opening two notebooks on different monitors.) If you prefer more literal screensharing, there’s also yuvipanda/jupyter-videochat which provides a server extension for proxying a Jitsi (WebRTC) powered video chat, which can also support screen sharing.

Opening Up Access to Jupyter Notebooks: Serverless Computational Environments Using JupyterLite

A couple of weeks ago, I started playing with jupyterlite, which removes the need for an external Jupyter server and lets you use JupyterLab or RetroLab (the new name for the JupyterLab classic styled notebook UI) purely in the browser using a reasonably complete Python kernel that runs in the browser.

Yesterday, I had a go at porting over some notebooks we’ve used for several years for some optional activities in first year undergrad equivalent course. You can try them out here: https://ouseful-demos.github.io/TM112-notebooks-jupyterlite.

You can also try out an online HTML textbook version that does require an external server, in the demo case, launched on demand using MyBinder, from here: https://ouseful-course-containers.github.io/ou-tm112-notebooks/Location%20Based%20Computing.html

The notebooks were originally included in the course as a low-risk proof of concept of how we might make use of notebooks in the course. Although Python is the language taught elsewhere in the module, engagement with it is through the IDLE environment, with no dependencies other than the base Python install: expecting students to install a Jupyter server howsoever, was a no-no. The optional and only limited use of notebooks meant we could also prove a hosted notebook solution using JupyterHub and Kubernetes in a very light touch way: authentication via a Moodle VLE LTI link gave users preauthenticated access to a JupyterHub server from where students could run the provided notebooks. The environment was not persistent though: if students want to save their notebooks to work on them at a future time, they had to export the notebooks then re-upload in their next session. We were essentially running just a temporary notebook server. The notebooks were also designed to take this into account, i.e. that the activities should be relatively standalone and self-contained, and could be completed in a short study session.

To the extent that the rest of the university paid no attention, it would be wrong to class this as innovation. On the one hand, the approach we used was taken off-the-shelf (Zero to JupyerHub With Kubernetes), although some contributed docs did result (using the JupyterHub LTI authenticator with Moodle). The deployment was achieved very much as a side project, using personal contacts and an opportunity to deploy it outside of formal project processes and procedures and before anyone realised what had actually just happened. (I suspect they still don’t). On the other, whilst it worked and did the job required of it, it influenced nothing internally and had zero internal impact other than meeting the needs of several thousand students. And whilst it set a precedent, it wasn’t really one we managed to ever build directly from or invoke to smooth some later campaign.

As well as providing the hosted solution, we also made the environment available via MyBinder, freeloading on that service to provide students with an environment that they could access ex- of university systems. This is important because it meant, and means, that access remains available to students at the end of the course. Unlike traditional print based models of distance education, where students get a physical copy of course materials they can keep for ever, the online first approach that dominates now means that students lose access to the online materials after some cut-off point. So much for being able to dig that old box out of the loft containing your lecture notes and university textbooks. Such is the life of millenials, I guess: a rented, physical artefactless culture. Very much the new Dark Age, as artist James Bridle has suggested elsewhere.

But is there a better way? Until now, to run a Jupyter notebook has placed a requirement on being able to access a Jupyter server. At this point, it’s worth clarifying a key point. Jupyter is not notebooks. At its core, Jupyter is a set of protocols that provide access to arbitrary computational environments, that can run arbitrary code in those environments, and that can return the outputs of that code execution under a REPL (read-eval-print loop) model. The notebooks (or JupyterLab) are just a UI layer. (Actually, they’re a bit more interesting than that, as are ipywidgets, but that’s maybe something for another post.)

So, the server and the computational environment. The Jupyter server is the thing that provides access to the computational environment. And the computational environment has typically needed to run “somewhere else”, as far as the the notebook or JupyterLab UI is concerned. This could be on a remote hosted server somewhere in the cloud or provided by your institution, or in the form of an environment that exists and runs from your own desktop or laptop computer.

What JupyterLite neatly does is bring all these components into the browser. No longer does the notebook client, the user interface, need to connect to a computational environment running elsewhere, outside the browser. Now, everything can run inside the browser. (There is one niggle to this: you need to use a webserver to initially deliver everything into the browser, but that can be any old webserver that might also be serving any other old website.)

Now, I see this as A Good Thing, particularly in open online edcuation where you want learners to be able to do computational stuff or benefit from interactions or activities that require some sort of computational effort on the back end, such as some intelligent tutoring thing that responds to what you’ve just done. But a blocker in open ed has always been: how is that compute provided?

Typically, you either need the learner to install and run something — and this is something that does not scale well in terms of the amount of support you have to provide, because some people will need (a lot of!) support —or you need to host or otherwise provide access to the computational environment. And resource it. And probably also support user authentication. And hence also user registration. And then either keep it running, or prevent folk from accessing it from some unspecified date in the future.

What this also means is that whilst you might reasonably expect folk who want to do computing for computing’s sake to take enough of an interest, and be motivated enough to install a computing evironment of their own to work in, for folk who want to use use computing to get stuff done, or just want to work through some materials without having to already have some skills in installing and running software (on their own computer), it all becomes a bit too much, a bit too involved and let’s just not bother. (A similar argument holds when hosting software: the skills required to deploy and manage end-user facing software on a small network, for example, (think of the have-a-go teacher who looks after the primary school computer network), or the more considerable skills (and resource) required to deploy environments in a large university with lots of formalised IT projects and processes. When just do it becomes a project with planning and meetings and all manner of institutional crap, it can quickly become “just what is the point in even trying to do any this?!”

Which is where running things in the browser makes it easier. No install required. Just publish files via your webserver in the same way you would publish any other web pages. And once the user has opened their page in the browser, that’s you done with it. They can run the stuff offline, on their own computer, on a train, in the garden, in their car whilst waiting for a boat. And they won’t have had to install anything. And nor will you.

Try it here: https://ouseful-demos.github.io/TM112-notebooks-jupyterlite.

Helping Learners Look at Their Code

One of the ideas I’m using the Subject Matter Authoring With Jupyter Notebooks online textbook to riff around is the notion (which will be familiar to long time readers of this blog) of sharing the means of asset production with students. In many cases, the asset production I am interested in relates to the creation of media assets or content that is used to support or illustrate teaching and/or learning material.

For example, one well known graphical device for helping explain a simple algorithm or process is a flow chart. Here’s an example from an OpenLearn unit on Computers and compute systems:

The image may have been drawn using a flowchart generation package, or “freehand” by an artist in a generic drawing package.

Here’s the same flow chart (originally generated in SVG form) produced by another production route, the flowchart.js Javascript package:

The chart was created from a script with a particular syntax:

st=>start: start
in1=>inputoutput: Accept data from sensor
op1=>operation: Transform sensor data to display data format
out1=>inputoutput: Send display data to display
e=>end: end

st->in1->op1->out1->e

To begin with, each element is defined using a construction of the form: uniqueIdentifier=>blocktype: label. (Note that the space after the colon following the blocktype is required or the diagram won’t be laid out properly.) The relationship describing how the various blocks are connected is then described using their unique identifiers.

Using a piece of simple IPython magic (flowchart_js_jp_proxy_widget), we can create a simple tool for rendering flowcharts created using flowchart.js in a Jupyter notebook, or defining the production of an image asset in an output document format such as an HTML textbook or a PDF document generated from the notebook “source” using a publishing tool such as Jupyter Book.

(In a published output created using Jupyter Book, we could of course hide or remove the originating script from the final document and just display the flow chart.)

To update the flowchart all that is required is that we update the script and rerun it (or reflow the document if we are creating a published format).

One other thing we notice about the OpenLearn document is that it links to a long description of the diagram, which is required for accessibility purposes. Only, in this case, it doesn’t…:

(I really do need to get around to hacking together an OU-XML quality report tool that would pick up things like that… I assume there are such tools internally — the OU-XML gold master format has been around for at least 15 years — but I’ve never seen one, and working on one of my own would help my thinking re: OU-XML2ipynb or OU-XML2MyST conversions anyway.)

A very literal long description might have taken the form of something like “A flow chart diagram showing a particular process. A rounded start block connects to an input/output block (a parallelogram) labeled ‘Accept data from sensor’…” and so on. In one sense, just providing the flowchart.js source would provide an unsighted user with all the information contained in the diagram, albeit in a slightly abstract form. But it’s not hard to see how we might be able to automate the creation of a simple text creation script from the flowchart description. (That is left as an exercise for the reader… Please post a link to your solution in the comments!;-) Or if you know of any reference material with best practice guidance for generating long descriptions of flow charts in particular, and diagrams in general, for blind users, please let me know, again via the comments.)

So, that’s an example of how we can use text based tools to generate simple flow charts. Here’s another example, taken from here:

Simple flow chart (via pyflowchart docs)

This diagram is not ideal: the if a block feels a bit clunky, we might take issue with the while operation block not being cast into a decision loop (related issue – and me not reading the docs! This is a feature, not a bug, an automated simplification that can be disabled…), and why is the print() statement a subroutine? But it’s a start. And given the diagram has been generated from a script, if we don’t like the diagram, we can easily make changes to it by editing the script and improving on it.

At this point, it’s also worth noting that the script the image was generated was itself generated from a Python function defintion using the pyflowchart Python package:

The script is slightly harder to read than the original example, not least in the way the unique identifiers are designed, but it’s worth remembering this is currently generated primarily for a machine, rather than a person, to read. (We could create simple unique identifiers, for example, if we wanted to make the script more palatable to human readers.)

Whilst the script may not be ideal, it does function as a quick first draft that an author can work up. As with many automation tools, they are most effective when used by a human performing a task in order to support that task, rather than instead of a human performing that task (in this case, the task of producing a flowchart diagram).

We can also generate the diagram from a function in a Jupyter notebook code cell using a bit of IPython magic:

Creating a flowchart with pyflowchart magic

On my to do list is to update the magic to alternatively display the flowchart.js diagram generating script as well as optionally execute the code in the magicked code cell (at the moment, the code is not executed in the notebook Python kernel environment). And having a look at automatically generating human readable text descriptions.

It’s not just functions that we can render into flowcharts either. We can also render flowcharts from simpler scripts:

(Thinks: it might be nice if we could add a switch to add start and stop blocks to top and tail this sort of flowchart. Or maybe allow a pass statement at the very start or end of very end of a code fragment to get rewritten as start/stop block as appropriate.

So, here we have a couple of tools that can be used:

  • to generate a generative text description of an algorithm defined using Python code from an abstract syntax tree (AST) representation of the code;
  • to render a media asset (a flow chart diagram) from a generative text description.

So that’s the means of production part.

We can use this means of production to help support the creation of flowcharts in a relatively efficient and straightforward way. The lower the overhead or cost to doing something the more likely we are to do it, or at least, the more likely we are to consider doing it because of the fewer blockers to actually doing it. So this sort of route, this means of production, makes it easier for us to make use of graphical flow charts to illustrate materials if we want to.

If we want to.

Or how about if a student wants to?

With an automated means of production available, it becomes easier to create additional examples that we might make optionally available (either in final rendered form, or in a generated form, from a generative scripts) that some students might find useful.

But if we make the means of production available to students, then it means that they can generate their own examples. And check their own work.

Take the case of self-assessment activities generated around flowcharts. We might imagine:

  • asking a learner to identify which flowchart of several describes a particular code fragment;
  • which code fragment of several implements a particular flowchart;
  • what a flowchart for a particular code fragment might look like;
  • what a piece of code implementing an alogirithm described in a provided flow chart might look like.

In each of the above cases, we can use a generative code2flowchart method to help implement such an activity. (Note that we can’t go directly from flowchart to code, though we may be able to use the code to flowchart route to workaround that in some cases with a bit of creative thinking…)

In addition, my providing learners with the means of production for generaing flowcharts, they can come up with their own exercises and visualise their own arbitrary code in a flowchart format to self-check their own work or understanding. Whether folk really can be said to be “visual learners” or not, sometimes it can help drawing things out, seeing them represented in visual form, and reading the story back from what the diagram appears to be saying.

And if we can also get a simple flowchart script to human readable text generator going, we can provide yet another different way of looking at (which is to say, reading), the code text. And as with many accessibility tools, that can be useful for everyone, not just the learner who needs an alternative format in order to be able to access the materials in a useful and useable (which is to say, a meaningful) way.

PS Finally, it’s worth noting the some of the layouts that flowchart.js generates are a bit broken. Some people will use this to argue that “because it breaks on these really weird edge cases and doesn’t do everything properly, we shouldn’t use it for anything, even the things it does work well on”. This is, unfortunately, an all-too-common response in an organisation with a formal process mentality that also belies a misunderstanding of using automation instead of rather than by and in support of. There are two very obvious retorts to this: firstly, the flowchart.js code is open, so if we spot something is broken we can fix it (erm, like I haven’t) and make it better for everyone (and which will in turn encourage more people to use it and more people to spot and fix broke things); secondly, the flowchart.js code which the author has presumably ensured is logically correct, can be used to generate a “first draft” SVG document that an artist could rearrange on the page so that it looks nice, without changing the shape, labels or connectedness of the represented objects which are a matter of correctness and convention and which the artist should not be allowed to change.

Fragment: Helping Learners Read Code

Picking up on Helping Learners Look at Their Code, where I showed how we can use the pyflowchart Python package to render a flowchart equivalent of code in a notebook code cell using the flowchart.js package, I started wondering about also generating text based descriptions of simple fragements of code. I half expected there to be a simple package out there that would do this — a Python code summariser, or human radable text description generator — but couldn’t find anything offhand.

So as a a really quick proof of concept knocked up over a coffee break, here are some sketches of a really naive way in to parsing some simple Python code (and that’s all we need to handle…) on the way to creating a simple human readable text version of it.

#  Have a look at the AST of some Python code

# Pretty print AST
#https://github.com/clarketm/pprintast
#%pip install pprintast

from pprintast import pprintast as ppast # OR: from pprintast import ppast

# 2. pretty print AST from a "string".
exp = '''
import os, math
import pandas as pd
from pprintast import pprintast2 as ppast
def test_fn(a, b=1, c=2):
    """Add two numbers"""
    out = a+b
    print(out)
    return out
def test_fn2(a, b=1):
    out = a+b
    if a>b:
        print(a)
    else:
        print(b)
    print(out)
'''

ppast(exp)

This gives a pretty printed output that lets us review the AST:

Module(body=[
    Import(names=[
        alias(name='os', asname=None),
        alias(name='math', asname=None),
      ]),
    Import(names=[
        alias(name='pandas', asname='pd'),
      ]),
    ImportFrom(module='pprintast', names=[
        alias(name='pprintast2', asname='ppast'),
      ], level=0),
    FunctionDef(name='test_fn', args=arguments(posonlyargs=[], args=[
        arg(arg='a', annotation=None, type_comment=None),
        arg(arg='b', annotation=None, type_comment=None),
        arg(arg='c', annotation=None, type_comment=None),
      ], vararg=None, kwonlyargs=[], kw_defaults=[], kwarg=None, defaults=[
        Constant(value=1, kind=None),
        Constant(value=2, kind=None),
      ]), body=[
        Expr(value=Constant(value='Add two numbers', kind=None)),
        Assign(targets=[
            Name(id='out', ctx=Store()),
          ], value=BinOp(left=Name(id='a', ctx=Load()), op=Add(), right=Name(id='b', ctx=Load())), type_comment=None),
        Expr(value=Call(func=Name(id='print', ctx=Load()), args=[
            Name(id='out', ctx=Load()),
          ], keywords=[])),
        Return(value=Name(id='out', ctx=Load())),
      ], decorator_list=[], returns=None, type_comment=None),
    FunctionDef(name='test_fn2', args=arguments(posonlyargs=[], args=[
        arg(arg='a', annotation=None, type_comment=None),
        arg(arg='b', annotation=None, type_comment=None),
      ], vararg=None, kwonlyargs=[], kw_defaults=[], kwarg=None, defaults=[
        Constant(value=1, kind=None),
      ]), body=[
        Assign(targets=[
            Name(id='out', ctx=Store()),
          ], value=BinOp(left=Name(id='a', ctx=Load()), op=Add(), right=Name(id='b', ctx=Load())), type_comment=None),
        If(test=Compare(left=Name(id='a', ctx=Load()), ops=[
            Gt(),
          ], comparators=[
            Name(id='b', ctx=Load()),
          ]), body=[
            Expr(value=Call(func=Name(id='print', ctx=Load()), args=[
                Name(id='a', ctx=Load()),
              ], keywords=[])),
          ], orelse=[
            Expr(value=Call(func=Name(id='print', ctx=Load()), args=[
                Name(id='b', ctx=Load()),
              ], keywords=[])),
          ]),
        Expr(value=Call(func=Name(id='print', ctx=Load()), args=[
            Name(id='out', ctx=Load()),
          ], keywords=[])),
      ], decorator_list=[], returns=None, type_comment=None),
  ], type_ignores=[])

We can now parse that into a dict, for example:

#https://www.mattlayman.com/blog/2018/decipher-python-ast/
import re
import ast
from pprint import pprint

# TO DO update generic_visit to capture other nodes
# A NodeVisitor can respond to any type of node in the Python AST.
# To visit a particular type of node, we must implement a method that looks like visit_.
class Analyzer(ast.NodeVisitor):
    def __init__(self):
        self.stats = {"import": [], "from": [], "function":[]}

    def visit_Import(self, node):
        for alias in node.names:
            import_ = {'name':alias.name, 'alias':alias.asname}
            self.stats["import"].append(import_)
        self.generic_visit(node)

    def visit_ImportFrom(self, node):
        imports = {'from': node.module, 'import':[]}
        for alias in node.names:
            imports['import'].append({'name':alias.name, 'as':alias.asname})
        self.stats["from"].append(imports)
        self.generic_visit(node)
    
    def visit_FunctionDef(self, node):
        ret = None
        args = [a.arg for a in node.args.args]
        args2 = [c.value for c in node.args.defaults]
        argvals = [a for a in args]
        for (i,v) in enumerate(args2[::-1] ):
            argvals[-(i+1)] = f"{args[-(i+1)]}={v}"
        for n in node.body:
            if isinstance(n, ast.Return):
                ret = re.sub('^return\s+' , '', ast.get_source_segment(exp, n))
        self.stats["function"].append({'name':node.name, 
                                       'docstring': ast.get_docstring(node),
                                       'returns': ret,
                                       'args': args, 'args2': args2, 'argvals':argvals,
                                       'src':ast.get_source_segment(exp,node)})
        self.generic_visit(node)

    def report(self):
        pprint(self.stats)
 

And that then generates output of the form:

tree = ast.parse(exp)
analyzer = Analyzer()
analyzer.visit(tree)
analyzer.report()

'''
{'from': [{'from': 'pprintast',
           'import': [{'as': 'ppast', 'name': 'pprintast2'}]}],
 'function': [{'args': ['a', 'b', 'c'],
               'args2': [1, 2],
               'argvals': ['a', 'b=1', 'c=2'],
               'docstring': 'Add two numbers',
               'name': 'test_fn',
               'returns': 'out',
               'src': 'def test_fn(a, b=1, c=2):\n'
                      '    """Add two numbers"""\n'
                      '    out = a+b\n'
                      '    print(out)\n'
                      '    return out'},
              {'args': ['a', 'b'],
               'args2': [1],
               'argvals': ['a', 'b=1'],
               'docstring': None,
               'name': 'test_fn2',
               'returns': None,
               'src': 'def test_fn2(a, b=1):\n'
                      '    out = a+b\n'
                      '    if a>b:\n'
                      '        print(a)\n'
                      '    else:\n'
                      '        print(b)\n'
                      '    print(out)'}],
 'import': [{'alias': None, 'name': 'os'},
            {'alias': None, 'name': 'math'},
            {'alias': 'pd', 'name': 'pandas'}]}
'''

It’s not hard to see how we could then convert that to various text sentences, such as:

# N packages are imported directly: os and math without any aliases, pandas with the alias pd
# The ppast package is loaded in from the pprintast module with alias ppast
# Two functions are defined: test_fn, which will add two numbers, and...
# The test_fn function takes two arguments TO DO N required and M optional

It would be trivial to create some magic to wrap all that together, the let user use a block cell magic such as %%summarise_this_code to generate the text description, or play it out using a simple text to speech function.

PS in passing, it’s also worth noting pindent.py (via) which will add #end of block comments at the end of each code block in a Python program. Backup gist: https://gist.github.com/psychemedia/2c3fe0466aca1f760d67d5ca4f6e00b1

With Permission: Running Arbitrary Startup Services In Docker Containers

In Running Arbitrary Startup Scripts in Docker Containers, I described a recipe cribbed from MyBinder/repo2docker, for running arbitrary scripts on startup of a Docker container.

One thing I hadn’t fully appreciated was the role of permissions in making sure that scripts and services called in the startup script had enough permissions to run. In the containers we are using, which are inspired by the official Jupyter stack containers, the containers are started with a specified user (in the Jupyterverse, this is user jovyan wuth UID 1000, by convention).

The start script runs under the user that is started into the container, so trying to start the postgres database service from my start script resulted in a permissions error.

One possible fix is to elevate the permissions of the user so that they can run the desired start commands. This is perhaps not as unreasonable as it might sound, at least in an educational context. The containers are single user environments, and when run from a multi-user JupyterHub environment (at least under Kubernetes, rather than for example in The Littlest JupyterHub). Whilst we don’t want learners to be in a position where they accidentally destroy their environment, my personal belief is that we should allow learners to have as much ownership of the environment as possible. (It should also be noted that if a student does mangle a containerised environment, respawning the environment from the original image as a new container should put everything back in place…)

So how do we go about elevating permissions? The approach I have used to date (for example, in here) is to allocate sudoers priviliges to the user in respect of at least the commands that are used to start up services in the start up script.

For example, the following Dockerfile command gives the user permission to start the postgresql service, run the mongo server and fire up an external start script:

RUN echo "$NB_USER ALL=(ALL:ALL) NOPASSWD: /sbin/service postgresql restart" >> /etc/sudoers && \
    echo "$NB_USER ALL=(ALL:ALL) NOPASSWD: /usr/bin/mongod" >> /etc/sudoers && \
    echo "$NB_USER ALL=(ALL:ALL) NOPASSWD: /var/startup/start_jh_extras" >> /etc/sudoers

The extra start script is actually provided as a place for additional startup items required when used in a JupyterHub environment, about which more in a later post. (The sudoers bit for that script should probably really be in the Dockerfile that generates the JupyterHub image, which slightly differs from the local image in my build.)

if [ -f "/var/startup/start_jh_extras" ]; then
    sudo /var/startup/start_jh_extras
fi

PS for a complementary approach to all this, see my colleague Mark Hall’s ou-container-builder.

Fragment: Remarkable Error Generation

It’s that time of year again when the vagaries of marking assert themselves and we get to third mark end of course assessment scripts where the first two markers award significantly different marks or their marks straddle the pass/fail boundary.

There is a fiddle function available to us where we can “standardise” marks, shifting the the mean of particular markers (and maybe the sd?) whose stats suggest they may be particularly harsh or lenient. Ostensibly, standardisation just fixes the distribution; ideally, this is probably something a trained statistiscian should do; pragmatically, it’s often an attempt to reduce the amount of third marking; intriguingly, we could probably hack some code to “optimise” stanadardisation to bring everyone in to line and reduce third marking to zero; but we tend to avoid that process, leaving the raw marks in all their glory.

I’ve never really understood why we don’t do a post mortem after the final award board and compare the marks awarded by markers in their MK1 (first marker) and MK2 (second marker) roles against the mark finally awarded to a script. This would generate some sort of error signal that modules teams, staff tutors and markers could use to see how effective any given marker is at “predicting” the final grade awarded to a script. But we don’t do that. Analytics are purely for applying to learners because it’s their fault. (I often wonder if the learning analytics folk look at marker identity as one of the predictors for a student’s retentin and grade; and if there is an effect; or maybe some things are best left under the stone…)

Anyway… third marking time.

In a sense, it’s a really useful activity because we get to see a full range of student scripts and get a feel for what they’ve got out of the course.

But the fact that we often get a large disparity between marks does, as ever, raise questions about the reliablity of the marks awarded to scripts we don’t third mark (for example, if two harsh or lenient markers mark the same scripts). I’m sure there are ways the numbers could be churned to give some useful and simple insights into individual marker behaviour, rather than the not overly helpful views we’re given over marker distributions. And I wonder if we just train a simple text classifier on raw scripts against the final awarded mark on a script how much it would vary compared to human markers. And maybe one that classifies based on screenshots of the report (a 20 second skim of how a report looks often gives me sense of which grade boundary it is likely as not to fall in…)

But ours not to reason why…

Tinkering With Selenium IDE: Downloading Multiple Files from One Page, Keyed by Another

It being the third marking time of year again, I get to enjoy the delights of having to use various institutional systems to access marks and student scripts. One system stores the marks, allocates me my third marking tasks (a table of student IDs and links to their marks, their marks, and a from to submit my marks). Another looks after the scripts.

To access the scripts, I need to go to another system, enter the course code and student ID, one at a time, (hmm, what happens if I try a list of IDs with various separators; could that give me multiple files?) to open a pop-up window from which I can click to collect a zipped file containing the student’s submitted work. The downloaded file is downloaded as a zip file with a filename of the form ECA-2021-06-28_2213.zip ; which is to say, a filename based on datetime.

To update a set of marks, I need to get a verification code from the pop up raised after entering the student ID on the second system into a form on the page associated with a particular student’s marks on the first system. Presumably, the thinking about workflow went something like: third marker looks at marks on first system, copies ID, gets script and code from second system, marks script, enters code from second system in first system, updates mark. For however many scripts you need to mark. One at a time. Rather than: download every script one at a time, do marking howsoever, then have to juggle both systems trying to figure out the confirmation code for a particular student to update the marks from a list you’ve scribbled onto a piece of paper against their ID (is that a 2 or a 7?). Or whatever.

Needless to say, several years ago I hacked a mechanicalsoup Python script to look up my assigned marking on the first system, along with the first and second marks, download all the scripts and confirmation codes from the second system, unzip the student script downloads and bundle everything into a directory tree. I also hacked some marking support tools that would display how the markers compared on each of the five marking criteria they scored scripts against and allow me to record my marks. I held off from automating the upload of marks back to the system and kept that as a manual step becacause I don’t want to get into the habit of hacking code to write to university systems just in case I mess something up… I did try to present my workflow and tools to exams and various others by sharing a Powerpoint review of it, but as I recall never got any reply.

Anyway… mechanicalsoup. A handy package combing mechanize and beautifulsoup, the first part mocked a browser and allowed you to automate it, and the second part provided the scraping utilities. But mechanize doesn’t do Javascript. Which was fine because the marks and scripts systems are old old HTML and easily scraped, pretty vanilla tables and web forms. And the old OU auth was pretty simple to automate your way through too.

But the new OU authenticator uses Javascript goodness(?) as part of its handshake so my timesaver third marking scraping tools are borked because I can’t get through the auth.

So: time to play with Selenium, which is a complete browser automation tool that automates an off-the-shelf browser (Chrome, or Firefox, or Safari etc) rather than mocking one up (as per mechanicalsoup). Intended as a tool for automated testing of websites, you can also use it as a general purpose automation tool, or to provide browser automation for screenscraping. I’ve tinkered with Selenium before, scripting it from Python to automate repetitive tasks (eg Bulk Jupyter Notebook Uploads to nbgallery Using Selenium) but there’s also a browser extension / Selenium IDE that lets you record steps as you work through a series of actions in a live website, as well as scripting in your own additional steps.

So: how hard can it be, I thought, to record a quick script to automate the lookup of student IDs and then step through each one? Surprisingly faffy, as it turns out. The first issue was simply how to iterate through the rows of the table containing each individual student reference to pick up the student ID.

The method I ended up with was to get a cound of rows in the table, then iterate through each row, picking up the student ID as link text (of the form STUDENT_ID STUDENT NAME), duly cleaned by splitting on the first space and grabbing the first element, and then manually creating a string of delimited IDs STUDENT_ID1::STUDENT_ID2::... . (I couldn’t seem to add IDs to an ID array but I was maybe doing something wrong… And trying to find any sensible docs on getting stuff done using the current IDE seems to be a largely pointless task.)

So, I now have a list of IDs, which means I can (automatically) click through the script download system and grab the scripts one at a time. Remember, this involves adding a course code and a student identifer, clicking a button to get a pop up, clicking a button to zip and download the student files, then closing the pop up.

Here’s the first part – entering the course code and student ID:

In the step that opens the new window, we need to flag that a new window has been opened and and generate a reference to it:

In the pop-up, we can then click the collect button, wait a moment for the download to start, then close the pop-up and return to the window where we enter the course code and student ID:

If I now run the script on a browser where I’m already logged in (so the browser already has auth cookies set), I can just sit back and watch it grab the student IDs from my work allocation table on the first system to generate a list of IDs I need scripts for, and then download each one from the second system.

So I have the scripts, but as a set of uselessly named zip files (some of them duplicates); and I don’t have the first and second marks scrpaed from the first system. Or the confirmation codes from the second system. To perform those steps, I probably do need a Python script automating the Selenium actions. the Selenium IDE is fine (ish) for filling in forms with simple scraped state and then clicking buttons that act on those values, but for scraping it’s not really appropriate.

Whilst the Selenium IDE doesnlt export Python code, it does produce an export JSON file that itemises the steps in scripts created in the IDE. This could be used to help boostrap the production of Python code. The Selenium IDE recorder provides a way of recording simple pointy-clicky sequences of action which could be really useful to help get those scripts going. But ideally, I need a thing that can replay the JSON exported scripts from Python then I could have the best of both worlds.

Finally, in terms of design pattern, this recipe doesn’t include any steps that interact directly with logging in: the automated browser uses cookies that have been set previously elsewhere. Rather, the script automates actions over a previously logged in browser. Which means this sort of script is something that could be easily shared within an organisation? So I wonder, are there orgs in which the core systems don’t play well but skunkworks and informal channels share automation scripts that do integrate them, ish?!

(Hmm… would the Python scripted version load a browser with auth cookies set, or does it load into a private browser in which authentication would be required?)

Bah… I really should be marking, not tinkering…

PS It looks like you can export to a particular language script:

…but when I try it I get an error message regarding an Unknown locator:

Appropriate/ing Knowledge and Belief Tools?

A few weeks ago, I got my invite to the OpenAI GPT-3 beta (Terms of Engagement With the OpenAI API). You know the one: give it a start phrase and it will then start to generate follow on text for you. The phrase can be anything:

It was a cold, grey, morning. I picked up the chain saw and headed off into the town.

Wasn’t long before I was joined by a couple of men I knew from the pub. We’d worked together for a few years before my accident, and they were just pleased to see me back at work.

It was enjoyable, once I’d got used to the hard physical work. I knew I was in good shape, and the cold air made me feel even better.

My arms and legs were a little stiff, but nothing that wouldn’t soon disappear. I’d stopped taking painkillers a couple of months after getting the cast off, and hadn’t had any problems since.

It had been almost three months since I’d been at work. It was good to be back.

We were working on the new school, which had been burnt down a few months before. It was almost complete, and all that was left was to do the roofing and masonry work.

The owner of the building company was a nice man, with a sense of humour. He placed a ladder against the outside wall for us to stand on while we worked on the roof. I knew it was safe as long as he was holding it.

We had almost finished, when we heard shouting from the ground. We turned around to see the boss waving his arms, and we realised that the ladder had slipped from under one of the men. He was dangling from the top rung

OpenAI machine generated text

Or from a slightly different start:

It was a cold, grey, morning. I mopped up the blood, picked up the chain saw, and headed off into the town.

My face and head were covered in blood, so I had to drive with the window down. The blood was beginning to dry and flake. I didn’t want anyone to see me, so I didn’t stop.

I pulled into the parking lot of a 7-Eleven. I went inside, pulled out a newspaper, and sat down at a table by the window.

“Joe,” the guy behind the counter called out.

“Damn…that’s me,” I thought. I looked up, and was relieved it was just him. I gave him a nod.

“Hey, Joe, you okay?”

There was no point in lying to him.

“No. Not really,” I said.

“Uh, what happened?”

I’d been here before. If I had a different face, that would have been the end of it. But I had blood on my face.

“You need to go to the hospital?”

He was getting nervous.

I shook my head and said, “No. It’s nothing. Just a fight with my wife.”

He smiled, relieved. He went back behind the counter. I continued reading the paper. A few minutes later, he came back over and stood by my table

OpenAI machine generated text

(Hmmm… product placement opportunities to help pay for these services and keep then “free”?!)

Now, the above may or may not be “plagiarised” text, lifted directly from a single source or recombined from multiple trash fiction sources. The second piece is perhaps interesing because it makes use of dialogue. I can’t help but wonder what the surrealists would have made of this, how they would have seen this as a mechanical riff on both the exquisite corpse and automatic writing, an automated “fetch the next word” algorithm built on top of machine based hallucination engine wrapping the GPT-3 model constructed from an exquisite and almost unimagible, exquisite even, text corpus. (Yes, it was me that wrote that; not the machine.)

So the attribution of the previous text may or may not be suspect. The text may or may not be original. We have no way of knowing.

But consider the following example, a declarative rephrasing of a simple recall question — what are Asimov’s Three laws of Robotics?— that we have asked year on year in an introductory module:

Asimov’s Three Laws of Robotics are:

The first law: A robot may not injure a human being or, through inaction, allow a human being to come to harm. The second law: A robot must obey the orders given it by human beings except where such orders would conflict with the First Law. The third law: A robot must protect its own existence as long as such protection does not conflict with the First or Second Laws.

The laws are implied in many of Asimov’s Robot books, but they are first stated explicitly in The Robots of Dawn. Other issues are also raised in the series. In later books, Asimov added a “zeroth law” of robotics: “A robot may not harm humanity, or, by inaction, allow humanity to come to harm”; this was introduced in Robots and Empire and again in The Caves of Steel.

The first three laws are similar to “The Three Laws of Eddorian Robotics” appearing in Eddore from the Lensman series by E. E. Smith. However, Smith’s laws are not a fundamental part of his future history, but are instead a throwaway item. Smith stated that his laws were merely a literary device to be used in his stories, and that they do not reflect any deep philosophical idea.

(More product placment?!)

In this case, it seems as if we get a factful respone. Compare, for example, the result of typing the interrogative form of the statement into the Google web search engine:

In this case, we get a similiar response, but with some form of attribution.

So, OpenAI can perhaps be used as a knowledge engine. But how trustworthy is it?

In formal logics of knowledge and belief, a distinction is often made between a belief statement that the system holds and uses to reason about the world and a knoweldge statement, a statement that the system holds and uses to reason about the world that is also true in and of the world.

In some respects, we might see Google as attempting to perform as a knowledge engine, returning facts that are true, and OpenAI as a belief engine, freewheeling a response based on what it’s just heard and what it’s heard others say before.

Let’s consider another example, the start of a declarative statement that we might be offered in response to an assessment questions aked in a simple databases module, for example:

The main differences between relational databases and document databases are

that document databases are:

Not relational. This means that relationships between data entities are not enforced in a strict sense. Document databases are designed to store and retrieve individual documents, not rows in a table.

Non-normalized. Document databases do not store data in multiple tables that are related by key fields. Instead, all the data for a given entity is stored in a single document.

In addition, there is no enforced integrity or consistency between entities in a document database.

Not ACID-compliant. ACID stands for Atomicity, Consistency, Isolation, Durability. These are the four basic properties of a transaction and are used to guarantee data integrity in a database.

Because document databases do not enforce relationships between entities, they do not meet the ACID test and thus are not transactional.

However, document databases address the same three basic CRUD (Create, Read, Update, Delete) scenarios as other non-relational databases.

They can also be queried and filtered using SQL queries and can scale to hundreds of nodes with a single database cluster.

Also, they can be hosted in a variety of cloud environments, including Amazon Web Services (AWS).

For more on why I refer to document databases as NoSQL databases, see this page on Wikipedia.

OpenAI machine generated text from initial prompt

(And even more product placement?! Is this gonna be a whole new SEO style battleground?!)

So… we get an answer, of a sort. (Computing academics are welcome to score the above answer out of 5 and submit marks as comments to the is post…). One think I note in particular is the reference (in text) to Wikipedia, but no link or page reference. At that point in the response, I had limited to the engine to produce a certain number of words, so who knows whether a link or page reference would have been generated as the next item. (I should have let it play on, I guess…)

One might also wonder what other Coleridgian interruptions our automatic writing engine might experience…?

It’s not just text that the models will generate. A recent announcement from Github (owned by Microsoft) and OpenAI introduces Github Copilot, “a new AI pair programmer that helps you write better code” which claims to be able to let you “write a comment describing the logic you want, and let GitHub Copilot assemble the code for you”, “let GitHub Copilot suggest tests that match your implementation code”, and let “GitHub Copilot show you a list of solutions [so you can] evaluate a few different approaches”.

In passing, I note an interesting UI feature in highlighting the the latter example, anudge, literally: a not-button is nudged, enticing you to click it, and if you do, you’re presented with another example:

The code as it currently stands is based on a model trained from submissions to Github. My immediate thought was: is it possible to licence code in a way that forbids its inclusion in machine learning/AI training sets (or will it be a condition of use of Github that public code repos at least must hand over the right for the code, and diffs, and commit comments to be used for machine training?). Another observation I saw several folk make on the Twitterz was whether we’ll start t see folk deliberately putting bad code or exploit code into Github in an attempt to try to pollute the model. As a quality check, I wondered what would happen if every Stack Overflow were provided with a machine generated answer based on OpenAI generated text and Copilot generated code and then used upvotes and downvotes as a error/training signal. Then @ultrazool/Jopointed out that a training signal can already be generated from suggested code that later appears in a git commit, presumably as a vote of confidence. We are so f****d.

It’s also interesting to ponder how this fits into higher education. In the maths and sciences, there are a wide range of tools that support productivity and correctness. If you want a solution to, or the steps in a proof of, a mathematical or engineering equation, Wolfram Alpha will do it for you. Now, it seems, if you want an answer to a simple code question, Copilot will offer a range of solution for your delectation and delight.

At this point, it’s maybe worth noting that code reuse is an essential part of coding practice, reusing code fragments you have found useful (and perhaps then adding them to the language in the form of code packages on PyPi), as for example described in this 2020 arXiv preprint on Code Duplication and Reuse in Jupyter Notebooks.

So when it comes to assessment, what are we to do: should we create assessments that allow learners to use knowledge and productivity tools, or should we be constraining them to do their own work, ex- of using mechanincal (Worlfram Alpha?) or statistical-mechanical (OpenAI) support tools for the contemporary knowledge worker?

PLAGIARISM WARNING – the use of assessment help services and websites

The work that you submit for any assessment/exam on any module should be your own. Submitting work produced by or with another person, or a web service or an automated system, as if it is your own is cheating. It is strictly forbidden by the University.

You should not:

– provide any assessment question to a website, online service, social media platform or any individual or organisation, as this is an infringement of copyright.

– request answers or solutions to an assessment question on any website, via an online service or social media platform, or from any individual or organisation. use an automated system (other than one prescribed by the module) to obtain answers or solutions to an assessment question and submit the output as your own work.

– discuss exam questions with any other person, including your tutor. The University actively monitors websites, online services and social media platforms for answers and solutions to assessment questions, and for assessment questions posted by students.

A student who is found to have posted a question or answer to a website, online service or social media platform and/or to have used any resulting, or otherwise obtained, output as if it is their own work has committed a disciplinary offence under Section SD 1.2 of our Code of Practice for Student Discipline. This means the academic reputation and integrity of the University has been undermined.

And when it comes to the tools, how should we view things like OpenAI and and Copilot? Should we regard them belief engines, rather than knowledge engines, and if so how should we then interact with them? Should we be starting to familiarise ourselves with the techniques descriebed in Automatic Detection of Machine Generated Text: A Critical Survey, or is that being unnecessarily prejudiced against the machine?

In skimming the OpenAI docs [Answer questions guide], one of the ways of using OpenAI is as “a dedicated question-answering endpoint useful for applications that require high accuracy text generations based on sources of truth like company documentation and knowledge bases”. The “knowledge” is provided as “additional context” uploaded via additional documents that can be used to top up the model. The following code fragment jumped out at me though:

{"text": "puppy A is happy", "metadata": "emotional state of puppy A"}
{"text": "puppy B is sad", "metadata": "emotional state of puppy B"}

The data is not added as a structured data object, such as `{subject: A, type: puppy, state:happy}`, it is added as a text sentence.

Anyone who has looked at problem solving strategies as a general approach in any domain is probably with familiar with the idea that the way you represent a problem can make it easier (or harder) to solve. In many computing (and data) tasks, solutions are often easier if you represent them in a very particular, structured way. The semantics are essentially mapped to syntax, so if you get the syntax right, the semantics follow. But here we have an example of taking structured data and mapping it into natural language, where it is presumably added to the model but with added weight to be applied in recall?

This puts me in mind of a couple of other things:

  • a presentation by a somone from Narrative Science or Automated Insights (I forget which) many years ago, commenting on how one use for data-to-text engines was to generate text sentences from every row of data in a database so that it could then be searched for using a normal text search engine, rather than having to write a database query;
  • the use of image based representations in a lot of a machine learning applications. For example, if you want to analyse an audio waveform, whose raw natural representation is a set of time ordered amplitude values, one way of presenting it to a machine learning system is to re-present it as a spectrogram, a two dimensional image with time along the x-axis and a depiction of the power of each frequency component along the y-axis.

It seems as if everyday we are moving away from mechanical algorithms to statistical-mechanical algorithms — AI systems are stereotype engines, with prejudiced beliefs based on the biased data they are trained on — embedded in rigid mechanical processes (the computer says: “no”). So. Completely. F****d.

Scoping Out JupyterLab Extensions

Over the years, we’ve added quite a few classic notebook extensions to the mix in the environment we distribute to students in our data management and analysis course.

One of the things that has put me off moving to JupyterLab has been the increased complexity in developing extensions, both in terms of the typescript source and the need to familiarise myself with the complex and bewildering JupyterLab framework API.

But if we are to move to JupyterLab components, or if I want to try to make more use of JupyterLite, we are going to need to find some JupyterLab extension equivalents to the extensions we currently use in classic notebooks; or find someone to create them. And on current form, given the chances of the latter are near zero, that means I need to update the extensions myself. So this post is a quick review of already available JupyterLab extensions that might serve as equivalents of the extensions we currently use, or that I could use as cribs for my own extensions.

“Official” cookie cutter and example extension repos are available: jupyterlab/extension-cookiecutter-ts, jupyterlab/extension-examples. The examples repos are filled with dozens and dozens of developer cruft files and it’s not clear which are actually required, of those that are required, whether they are/can be/should be automatically generated (and how), or whether not-essential files can break things if they are laying around and not correct etc etc. This is a confusing and a massive blocker to me trying to get started from a position of wanting to have as little to do as possible with formal developer IDEs, build crap, test crap, etc etc. Productivity tool support for “proper devs” in “proper IDEs” is all very well for “proper devs”; but for the rest of us, it’s a hostile “you’re not welcome” signal (think: hostile architecture) and goes very much against a minimum viable example principle where only the bare essentials are included in the repo…

I’m not sure what extension, if anything, is powering the cell tags, which appear to only be accessed via the settings gear in the right hand margin tab list:

If you click on a tag, it seems to disappear, which is a usability nightmare, in my opinion…

Collapsible_Headings: JupyterLab equivalent of classic notebook nbextensions/collapsible_headings extension; note that the value assigned to heading_collapsed metadata is *not* the same in the JupyterLab and classic extensions. In the classic extension, the assignment is to the boolean true; in the JupyterLab extension, the assignment is to the string "true". See this related issue requesting metadata parity across extensions.

jupyterlab-skip-traceback: JupyterLab version of classic notebook nbextensions/skip-traceback extension for collapising error messages beneath error message name header; dozens of files in the repo, no idea which are necessary and which are just stuff;

jupyterlab-system-monitor: “display system information (memory and cpu usage)”, a bit like the classic notebook jupyter-resource-usage (nbresuse as was?) extension;

jupyterlab-execute-time: display cell execution times, similar to classic notebook nbextensions/execute_time extension;

spellchecker: “highlight misspelled words in markdown cells within notebooks and in the text files”, cf. classic notebook nbextensions/spellchecker extension;

jupyterlab_code_formatter (docs): format one or more code cells; cf. classic notebook nbextensions/code_prettify extension;

jupyterlab-cell-flash: “show a flash effect when a cell is executed”; this could be a useful crib for replicating some of the cell run status indicators that are provided for the class notebook UI by nb_cell_execution_status (pending, running, completed cell activity).

clear-cell-outputs : what looks like a simple extension to clear all cell outputs from a toolbar; of the dozens of files in the repo, I’m not sure what the smallest subset you actually need to get this to build/install actually is. The demo shows outputs in an empty notebook being cleared, so I have no idea if it actually does anything. This might be useful as a crib for: adding a toolbar button; iterating through code cells; clearing cell output;

jupyter-scribe: “transforms Markdown cells into rich-text-editing cells, powered by ProseMirror”; cf. classic notebook jupyter-wysiwyg or livemdpreview extensions.

jlab-hide-code: very old (does it still work, even?) extension to provide two toolbar buttons to hide/unhide all code cells; cf. classic notebook nbextensions/hide_input_all extension;

jupyterlab-hide-code: provides a JupyterLab toolbar button “to run the code cells and then to hide the code cells”;

jupyterlab-show-cell-tags: show cell tags within notebook UI;

jupyterlab-codecellbtn: add a run button to the footer of each code cell;

jlab-enhanced-cell-toolbar: enhance cell toolbar with a toolbar for selected cells allowing cell type selection tool, code cell run button, tag display and tag editor tools;

jupyterlab-custom-css (about; not updated to JupyterLab 3.0?): add custom css rules in the settings in the Advanced Settings Editor;

visual-tags: “more easily choose which cells get executed”, apparently, but the README is just boilerplate cruft and I can’t see what this does (if anything), how to do it, etc etc.

jupyterlab_templates: create a new notebook from a notebook template.

There’s nothing I’ve found yet that provides anything like the nb_extension_empinken or nb_extension_tagstyler extensions for styling cells based on tags (if you know of an example, please suggest it via the comments).

There have been a couple of aborted(?)/stale-d attempts to add DOM attributes based on tags (eg here and as originally referenced here) but this enabling feature never seems to get enough traction to make it as a merged PR. There is also a community contributed celltag2dom, but it also looks like it may be unmaintained/stale (IIRC, I couldn’t make sense of the code, and as with many extensions built presimably from examples, there’s a stack of files and I have no idea which are necessary, what any of them do, if any of the essentially superfluous ones can break things if they are laying around but not right, etc etc.).

VS Code as an Integrated, Extensible Authoring Environment for Rich Media Asset Creation

One of the things I’m a bit conflicted about in the context of asset creation tools is the extent to which they should be scripted. The examples I started pulling together in my Subject Matter Notebooks demos (an effort which has stalled again…:-( are essentially scripted: the asset generating examples are generated from script. But there are other ways of creating assets within a Jupyter environment, using interactive editors such as the draw.io editor:

One of the concerns I have with tools like this in the JupyterLab context is that you can start to step away from the narrated description of how an asset was created. You can see this even more clearly in things like the plotly chart editor:

editor.gif

Yes, you can use this tool to create a diagram or source file from which you can generate a png or interactive chart asset, for example. And yes, you can edit that source file via the UI. But the linear, literate, narrated construction of the asset is lost.

But that’s a debate for another post… This post is about VS Code.

Just like a Jupyter user interface environment can be used as a rich (generative) interactive asset authoring and display environment (for example, OpenJALE, my Open Jupyter Authoring and Learning Environment demo), so too can VS Code. And with a much larger developer community, VS Code currently provides support for a wider range of tools than the Jupyter ecosystem, and competition between them that drives further improvement.

So here’s a quick review of just some of the tools that VS Code already integrates via extensions that support the authoring of rich media (I’ll review various extensions that support teaching computing related subjects in another post).

As we’ve mentioned draw.io already, in the context of its availability as JupyterLab extension, let’s start there: the hediet.vscode-drawio extension provides .drawio extension senstivity in the form of an embedded drawio provided in much the same way as the JupyterLab extension. I donlt know if the JupyterLab extension works in a collaborative mode, but the VS Code extension claims to, via VS Code Liveshare. Another handy feature is the ability to edit the diagram and its XML source code side by side: edit the diagram and the XML updates, edit the XML and the diagram updates (does this approach also work in JupyterLab? I guess it should if the diagram is reactive to changes in the source file?)

When it comes to editing markdown files, a markdown side-by-side previewer is offered natively, with support for things like embedded LaTeX maths expressions provided by simple extensions such as goessner.mdmath, or with SMILES chemical molecule rendering support too, mathpix.vscode-mathpix-markdown, or as part of more comprehensive extensions such as shd101wyy.markdown-preview-enhanced. The latter (docs) adds support for maths equations, PDF generation, scripted diagram generation (including flowchart.js, Mermaid, PlantUML, Graphviz, Vega/Vega-Lite, Ditaa (ASCII art)). The extension can also execute code and render code outputs into the document (cf. Rmd in RStudio).

Other extensions that provide additional support to the markdown editing include spellchecking (for example, swyphcosmo.spellchecker), linting (for example, DavidAnson.vscode-markdownlint), pasting images into a markdown document from the clipboard (for example, telesoho.vscode-markdown-paste-image) and standalone diagram generating extensions (for example, gebv.pikchr (pikchr syntax editing and previews; render server required) or qiqiworld.vscode-markdown-tinymind (mindmaps); SimonSiefke.svg-preview for SVG previews; vstirbu.vscode-mermaid-preview for mermaid.js diagrams; several previewers for Graphviz/dot language, including tintinweb.graphviz-interactive-preview; the markdown-preview-enhanced extensions seems to cover several of these). I haven’t found a tikz renderer/previewer extension yet although there are lots of LaTeX related extensions, including the James-Yu.latex-workshop extension, which I notice supports some tikz snippets so maybe it does do tikz diagram rendering more genrally too.

Support is also available (currently in the VS Code insiders preview edition) for Jupytext (donjayamanne.vscode-jupytext), which will provide VS Code notebook style editing of markdown documents.

Musical scores / music typesetting in VS Code is also available using extensions such as lhl2617.VSLilyPond (a LilyPond wrapper) and ABC notation (softaware.abc-music).

Live-Preview

I haven’t spotted any extensions yet to support the creation and editing of electronic circuit diagram schematics.

3D model previews are available using extensions such as slevesque.vscode-3dviewer or michead.vscode-mesh-viewer. (In terms of image viewers, I haven’t found a FITS image viewer extension yet, or extensions that provide lookup and previewing lidar or satellite imagery? Maybe the astronomers et al. don’t do VS Code?)

Developer RandomFractalsInc publishes an interesting set of extensions that support interactive exploration of data and maps: the RandomFractalsInc.vscode-data-preview extension provides exploratory charting, the RandomFractalsInc.vscode-vega-viewer and RandomFractalsInc.vscode-chartjs extensions provide previews of Vega and Chart.js charts respectively, and the RandomFractalsInc.geo-data-viewer provides support for rendering a range of geo data formats including GeoJSON, TopoJSON, KML, GPX, and shapefiles. Other map preview extensions include jumpinjackie.vscode-map-preview. (I haven’t spotted a 3D map/elevation raster previewer yet.)