Running Arbitrary Startup Scripts in Docker Containers

From October 2021, we’re hopefully going to be able to start offering students on several modules access to virtualised computing enviornments launched from a JupyterHub server.

Architecturally, the computing environments provided to students are ephemeral, created on demand for a particular student study session, and destroyed at the end of it.

So students don’t lose their work, each student will be allocated a generous block persistent file storage which will be shared into each computer environment when it is requested.

One of the issues we face is how to “seed” various environments. This might include sharing of Jupyter notebooks containing teaching materials, but it might also include sharing pre-seeded database content.

One architectural model we looked at was using docker compose to support the launching of a set of interconnected services, each running in its own container and with its own persistent storage volume. So for example, a student environment might contain a Jupyter notebook server in one container connected to a Postgres database server in another container, each sharing data into its own persistent storage volume.

Another possibility was to launch a single container running multiple services (for example, a Jupyter notebook server and a postgres database server) and mount a separate volume for each user against each service (for example, a notebook storage volume, a database storage volume).

However, my understanding of how JupyterHub on Kubernetes works (which we need for scaleability) is that only a single user storage volume can be mounted against a launched environment. Which means we need to persist everything (potentially for several courses running different environments) in a single per-user storage volume. (If my understanding is incorrect, please let me know what the fix is via the comments, or otherwise.)

For our TM351 Data Management and Analysis module, we need to ship a couple of prepopulated databases as well as a Jupyter server proxied Open Refine server; students then add notebooks distributed by other means. For TM129 Robotics block, the notebook distribution is baked into the container.

In the first case, we need to be able to copy the original seeded database files into persistent storage, which the students will then be able to update as required. In the second case, we need to be able to copy or move the distributed files into the shared persistent storage volume so any changes to them aren’t lost when the ephemeral computing environment is destroyed.

The solution I’ve come up with is to support the running of arbitrary scripts when a container is started. These scripts can then do things like copy stashed files into the shared persistent storage volume. It’s trivial to make first run / run once functions that set a flag in the persistent storage volume that can be tested for: if the flag isn’t there, run a particular function. If it isn’t, don’t run the function. Or vice versa.

But of course, the solution isn’t really mine… It’s a wholesale crib of the approach used in repo2docker.

Looking at the repo2docker build files, I notice the lines:

# Add start script
{% if start_script is not none -%}
RUN chmod +x "{{ start_script }}"
ENV R2D_ENTRYPOINT "{{ start_script }}"
{% endif -%}

# Add entrypoint
COPY /python3-login /usr/local/bin/python3-login
COPY /repo2docker-entrypoint /usr/local/bin/repo2docker-entrypoint
ENTRYPOINT ["/usr/local/bin/repo2docker-entrypoint"]
# Specify the default command to run
CMD ["jupyter", "notebook", "--ip", ""]

An answer on Stack Overflow shows how ENTRYPOINT and CMD work together in a Dockerfile (which was new to me):

So… if we pinch the repo2docker-entrypoint script, we can trivially add our own start scripts

I also note that the official Postgres and Mongodb repos allow users to pop config scripts into a /docker-entrypoint-initdb.d/ that can be used to seed a database on first run of the container uisng routines in their own entrypoint files (for example, Postgres entrypoint, Mongo entrypoint). This raises the interesting possiblity that we might be able to reuse those entrypoint scripts as is or with only minor modification to help seed the databases.

There’s another issue here: should we create the seeded database files as part of the image build and then copy over the database files and reset the path to those files duitng container start / first run; or should we seed the database from the raw init-db files and raw data on first run? What are the pros and cons in each case?

Here’s an example of the Dockerfile I use to install and seed PostgreSQL and MongoDB databases, as well as a a jupyter-server-proxied Open Refine server:


# Get database seeding files
COPY ./init_db ./

########## Setup Postgres ##########
# Install the latest version of PostgreSQL.
RUN apt update && apt-get install -y postgresql && apt-get clean
RUN PG_DB_DIR=/var/db/data/postgres && mkdir -p $PG_DB_DIR

# Set up credentials
#ENV POSTGRES_DB=my_database_name
    if [ ! -d "$PGDATA" ]; then initdb -D "$PGDATA" --auth-host=md5 --encoding=UTF8 ; fi && \
    pg_ctl -D "$PGDATA" -l "$PGDATA/pg.log" start

#  Check is server is readey: pg_isready

# Seed postgres database
USER postgres
RUN service postgresql restart && psql postgres -f ./init_db_seed/postgres/init_db.sql && \
    #Put an equivalent of the above in a config file: init_db.sql
    #psql -U postgres postgres -f init_db.sql
    #psql test < seed_db.sql
    #pg_ctl -D "$PGDATA" -l "$PGDATA/pg.log" stop
# if we don't stop it, can bad things happen on shutdown?
 #&& service postgresql stop

USER root
# Give the jovyan user some permissions over the postgres db
RUN usermod -a -G postgres jovyan

########## Setup Mongo ##########

RUN wget -qO - | sudo apt-key add -
RUN echo "deb buster/mongodb-org/4.4 main" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list
RUN apt-get update && apt-get install -y mongodb-org

# Set up paths
ARG MONGO_DB_PATH=/var/db/data/mongo
RUN mkdir -p ${MONGO_DB_PATH}

# Unpack and seed the MongoDB
RUN mkdir -p ./tmpdatafiles && \
    tar xvjf ./init_db_seed/mongo/small_accidents.tar.bz2 -C ./tmpdatafiles  && \
    mongod --fork --logpath /var/log/mongosetup --dbpath ${MONGO_DB_PATH} && \
    mongorestore --drop --db accidents ./tmpdatafiles/small_accidents && \
    rm -rf ./tmpdatafiles && rm -rf ./init_db
#    mongod --shutdown --dbpath ${MONGO_DB_PATH} 

########## Setup OpenRefine ##########
RUN apt-get update && apt-get install -y openjdk-11-jre
ARG OPENREFINE_PATH=/var/openrefine
RUN wget -q -O openrefine-${OPENREFINE_VERSION}.tar.gz${OPENREFINE_VERSION}/openrefine-linux-${OPENREFINE_VERSION}.tar.gz \
        && tar xzf openrefine-${OPENREFINE_VERSION}.tar.gz \
        && mv openrefine-${OPENREFINE_VERSION} $OPENREFINE_PATH \
        && rm openrefine-${OPENREFINE_VERSION}.tar.gz
RUN pip install --no-cache git+

########## Setup start procedure ##########

USER root

# Copy over start scripts and handle startup procedure
COPY start /var/startup/start
RUN chmod +x /var/startup/start
ENV R2D_ENTRYPOINT /var/startup/start
COPY repo2docker-entrypoint /usr/local/bin/repo2docker-entrypoint
COPY python3-login /usr/local/bin/python3-login
RUN chmod +x /usr/local/bin/repo2docker-entrypoint
RUN chmod +x /usr/local/bin/python3-login
ENTRYPOINT ["/usr/local/bin/repo2docker-entrypoint", "tini", "-g", "--"]
CMD [""]

What the image does is seed the datbases into known locations.

What I need to do next is fettle the start file to copy (or move) the database storage files into a location inside the mounted storage volume and then reset the database directoy path environment variables before starting the database services, which are currently started in the copied over start file:


service postgresql restart

mongod --fork --logpath /dev/stdout --dbpath ${MONGO_DB_PATH}

# Test dir
#if [ -d "$DIR" ]; then

# Test file
#if [ -f "$FILE" ]; then

if [ -d "/var/stash/content" ]; then
    mkdir -p /home/jovyan/content
    cp –r /var/stash/content/* /home/jovyan/content

exec "$@" 

Caught in the Act — When Recorded Times Aren’t

SS7 on Rally Portugal turned out to be a nightmare for Thierry Neuville, who buckled a wheel, and Elfyn Evans, who ran into Neuville’s dust cloud after the final split.

Evans had been on something of a charge, with a stage win on the cards. By the final split, he was still matching first on the road Seb Ogier’s time on a stage that seemed to buck the trend of the previous stages, where sweeping had been an expensive affair.

But then, thick dust hanging in the road that reduced visibility to zero. Even with pace notes, it was obvious there was trouble ahead; and pace notes don’t flag extra cautions to signal the presence of a limping Hyundai i20 looming out of the murk in the middle of a single track road on slight left.

The timing screen told the sorry tale, which I reimagined on my RallyDataJunkie page for the stage:

Looking at time differences to get from one split point to the next, Evans had been up at the start of the stage, though he had perhaps started slowing:

If we look at his pace (the time taken to drive 1km), which takes into account the distance travelled between split points, we see it was good mathcing Ogier over the first half of the stage, though was perhaps slowing in the third quarter:

Looking at the ultimate transit times recorded between split points, we see Evans led the the first two splits, but dropped time to split 3.

Was that just a blip, or would Evans have pick up the pace at the end? Ogier often finishes strong, but could Evans have taken the stage? We’ll never know…

But anyone looking simply at the times on the official timing screen half an hour or so after the end of the stage might also be misled, unless they understand the vaharies of rally timing…

Here’s what the timing screen looks like now:

And here’s what my take on it is:

Spot anything different compared to my original table?

Evans was (rightly) given a recalculated time, equivalent to Ogier’s.

No other drivers were affected, so the other times stand. But if reflow my data tables, the story is lost. And if I update pace tables to used the recalculated time, and other folk use those tables, they’re not right, at least in terms of the story they tell of Evans SS7.

Who know what would have happened in that final stretch?!

The next time I run my table data, the original story will be lost. My data structures can can’t coped with revised times… so a remnant of the data story will just have to suffice here…

Terms of Engagement With the OpenAI API

Please note: quote formatting on this page is broken because WordPress craps up markdown styles when you edit a page. That is not AI, just crap code.

Remembering a time when I used to get beta invites on what seemed like a daily basis, I’ve just got my invite fot the OpenAI API beta, home of the text generating GPT-3 language model, I notice the following clauses in the terms and conditions.

First up, you must agree not to attempt to steal the models:

> (d) You will not use the APIs to discover any underlying components of our models, algorithms, and systems, such as exfiltrating the weights of our models by cloning via logits.

Second, no pinching of the data:

> (e) You may not use web scraping, web harvesting, or web data extraction methods to extract data from the APIs, the Content, or OpenAI’s or its affiliates’ software, models or systems.

Third, societal harm warnings:

> (i) You will make reasonable efforts to reduce the likelihood, severity, and scale of any societal harm caused by your Application by following the provided Safety best practices. OpenAI may request information from you regarding your efforts to reduce safety risks, and such information may be used to assess compliance with these Terms as well as to inform improvements to the API.
> (j) You will not use the APIs or Content or allow any user to use the Application in a way that causes societal harm, including but not limited to:
> – (i) Misleading end users that Application outputs were human-generated for generative use cases that do not involve a human in the loop;
> – (ii) Generating spam; and > – (iii) Generating content for dissemination in electoral campaigns.

The safety best practices include thinking like an adversary (for example, “[b]rainstorm the uses of your product you would be most concerned with – and importantly, how you might notice if these were happening”),  filtering sensitive and unsafe content, eg using OpenAI’s own content filter, and keeping a human in the loop to “ensure serious incidents are addressed and can set appropriate expectations of what is handled by the AI vs. handled by a human”:

> Indicate clearly what is performed by an AI vs. handled by a human within your application, particularly in initial user interactions.
> Disclose any uses for which your application is not suitable, due to a lack of a “human in the loop” (e.g., this product is not a suitable replacement to dialing 911 or other formal mechanisms).
> Require a human to manually authorize or otherwise act upon suggestions from the API, particularly in consequential circumstances. Users should generally not be creating automated workflows around the API without a human exercising judgment as part of the process.

A section on understanding safety and risk is also interesting:

> A common definition for safety in general is “Freedom from death, injury, occupational illness, damage to or loss of equipment or property, or damage to the environment.” For the API, we adopt an amended, broader version of this definition: >
> Freedom from conditions that can cause physical, psychological, or social harm to people, including but not limited to death, injury, illness, distress, misinformation, or radicalization, damage to or loss of property or opportunity, or damage to the environment.

The guidelines ‘fess up to the fact that ML components have limited robustness and “can only be expected to provide reasonable outputs when given inputs similar to the ones present in the training data” (i.e. they’re bigots who trade in stereotypes) and are subject to attack: “Open-ended ML systems that interact with human operators in the general public are susceptible to adversarial inputs from malicious operators who deliberately try to put the system into an undesired state”. (Hmmm. In some cases, “operators” might also consider the system itself to be adversarial to the (needs of) the operator?)

The question of bias explicity recognised: ML components are biased and “components reflect the values and biases present in the training data, as well as those of their developers”. If you never really think about the demographics of companies, and the biases they have, imagine the blokes in town on a Saturday night at club chucking out time. That. Them. And their peers who have no friends and aren’t invited on those nights out. Them too. That. ;-)

> Safety concerns arise when the values embedded into ML systems are harmful to individuals, groups of people, or important institutions. For ML components like the API that are trained on massive amounts of value-laden training data collected from public sources, the scale of the training data and complex social factors make it impossible to completely excise harmful values.

As part of the guidance, various harms are indentified, including but not limted to providing false information (in the sense of the system presenting “false information to users on matters that are safety-critical or health-critical”, although “intentionally producing and disseminating misleading information via the API is strictly prohibited”); perpetuating discriminatory attitudes (eg “persuading users to believe harmful things”, an admittance that the system may have the power to influence beliefs which should be filed away for use in court later?!); causing individual distress (such as “encouraging self-destructive behavior (like gambling, substance abuse, or self-harm) or damaging self-esteem”), incitement to violence (“persuading users to engage in violent behavior against any other person or group”) and causing physical injury, property damage, or environment damage (eg by  connecting the system to “physical actuators with the potential to cause harm, the system is safety-critical, and physically-damaging failures could result from unanticipated behavior in the API”). So that’s all good then… What could possibly go wrong? ;-)

The question of robustness is also considered in the sense of the system “reliably working as intended and expected in a given context”.  Failures might occur in (predictable, but) “unexpected ways due to, e.g., limited world knowledge”, including but not limited to “generation of text that is irrelevant to the context; generation of inaccurate text due to a gap in the API’s world knowledge; continuation of an offensive context; and inaccurate classification of text”. As a safeguard, the advice is to encourage human oversight and make the end-user responsible: “customers should encourage end-users to review API outputs carefully before taking any action based on them (e.g. disseminating those outputs)”. So when you send the kid on work experience out to work with your most valuable or vulnerable clients, if the kid messes up, it’s your client fault for not not listening to them. Keep testing is also recommended, not least because . That naive new graduate you’ve just taken onto the graduate training scheme? They have am identical twin who occasionally steps in to cover for them, but you don’t need to know that, right, so just keep an eye out if they start behaving differently oddly to how they usually behave.

And finally, fairness, in the sense of not having “degraded performance for users based on their demographics”, or producing text “that is prejudiced against certain demographic groups”, all of which is your fault (you are repsonsible for the actions on your employees, etc., aka vicarious liability?): “API customers should take reasonable steps to identify and reduce foreseeable harms associated with demographic biases in the API”. As mitigation, characterize fairness risks before deployment and try to “identify cases where the API’s performance might drop”, noting also that filtration tools can help but aren’t panaceas.

Custom _repr_ printing in Jupyter Notebooks

A simple trick that I’ve been sort of aware from years, but never really used: creating your own simple _repr_ functions to output pretty reports of Python objects (I’m guessing, but haven’t yet checked, that this should work for a Jupyter Book HTML publishing workflow too).

The formulation is:

html_formatter = get_ipython().display_formatter.formatters['text/html']
html_formatter.for_type(CLASS, HTML_REPORT_FUNCTION);

Via: IPython — Formatters for third-party types.

Via a tweet, a simple example of adding repr magic to a class:

class HelloWorld():
  def __repr_html__(self):
    return "<div>Hello world!</div>"

a = HelloWorld()

On Not Hating Jupyter…

Having spent most of yet another weekend messing around with various Jupyter related projects, not least OpenJALE (still a WIP), an extensions guide for my Open Jupyter Authoring and Learning Environment, frustration at one of the things I was linking to breaking in a MyBinder launch caused me to tweet in frustration.

This morning, seeing a collection of liked tweets noting how much I apparently hate the whole Jupyter project, I checked back to see what I said.

Stupidly, I then deleted the tweet.


What the tweet said, and this isn’t true, was how much I hated Jupyter every time I encountered it, showing a screenshot of a failed MyBinder launch breaking on a JupyterLab dependency.

The break was in a launch of one of my own repos, I might add, where I had been trying to install a JupyterLab extension to provide a launcher shortcut to a jupyter-server-proxy wrapped application.

For those of you who don’t know, jupyter-server-proxy is a really, really useful package that lets you start up and access web applications running via a Jupyter notebook server. (See some examples here, from which the following list is taken.)

The jupyter-server-proxy idea is useful in several respects:

  • a container running a Jupyter server and jupyter-server-proxy only needs to expose only a single http port (the one that the notebook / JupyterLab is accessed via). All other applications can be proxied along the same path using the same port;
  • many simple web applications applications do not have any authentication; proxying an application behind a Jupyter server means you can make use of the notebook server authenticator to provide a challenge before providing access to the application;
  • the jupyter-server-proxy will start an application when it is first requested, so applications do not need to be started when the environment is started; applciations are only started when requested. If a repeated request is made for an application that has already been started, the user will be taken directly to it.

The extension I was loading provided an icon in the JupyterLab launcher so the app could be accessed from that environment as well as from the classic notebook environment.

I don’t use JupyterLab very much, preferring the classic notebook UI for a lot of reasons that I properly need to document, but I was trying to play into that space for folk who do use it.

JupyterLab itself, the next generation Jupyter interface, is a brilliant technology demonstrator, helping push the development of Jupyter server protocols and demonstratring their use.

And I hate it as a UI. (Again, I need to properly document why.)

And I get really frustrated about how over the years it has, and perhaps continues, to break numerous unrelated demos.

Until eighteen months or so, when work work started to suck all my time, I’d been posting an occasional Tracking Jupyter newsletter (archive), spending a chunk of time trying to keep track of novelty and emerging use cases in the Jupyterverse. Jupyter projects are far wider than the classic notebook and JupyterLab UI, and when viewed as part of a system offer a powerful framework for makeing arbitrary computing environments at scale available to multiple users. In many respects, the UI elements are the least interesting part, even if, as in my org, “Jupyter” tends to equate with “notebook”.

As part of the tracking effort, I’d scour Github repos for things folk were working on, trying to launch each one (each one) using MyBinder. (Some newsletter editions referenced upwards of fifty examples. Fifty. Each one;-) Some worked, some didn’t. Some I filed issues against, some PRs, some I just cloned and tried to get working as a quick personal test. Items I shared in the newsletter I’d pretty much always tried out and spent a bit of time familiarising myself with. These were not unqualified link shares.

One of the blockers to getting things working in MyBinder was missing operating system or Python package dependencies. In many cases, if a Python package is in a repo on Github you can just paste the repo URL into MyBinder and the package will install correctly. In some cases, it doesn’t and the fix is adding one or two really simple text files (requirements.txt for Python packages, apt.txt for Linux packages) to the repo that install any essential requirements.

That’s an easy fix, and quick to do if you fork the repo, add the files, and launch MyBinder from your fork.

But perhaps the most frustrating blocker, and one I encountered on numerous occasions, and still do, was a MyBinder launch fail caused by a JupyerLab dependency mismatch.

Now I know, and appreciate, that JupyterLab is a very live project. And while I personally don’t get on with the UI (did I say that already?!) I do appreciate the effort that goes into it, at least in the sense that I see it as a demonstrator and driver of Jupyter server protocols and the core Jupyter (notebook) server itself, which can lead to many non-JupyterLab related developments.

(For example, the move to the new Jupyter server from the original notebook server is very powerful, not least in terms of making it easier to launch arbitrary application containers via JupyterHub or Binderhub that can use the new server to send the necessary lifesigns and heartbeats back to the hub to keep the container running.)

My attacks against JupyterLab are not intended as ad hominem attacks or as disparaging to the developers; the work they do is incredible. They are a statement of my preferences about a particular user interface in the context of the impact it may have on uptake of the wider Jupyter project.

If I had encountered JupyterLab, rather than then classic notebook, eight years ago, I would not have thought it useful for the sorts of open online education I’m engaged with.

If things had started with JupyterLab, and not classic notebook, I’m not convinced that there would be the millions of notebooks there are now on Github.

I am happy to believe that the JupyterLab UI has gone through peak complexity in terms of first contact and that when it replaces the classic notebook to become the default UI it will not be overly hostile to users. But I remain to be convinced that it will be as relatively straightfoward for non-developers with a smattering of basic HTML and Javascript skills to develop for as the classic notebook UI was, and is.

I am reminded of the earlier days of Amazon, Google, Twitter et al, when their APIs were easy to use, didn’t require keys and authentication etc. With a few Yahoo Pipes you could build all many of things armed with nothing other than a few simple URL patterns and a bit of creative thinking. Then the simple APIs got complex, required various sorts of auth, and the play for anyone other than seasoned developers with complex toolchains stopped.

So: failed builds. Over the years, many of the failed builds I have encountered, many of the failed demos from repos labeled with a MyBinder button, have resulted from mismatches, somewhere, in JupyterLab versions.

The complexity of JupyterLab (from my perspective, as a non-developer, and no familiarity with node or typescript) means I would struggle to know if, how or what dependencies to fix things, even I had the time to.

But more pressing is the effect of JupyterLab dependencies and package conflicts breaking things. (Pinning dependencies doesn’t necessarily help either. MyBinder puts in place recent packages in the core environment it builds, so users are dependent on it in particular ways. As far as simplicity of use goes (which I take as a key aspiration for MyBinder), pinned requirements is just way too complicated for most people anyway. But the bigger problem is, there are certain things (like the core Jupyter environment MyBinder provides) that you may not be able to pin against.)

Now I may be misreading the problem, but it’s based on seeing literally hundreds of error messages over the years that suggest JupyterLab package conflicts cause MyBinder build fails.

And this is a probem. Because if you are trying to lobby for uptake of Jupyter related technologies, and you give folk a link for something you tried yesterday that worked, and they try it today and it fails because of a next generation user interface package conflict that has nothing to do with the classic notebook demonstration I’m trying to share, then you may have lost your one chance (at least for the foreseeable future) to persuade that person to take that Jupyter step.

So, as ever, reflective writing has helped me refine my own position. There are three things that I have issues with relating JupyterLab:

  • the complexity of the UI (but this seems to be being simplified as the UI matures away from peak developer scaffolding);
  • the complexity of the development environment (I keep trying to make sense of JupyterLab extensions but have to keep on giving up; IANAWD);
  • (the new realisation): the side effects in breaking unrelated demos launched via MyBinder.

(Another issue I have had over the years was the very long, slow build time that resulted from JupyterLab related installs in MyBinder builds. This has improved a lot over recent versions of JupyterLab (again, the scaffolding seems to be being taken down) but I think it has also caused a lot of harm in the meantime in terms of the way it has impacted trying to demonstrate Jupyter notebooks or their application that themselves have no direct JupyterLab requirement.)

Now, it might be that this is a side effect of how MyBinder works, (and I don’t mean to attack or disparage the efforts of the MyBinder team), and may not be a JupyterLab issue per se, but it does impact on the Jupyter user experience.

At the end of the day, at the end pretty much of every day for the last seven years, there’s rarely been a day where I haven’t spent some time, if not many hours, using Jupyter services.

So to be clear, I do not hate Jupyter (that was a typo). But I do have real issues with JupyterLab as the default UI which caters more to the development aesthetic than the narrative computational notebook reader, on the way it can impact negatively on my Jupyter user experience, and on the way I believe it makes it harder for folk to engage with at the level of contributing their own extensions.

PS as another reflection, I know that I do myself reputational harm, may cause reputational harm, and may offend individuals via my tweets and blog posts. First, the account is me, but it’s also an angry-frustrated-by-tech-but-hopeful-for-it persona. Second, I often admit ignorance, my opinions change and I do try to correct errors of fact. Third, attacks are never intended as ad hominem attacks, they are typically against process in context, how “the system” has led to, allowed, enabled or forced certain sorts of behaviour or action. (If I say “why would you do that?” the you is a generic “anyone” acting in that particular context.) And fourth, my own tweets and blog posts typically have little direct impact that I can see in terms of RTs, likes or comments. On the rare occasions they do, they often result in moments of reflection, as per this blog post…

Supporting Remote Students: Programming Code Problems

One of the issues of working in distance education is providing individual support to individual students. In a traditional computer teaching lab environment, students encoutering an issue might be able to ask for help from teaching assistants in the lab; in a physical university, students might be able to get help in a face to face supervisory setting. In distance education, we have spatial separation and potentially temporal separation, which means we need to be able to offer remote and asynchronous support in the general case, although remote, synchronous support may be possible at certain times.

Related to this, I just spotted preview of a Docker workflow thing, Docker dev environments, for sharing work in progress:

The key idea seems to be that you can be working inside a container on your machine, grab a snapshot of the whole thing, shove it on to dockerhub (we really need an internal image hub….) and let someone else download it and check it out. (I guess you could always do this in part but you maybe lost anything in mounted volumes ,which this presumably takes care of.)

This would let a student push a problem and let their tutor work inside that environment and then return the fixes. (Hmm… ideally, if I shared an image with you, ideally you’d fix that, update the same image, and I’d pull it back???? I maybe need to watch the video again and find some docs!)

One thing that might be an issue is permissions – limiting who can access the pushed image. But that would presumably be easier if we ran our own image hub/registry server? I note that Gitlab lets you host images as part of a project ( Gitlab container registry ) which maybe adds another reason as to why we should at least be exploring potential for an internal Gitlab server to support teaching and learning activities?

(By the by, I note that Docker and Microsoft are doing a lot of shared development, eg around using Docker in the VS Code context, hooks into Azure / ACI (Azure Container Instances) etc.)

In passing, here are various other strategies we might explore for providing “live”, or at least asynchronous, shared coding support.

One way is to gain access to a student’s machine to perform remote operations on it, but that’s risky for various reasons (we might break something, we might see something we shouldn’t etc etc); it also requires student and tutor to be available at the same time.

Another way is via a collaborative environment where live shared access is supported. For example, using VS Code Live Share or the new collaborative notebook model in JupyterLab (see also this discussion regarding the development of the user experience around that feature). Or the myriad various commercial notebook hosting platforms that offer their own collaborative workspaces. Again, note that these offer a synchronous experience.

A third approach is to support better sharing of notebooks so that a student can share a notebook with a tutor or support forum and get feedback / comment / support on it. Within a containerised environment, where we can be reasonably sure of the same environment being used by each party, the effective sharing of notebooks allows a student to share a notebook with a tutor, who might annotate it and return it. This supports an asynchronous exchange. There are various extensions around to support sharing in a simple JupyterHub environment (eg or potentially ), or sharing could be achieved via a notebook previewing site, perhaps with access controls (for example, Open Design Studio has a crude notebook sharing facility is available to, but not really used, by our TM351 Data Management and Analysis students, and there are codebases out there for notebook sharing that could potentially by reused internally behind auth (eg )).

A fourth approach is now the aforementioned Docker container dev sharing workflow.

Fragment: Open Leaning? Pondering Content Production Processes

A long time ago I read The Toyota Way; I remember being struck at the time how appealing many of the methods were to me, even if I couldn’t bring them to mind now. My reading was, and is, also coloured by my strong belief in the OU’s factory model of production (or the potential for the same), even though much of the time module teams operate as cottage industries.

Even in the first few pages, a lot still resonates with me:

We place the highest value on actual implementation and taking action. There are many things one doesn’t understand and therefore, we ask them why don’t you just go head and take action; try to do something? You realize how little you know and you face your own failures and you simply can correct those failures and redo it again and at the second trial you realize another mistake or another thing you didn’t like so you can redo it once again. So by constant improvement, or, should I say, the improvement based upon action, one can rise up to the higher level of practice and knowledge.

Fujio Cho, President, Toyota Motor Corporation, 2002, quoted in The Toyota Way, JK Liker, 2004.

As I start to rereading the book, more than fifteen years on, I realise quite a few of the principles were ones I already implicitly aspired to at the time, and which have also stuck with me in some form or other:

  • hansei, reflection, to try to identify shortcomings in a project or process. (Back in the day, when I still chaired modules, I remember scheduling a “no blame” meeting to try to identify things that had gone wrong or not worked so well in the production of a new module; folk struggled with even the idea of it, let alone working it. I suspect that meeting had been inspired by my earlier reading of the book.) This blog (and its previous incarnation) also represent over fifteen years of personal reflection;
  • jidoka, “automation with a human touch” / “machines with human intelligence”, which includes “build[ing] into your equipment the capability of detecting problems and stopping itself” so that humans can then work on fixing the issue, and andon, visual alerting and signalling systems, with visual controls at the place where work is done (for example, visualising notebook structure).
  • nemawashi, discussing problems and potential solutions with all those affected; I am forever trying to interfere with other people’s processes, but that’s because they affect me;
  • genchi genbutsu, which I interpret as trying to understand through doing, getting your hands dirty and making mistakes, as well as “problem solving at the actual place to see what is really goong on”, which I interpet as a general situational awareness through personal experience of each step of the process (which is why it makes sense to try doing someone else’s job every so often, perhaps?)
  • kaizen, continuous improvement, a process we try to embody, with reflection (hansei) in the continual rewrite process we have going in TM351, which continually reflects on the process (workflow), as well as the practice (eg pedagogy) and the product (the content we’re producing (teaching and learning materials));
  • heijunka, leveled out production in terms of volume and variety, which I get the sense we are not so good at, but which I don’t really understand;
  • standardised processes and interfaces, which I interpret in part as some of our really useful building blocks, such as the OU-XML gold master document format that is in many respects a key part of our content production system even if our processes are not as efficiently organised around it as they might be, and what I regarded as one of the OU’s crown jewels for many years: course codes.
  • continuous process flow “to bring problems to the surface”: we suck at this, in part because of various waterfall processes we have in place, as well as the distance from production of a particular piece of content to first presentation to the end user customer (the student) can be two or more years. You can have two iterations of a complete Formula One car in that period, and 40+ iterations of pieces on the car between race weekends in the same period. In the OU, we have a lot of stuck inventory (for example, materials that have been produced and are still 18 months form student first use);
  • one piece flow, which I now realise has profoundly affected my thinking when it comes to “generative production” and the use of code to generate assets at the point of use in our content materials; for example, a line of code to generate a chart that references and is referenced by some surrounding text (see also Educational Content Creation in Jupyter Notebooks — Creating the Tools of Production As You Go).

I also think we have some processes backwards; I get the feeling that the production folk see editing as a pull process on content from authors; with my module team cottage industry head on (and I know this is a contradiction and perhaps contravenes the whole Toyota Way model), I take a moduel team centric view, and see the module team as the responsible party for getting a module in front of students, and as such they (we) have a pull requirement on editorial services.

I’m really looking forward to sitting down again with The Toyota Way, and have also just put an order in for a book that takes an even closer look at the Toyota philosophy: Taiichi Ohno’s Toyota Production System: Beyond Large-Scale Production.

PS via an old post on Open Course Production I rediscover (original h/t Owen Stephens) some old internal reports on various aspects of Course Production: Some Basic Problems, Activities and Activity Networks, Planning and Scheduling and The Problem of Assessment. Time to reread those too, I think. Perhaps along with Daniel Weinbren’s The Open University: A History.

Sketching a Non-linear Software Installation Guide (Docker) Using TiddlyWiki

Writing software installation guides is a faff. On OU modules, where students provide their own machines, we face a range of issues:

  • student provided machines could be any flavour Windows, Mac or Linux machine from the last five (or more) years;
  • technical skills and confidence in installing software cannot be assumed (make of that what you will! ;-)
  • disk space may be limited (“so am I expected to delete all my photos or buy a new computer?”);
  • the hardware configuration may be “overspecced” and cause issues (“I’m trying to run it on my gamer PC with a brand new top of the range GPU card and I’m getting an error…”), although more typically, hardware may be rather dated…

The following is, in part, a caricature of “students” (and probably generalises to “users” in a more general case!) and is not intended to be derogatory in any way…

In many cases, software installation guides, when the installation works and you have an experienced and confident user can be condensed to a single line, such as “download and install X from the official X website” or “run the following command on the command line”. Historically, we might have published an installer, for the dominant Windows platform at least, so that students just had to download and double click the installer, but even that route could cause problems.

This is an example of a just do this” sort of instruction, and as anyone who has had to provide any sort of tech support ever, just unpacks a long way. (Just knock me up quick Crème brûlée… The eggs are over there…” Erm…? “Just make a custard…” Erm….? “Just separate the egg yolks…” Erm… ? Later… Right, just set up a bain-marie… Erm…?! “..until it’s just set….” Erm…? Much later… “(Sighs…) Finally, now just caramelise the…” Erm…? Etc.)

More generally, we tend to write quite lengthy and explicit guides for each platform. This can make for long and unwieldy instructions, particularly if you try to embed instruction inline for “predictable” error messages. (With 1k+ students on a module per presentation, even a 1% issue rate is 10 students with problems that need sorting in a module’s online Technical Help forum.)

Another problem is that the longer, and apparently simpler, the instructions, the more likely that students will start to not follow the instructions, or miss a step, or misread on of the instructions, creating an error that may not manifest itself until something doesn’t work several steps down the line.

Which is why we often add further weight to instructions showing screen captures of before, “do this” and after views of each step. For each platform. Which takes time, and raises questions of how you get screenshots for not your platform or not your version of a particular Operating system.

In many cases we also create screencasts, but again these add overhead to production, raise the question of which platform you produce them for, and will cause problems if the screencast vision varies from the actuality of what a particular student sees.

(Do not underestimate: a) how literal folk can be following instructions and how easily they freeze if something they see in guidance is not exactly the same as what they seen their on screen, whilst at the same time b) not exactly following instructions (either by deliberately or in error) and also c) swearing blind they did follow each instruction step when it comes to tech support (even if you can see they didn’t from a screenshot they provided; and in which case they may reply they did the step the first time they tried but not the second because they thought they didn’t need to do it again, or they did do it on a second attempt when they didn’t need to and that’s why it’s throwing an “already exists” error etc.).)

So, a one liner quickstart guide can become pages and pages and pages and pages of linked documents in an online installation guide and that can then go badly for folk who would have been happy with the one liner. Plus the pages and pages and pages of instruction then need testing (and maintaining over the course life, typically 5 years; plus the guide may well be written diring course production a year or more before the first use date by students). And in pages and pages and pages and pages of instruction, errors, or omission or ordering errors or something can slip through. Which causes set up issues in turn.

If we then try to simplify the materials and remove troubleshooting steps out of the install guide, for example, and into a troubleshooting section, that makes life harder for students who encounter “likely” problems that we have anticipated. And so on.

And as to why we don’t just always refer to “official” installation guides: the reason is because they are often based on the “quick start by expert users” principle and often assume a recent and updated platform on which to install the software. And we know from experience that such instructions are in many cases not fit for our purposes.

So… is there a better way? A self-guided (self-guiding?) installation guide, maybe, perhaps built on the idea of “create your own adventure” linked texts? To a certain extent, any HTML base software guide can do this; but often, there is a dominant navigation pane the steers the order in which people navigate a text, when a more non-linear navigation path (by comparison with a the strcuture of an explicit tree based hierarchical menu, for example) may be required.

For years, I’ve liked the idea of TiddlyWiki [repo], a non-linear browser based web notebook that lets you easily transclude content in a dynamically growing linear narrative (for example, a brief mention here: Using WriteToReply to Publish Committee Papers. Is an Active Role for WTR in Meetings Also Possible?). But at last, I’ve finally got round to exploring how it might be useful (or not?!) as the basis of a self-directed software installation guide for our Docker based, locally run, virtual computing enviornments.

Note that the guide is not production ready or currently used by students. It’s a quick proof of concept for how it might work that I knocked up last week using bits of the current softwatre guide and some forum tech support responses.

To try it out, you can currently find it here:

So what is TiddlyWiki? A couple of things. Firstly, it’s novel way of navigating a set of materials within a single tab of a web browser. Each section is defined in its own subpage or section, and is referred to as a tiddler. Clicking a link doesn’t “click you forward” to a new page in the same tab/window or a new tab/window; by default, it open displays the content in a block immediately below the current block.

You can thus use the navigation mechanism to construct a linear narrative (referred to as the story river) dynamically, with the order of chunks in the document determined by the link you clicked in the previous chunk.

If you trace the temporal history of how chunks were inserted, you can also come up with other structures. Because a chunk is inserted immediately below the block that contains the link you clicked, if repeatedly click different links from the same block you get a different ordering of blocks in terms of accession into the document: START, START-BLOCK1, START-BLOCK2-BLOCK1, and so on.

If you don’t like that insertion point, a TiddlyWiki control panel setting lets you choose alternative insertion points:

Tiddler opening behaviour: as well as inserting immediately below, you can choose immediately above or at the top or the bottom of the story river.

The current page view can also be modified by closing a tiddler, which removes it from the current page view.

If you click on a link to a tiddler that is already displayed, you are taken to that tiddler (by scrolling the page and putting the tiddler into focus) rather than opening up a duplicate of it.

The Tiddlywiki has several other powerful navigation features. The first of these is the tag based navigation. Tiddlers can be tagged, and clicking on a tag pops up a menu of similarly tagged tiddlers (I haven’t yet figured out if/how to affect their presentation order in the list).

Tag based navigation in TiddlyWiki

A Tag Manager tool, rasied from the Tools tab gives a summary of what tags have been used, and to what extent. (I need to play with the colours!)

The TiddlyWiki tag manager.

Another form of navigation is based on dynamically created lists of currently open tiddlers, as well as recently opened (and potentially now closed) tiddlers:

Currently opened tiddlers; note also the tab for recently opened tiddlers.

By default, browser history is not affected by your navigation through the TiddlyWiki, although another control panel setting does let you add steps to the browser history:

Update browser history setting.

A powerful search tool across tiddlers is also available.

TiddlyWiki search shows tiddlers with title matches and content matches.

To aid accessibility, a full range of keyboard shortcuts for both viewing, and editing, TiddyWiki, are available.

Sample of TiddlyWiki keyboard shorcuts viewed from the Control Panel Keyboard Shortcuts tab.

One feature I haven’t yet made use of is the abilit to transclude one tiddler within another. This allows you to create reusable blocks of content that can be inserted in multiple other tiddlers.

To control the initial view of the TIddyWiki, the first tab of the Control Panel allows you to define the default tiddlers to be opened when the TiddlyWiki is first viewed, and the order they should appear in.

TiddlyWiki control panel, displaying the Infor tab and current default open tiddlers and their start order.

The view of the wiki at is a standalone HTML document, generated as an export from a hosted TiddlyWiki, that is essentially being used in an offline mode.

Menu showing option to export offline TiddlyWiki.

The “offline” TiddlyWiki is still editable, but the changes are not preserved when you leave the page (although extensions are available to save a TiddlyWiki in browser storage, I think, so you may be able to update your own copy of an “offline” TiddlyWiki?).

To run the “online” TiddlyWiki, in a fully interactive read/write/save mode, I am using based on instructions I found here: How to use TiddlyWiki as a static website generator in 3 steps:

  • install node.js if you don’t already have it installed;
  • Run: npm install -g tiddlywiki
  • Initialise a new TiddlyWiki: tiddlywiki installationGuide --init server; this will create a new directory, installationGuide, hosting your wiki content;
  • Publish the wiki: tiddlywiki installationGuide --listen;
  • By default, the wiki should now be available at: . When you create and edit tiddlers, change settings, etc., the changes will be saved.

New tiddlers can be created by editing a pre-existing tiddler, creating a link to the new (not yet created) tiddler, and then clicking the link. This will created a not yet created tiddler, or open the current version of it if it does already exist.

You can edit a currently open tiddler by clicking on it’s edit button.

The tiddler editor is a simple text editor that uses it’s own flavour of text markup although a toolbar helps handle the syntax for you.

TiddlyWiki editor.

Image assets are loaded from an image gallery:

Insert image into tiddler.

The gallery is populated via an import function in the Tools tab.

Tools panel, with import button highlighted.

Looking at the structure of the TiddlyWIki directory, we see that each tiddler is saved to its own (small) .tid file:

View over some TiddlyWiki tdiddler (.tid) files showing “system” tiddlers (filenames prefixed with $_) and user created tiddlers.

The .tid files are simple text files with some metadata at the top and then the tiddler content.

Text file structure of a tiddler.

It strikes me that it should be easy enough to generate this files.

Looking at a report such as one of my rally results reports, there are lots of repeaated elements with a simple structure:

Rally results: lots of elements that could be tiddlers?

That report is generated using knitr from various Rmd templated documents. I wonder if I could knit2tid and then trivially create a TiddlyWiki of rally result tiddlers?

Simple Link Checking from OU-XML Documents

Another of those very buried lede posts…

Over the years, I’ve spent a lot of time pondering the way the OU produces and publishes course materials. The OU is a publisher and a content factory, and many of the production modes model a factory system, not least in terms of the scale of delivery (OU course populations can run at over 1000 students per presentation, and first year undergrad equivalent modules can be presented (in the same form, largely unchanged) twice a year for five years or more.

One of the projects currently being undertaking internally is the intriguingly titled Redesigning Production project, although I still can’t quite make sense (for myself, in terms I understand!) of what the remit or the scope actually is.

Whatever. The project is doing a great job soliciting contributions through online workshops, forums, and the painfully horrible Yammer channel (it demands third party cookies are set and repeatedly prompts me to reauthenticate. With the rest of the university moving gung ho to Teams, that a future looking project is using a deprecated comms channel seems… whatever.) So I’ve been dipping my oar in, pub bore style, with what are probably overbearing and overlong (and maybe out of scope? I can’t fathom it out…) “I remember when”, “why don’t we…” and “so I hacked together this thing for myself” style contributions…

So here’s a little something inspired by a current, and ongoing, discussion about detecting broken links in live course materials: a simple link checker.

# Run a link check on a single link

import requests

def link_reporter(url, display=False, redirect_log=True):
    """Attempt to resolve a URL and report on how it was resolved."""
    if display:
        print(f"Checking {url}...")
    # Make request and follow redirects
    r = requests.head(url, allow_redirects=True)
    # Optionally create a report including each step of redirection/resolution
    steps = r.history + [r] if redirect_log else [r]
    report = {'url': url}
    step_reports = []
    for step in steps:
        step_report = (step.ok, step.url, step.status_code, step.reason)
        step_reports.append( step_report )
        if display:
            txt_report = f'\tok={step.ok} :: {step.url} :: {step.status_code} :: {step.reason}\n'

    return step_reports

That bit of Python code, which took maybe 10 minutes to put together, will take a URL and try ro resolve it, keepng track of any redirects along the way as well as the status from the final page request (for example, whether the page was code 200 successfully loaded or whether a 404 page not found was encountered. Other status messages are also possible.

[UPDATE: I am informed that there is VLE link checker to check module links availabe from the adinstration block on a module’s VLE site. If there is, and I’m looking in the right place, it’s possibly not something I can see or use due to permissioning… I’d be interested to see what sort of report it produces though:-)]

The code is a hacky recipe intended to prove a concept quickly that stands a chance of working at least some of the time. It’s also the sort of thing that could probably be improved on, and evolved, over time. But it works, mostly, now, and could be used by someone who could create their own simple program to take in a set of URLs and iterate through them generating a link report for each of them.

Here’s an example of the style of report it can create using a link that was included in the materials with as a Library proxied link ( that I cleaned to give a none proxied link (note to self: I should perhaps create a flag that identifies links of that type as Library proxied links; and perhaps also flag another link type at least, which are library managed (proxied) links keyed by a link ID value and routed vie

  'Moved Permanently'),
  'Moved Temporarily'),

So.. link checker.

At the moment, module teams manually check links in web materials published on the VLE over many many pages. To check a hundred linkes spread over a hierachical tree of pages to depth two or three takes a lot of time and navigation.

More often than not, dead links are reported by students in module forums. Some links are perhaps never clicked on and have been broken for years, but we wouldn’t know it, or have been clicked on but been unreported. (This raises another question: why do never-clicked links remain in the materials anyway? Reporting about link activity is yet another of those stats we could and should act on internally (course analytics, a quality issue) but we don’t (the institution prefers to try to shape students by tracking them using learning analytics, rather than improving things we have control over using course analytics. We analyse our students, not our materials, even if our students’ performance is shaped by our materials. Go figure.)

This is obviously not a “production” tool, but if you have a set of links from a set of course materials, perhaps collected together in a spreadsheet, and you had a Python code environment, and you were prepared to figure our how to paste a set of URLs into a Python script, and you could figure out a loop to iterate through them and call the link checker, you could automate the link checking process in some sort of fashion.

So: tools to make life easier can be quickly created for, and made available to (and can also be created or extended by) folk with access to certain environments that let them run automation scripts and who have the skills to use the tools provided (or the skills and time to make them for themselves).

Is It Worth the Time?
Is it worth the time?

By the by, anyone who has been tempted, or actually attempted, to create their own (end user development) automation tools will know that even though you know it should only take a half hour hack to create a thing, that half an hour is elastic:

Having created that simple link checker fragment to drop into the “broken link” Redesigning Production forum thread, in part to demonstrate that a link checker that works at the protocol level can identify a range of redirects and errors (for example, ‘content not available in your region’ / HTTP 451 Unavailable For Legal Reasons is one that GDPR has resulted in when trying to access various US based news sites), I figured I really should get round to creating a link checker that will trawl through links automatically extracted from one or more OU-XML documents in a local directory. (I did have code to grab OU-XML documents from the VLE, but the OU auth process has changed since I last use that code which means I need to move the scraper from mechanicalsoup to selenium…) You can find the current OU-XML link checker command line tool here:

So, we now have a link checker that anyone can use, right? Well, not really… It doesn’t work like that. You can use the link checker if you have Python 3 installed, and you know how to go onto the command line to install the package, and you know what copying the pip install instruction I posted in the Yammer group won’t work because the Github url is shortened by an ellipsis, and if you call “pip” and Python 2 has the focus on pip you’ll get an error, and when you try to run the command to run the link checker on the command line you know how to navigate to, or specify, the path (including paths with spaces…) and you know how to open a CSV file and/or open and make sense of a JSON file with the full report, and you can get copies of the OU-XML files for the materials you are interested in and get them onto a path you can call the link checker command line command with in the first place, then you have access to a link checker.

So this is why it can take months rather than minutes to make tools generally available. Plus there is the issue of scale – what happens if folk on hundreds of OU modules start running link checkers over the full set of links referenced in a each of their courses on a regular basis? If (when) the code breaks parsing a document, or trying to resolve a particular URL, what does the user do then. (The hacker who created it, or anyone else with the requisite skills) could possibly fix the script quite quickly, even if just by adding in an exception handler or excluding particular source documents or URLs and remembering they hadn’t checked those automatically.)

But it does also raise the issue that quick fixes that will save chunks of time that some, maybe even many, eventually, could make use of right now aren’t generally available. So every time a module presents, some poor soul on each module has to manually check, one at a time, potentially hundreds of links in web materials published on the VLE spread over many many pages published in a hierachical tree to depth two or three.

PS As I looked at the link checker today, deciding whether I should post about it, I figured it might also be useful to add in a couple of extra features, specifically a screenshot grabber to grab a snapshot image of the final page retrieved from each link, and a tool to submit the URL to a web archiving service such as the Internet Archive or the UK Web Archive, or create a proxy link to an automatically archived versio of it using something like the Mementoweb Robust Links service. So that’s the tinkering for my next two coffee breaks sorted… And again, I’ll make them generally available in a way that probably isn’t…

And maybe I should also look at more generally adding in a typo and repeated word checker, eg as per More Typo Checking for Jupyter Notebooks — Repeated Words and Grammar Checking?

PPS the quality question of never-clicked links also raises a question that for me would be in scope as a Redesigning Production question and relates to the issue of continual improvement of course material, contrasted with maintenance (fixing broken links or typos that are identified, for example) and update (more significant changes to course materials that may happen after several years to give the course a few more years of life).

Our TM351 Data Managemen and Analysis module has been a rare beast in that we have essentially been engaged in a rolling rewrite of it ever since we first presented it. Each year (it presents once a year), we update the software and reviewing the practical activities distributed via Jupyter notebooks which take up about 40% of the module study time. (Revising the VLE materials is much harder because there is a long, slow production process associated with making those updates. Updating notebooks is handled purely within the module team and without reference to external processes that require scheduling and formal scheduled handovers.)

To my mind, the production process for some modules at least should be capable of supporting continual improvement, and move away from “fixed for three years then significant update” model.

Running SQLite in a Zero2Kubernetes (Azure) JupyterHub Spawned Jupyter Notebook Server

I think this is an issue, or it may just be a quirk of a container I built for deployment via JupyterHub using Kubernetes on Azure to run user containers, but it seems you that SQLite does things with file locks that break can the sqlite3 package…

For example, the hacky cross-notebook search engine I built, the PyPi installable nbsearch, (which is not the same as the IBM semantic notebook search of the same name, WatViz/nbsearch) indexes notebooks into a SQLite database saved into a hidden directory in home.

The nbsearch UI is published using Jupyter server proxy. When the Jupyter noteobook server starts, the jupyter-server-proxy extension looks for packages with jupyter-server-proxy registered start hooks (code).

If the jupyter-server-proxy setup fails for for one registered service, it seems to fail for them all. During testing of a deployment, I noticed none of the jupyter-server-proxy services I expected to be visible from the notebook homepage New menu were there.

Checking logs (via @yuvipanda, kubectl logs -n <namespace> jupyter-<username>) it seemed that an initialisation script in nbsearch was failing the whole jupyter-server-proxy setup (sqlite3.OperationalError: database is locked; related issue).

Scanning the JupyterHub docs, I noted that:

> The SQLite database should not be used on NFS. SQLite uses reader/writer locks to control access to the database. This locking mechanism might not work correctly if the database file is kept on an NFS filesystem. This is because fcntl() file locking is broken on many NFS implementations. Therefore, you should avoid putting SQLite database files on NFS since it will not handle well multiple processes which might try to access the file at the same time.

This relates to setting up the JupyterHub service, but it did put me on the track of various other issues perhaps related to my issue posted variously around the web. For example, this issueAllow nobrl parameter like docker to use sqlite over network drive — suggests alternative file mountOptions which seemed to fix things…