Fragment: Software Decay From Inside and Out

Over the last couple of weeks, I’ve been dabbling with a new version of the software environment we use for our TM351 Data Management and Analysis course, bundling everything into a single monolithic docker container (rather than a more elegant docker compose solution because we haven’t yet figured out how to mount multiple personal volumes from a JupyterHub/k8s config),

Hmm… in a docker compose set up, where I mount a persistent volume onto container A at $SHAREDPATH, can I mount from a path $SHAREDPATH/OTHER in that container into another, docker compose linked container?

At the final hurdle, have fought with various attempts to build a docker container stack that works, I hit an an issue when trying to open a new notebook:

Crap.

The same notebook works fine in JupyterLab, so there is something wrong, somewhere, with launching notebooks in the classic Jupyter notebook UI.

Which made me a bit twitchy. Because the classic notebook is the one we use for teaching in several courses, and we use a wide variety of off-the-shelf extensions, as well as a range of custom developed extensions to customise our notebook authoring and presentation environment (examples). And these customisations are not available in JupyterLab UIs. For a related discussion, see this very opinionated post.

For folk who follow these things, and for folk who have a stake in the classic notebook UI, the question of long term support for the classic UI should be a consideration and a concern. Support for the classic notebook UI is not the focus area for the core Jupyter UI project developers effort.

And here’s another weak signal of a possible fork in the road:

The classic notebook user community, of which I consider myself a part, which includes education and could well extend into publishing more generally as tools like Jupyter Book mature even further, need to be mindful that someone needs to look after this codebase. And it would be a tragedy if that someone turned out to be someone who forked the codebase for their own (commercial) publishing platform. An Elsevier, for example, or a Blackboard.

Anyway, back to my 500 server error.

Here’s how the error starts to be logged in by the Jupyter server:

And here’s where the problem might be:

In a third party package that provides and an “export to docx (Microsoft Word)” feature implemented as a Jupyter notebook custom bundler.

Removing that package seemed to fix things, but it got me wondering about whether I should treat this as a weak signal of software rot in Jupyter notebook. I tweeted to the same effect — with a slight twinge of uncertainty about whether folk might think I was dissing the Jupyter community again! — but then started pondering about what that might actually mean.

Off the top of my head, it seems that one way of slicing the problem is consider rot that comes from two different directions:

• inside: which is to say, as packages that the notebook depends on update, do things start to break inside the notebook server and its environment. Pinning package versions may help, or making sure you always run the notebook server in its own, very tightly controlled Python environment and always serve kernels from a separate environment. But if you do need to install other things in the same environment as the notebook server, and there is conflict in the dependencies of those things and the notebook server’s dependencies, things might break;

• outside: which is to say, things that a user or administrator might introduce into the notebook environment to extend it. As in the example of the extension I installed that in its current version appears to cause the 500 server error noted above.

Note that in the case of the outside introduced breakage, the error for the user appears to be that something inside the notebook server is broken: for the user, they draw the system boundary around the notebook server and its extensions whilst for the developer (core notebook server dev, or the extension developer), they see the world a bit differently:

There are folk who make an academic career out of such concerns of course, who probably have a far more consdered take on how software decays and how software rot manifests itself, so here are a few starters for 10 that I’ve added to my reading pile (no idea how good they are: this was just a first quick grab):

• Le, Duc Minh, et al. “Relating architectural decay and sustainability of software systems.” 2016 13th Working IEEE/IFIP Conference on Software Architecture (WICSA). IEEE, 2016.
• Izurieta, Clemente, and James M. Bieman. “How software designs decay: A pilot study of pattern evolution.” First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007). IEEE, 2007.
• Izurieta, C., Vetrò, A., Zazworka, N., Cai, Y., Seaman, C., & Shull, F. (2012, June). Organizing the technical debt landscape. In 2012 Third International Workshop on Managing Technical Debt (MTD) (pp. 23-26). IEEE.
• Hassaine, S., Guéhéneuc, Y. G., Hamel, S., & Antoniol, G. (2012, March). Advise: Architectural decay in software evolution. In 2012 16th European Conference on Software Maintenance and Reengineering (pp. 267-276). IEEE.
• Hochstein, Lorin, and Mikael Lindvall. “Combating architectural degeneration: a survey.” Information and Software Technology 47.10 (2005): 643-656.