On Not Hating Jupyter…

Having spent most of yet another weekend messing around with various Jupyter related projects, not least OpenJALE (still a WIP), an extensions guide for my Open Jupyter Authoring and Learning Environment, frustration at one of the things I was linking to breaking in a MyBinder launch caused me to tweet in frustration.

This morning, seeing a collection of liked tweets noting how much I apparently hate the whole Jupyter project, I checked back to see what I said.

Stupidly, I then deleted the tweet.

Crap.

What the tweet said, and this isn’t true, was how much I hated Jupyter every time I encountered it, showing a screenshot of a failed MyBinder launch breaking on a JupyterLab dependency.

[UPDATE:] At the risk of upsetting an overlapping part of the community, I suspect I probably meant to type “I hate JupyterLab”…

The break was in a launch of one of my own repos, I might add, where I had been trying to install a JupyterLab extension to provide a launcher shortcut to a jupyter-server-proxy wrapped application.

For those of you who don’t know, jupyter-server-proxy is a really, really useful package that lets you start up and access web applications running via a Jupyter notebook server. (See some examples here, from which the following list is taken.)

The jupyter-server-proxy idea is useful in several respects:

• a container running a Jupyter server and jupyter-server-proxy only needs to expose only a single http port (the one that the notebook / JupyterLab is accessed via). All other applications can be proxied along the same path using the same port;
• many simple web applications applications do not have any authentication; proxying an application behind a Jupyter server means you can make use of the notebook server authenticator to provide a challenge before providing access to the application;
• the jupyter-server-proxy will start an application when it is first requested, so applications do not need to be started when the environment is started; applciations are only started when requested. If a repeated request is made for an application that has already been started, the user will be taken directly to it.

The extension I was loading provided an icon in the JupyterLab launcher so the app could be accessed from that environment as well as from the classic notebook environment.

I don’t use JupyterLab very much, preferring the classic notebook UI for a lot of reasons that I properly need to document, but I was trying to play into that space for folk who do use it.

JupyterLab itself, the next generation Jupyter interface, is a brilliant technology demonstrator, helping push the development of Jupyter server protocols and demonstratring their use.

And I hate it as a UI. (Again, I need to properly document why.)

And I get really frustrated about how over the years it has, and perhaps continues, to break numerous unrelated demos.

Until eighteen months or so, when work work started to suck all my time, I’d been posting an occasional Tracking Jupyter newsletter (archive), spending a chunk of time trying to keep track of novelty and emerging use cases in the Jupyterverse. Jupyter projects are far wider than the classic notebook and JupyterLab UI, and when viewed as part of a system offer a powerful framework for makeing arbitrary computing environments at scale available to multiple users. In many respects, the UI elements are the least interesting part, even if, as in my org, “Jupyter” tends to equate with “notebook”.

As part of the tracking effort, I’d scour Github repos for things folk were working on, trying to launch each one (each one) using MyBinder. (Some newsletter editions referenced upwards of fifty examples. Fifty. Each one;-) Some worked, some didn’t. Some I filed issues against, some PRs, some I just cloned and tried to get working as a quick personal test. Items I shared in the newsletter I’d pretty much always tried out and spent a bit of time familiarising myself with. These were not unqualified link shares.

One of the blockers to getting things working in MyBinder was missing operating system or Python package dependencies. In many cases, if a Python package is in a repo on Github you can just paste the repo URL into MyBinder and the package will install correctly. In some cases, it doesn’t and the fix is adding one or two really simple text files (requirements.txt for Python packages, apt.txt for Linux packages) to the repo that install any essential requirements.

That’s an easy fix, and quick to do if you fork the repo, add the files, and launch MyBinder from your fork.

But perhaps the most frustrating blocker, and one I encountered on numerous occasions, and still do, was a MyBinder launch fail caused by a JupyerLab dependency mismatch.

Now I know, and appreciate, that JupyterLab is a very live project. And while I personally don’t get on with the UI (did I say that already?!) I do appreciate the effort that goes into it, at least in the sense that I see it as a demonstrator and driver of Jupyter server protocols and the core Jupyter (notebook) server itself, which can lead to many non-JupyterLab related developments.

(For example, the move to the new Jupyter server from the original notebook server is very powerful, not least in terms of making it easier to launch arbitrary application containers via JupyterHub or Binderhub that can use the new server to send the necessary lifesigns and heartbeats back to the hub to keep the container running.)

My attacks against JupyterLab are not intended as ad hominem attacks or as disparaging to the developers; the work they do is incredible. They are a statement of my preferences about a particular user interface in the context of the impact it may have on uptake of the wider Jupyter project.

If I had encountered JupyterLab, rather than then classic notebook, eight years ago, I would not have thought it useful for the sorts of open online education I’m engaged with.

If things had started with JupyterLab, and not classic notebook, I’m not convinced that there would be the millions of notebooks there are now on Github.

I am happy to believe that the JupyterLab UI has gone through peak complexity in terms of first contact and that when it replaces the classic notebook to become the default UI it will not be overly hostile to users. But I remain to be convinced that it will be as relatively straightfoward for non-developers with a smattering of basic HTML and Javascript skills to develop for as the classic notebook UI was, and is.

I am reminded of the earlier days of Amazon, Google, Twitter et al, when their APIs were easy to use, didn’t require keys and authentication etc. With a few Yahoo Pipes you could build all many of things armed with nothing other than a few simple URL patterns and a bit of creative thinking. Then the simple APIs got complex, required various sorts of auth, and the play for anyone other than seasoned developers with complex toolchains stopped.

So: failed builds. Over the years, many of the failed builds I have encountered, many of the failed demos from repos labeled with a MyBinder button, have resulted from mismatches, somewhere, in JupyterLab versions.

The complexity of JupyterLab (from my perspective, as a non-developer, and no familiarity with node or typescript) means I would struggle to know if, how or what dependencies to fix things, even I had the time to.

But more pressing is the effect of JupyterLab dependencies and package conflicts breaking things. (Pinning dependencies doesn’t necessarily help either. MyBinder puts in place recent packages in the core environment it builds, so users are dependent on it in particular ways. As far as simplicity of use goes (which I take as a key aspiration for MyBinder), pinned requirements is just way too complicated for most people anyway. But the bigger problem is, there are certain things (like the core Jupyter environment MyBinder provides) that you may not be able to pin against.)

Now I may be misreading the problem, but it’s based on seeing literally hundreds of error messages over the years that suggest JupyterLab package conflicts cause MyBinder build fails.

And this is a probem. Because if you are trying to lobby for uptake of Jupyter related technologies, and you give folk a link for something you tried yesterday that worked, and they try it today and it fails because of a next generation user interface package conflict that has nothing to do with the classic notebook demonstration I’m trying to share, then you may have lost your one chance (at least for the foreseeable future) to persuade that person to take that Jupyter step.

So, as ever, reflective writing has helped me refine my own position. There are three things that I have issues with relating JupyterLab:

• the complexity of the UI (but this seems to be being simplified as the UI matures away from peak developer scaffolding);
• the complexity of the development environment (I keep trying to make sense of JupyterLab extensions but have to keep on giving up; IANAWD);
• (the new realisation): the side effects in breaking unrelated demos launched via MyBinder.

(Another issue I have had over the years was the very long, slow build time that resulted from JupyterLab related installs in MyBinder builds. This has improved a lot over recent versions of JupyterLab (again, the scaffolding seems to be being taken down) but I think it has also caused a lot of harm in the meantime in terms of the way it has impacted trying to demonstrate Jupyter notebooks or their application that themselves have no direct JupyterLab requirement.)

Now, it might be that this is a side effect of how MyBinder works, (and I don’t mean to attack or disparage the efforts of the MyBinder team), and may not be a JupyterLab issue per se, but it does impact on the Jupyter user experience.

At the end of the day, at the end pretty much of every day for the last seven years, there’s rarely been a day where I haven’t spent some time, if not many hours, using Jupyter services.

So to be clear, I do not hate Jupyter (that was a typo). But I do have real issues with JupyterLab as the default UI which caters more to the development aesthetic than the narrative computational notebook reader, on the way it can impact negatively on my Jupyter user experience, and on the way I believe it makes it harder for folk to engage with at the level of contributing their own extensions.

PS as another reflection, I know that I do myself reputational harm, may cause reputational harm, and may offend individuals via my tweets and blog posts. First, the account is me, but it’s also an angry-frustrated-by-tech-but-hopeful-for-it persona. Second, I often admit ignorance, my opinions change and I do try to correct errors of fact. Third, attacks are never intended as ad hominem attacks, they are typically against process in context, how “the system” has led to, allowed, enabled or forced certain sorts of behaviour or action. (If I say “why would you do that?” the you is a generic “anyone” acting in that particular context.) And fourth, my own tweets and blog posts typically have little direct impact that I can see in terms of RTs, likes or comments. On the rare occasions they do, they often result in moments of reflection, as per this blog post…