What’s On the Horizon in the Jupyter Ecosystem?

Having been given a “visioning” role for a new level 1 course in production, I’ve started trying to make sense of what an online, or at least, virtual, computing and IT lab might look like for use in an OU context.

One of the ways I’ve tried to carve up the problem is in terms of support tools (the sorts of things a classroom management system might offer – chat rooms, screen sharing, collaborative working, etc) and end-user, task related applications. Another is to try to get a feel for how ecosystems might develop around particular technologies or communities.

It probably won’t surprise regular readers that one of the communities I’ve been looking at is the one growing up around Jupyter notebooks. So here’s a quick summary of some of the Jupyter related projects currently under development that have caught my eye.

Dashboards and Alternative Browser Based UIs

Although I’ve still to start playing with Jupyter widgets, the Jupyter incubator dashboards project seems to be offering support for a structured way o using them in the form of grid-based dashboards generated directly from notebooks. (I guess this is a variant of creating interactive slide decks, eg using nbconvert –to slides, from notebooks?)

It seems as if the dashboard project came out of the IBM Cloud Emerging Technology group (Dynamic Dashboards from Jupyter Notebooks) which suggests that as a tool Jupyter notebooks might have some appeal for business, as well as education and research…

Another company that seems to have bought into the Jupyter ecosystem is technical book publisher O’Reilly. Their thebe code library claims to provide “an easy way to let users on a web page run code examples on a server”, such as a simple HTML UI for a Jupyter served process, as this thebe demo illustrates.

(The Jupyter project itself seems to be developing several Javascript libraries that could be handy, such as Jupyter JS Services, a “Javascript client for the Jupyter services REST APIs”, and maybe jupyter-js-utils, “JavaScript utilities for the Jupyter frontend”; but as with many Jupyter projects in active development, if you’re not on the developer channel it can be tricky working out what the code is for or how to use it!)

One thing I’ve been wondering about for a rewrite of out level 1 residential school robotics activity is whether we might be able to produce a browser or electron app based desktop or tablet based editor, inspired by the look and feel of the RobotLab drag’n’drop text based editor we’ve used in the course previously, to connect to a Jupyer server running on a Lego EV3 brick; and the thebe demo suggests to me that we might…

Collaboration

Collaboration around Jupyter notebooks comes in two forms: realtime collaborative editing within the same notebook (where two users have a copy of the same notebook open in separate editors and see each others updates in realtime), and collaboration around documents in a shared/social repository.

SageMathCloud already offers realtime collaboration within Jupyter notebooks, but official Jupyter support for this sort of feature is still on the official Jupyter project roadmap (using Google Drive as the backbone).

Realtime collaboration within notebooks is also available in the form of Livebook [code], which lives outside the main Jupyter project; the live demo site allows you to create – and collaborate around – temporary notebooks (pandas included): try opening a couple of copies of the same notebook (same URL) in a couple of browsers to get a feel for how it works…

In terms of asynchronous collaboration, this independent Commit-and-Push to GitHub from Jupyter Notebooks notebook extension looks interesting in terms of its ability to save the current notebook as a git commit (related issue here). The original nbdiff project [code] appears to have stalled, but there again, the SageMathCloud environment provides a history slider that lets you play through a whole series of (regular) saves of a notebook to show how it evolved and get access to “interim” versions of it.

There seems to be an independent NotebookDiff extension for comparing the state of notebook checkins, though I haven’t used it. I’m guessing the GitCheckpoints extension from the same developers (which I also haven’t tried) saves checkpoints as a git commit?

Jupyter on the Desktop

One of the “problems” of current Jupyter notebook usage is that the application does not run as a standalone app; instead, a server is started and then notebooks are accessed via a browser.

The Jupyter/Atom-notebook project takes a cross platform Atom editor (an electron framework app) and embeds Jupyter notebooks inside it. Double-clicking a notebook file will open it into the editor.

The nteract/composition app is a desktop based electron app, currently under development (I couldn’t get it to build with my node.js installation).

See also: this earlier, independently produced, proof of concept IPython Desktop project that offers a cleaner experience; the independent, proof-of-concept Jupyter sidecar electron app, that displays rich Jupyter kernel output from commands issued in a command line shell in an HTML presenting side display; and the Atom Hydrogen extension, which allows code to be executed against Jupyter kernels, Light Table style.

Summary

A quick scout around Jupyter related projects in progress shows much promise in the development of end-user tools that will make Jupyter notebooks easier to use, as well as tools that support collaborative working around a particular notebooks.

The Jupyter project has an active community around it and recently advertised for a full time project manager.

Jupyter notebooks feature in the IBM Data Scientist Workbench (as well as things like Wakari and Domino Data Lab) and IBM also seemed to bootstrap the dashboard components. Technical book publisher O’Reilly use Jupyter notebooks as a first-class authoring environment for the O’Reilly publishing program and Github recognises the .ipynb file format a first class document type, rendering HTML previews of .ipynb files uploaded to Github or as Github gists.

In a university context, Jupyter notebooks offer much potential for both teaching and research. It will be interesting to see how university IT departments react to this style of computing, and whether they try to find ways of supporting their community in the use of such systems, or whether their users will simply decide to go elsewhere.

See also: Seven Ways of Running IPython / Jupyter Notebooks.

PS I think this is probably going to become a living post…

• nbpresent: next generation slideshows from notebooks, apparently…
• nbbrowserpdf: “LaTeX-free PDF generation for Jupyter Notebooks”

Both of those come from the Anaconda developers, so it seems like Continuum are buying into the Jupyter ecosystem…

And some more from IBM: Jupyter Notebooks as RESTful Microservices that “turn notebooks into RESTful web APIs”. Hmm, literate API definitions than can be consumed by literate API consumer notebooks?

PS [March 2016] For a more recent round-up, see the IBM Emerging Tchnologies blog post: Powered By Jupyter: A Survey of the Project Ecosystem.