Notes on the JupyterLab Notebook HTML DOM Model, Part 8.5: A Reproducible Development Process

Across the course of the earlier posts in this series, I’ve been trying to place a series of stepping stones that demonstrate in a reasonably complete way how to create a simple JupyterLab notebook extension. At this point, I was hoping to share a repo that would demo the extension in a MyBinder environment, and act as an endpoint for installing a pre-built extension via a pip install (and then it’d be a simple step to pop it onto PyPi (at this point, I still donlt know how to build a pre-built extension). The next step would then be to also figure out a way of bundling the extension into a JupyterLite distribution. But. there’s always a but…

Here’s what my local file browser tells me my working directory looks like:

The directory was created by the Github App synch-ing a new repo I’d created from Github. This approach has the benefit that I don’t have to worry about any of the Github settings – I create a simple repo on Github then let the app figure out how to set up a directory I can work from on my local machine.

Here’s what Github tells me the structure looks like:

That is, top level is a jupyterlab_empinken directory. But that is not what I was expecting to see, because the ./Github/jupyerlab_empinken directory is supposed to map onto the top-level of the repo. And looking at my desktop file browser, that’s what looks like should be visible. I also note that lots of files are not added to the repo, and are ignored in the Github app file viewer: seems likes there’s a new .gitignore file in there somewhere too…

The nested directory structure visible in the Github repo is easily explained: when I ran the cookiecutter to seed the project (cookiecutter https://github.com/jupyterlab/extension-cookiecutter-ts), I did so in the ./Github/jupyerlab_empinken directory. I don’t recall if the cookiecutter had the option to install the files into the ./ directory, but it would be useful if it did.

As for how I managed to map the files to give the view in the local file browser, presumably setting an alias somewhere, I’m not sure what I did do?!

But I some point I did the following, which may or may not make sense…

# ================================================
#                  DON'T TRY THIS...
# ================================================

# I'm just trying to keep a note of what I originally did...

# I'm guessing is is where files were popped into
# the top level directory?
pip3 install --upgrade ./jupyterlab_empinken/

# ...

# At some point, I then went into the subdir
# created by the cookiecutter
cd jupyterlab_empinken/

# At this point, I was floundering because I 
# couldn't find a way to update the extension
# in my JupyterLab environment...

# So I tried everything I could find...
jupyter labextension develop . --overwrite
jlpm run build
jlpm run rebuild
python3 -m pip install -e .
jupyter labextension develop . --overwrite

# Something in that chain appeared to do the trick,
# because from this point, and ny whatever process,
# the following steps then reliably rebuilt 
# the extension and ran a version of JupyterLab 
# with the updated extension installed...
jlpm run build
jupyter lab

I also had an issue of being unable to uninstall the extension. This may or may not relate to this issue which hints at the need to manually remove files but doesn’t give a path as to where to remove files from. If the recommended dev route installs uninstallable files, it would be useful to provide a set of explicit manual removal instruction steps. In the end, I found a set of extension related files left over in /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/share/jupyter/labextensions (the jupyter --paths command gives a list of places to search amongst the data: paths. Removing these (rm -r /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/share/jupyter/labextensions/jupyterlab-empinken) and reloading JupyterLab seemed to do the trick of removing the extension from the running application.

In passing, I note this recent JupyterLab issue — extension-cookiecutter-ts out of sync with doc. I use plugin, and things seem to break if I just change it to extension if I do nothing else? O rmaybe I missed something… From the issue, I also don’t know if this means there’ll be a breaking change at some point?

So, let’s start again from the beginning, and see if the extension tutorial docs give us a minimal, exact and reproducible set of commands that will:

• put the files in new directory in the locations I want them;
• let me rebuild and preview the extension in a JupyterLab environment each time I edit the src/index.ts file, or the style/base.css file, or the schema/plugin.json file.

I’ll then try to remember to retrofit instructions back into earlier posts in this series…

Then, as I’d intended to do for this post before it became an interstitial post, I’ll have a go at creating a prebuilt extension that can be easily distributed in the next post.

But before we start, let’s just review the architecture, at least as I understand it. The core JupyterLab application runs in a browser and can be extended using JupyterLab extensions. An extension is a Javascript package that exports one more plugins that extend JupyterLab’s behaviour. The cookiecutter Python package provides a way of distributing a build environment for a JupyterLab extension. In part, installing the Python package installs front end component files into JupyterLab (somehow!). The cookiecutter package also bundlles various project files that support the compilation of JupyterLab extension Javascript files from source Typescript files. Building the JupyterLab extension creates the files that JupyterLab needs from the source files. When we actually distribute a JupyterLab extension, we can use the Python package machinery to bundle and install the JupyterLab extension Javascript, CSS and other sundry files into JupyterLab.

So let’s go back to the beginning, and to the cookiecutter package. The first question I need to answer is a really simple one: how do I get the cookiecutter to unpack its wares into the top level of the directory into which I want the files to go, rather than in a subdirectory?

Running the cookiecutter, it seems by default as if it won’t overwrite a pre-existing directory. However the -for --overwrite-if-exists flag will allow you to write over the pre-existing directory (the .git files are left untouched). If you don’t want files in the directory that already exists (such as a LICENSE file) overwriting, then also add the -s or--skip-if-file-exists flag. I note that the default license generated by the cookiecutter is the BSD 3-Clause License.

So if I am in my Github directory (assuming I used a repo (directory) name with underscores rather then - separators: the cookie-cutter converts all dashes to underscores…), I can run the cookiecutter with and then specify the repo directory name as the extension name to unpack the files in that directory.

cookiecutter https://github.com/jupyterlab/extension-cookiecutter-ts --overwrite-if-exists--skip-if-file-exists

Or more concisely, cookiecutter https://github.com/jupyterlab/extension-cookiecutter-ts -fs

When prompted for what sort of extension we want, select the default extension type, [1] - frontend; also set settings as y which will presumably seed a settings file for us.

Note that the cookiecutter adds a rather comprehensive .gitignore file to the directory. If you are looking to build and distribute an extension via the repo (see the next post in this series, hopefully!), then you will probably want to remove the dist/ entry from the automatically created .gitignore file so you can commit the disribution wheel to the repository.

The next step is to see if we can set the directory up so that we can easily update JupyterLab. The extension tutorial docs suggest the following approach.

First, we need to install the extension using pip. Installing the package “copies the frontend part of the extension into JupyterLab”. The docs suggest the flags -ve, but from the pip docs I can’t see the -v flag at all. The -e flag (--editable) installs the project in an editable mode (docs). This is like a faux install, in that rather than installing the package, a special .egg-link file is created that allows Python to treat the development repo as if it were the installed Python package. As the package files are updated in the directory, the changes are available as if the package had been updated via pip.

pip3 install -e .

Installing the package creates and downloads a large number of files to the package directory that are used to support the building of the JupyterLab extension package.

When I ran this command, it took a ridiculously long time (several minutes, or long enough to think it must have got stuck or failed somewhow) to install.

The development mode package can supposedly be uninstalled in the normal way:

pip3 uninstall PACKAGENAME

However, files may still be languishing… See the note above about tracking down copies of the extension files that are not removed by the uninstaller.

The docs suggest that “[w]e can run this pip install command again every time we make a change to copy the change into JupyterLab” which suggests that simply making “live” editable changes to the Python package (via the --editable mode) is not enough for the changes to be reflected in the JupyterLab environment. What we additionally need to to do is create a way that allows JupyterLab to make use of any updated JupyterLab package files, akin to the way we give the Python access to the “live” --editable package updates.

It’s not totally clear to me what files we need to “make live” in the Python package, or how and when the Python environment interacts with the cookiecutter generated Python package files.

The following command ensures that as we rebuild JupyterLab extension package files, they become available to JuptyerLab:

jupyter labextension develop --overwrite .

We now have a “live” development environment. As we update the extension package files, we run the following command to (re)buid the extension:

jlpm run build

Refreshing JupyterLab in the browser should now display the updated extension.

The JupyterLab Package Manager (jlpm) is a JupyterLab-provided, locked version of the yarn package manager intended to simplify JuptyerLab development. Handy commands include jlpm install to ensure that required node packages are installed and jlpm run build to build extension files.

In the next post (and hopefully the final post!) in this series, I’ll try to pull together all the pieces to show how to build and disitrubute a pre-built JupyterLab extension.