My JupyterLite Blockers Are Rapidly Being Addressed…

Three months or so ago, in My Personal Blockers to Adopting JupyterLite for Distance and Open Educational Use, described several issues that I saw as blocking when it came to using JupyterLite in an educational context. These included the inability to distribute pre-installed packages as part of the JupyterLite distribution, the inability read and write files programmatically, and the inability to work on files in the local filesystem.

Many of my personal blockers have now been addressed, to such an extent that I think I should probably get round to updating our simple Learn to Code for Data Analysis open educational course to use JupyterLite or RetroLite (I suspect I might actually have to invoke the open license and publish a modified version of my own: the way we institutionally publish content is via a not very flexible Moodle platform and trying to accommodate the JupyterLite environment could be a step too far!).

So: what can we now do with JupyterLite that we couldn’t three months ago that make it more acceptable as an environment we can use to support independent distance and open educational users?

Perhaps the biggest blocker then was the inability to read and write files into the JupyterLite filesystem. This meant that workarounds were required when running packages such as pandas to open and save files from and to JupyterLite file storage. This has now been addressed, so packages such as pandas are now able to read some data files shipped as part of the JupyterLite distribution and also save and read back files into the JupyterLite file system. The JupyterLite file system support also means you can access a list directory contents, for example, from code within a notebook. (Data file type read/writes that aren’t currently supported by pandas, including SQLite file read/writes, are being tracked via this issue.) Indeed, the pandas docs now include a JupyterLite console that allows you to try out pandas code, including file read/writes, directly in the browser.

Another issue was the disconnect between files in the browser and files and the desktop. If you are working with the files in the default JupyterLite file panel, modifications to these files are save into browser storage. You can add and create new files saved to this file area, as well as deleting those files from browser storage. If files are shipped as part of the JupyterLite distribution, deleting those files, for example, after modification, resets the file to the originally distributed version. (A recent issue raises the possibility of how to alert users to an updated version of the notebook on the host repository.)

In many educational settings, however, we may want students to work from copies of notebooks that are stored on their local machine, or perhaps on a USB drive plugged into it. A recent extension that works in Chrome, Edge and Opera browsers — jupyterlab-contrib/jupyterlab-filesystem-access — added the ability to open, edit and save files on the local desktop from the JupyterLite environment. (In use, it’s a little bit fiddly in the way you need to keep granting permission to the browser to save files; but better that than the browser grabbing all sorts of permissions over the local file system without the user’s knowledge.)

In passing, I’m not sure if there’s support or demos yet for mounting files from online drives such as OneDrive, Dropbox or Google Drive, which would provide another useful way of persisting files, albeit one that raises the question of how to handle credentials safely.

When it comes to producing a JupyterLite distribution, which is to say, publishing a JupyterLite environment containing a predefined set of notebooks and a set of preinstalled packages, this has been non-trivial when it comes to adding additional packages to the distribution. The practical consequence of this is that packages need to be explicitly installed from notebooks using micropip or piplite, which adds clutter to notebooks, as well as code that will not run a “traditional”, non-JupyterLite envronment unless you add the appropriate code guards. However (and I have yet to test this), it seems that the new jupyterlite/xeus-python-kernel can be built relatively easily under automation to include additional Python packages that are not part of the default pyodide environment (presumably this requires that platform independent PyPi wheels are available, or custom build scripts that can build an appropriate Emscripten target built wheel?): minimal docs. (I note that this kernel also has the benefit that from time import sleep works!) The ipycanvas docs apparently demo this, so the answer to the question of how to create a JupyterLite distribution with custom additional pre-installed packages is presumably available somwhere in the repo (I think these commits are related: setup, package requirements.) It would be really handy if a minimal self-documenting jupyterlite-custom-package-demo were available to complement jupyterlite/demo with a minimal yet well commented/ narrated example of how to add a custom package to a JupyterLite distribution.

I would have tried a demo of bundling a custom package as a demo for this post, but from reading the docs and skimming the ipycanvas repo, I wasn’t sure what all-and-only the necessary steps are, and I don’t have time to go rat/rabbit-holing atm!

Installing JupyterLite in Chrome as a Browser Web App for Offline Use

If you are working in the Chrome browser on a desktop machine, you can install JupyterLite as a web application. (Firefox discontinued support for progressive web apps in the desktop version of Firefox in 2021.) A benefit of doing this is that you can then used the application in an offline mode, without the need for a web connection.

With a web connection available, if you launch a JupyterLite instance from the JupyterLite homepage, or from the JupyterLite demo page, you will see an option to install the environment as a progressive web application:

In the Chrome browser, you can view your Chrome installed web applications from the chrome://apps URL:

The application will open into its own browser window and can be used with or without a web connection. Files are saved into local browser storage, just as they would be if you were running the application from the original website. This means you can work against the website or the local web app, and the files you have saved into local browser storage will be available in both contexts.

If you do want to work in an offline mode, you need to ensure that all the packages you might want to call on have been “progressively” downloaded and cached by the app. Typically, JupyterLite will only download a Python package when it is first imported. This means that if your notebooks import different packages, you may find in offline use that a required package is not available. To get around this, you should create a single notebook importing all required packages and run that when you do have a network connection to make sure that all the packages you are likely to need are available for offline use.

Python Package Use Across Course Notebooks

Some time ago, I started exploring various ways of analysing the structure of Jupyter notebooks as part of an informal “notebook quality” unproject (innovationOUtside/nb_quality_profile).

Over the last week or two, for want of anything else to do, I’ve been looking at that old repo again and made a start tinkering with some of the old issues, as well as creating some new ones.

One of the things I messed around with today was a simple plot showing how different packages are used across a set of course notebooks. (One of the quality reports lists the packages imported by each notebook, and can flag if any packages are missing from the Python environment in which the tool runs.)

The course lasts ~30 weeks, with a set of notebooks most weeks and the plot shows the notebooks, in order of study, along the x-axis, with the packages listed as they are first enountered on the y-axis.

This chart is essentially a macroscopic view of package usage throughout the course module (and as long term readers will know, I do like a good mascroscope:-).

In passing, I note that I could also add colour and/or shapes or size to identify whether a package is in the Python standard library or whether it is imported from a project installed from PyPi, or highlight whether the package is not available in the Pyhton environment the tool that generates the chart is run in.

A quick glance at it reveals several things:

  • pandas is used heavily throughout (you might guess this is a data related course!), as we can see from the long horizontal run throughout the course;
  • several other packages are used over consecutive notebooks (short, contiguous horizontal runs of dots), suggesting a package that has a particular relevance for the subject matter studied that week;
  • vertical runs show that several new packages are used for the first time in the same notebook, perhaps to acheive a particular task. If the same vertical run appears in other notebooks, perhaps a similar task is being performed in each of those notebooks;
  • there is a steady increase in the number of packages used over the course. If there is a large number of packages introduced for the first time in a single notebook (i.e. a vertical run of dots), this might suggest a difficult notebook for students to work through in terms of new packages and new functions to get their head round;
  • if a package is used only one notebook (which is a little hard to see — I need to explore gridlines in a way that doesn’t overly clutter the graphic), it might be worth visiting that notebook to see if we can simplify it and remove the singleton use of that package, or check the relevance of the topic it relates to to the course overall;
  • if a notebook imports no modules (or has no code cells), it might be worth checking to see whether it really should be a notebook;
  • probably more things…

I’m now wondering what sort of tabular or list style report listing might be useful to identify the notebooks each module appears in, at least for packages that only appear once or twice, or are widely separated in terms of when they are studied.

I also wonder if there are any tools out that I can use to identify package functions used in each notebook to see how they are distributed over the module. (This all rather makes me think of topic analysis!)

In Passing, Noting Python Apps, and Datasette, Running Purely in the Browser

With pyodide now pretty much established, and from what I can tell, possibly with better optimised/lighter builds on the roadmap, I started wondering again about running Python apps purely in the browser.

One way of doing this is to create ipywidget powered applications in a JuptyerLite context (although I don’t think you can “appify” these yet, Voila style?) but that ties you into the overhead of running JupyterLite.

The React in Python with Pyodide post on the official pyodide blog looked like it might provide a way in to this, but over the weekend I noticed an annoucement from Anaconda regarding Python in the Browser using a new framework called PyScript (examples). This framework provides a (growing?) set of custom HTML components that appear to simplify the process of building pyodide Python powered web apps that run purely in the browser.

I also noticed over the weekend that sqlite_utils and datasette now run in a pyodide context, the latter providing the sql api run against an in-memory database (datasette test example).

The client will also return the datasette HTML, so now I wonder: what would be required to be able to run a datasette app in the JuptyerLite/JupyterLab context? The datasette server must be intercepting the local URL calls somehow, but I imagine that the Jupyter server is ignorant of them. So how could datasette “proxy” its URL calls via JupyterLite so that the local links in the datasette app can be resolved? (We surely wouldn’t want to have to make all the links handled button elements?)

UPDATE 5/5/22: It didn’t take Simon long… Datasette now runs as a full web app in the browser under pyodide. Announcement post here: Datasette Lite: a server-side Python web application running in a browser.

So now I’m wondering again… is there a way to “proxy” a Python app so that it can power a web app, running purely in the browser, via Pyodide?

Extracting geoJSON Data From Leaflet Maps with shot-scraper

The shot-scraper package is a crazy piece of command-line magic from Simon Willison that, among other things, lets you grab a web page, and all its attendant Javascript state, into a headless browser, inject a bit of scraper JavaScript into it, and return the result.

For some time, I’ve been wondering how to grab rally route data from the rather wonderful Rally Maps website (the last time I looked, the route info seemed to be baked into the page rather than being pulled in as data from its own easy to grab URI). One approach I looked at was a related technique described in Grabbing Javascript Objects Out of Web Pages And Into Python but IIRC, I’d got a little stuck in getting a clean set of route features out.

Anyway, when reading about Web Scraping via Javascript Runtime Heap Snapshots (again via @simonw), it struck me again that the route info must be in the leaflet map somewhere, so could we get it out? Thinking to search this time for how to export route leaflet I found a simple trick in a Stack Overflow question here that gives the following recipe (I think) for grabbing the route info from a leafelt map (assuming the map object is in the variable map):

shot-scraper javascript https://www.rally-maps.com/Rallye-Festival-Hoznayo-2022 "var collection = {'type':'FeatureCollection','features':[]}; map.eachLayer(function (layer) {if (typeof(layer.toGeoJSON) === 'function') collection.features.push(layer.toGeoJSON())}); collection" > scraped-routes.geojson

Having a quick peek in geojson viewer, and it seems to work (I just need to scrape some of the other data too, such as marker labels etc.)

Presumably, I could automate looking for variables of the leaflet map type in order to make this recipe even easier to use?

Custom Branded Logos for JupyterLab and RetroLab (Jupyter notebook v.7)

Of the three main blockers in terms of look feel that I’ve used as an excuse to not to start thinking about moving course materials over to JupyterLab/RetroLab, I’ve now got hacky extensions for styling notebooks, empinken style, and enable cell run status indicators. The next one is purely cosmetic – adding custom logos.

The proper way to do this (?!) is probably to use a custom theme. See https://github.com/g2nb/jupyterlab-theme/ for an example of adding custom logos to to a custom theme.

Whilst it doesn’t appear that there is straightforward “parameter configurable” way of doing this, and there is zero help on the various forums and issues trackers for anyone who does want to achieve this (because believe it or not, the setting does matter sometimes, and learners particularly, often benefit from thinking they’re in a “safe space” that is suggested by branded environments), I finally had a poke around for some hacky ways of doing this.

It does, of course, require all the pain of building an extension, but we don’t need much more than some simple CSS and some simple JS, so we can get away with using the the JupyterLab Javascript cookiecutter extension.

To open the cookiecutter files into a pre-existing directory, such as a one created by cloing a remote Gihub repo onto your desktop, run the command:

cookiecutter https://github.com/jupyterlab/extension-cookiecutter-js -fs

You can then set up the environment:

cd my_extension/`

python -m pip install .

And do a test build:

jlpm run build && python3 -m build

You can then install the resulting package:

pip3 install ./dist/my_extension-0.1.0-py3-none-any.whl

To customise the logos, in the ./style/base.css file, we can hide the default JupyterLab logo and add our own:

#jp-MainLogo {
    background-image: url(./images/OU-logo-36x28.png);
    background-repeat: no-repeat;
}

#jp-MainLogo > svg {
    visibility: hidden;
}

#jp-RetroLogo {
    background-image: url(./images/OU-logo-53x42.png);
    background-repeat: no-repeat;
}

#jp-RetroLogo > svg {
    visibility: hidden;
}

The images should be place in a new ./style/images/ folder.

To have a go at hacking the favicon (which works on a “full server, ish, but not in JupyterLite?), we need some simple Javascript in ./style/index.js:

import './base.css';

// Via: https://discourse.jupyter.org/t/changing-favicon-with-notebook-extension/2721

let head = document.head || document.getElementsByTagName('head')[0];

let link = document.createElement('link')
let oldLink = document.getElementsByClassName('favicon');
link.rel = 'icon';
link.type = 'image/x-icon';
link.href = 'https://www.open.ac.uk/oudigital/v4/external/img/favicons/favicon.png';
if (oldLink) {
    link.classList = oldLink[0].classList;
    head.removeChild(oldLink[0]);
}
head.appendChild(link);

I’m not sure how to reference a local, statically packed favicon shipped with the extension, so for now I pull in a remote one.

Rebuild the extension, and reinstall it:

jlpm run build && python3 -m build && pip3 install --upgrade./dist/my_extension-0.1.0-py3-none-any.whl

To make it easier to distribute, I remove the dist/ element from the .gitignore file and push everything to a repo.

The following Github action acan be manually triggered to build a JupyterLite enviornment pushed to Github Pages (in your Github repo, you need to go to Settings > Pages and select the gh-pages branch as the target for the site.

I include the custom extension in the JupyerLite build via a requirements-jupyterlite.txt file which includes the following:

./dist/my_extension-0.1.0-py3-none-any.whl

name: JupyterLite Build and Deploy

on:
  release:
    types: [published]
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Setup Python
        uses: actions/setup-python@v2
        with:
          python-version: 3.8
      - name: Install the dependencies
        run: |
          python -m pip install -r requirements-jupyterlite.txt
      - name: Build the JupyterLite site
        run: |
          cp README.md content
          jupyter lite build --contents content
      - name: Upload (dist)
        uses: actions/upload-artifact@v2
        with:
          name: jupyterlite-demo-dist-${{ github.run_number }}
          path: ./_output

  deploy:
    if: github.ref == 'refs/heads/main'
    needs: [build]
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2.3.1
      - uses: actions/download-artifact@v2
        with:
          name: jupyterlite-demo-dist-${{ github.run_number }}
          path: ./dist
      - name: Deploy
        uses: JamesIves/github-pages-deploy-action@4.1.3
        with:
          branch: gh-pages
          folder: dist

A demo of the extension can then be tried purely withing the browser via JupyterLite.

Repo is here: https://github.com/innovationOUtside/jupyterlab_ou_brand_extension

Demo is here: http://innovationoutside.github.io/jupyterlab_ou_brand_extension/

Files changed compared to cookiecutter generated pages here: https://github.com/innovationOUtside/jupyterlab_ou_brand_extension/commit/688863cb79557920b1950a9a9b0331ccedcdac39

Tinkering Towards a JupyterLab Cell Status Indicator Extension

Having managed to build one extension (a JupyterLab version of empinken), I thought I’d have ago an another, a JuptyerLab version of the classic notebook innovationoutside/nb_cell_execution_status extension which colours a code cell run indication according to whether the cell is running (or queued for execution), has run successfully, or ran with an error.

You can try it, via JuptyerLite (howto), here: innovationoutside/nb_cell_execution_status

The code in its minimal form is really simple, although that isn’t to say it didn’t take me a disproportionate amount of time to find the methods I guessed might help implement the feature and then try to get the actual extension compiled and working. I still really, really, really struggle with the whole JupyterLab thing. The more I use it, the more I don’t want to, if that were possible…!;-)

As a starting point, I used the JupyterLab Typescript extension cookiecutter (e.g. as per here). I needed to import one additional package, import { NotebookActions } from '@jupyterlab/notebook';, which also need adding in the dependencies of the package.json file as "@jupyterlab/notebook": "^3.3.4".

I would have had a nice git diff from a single check-in after the “base” cookiecutter check-in, showing just what was required. But the reality is that there are dozens of commits in the repo showing the numerous craptastic and ignorant end-user-dev attempts I kept making to try to get things to work. Have I mentioned before how much I dislike eveything about JupyterLab, and its development process?!;-)

Essentially, all the extension requires that you subscribe to signals that fire when a code cell has been queued for execution, or when it completes execution.

Specifically, when one or more code cells are selected and run, a NotebookActions.executionScheduled signal is issued. We can subscribe to this signal, remove any classes previously added to the cell DOM in the notebook HTML that relate to the status of previous successful or unsuccessful cell execution (cell.inputArea.promptNode.classList.remove()), and add a new class (cell.inputArea.promptNode.classList.add()) to indicate a scheduled status.

When the cell has completed execution, a NotebookActions.executed signal raised. This also carries with it information regarding whether the cell executed successfully or not; we can remove the scheduled-or-executing class from the cell DOM in the notebook HTML, and then use this cell executed status to set a class identifying the executed status.

Note, there are also various notebook API methods, but I’m not really sure how to tap into those appropriately….

import {
  JupyterFrontEnd,
  JupyterFrontEndPlugin
} from '@jupyterlab/application';

//https://jupyterlab.readthedocs.io/en/3.1.x/api/classes/notebook.notebookactions-1.html
import { NotebookActions } from '@jupyterlab/notebook';
//import { IKernelConnection } from '@jupyterlab/services/Kernel IKernelConnection';
import { ISettingRegistry } from '@jupyterlab/settingregistry';

/**
 * Initialization data for the jupyterlab_cell_status extension.
 */
const plugin: JupyterFrontEndPlugin<void> = {
  activate,
  id: 'jupyterlab_cell_status_extension:plugin',
  autoStart: true,
  optional: [ISettingRegistry],//, IKernelConnection]
};

function activate (
  app: JupyterFrontEnd, settingRegistry: ISettingRegistry | null): void {
    console.log("jupyterlab_cell_status_extension:plugin activating...");
    /*
    // We can use settings to set the status colours
    if (settingRegistry) {
      settingRegistry
        .load(plugin.id)
        .then(settings => {
          console.log("jupyterlab_cell_status_extension:plugin: loading settings...");
          const root = document.documentElement;
          const updateSettings = (): void => {
            const queue_color = settings.get('status_queue').composite as string;
            const success_color = settings.get('status_success').composite as string;
            const error_color = settings.get('status_error').composite as string;
            
            root.style.setProperty('--jp-cell-status-queue', queue_color);
            root.style.setProperty('--jp-cell-status-success', success_color);
            root.style.setProperty('--jp-cell-status-error', error_color);
            
          };
          updateSettings();
          console.log("jupyterlab_cell_status_extension:plugin: loaded settings...");
          // We can auto update the color
          settings.changed.connect(updateSettings);
        })
        .catch(reason => {
          console.error('Failed to load settings for jupyterlab_cell_status_extension.', reason);
        });
    }
    */

    /*
    // This was a start at sketching whether I could
    // reset things if the kernel was restarted.
    // Didn't get very far though?
    IKernelConnection.connectionStatusChanged.connect((kernel, conn_stat) => {

    console.log("KERNEL ****"+conn_stat)
    });
    */
    NotebookActions.executed.connect((_, args) => {
      // The following construction seems to say 
      // something akin to: const cell = args["cell"]
      const { cell } = args;
      const { success } = args;
      // If we have a code cell, update the status
      if (cell.model.type == 'code') {
        if (success)
          cell.inputArea.promptNode.classList.add("executed-success");
        else
          cell.inputArea.promptNode.classList.add("executed-error");
          cell.inputArea.promptNode.classList.remove("scheduled");
      }
    });

    NotebookActions.executionScheduled.connect((_, args) => {
      const { cell } = args;
      // If we have a code cell
      // set the status class to "scheduled"
      // and remove the other classes
      if (cell.model.type == 'code') {
        cell.inputArea.promptNode.classList.remove("executed-success");
        cell.inputArea.promptNode.classList.remove("executed-error");
        cell.inputArea.promptNode.classList.add("scheduled");
      }
    });  

    console.log("jupyterlab_cell_status_extension:plugin activated...");
  }

export default plugin;

To build a Python wheel for an installable version of the extension, I use the build command:

jlpm clean && jlpm run build && python3 -m build

When cell is running, or queued, by default the cell status area now highlights a blie. Successful cell execution is denote green, and unsuccessful cell execution is red.

Another disproportionate amount of time was spent trying to figure out why the settings weren’t being loaded. I tracked this down to a mismatch between the extension id defined in the index.ts file and the name in the schema/pugin.json file. It still didn’t work, but a repeated “comment out all the settings code, rebuild, commit to Github, republish the JupyerLite distro” process, uncommenting a line at a time revealed: no errors, and it had started working. So I have no idea what went wrong.

If the itereated process sounds faintly stupid, and it took maybe 5 minutes each iteration (two to build the extension, two to republish the JupyterLite distribution). That’s partly because I was suspicious that my local build process wasn’t looking on the correct path. (I know that JupyterLite builds don’t respect paths on my local machine but work fine when run on Github via a Github Action.)

Anyway, the settings seem to work now to allow you to set the status indicator colours, though after a change you need to close a notebook then open it again to see the change reflected.

If you close a notebook and open it again, whether or not you stop the kernel, then the status colouring is lost. I did start to wonder whether I should persist the status as notebook metadata (eg how to set tags can be found here) to allow a notebook opened against a still running kernel to show the cell status (classing the cell based on cell metadata), and then managing persistent tags and classes based on cell execution status and notebook kernel signals (eg removing all status indications if the kernel is stopped. I’m not sure how to access the notebook/cells from inside a IKernelConnection.connectionStatusChanged connected handler though? (My thinking is the cell status indicators should give you a reasonable indication of whether a particular cell has contributed to the kernel state. I wonder if there’s any code in nbsafety-project/nbsafety that might help me think this through a bit more?)

I guess the next step is to see if I can also add the audio indicators that I’d added to the classic notebook extension too…

Storytelling…

Having been going to storytelling festivals and events for 20+ years, and finally joining a stroytelling group in the summer before the first lockdown, I started telling tales at folk clubs on the island at the end of last Autumn.

last month, along with a couple of other Island Stroytellers, Sue and Holly, we did a charity gig as Three Island Storytellers:

Last night, I had a feature slot at the Waverley folk night in Carisbrooke, and got to do my first hand written set list…

More dates will hopefully be announced soon… ;-)

Imagining “JupyterLab Magic Tags”: Tag Based Code Cell Execution Modifiers in JupyterLab

In Pondering — Admonition Tags for JupyterLab Markdown Cells? I finally for round to describing a feature I’d like to see in JupyterLab, specifically the ability to tweak the execution of markdown cells based on cell tags. I’ve already managed to tweak the style of cells based on cell tags (a missing feature which was one of the blockers I have had on adopting JupyerLab for several years) so this represents the next step.

In this post, I thought I’d make a note of another tag based feature I’ve wanted for a couple a years: the ability to modify cell execution based on cell tag metadata.

This might sound a bit odd, but it’s really just another take on cell magics, at least as far as the user might be concerned.

I make use of IPython magics block cell heavily to modify the behaviour of code cells, often in context of using the magic to invoke a Python function that will execute a function over content contained in the cell. This essentially allos you to define a sort of custom code cell type. A good example is provided by the catherinedevlin/ipython-sql extension which installs a sql magic that lets you connect to a database and then run SQL queries directly from a magicked code cell:

This is fine insofar as it goes, but I’d quite like to be able to abstract that a bit further into the UI, and use a cell tag, rather than a magic, to modify the cell behaviour.

Tagging a cell with ipython-magic-sql would have the same effect as adding %%sql to the cell. (Yes, I know there may be an issue with handling magic parameters; that’s left as a on open question for now…) A tagstyle could automatically render the cell in a way that highlights it has been magicked, making it clear in the UI that modified behaviout on the cell is to be expected.

In terms of handling the modified execution, one approach I can imagine would be to define custom code cell handler, similar to the custom markdown cell handler defined in the agoose77/jupyterlab-imarkdown extension (see also this related (as yet unanswered) query about Creating a Custom JupyterLab Notebook Markdown Cell Renderer; maybe I should try Stack Overflow again…).

A jupyterlab-magic-tag-template extension might provide one way of helping end-users create their own magic tags. For a magic with out paramteers, we might image a simple definition file:

tag: SQL-MAGIC-TAG
block_magic: %%sql
package: ipython-sql

Note that in the case of the ipython-sql package, we would elsewhere have had to define the sql connection. Alternatively, this could be pre-loaded and enabled by the extension, possibly even in a separate kernel. If the extension supported user configuration settings, that might also provide a route for setting global magic parameters and having them added to the magic command whenever it is called.

Tagging a cell with SQL-MAGIC-TAG would then:

  • cause the cell to have slightly different style applied; this might be as simple as a coloured border, which could have a default value that is over-riden via a config setting;
  • when a cell is executed, the contents of the cell are prefixed by the magic invocation;
  • cell outputs are rendered as per normal.

It’s not hard to imagine tags also being used to support an element of polyglot language execution behaviour; for example, magic can be used to modify a cell to allow the execution of R code in an IPython notebook. But why not just let the user use a cell tag to invoke that behaviour?

PS I’m happy for folk to tell me why they think this is a ridiculous idea..! Feel free to post in the comments…

Demoing JupyterLab Extensions from an Extension Repo Using Github Pages and JupyterLite

Following on from Notes on the JupyterLab Notebook HTML DOM Model, Part 9: Building and Distributing a Pre-Built Extension, in which the build process generated a platform independent Python package wheel, jupyterlab_empinken_extension-0.1.1-py3-none-any.whl, I pushed it to pypi using a simple pypi-publish Github Action (you need to set a PYPI_PASSWORD API key (minted in your PyPi account from Account Settings (scroll down..) API tokens ) in the repo Secrets > Actions > Repository secrets settings in order to push packages to PyPi.

I generally set a PyPi token with permissions on all repos, push the package for the first time using that token, then mint a repo specific token and update the repo settings; I keep thinking there must be a more sensible way?!)

name: Publish Python distributions to PyPI

on:
  release:
    types: [published]
  workflow_dispatch:

jobs:
  build-n-publish:
    name: Build and publish Python distribution to PyPI
    runs-on: ubuntu-18.04
    
    steps:
    - uses: actions/checkout@master
    - name: Set up Python 3.8
      uses: actions/setup-python@v1
      with:
        python-version: 3.8
        
    - name: Build a binary wheel and a source tarball
      run: |
        python3 -m pip install --user --upgrade setuptools wheel
        python3 -m pip install build
        python3 -m build
    - name: Publish distribution to PyPI
      # if: startsWith(github.event.ref, 'refs/tags')
      uses: pypa/gh-action-pypi-publish@master
      with:
        password: ${{ secrets.pypi_password }}

I also use a jupyterlite-pages action to push a JupyterLite distribution demoing the extension to Github Pages:

name: JupyterLite Build and Deploy

on:
  release:
    types: [published]
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Setup Python
        uses: actions/setup-python@v2
        with:
          python-version: 3.8
      - name: Install the dependencies
        run: |
          python -m pip install -r requirements-jupyterlite.txt
      - name: Build the JupyterLite site
        run: |
          cp README.md content
          jupyter lite build --contents content
      - name: Upload (dist)
        uses: actions/upload-artifact@v2
        with:
          name: jupyterlite-demo-dist-${{ github.run_number }}
          path: ./_output

  deploy:
    if: github.ref == 'refs/heads/main'
    needs: [build]
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2.3.1
      - uses: actions/download-artifact@v2
        with:
          name: jupyterlite-demo-dist-${{ github.run_number }}
          path: ./dist
      - name: Deploy
        uses: JamesIves/github-pages-deploy-action@4.1.3
        with:
          branch: gh-pages
          folder: dist

To use Github Pages (a website published from a specified repo branch and directory), you need to ensure that they are enabled from your repo Settings:

My actions are triggered by a release or (more usually) manually:

I really should make sure that the JupyterLite build follows on from a package release or an update to the content/ directory.

You might notice the JupyterLite distribution action references a requirements file (requirements-jupyterlite.txt). This file needs to include the JupyterLab and JuptyerLite packages, as well as any other (prebuilt) packages you want to install:

# Base install
ipywidgets>=7.7,<8
jupyterlab~=3.3.0
jupyterlite==0.1.0b4

# Extension package we want to demo
jupyterlab-empinken-extension

If the package you want to demo in JupyterLite is not available from PyPi, I wonder, can you specfy the wheel URL in the requirements file (eg https://raw.githubusercontent.com/innovationOUtside/jupyterlab_empinken_extension/main/dist/jupyterlab_empinken_extension-0.1.1-py3-none-any.whl? If not, add a line to the jupyterlite distribution action along the lines of pip3 install https://raw.githubusercontent.com/innovationOUtside/jupyterlab_empinken_extension/main/dist/jupyterlab_empinken_extension-0.1.1-py3-none-any.whl).

Any files you want included as part of the distribution, such as a demo notebook, should be placed in the repo content/ directory.

When the site is published, (and you might need to check in the repo settings that you have enabled Pages appropriately), you should be able to test your extension running in JupyterLab in your browser. For example, my innovationOUtside/jupyterlab_empinken_extension demo is at https://innovationoutside.github.io/jupyterlab_empinken_extension/

Pondering — Admonition Tags for JupyterLab Markdown Cells?

In Fragments – Previewing Richly Formatted Jupyter Book Style Content Authored Using MyST-md I noted the JupyterLab-MyST extension which can render MyST admonition blocks included in a Markdown cell.

At the time, I wondered how easy it would be to used a cell tag on a markdown block instead of cluttering the markdown with the MyST admonition block code fences.

However, in rediscovering the agoose77/jupyterlab-imarkdown extension, which replicates the classic notebook Python Markdown extension that lets you embed and render Python variables within a markdown cell, I wonder how easy it would be to crib that extension to create a custom markdown cell renderer that could either:

  • parse the cell tags;
  • wrap the markdown cell content in an admonition block code fence;
  • continue with the JupyterLab-MyST rendering;

Or:

  • parse the cell tags;
  • wrap the markdown cell content in some admonition block HTML (resembling the output from the MyST extension, for example);
  • parse the HTML wrapped markdown content in the normal way;

Or:

  • parse the cell tags;
  • wrap the rendered cell output in appropriate admonition block HTML;
  • update the rendered cell output with the updated output.

In terms of getting a feel for working with the notebooks, it might be instructive to see how each of these appproaches could actually be implemented.

Poking around the JupyterLab-MyST extension, it seems as if it makes use of the @agoose77/jupyterlab-markup extension (import { simpleMarkdownItPlugin } from '@agoose77/jupyterlab-markup';) by extending the markdown-it pipeline it uses? One question that remains is how we get access to the cell metadata so that we can use that as the basis for wrapping the original cell markdown content appropriately? However, it seems that this isn’t possible in the jupyterlab-markup extension, which only has access to the original markdown cell content. Instead, we’d need a custom cell renderer such as the one provided by the jupyterlab-imarkdown extension (for a related query on how to reate custom markdown cell renderers, see Creating a Custom JupyterLab Notebook Markdown Cell Renderer).

For a discussion of the jupyterlab-markup plugin mechanism, see How do you add plugins?.

PS Ooh.. looking in https://github.com/agoose77/jupyterlab-markup/blob/main/src/builtins I notice that there are some other neat integrations… like mermaid diagram rendering in markdown cells:

The extension also looks to use some WASM powered support for rendering svgbob ascii diagrams to SVG: