Fragment: On Failing to Automate the Testing of a Jupyter Notebook Environment Using playwright and Github Actions

One of those days (again) when I have no idea whether what I’m trying to do is possible or not, or whether I’m doing something wrong…

On my local machine, everything works fine:

  • a customised classic notebook Jupyter environment is running in a Docker container on my local machine, with a port mapped on the host’s localhost network;
  • the playwright browser automation framework is installed on the host machine, with a test script that runs tests against the Jupyter environment exposed on the localhost network;
  • the tests I’m running are to login to the notebook server, grab a screenshot, then launch a Jupyter server proxy mapped OpenRefine environment and grab another screenshot.

So far, so simples: I can run tests locally. But it’d be nice to be able to do that via Github Actions. So my understanding of Github Actions is that I should be able to run my notebook container as a service using a service container, and then use the same playwright script as for my local tests.

But it doesn’t seem to work:-( [UPDATE: seems like was image was borked somehow; updated that image (using same tag)… Connection to server works now..]

Here’s my playwright script:

import { test, expect } from '@playwright/test';

// Create test screenshots
// npx playwright test --update-snapshots


// Allow minor difference - eg in the new version
const config: PlaywrightTestConfig = {
  expect: {
    toHaveScreenshot: { maxDiffPixels: 100 },
  },
};
export default config;

test('test', async ({ page }) => {
  // Go to http://localhost:8888/login?next=%2Ftree%3F
  await page.goto('http://localhost:8888/login?next=%2Ftree%3F');
  await expect(page).toHaveScreenshot('notebook-login-page.png');
  // Click text=Password or token: Log in >> input[name="password"]
  await page.locator('text=Password or token: Log in >> input[name="password"]').click();
  // Fill text=Password or token: Log in >> input[name="password"]
  await page.locator('text=Password or token: Log in >> input[name="password"]').fill('letmein');
  // Press Enter
  await page.locator('text=Password or token: Log in >> input[name="password"]').press('Enter');
  await expect(page).toHaveURL('http://localhost:8888/tree?');
  await expect(page).toHaveScreenshot('notebook_homepage-test.png');
  // Click text=New Toggle Dropdown
  await page.locator('text=New Toggle Dropdown').click();
  await expect(page).toHaveScreenshot('notebook_new-test.png');
  // Click a[role="menuitem"]:has-text("OpenRefine")
  const [page1] = await Promise.all([
    page.waitForEvent('popup'),
    page.locator('a[role="menuitem"]:has-text("OpenRefine")').click()
  ]);
  await page1.waitForSelector('#right-panel');
  await expect(page1).toHaveScreenshot('openrefine-homepage.png');
});

And here’s my Github Action script:

name: test

on:
  workflow_dispatch:

# This job installs dependencies and runs the tests
jobs:
  notebook-tests:
    runs-on: ubuntu-latest
    services:
      jupyter-tm351:
        image: ouvocl/vce-tm351-monolith:22j-b2
        ports:
        - 8888:8888
    steps:
    - uses: actions/checkout@v3

    - uses: actions/setup-node@v3
      with:
        node-version: 16
    - name: Install playwright dependencies
      run: |
        npx playwright install-deps
        npx playwright install
        npm install -D @playwright/test
    # Test setup
    - name: Test screenshots
      run: |
        npx playwright test

It’s been so nice not doing any real coding over the last few weeks. I realise again I don’t really have a clue how any of this stuff works, and I don’t really find code tinkering to be enjoyable at all any more.

FWIW, test repo I was using is here. If you spot what I’m doing wrong, please let me know via the comments… FIXED NOW – SEEMS TO WORK

PS hmmm… maybe that container was broken in starting up… not sure how; I have a version running locally?? Maybe my local tag version is out of synch with pushed version somehow? FIXED NOW – SEEMS TO WORK

PPS I just added another action that will generate gold master screenshots and temporarily stash them as action artefacts. Much as the above except the npx playwright test --update-snapshots line to create gold master screenshots (which assumes things are working as they should….) and then a step to save the screenshot image artefacts.

name: generate-gold-master-images

on:
  workflow_dispatch:

# This job installs dependencies, builds the book, and pushes it to `gh-pages`
jobs:
  notebook-tests:
    runs-on: ubuntu-latest
    services:
      jupyter-tm351:
        image: ouvocl/vce-tm351-monolith:22j-b2
        ports:
        - 8888:8888
    steps:
    - uses: actions/checkout@v3

    - uses: actions/setup-node@v3
      with:
        node-version: 16
    - name: Install playwright dependencies
      run: |
        npx playwright install-deps
        npx playwright install
        npm install -D @playwright/test

    # Generate screenshots
    - name: Generate screenshots
      run: |
        npx playwright test --update-snapshots

    - uses: actions/upload-artifact@v3
      with:
        name: test-images
        path: tests/openrefine.spec.ts-snapshots/

Author: Tony Hirst

I'm a Senior Lecturer at The Open University, with an interest in #opendata policy and practice, as well as general web tinkering...

%d bloggers like this: