Extending the folium Magic…

A couple of days ago I posted about some IPython magic for embedding interactive maps in Jupyter notebooks.

I had a bit more of a play yesterday, and then into the night, and started to add support in for other things. For example, you can now build up a map using several pieces of magic, for example by assigning a magic generated map to a variable and then passing that map in as the basemap to another piece of magic.

If the output of the previously run code cell is a folium map, the magic will use that map as the starting map by default – there is a switch (-b None), or some alternative magic (%folium_new_map) that forces a fresh map if you want to start from scratch.

The magic now also has some support for trying to guess where to centre a map, as well as logic that tries to guess at what columns you might want to use when generated a choropleth map from some data and a geojson file. This is best exemplified by some helper magic:

The same routines are used to try to guess column names if you omit them when trying to plot a choropleth. For example, here we only pass in a reference to the data file, the geojson file, and the numeric data column to colour the map. The columns used to plot the boundaries are guessed at.

This is likely to be very ropey – I’ve only tested it with a single pair of data/geosjon files, but it might work for limited general cases…

You can play with it here: Binder

Code is…

…something that everyone is supposed to learn, apparently, but I’m not sure why; and you don’t have to if you’re an adult learner, compared to someone in school?

So here are a couple of reasons why I think knowledge of how to use code is useful.

Code is a means for telling computers, rather than computer users, what to do

Writing instructional material for students that require them to use computer application is a real pain. You have to write laborious From the file menu, select…, in the popup wizard, check the… instructions, or use screenshots (which have to look exactly the same as what the learner is likely to see or they can get worried) or screencasts (which have to look exactly the same as what the learner is likely to see or they can get worried).

The problem is, of course, you don’t want the learner to really do any of that. You want them to be apply to apply a series of transformations to some sort of resource, such as a file.

On the other hand, you can write a set of instructions for what you want the computer to do to the resource, and give that to the learner. The learner can use that medium for getting the computer to transform the resource rather more literally…

Code is a tool for building tools

Code lets you build your own tools, or build your own tools on top of your own tools.

One huge class of tools you can build are tools that automate something else, either by doing a repetitive task multiple times over on your behalf, or by abstracting a lengthy set of instructions into a single instruction with some “settings” passed as parameters. And yes, you do know what parameters are: in the oven for thirty minutes at gas mark 5, which is to say, in “psuedo-code”, oven(gas_mark=5, time=30).

Here’s one tool I built recently – some magic that makes it easy to embed maps in Jupyter notebooks.

Code is a tool for extending other tools

If some you have access to the code that describes a tool, you can extend it, tune it, or customise it with your own code.

For example, here’s that  previous magic extended.

Code is something that can be reused in, or by, other bits of code

If you’ve written some code to perform a particular task in order to achieve something in one tool, you don’t need to write that code again if you want to perform the same task in another tool. You can just reuse the code.

Lots of code is “algorithmic boilerplate”, in the sense that it implements a recipe for doing something. Oftentimes, you may want to reuse the “algorithmic boilerplate” that someone else has got working in their application / tool, and reuse it, perhaps with slight remodification (swapping dataframes for datafiles, for example).

Disclaimer

In my student days, many nights were spent between the computer room, and the Student Union playing bridge. In the bridge games, the bidding was ad hoc because no-one could ever remember any of  the bidding systems you’re “supposed to use” properly… Obviously, it wasn’t proper bridge, but it passed the time amicably…

So unlike folk who (think code should be taught properly and don’t think to use code themselves for daily tasks), I have more pragmatic approach and ((think my code only has to work enough to make it worthwhile using it) or (because it’s an applied recreational activity like doing a crossword)).

Embedding folium Maps In Jupyter Notebooks Using IPython Magic

Whilst trying to show how interactive maps can be embedded in a Jupyter notebook, one of the comments I keep backing back is that “It’s too hard” because you have to write two or three lines of code.

So I’ve tried to simplify things by wrapping the two or three lines of code up as IPython magic, which means you can use a one liner.

The code can be found in this Githib repo: psychemedia/ipython_magic_folium.

To install:

pip install git+https://github.com/psychemedia/ipython_magic_folium.git

To load the magic in a Jupyter notebook:

%load_ext folium_magic

Then call as: %folium_map

The magic currently only works as line magic.

See the folium_magic_demo.ipynb notebook for examples, or run using Binder.

Binder

Display Map

  • -l, --latlong: latitude and longitude values, comma separated. If no value is provided a default location will be used;
  • -z, --zoom (default=10): set initial zoom level;

Add markers

  • -m, --marker: add a single marker, passed as a comma separated string with no spaces after commas; eg 52.0250,-0.7084,"My marker"

-M,--markers: add multiple markers from a Python variable; pass in the name of a variable that refers to:
– a single dict, such as markers={'lat':52.0250, 'lng':-0.7084,'popup':'Open University, Walton Hall'}
– a single ordered list, such as markers=[52.0250, -0.7084,'Open University, Walton Hall']
– a list of dicts, such as markers=[{'lat':52.0250, 'lng':-0.7084,'popup':'Open University, Walton Hall'},{'lat':52.0, 'lng':-0.70,'popup':'Open University, Walton Hall'}]
– a list of ordered lists, such as markers=[[52.0250, -0.7084,'Open University, Walton Hall'], [52., -0.7,'Open University, Walton Hall']]

If no -l co-ordinate is set to centre the map, the co-ordinates of the single marker, or the mid-point of the multiple markers, are used instead.

Display `geojson` file

  • -g, --geojson: path to a geoJSON file

If no -l co-ordinate is set to centre the map, the mid-point of the geojson boundary is used instead.

Display a Choropleth Map

A choropoleth map is displayed if enough information is provided to disaplay one.

  • -g/ --geojson: path to a geoJSON file
  • -d, --data: the data source, either in the form of a pandas dataframe, or the path to a csv data file
  • -c, --columns: comma separated (no space after comma) column names from the data source that specify: column to match geojson key,column containing values to display
  • -k, --key: key in geojson file to match areas with data values in data file;
  • optional:
  • -p, --palette: default='PuBuGn'
  • -o, --opacity: default=0.7

For example, load data from a pandas dataframe:

Or load from a data file:

 

This is still a bit fiddly because it requires you to add lat/longs for the base map and/or markers. But this could probably be addressed (ha!) by building in a geocoder, if I can find one that’s reliable and doesn’t require a key.

Scratch Materials – Using Blockly Style Resources in Jupyter Notebooks

One of the practical issues associated with using the Scratch desktop application (or it’s OU fork, OUBuild) for teaching programming is that runs on the desktop (or perhaps a tablet? It’s an Adobe Air app which I think runs on iOS?). This means that the instructional material is likely to be separated from the application, either as print or as screen based instructional material.

OUBuild

If delivered via the same screen as the application, there can be a screen real estate problem when trying to display both the instructional material and the application.

In OU Build, there can also be issues if you want to have two projects open at the same time, for example to compare a provided solution with your own solution, or to look at an earlier project as you create a new one. The solution is to provide two copies of the application, each running its own project.

Creating instructional materials can also be tricky, requiring the capturing of screenshots from the application and then inserting them in the materials, along with the attendant risk when it comes to updating the materials that screenshots as captured in the course materials may drift from the actuality of the views in the application.

So here are a couple of ways that we might be able to integrate Scratch like activities and guidance into instructional materials.

Calysto/Metakernel Jigsaw Extension for Jupyter Notebooks

The Calysto/Metakernel* Jigsaw extension for Jupyter notebooks wraps the Google Blockly package for use in a Jupyter notebook.

Program code is saved as an XML file, which means you can save and embed multiple copies of the editor within the same Jupyter notebook. This means an example programme can be provided in one embed, and the learner can build up the programme themselves in another, all in the same page.

The code cell input (the bit that contains the %jigsaw line) can be hidden using the notebook Hide Input Cell extension so only the widget is displayed.

The use of the editor is a bit tricky – it’s easy to accidentally zoom in and out, and I’m guessing not very accessible, but it’s great as a scratchpad, and perhaps as an instructional material authoring environment?

Live example on Binderhub

For more examples, see the original Jigsaw demo video playlist.

For creating instructional materials, we should be able to embed multiple steps of a programme in separate cells, hiding the code input cell (that is, the %jigsaw line) and then export or print off the notebook view.

LaTeX Scratch Package

The LaTeX Scratch package provides a way of embedding Blockly style blocks in a document through simple LaTeX script.

Using a suitable magic we can easily add scripts to the document (the code itself could be hidden using the notebook Hide Code Cell Input extension.

(Once again, the code cell input (the cell that contains the lines of LaTeX code) can be hidden using the notebook Hide Input Cell extension so only the rendered blocks are displayed.)

We can also create scripts in strings and then render those using line magic.

Live example on Binderhub

One thing that might be quite interesting is a parser that can take the XML generated from the Jigsaw extension and generate LaTeX script from it, as well as generating a Jigsaw XML file from the LaTeX script?

Historical Context

The Scratch rebuild – OU Build – used in the OU’s new level 1 introductory computing course is a cross platform, Adobe Air application. I’d originally argued that if the earlier taken decision to use a blocks style environment was irreversible, the browser based BlockPy (review and code) application might be a more interesting choice: the application was browser based, allowed users to toggle between blocks and Python code views, displayed Python errors messages in a simplified form, and used a data analysis, rather than animation, context, which meant we could also start to develop data handling skills.

BlockPy

One argument levelled against adopting BlockPy was that it looked to be a one man band in terms of support, rather than the established Scratch community. I’m not sure how much we benefit from, or are benefit to, the Scratch community though? If OU Build is a fork,  we may or may not be able to benefit from any future support updates to the Scratch codebase directly. I don’t think we commit back?

If the inability to render animations had also been a blocker, adding an animation canvas as well as the charting canvas would have been a possibility? (My actual preference was that we should do a bigger project and look to turn BlockPy into a Jupyter client.)

Another approach that is perhaps more interesting from a “killing two birds with one stone” perspective is to teach elementary programming and machine learning principles at the same time. For example, using something like Dale Lane’s excellent Scratch driven Machine Learning for Kids resources.

PS the context coda is not intended to upset, besmirch or provoke anyone involved with OUBuild. It’s self-contempt / self-critical, directed at myself for not managing to engage/advocate my position/vision in a more articulate or compelling way.

Fragment – Breaking Enigma Was Only Part of the Story…

Reading Who, me? They warned you about me? on ethics associated with developing new technologies, this quote jumped out at me: [m]y claim is that putting an invention into a public space inevitably makes that invention safer (Mike Loukides, December 7, 2017).

In recent years, the UK Government had several goes at passing bills that referred to the collection of communications data, the who, where, when and how of a communication but not its content .

[N]ot its content.

Many folk are familiar with stories of the World War Two codebreakers, the boffins, Alan Turing among them, who cracked the German enigma code. How they helped win the war by reading the content of enemy communications.

So given it was the content wot won it, we, cast as “enemies”, might conclude that the protecting the content is key. That the communications data is less revealing.

But that’s not totally true. Other important intelligence can be derived from traffic analysis, looking at communications between actors even if you don’t know the content of the messages.

If I know that X sent a message to Y  and Z five minutes before they committed a robbery on several separate connections, I might suspect that X  knew Y and Z, and was implicated in the crime, even if I didn’t know about the content of the messages.

Location data can also be used to draw similar inferences. For example, the Bloomberg article Mobile-Phone Case at U.S. Supreme Court to Test Privacy Protections describes a recent US Supreme Court case reviewing an appeal from a convicted armed robber who was in part convicted on the basis of evidence that data obtained from [his] wireless carriers to show he was within a half-mile to two miles of the location of four  … robberies when they occurred.

So what has this to do with “putting an invention into a public space”? Perhaps if the stories about how military intelligence made and makes use of traffic analysis and location analysis, and not just the content of decrypted messages, the collection of such data may not seem so innocuous…

When invention takes place in public, we (the public) know that it exists. We can become aware of the risks. Mike Loukides, December 7, 2017.

Just sayin’…

OERs in Practice: Repurposing Jupyter Notebooks as Presentations

Over coffee following a maps SIG meeting last week, fellow Jupyter notebooks enthusiast Phil Wheeler wondered about the extent to which tutors / Associate Lecturers might be able to take course materials delivered as notebooks and deliver them in tutorials as slideshows.

The RISE notebook extension allows notebooks to be presented using the reveal.js HTML presentation framework. The slides essentially provide an alternative client to the Jupyter notebook and can be autolaunched using Binder. (I’m not sure if the autostart slideshow can also be configured to Run All cells before it starts?)

To see how this might work, I marked up one of the notebooks in my showntell/maths demo setting some of the cells to appear as slides in a presentation based on the notebook.

I also used the Hide Input Jupyter extension to hide code cell inputs so that the code used to generate an output image or interactive could be hidden from the actual presentation.

Get into the slideshow editor mode from the notebook View menu, select Cell Toolbar and then Slideshow. Reset the notebook disable using View > Cell Toolbar > None.

To run the presentation with code cell outputs pre-rendered, you first need to run all the cells. From the notebook Cell menu select Run All to execute all the cells.  You can now enter the slideshow using the Enter/Exit RISE Slideshow toolbar buttom (it looks like a bar chart). Exit the presentation using the cross in the topleft of the slideshow display.

[Live demo on Binderhub]

PS building on the idea of using mapping notebook cells into a reveal.js tagged display, I wonder if we could do something similar using a scrollytelling framework such as scrollama, scrollstory or idyll?

OERs in Practice: Re-use With Modification

Over the years, I’ve never really got my head round what other people mean by OERs (Opern Educational Resources) in terms of how they might be used.

From my own perspective, wholesale reuse (“macro reuse”) of a course isn’t relevant to me. When tasked with writing an OU unit, if I just point to a CC licensed course somewhere else and say “use that”, I suspect it won’t go down well.

I may want to quote a chunk a material, but I can do that with books anyway. Or I may want to reuse an activity, and then depending on how much rework or modification is applied, I may reference the original or not.

Software reuse is an another possibility, linking out to or embedding a third party application, but that tends to fall under the banner of openly licensed software reuse as much as OER reuse. Sometimes the embed may be branded; sometimes it may be possible to remove the branding (depending on how the asset is created, and the license terms), sometimes the resource might be a purely white label resource that can be rebranded.

Videos and audio clips are another class of resource that I have reused, partly because they are harder to produce. Video clips tend to come in various forms: on the one hand, things like lectures retain an association with the originator (a lecture given by Professor X of university Y is very obviously associated with Professor X and university Y); on the other hand, an animations, like software embeds, might come in a branded form, white labelled, or branded as distributed but white label licensed so you can remove/rebrand if you want to put the effort in.

Images are also handy things to be able to reuse, again because they can be hard to produce in at least two senses: firstly, coming up with the visual or graphical idea, i.e. how to depict something in a way that supports teaching or learning; secondly, actually producing the finished artwork. One widely used form of image reuse in the OU is the “redrawing” of an image originally produced elsewhere. This represents a reuse, or re-presentation, of an idea. In a sense, the image is treated as a sketch that is then redrawn.

This level of “micro reuse” of a resource, rather than the “macro reuse” of a course, is not something that was invented by OERs – academics have always incorporated and referenced words and pictures created by others – but it can make reuse easier by simplifying the permissions pathway (i.e. simplifying what otherwise might be a laborious copyright clearance process).

One of the other ways of making use of “micro” resources is to reuse them with modification.

If I share a text with you as a JPG of a PDF document, it can be quite hard for you to grab the text and elide a chunk of it (i.e. remove a chunk of it and replace it with … ). If I share the actual text as text, for example, in a Word document, you can edit it as you will.

Reuse with modification is also a fruitful way of reusing diagrams. But it can be harder to achieve in practical terms. For example, in a physics or electronics course, or a geometry course, there are likely to be standard mechanical principle diagrams, electrical circuits or geometrical proofs that you are likely to want to refer to. These diagrams may exist as openly licensed resources, but… The numbers or letters you want to label the diagram with may not be the same as in the original. So what do you do? Redraw the diagram? Or edit the original, which may reduce the quality of the original or introduce some visual artefact the reveals the edit (“photocopy lines”!).

But what if the “source code” or means of producing the diagram. For example, if the diagram is created in Adobe Illustrator or CorelDRAW and the diagram made available as an Adobe Artwork .ai file or a  CorelDRAW .cdr file, and you have an editor (such as the original, or an alternative such as Inkscape) that imports those file formats, you can edit and regenerate a modified version of the diagram at the same level of quality as the original. You could also more easily restyle the diagram, even if you don’t change any of the content. For example, you could change line thickness, fonts or font sizes, positioning, and so on.

One of the problems with sharing image project files for particular applications is that the editing and rendering environment for working with project file is likely separate from your authoring environment. If, while writing the text, you change an item in the text and want to change the same item as referenced in the image, you need to go to the image editor, make the change, export the image, copy it back into your document. This makes document maintenance hard and subject to error. It’s easy for the values of the same item as referenced in the text and the diagram to drift. (In databases, this is why you should only ever store the value of something once and then refer to its value by reference. If I have your address stored in two places, and you change address, I have to remember to change both of them; it’s also quite possible that the address I have for you will drift between the two copies I have of it…)

One way round this is to include the means for creating and editing the image within your text document. This is like editing a Microsoft Word document and including a diagram by using Microsoft drawing tools within the document. If you share the complete document with someone else, they can modify the diagram quite easily. If you share a PDF of the document, they’ll find it harder to modify the diagram.

Another way of generating diagrams is to “write” it, creating a “program” that defines how to draw the diagram and that can be run in a particular environment to actually produce the diagram. By changing the “source code” for the diagram, and rerunning it, you can generate a modified version of the diagram in whatever format you choose.

This is what packages like TikZ support [docs].

And this is what I’ve been exploring in Jupyter notebooks and Binderhub, where the Jupyter notebook contains all the content in the output document, including the instructions to create image assets or interactives, and the Binder container contains all the software libraries and tools required to generate and embed the image assets and interactives within the document from the instructions contained within the document.

That’s what I was trying to say in Maybe Programming Isn’t What You Think It Is? Creating Repurposable OERs (which also contains a link to a runnable example).

PS by the by, I also stumbled across this old post, an unpursued bid, today, that I have no recollection of at all: OERs: Public Service Education and Open Production. Makes me wonder how many other unfinished bids I started…