Fragment – ROER: Reproducible Open Educational Resources

Fragment, because I’m obviously not making sense with this to anyone…

In the words of David Wiley (@opencontent), in defining the “open” in open content and open educational resources [link], he identifies “the 5R activities” that are supported by open licensing:

  1. Retain – the right to make, own, and control copies of the content (e.g., download, duplicate, store, and manage)
  2. Reuse – the right to use the content in a wide range of ways (e.g., in a class, in a study group, on a website, in a video)
  3. Revise – the right to adapt, adjust, modify, or alter the content itself (e.g., translate the content into another language)
  4. Remix – the right to combine the original or revised content with other material to create something new (e.g., incorporate the content into a mashup)
  5. Redistribute – the right to share copies of the original content, your revisions, or your remixes with others (e.g., give a copy of the content to a friend)

Whilst the legal framework is the one that has to be in place for educational institutions as publishers of (third party) content, where there is particular emphasis on citing others and not reusing content in an unacknowledged way, I have always been more interested in the practice of reusing content, particular when that means reuse with modification.

Various others have suggested sixth Rs. For example, Chris Aldridge’s The Sixth “R” of Open Educational Resources identifies “Request update (or maybe pull Request, Recompile, or Report to keep it in the R family?)“, in the sense of keeping stuff in a repo so people can fork the report, update it and track changes (maybe Revisions is a better R for that?). Rather than be an R relating to a right you can assert, Revisions is more about the practice.

As a trawl through the history of this blog suggests (for example, Open Content Anecdotes from nigh on a decade ago), I’ve also been less interested in the legal framework around OERs than I am in the practical reuse (with modification) of particular (micro? atomic?) assets, such as diagrams, or problem sets. That is, the things you are more likely to spot as relevant, useful or interesting and weave into your own materials, or replace your own crappy diagrams.

To a large extent, the legal bit doesn’t stop me, particularly if no-one finds out. The blocker is in the practicalities associated with reversioning the physical asset, and making actual changes or modifications to it, that makes it appropriate for my course.

(You can always redraw diagrams, which can also help you get round copyright on non-openly licensed works, but that takes time, skill, and maybe a drawing package you don’t have access to.)

So the idea that I started trying to crystallise out almost a year ago now  — OERs in Practice: Re-use With Modification — is based around another R, a leading R, or a +1 R, which (as with Aldridge’s suggested R) is a practice based R: Reproducibility. (Hmm.. maybe this post should have been The 5+2Rs of Open Content?)

By their very nature, these resources are resources that include their own “source code” for creating assets, so that if you want to create a modified version of the asset, you can modify the source and then regenerate the asset. A trivial example is to use diagrams that are “written diagrams” – diagrams generated from textual, written descriptions or codified versions of them and rendered from the description by a particular chart generating tool (for example, Writing Diagrams, Writing Diagrams – Boxes and Arrows and Writing Diagrams (Incl. Mathematical Diagrams)).

As to why this is a fragment, I’m stopping here… discussion about reproducibility is elsewhere, and will be to follow too, along with why this approach is opens up new opportunities for educators as well as learners. For now, see these other fragments on that topic in date order.

[Update: via a comment, @opencontent reminds me that he also made the distinction between legal and practical issues, with practical concerns raised in the ALMS framework – I should have read on from the legal issues to the Poor Technical Choices Make Open Content Less Open section… See the comment thread to this post for more, as well as this related post from the previous time the ALMS model was raised to my attention: Open ALMS.  I also note this recent IRRODL paper on Defining OER-Enabled Pedagogy, which I need to read through…]

PS some more related fragments from a hastily written, unsuccessful internal Esteem Project bid:


One problem associated with producing rich educational materials is that inconsistencies can occur when cross referencing text with media assets such as charts, tables, diagrams and computer code produced via different production routes. The project will explore and demonstrate how emerging technologies and workflows developed to support reproducible work practices can be adopted for the development of reproducible educational resources, including but not limited to educational materials rich in mathematical content, scientific / engineering diagrams, exploratory and explanatory statistics, maps and geospatial analysis, music theory and analysis, interactive browser based activities, animations and dynamically created audio assets.

The aim is to demonstrate:

  • The range of assets that can be produced / directly authored by academics including static, animated and interactive elements such as print quality drawings, animated scientific diagrams, and interactive web activities and applications (eg interactive maps, 3D models, etc.)
  • The workflows associated with demonstrating the production, maintenance, and reuse with modification of the assets and works derived from them, including but not limited variations on a theme in the production of parameterised assessment materials
  • The potential for using reproducible materials to facilitate maintenance, reversioning / updating and reuse with modification of module materials The outputs will include:
    • A range of OU module and OpenLearn unit materials reworked using the proposed technologies and workflows
    • A library of reproducible educational resource templates for a range of topic areas capable of being reused with modification in order to produce a range of assets from a common template.

The project will demonstrate how freely available, open source technologies can be used to support the direct authoring of rich and interactive media assets in a reproducible way.


Reproducible research tools increasingly support the direct authoring of rich documents that blend text, data, code, code outputs, with media assets (audio, video, static and animated images, interactives) generated from text based computer scripts.

The project proposes the co-option of such tools for use as authoring tools for reproducible educational materials. A single “source document” can include text as well as scripts for generating tables and charts, for example, from data contained within the document itself, minimising the distance between the production of assets from the materials they are used in.

The resulting workflow supports consistency in production and maintenance as well as reuse with modification thereafter by allowing updates in situ that can be used to recreate modified assets (diagrams created dynamically using updated values, for example). Materials will also be modifiable by ALs for tutorial use.

Authoring tools also support the direct authoring and creation of interactive components based around templated third party widgets such as 3D molecule viewers, or interactive maps, allowing authors to directly author interactive components that can be embedded in online / browser accessed materials.

Examples of the sorts of assets I had in mind to rework can be found in several of the notebooks available here.

Author: Tony Hirst

I'm a Senior Lecturer at The Open University, with an interest in #opendata policy and practice, as well as general web tinkering...

4 thoughts on “Fragment – ROER: Reproducible Open Educational Resources”

  1. Tony, I always love reading what you write. I, too, am interested in the practical problems around reuse of open content. In the page you link two at the top of your post, if you scroll further down, you will find the ALMS framework. This is my framework for thinking about the practical problems associated with reuse. I maintain the two framework separately, one dedicated to legal issues and one to practical issues, in order to keep things tidy. I would love to hear your thoughts about the relationship between reproducibility and what I described in that second framework.

    1. HI David,
      Thanks for the comment; oops on me for not reading through – s’what makes me not a scholar, I guess ;-) Thanks for the link – it reminds me I did chase a link to ALMS before:

      That post is actually a complement to this one, and considers another important part of the jigsaw, specifically providing the environment required to transform “source” documents into “rendered” content. (A metaphor I started to ponder whilst walking the dog this afternoon was HTML: the web took off in part because people could take “rendered” pages – what you see in a web browser – and then “View Source” on them to see the raw HTML that actually defined the page. The raw HTML could be tweaked in a simple text editor and the re-rendered in the browser, giving you your own, modified page. That users could also make enough sense of the HTML to change the bits they wanted to change (the content bits, rather than the code bits) without needing to be programmers.
      In this post, and the notebooks linked to, I’m focussing on making the “HTML”/View source available; that previous post, looking at Binder, etc, is more about providing the “browser” that renders the source to the finished product.
      Azure notebooks, plus some boilerplate package install code cells in my notebooks, play the same role in providing a complete execution environment as does a configured Binder container.

      Thanks again for commenting by, it gets lonely here sometimes!

  2. Hi Tony! Some feedback from me: I think it more likely that people will just continue to search online for a resource they can use than redraw/remake assets. When they do remake/redraw assets they are more likely to use a familiar tool (Excel graph, Paint graphic, photo of hand-written diagram, etc.) to reproduce rather than delve into Jupyter.

    I’m pretty impressed with Jupyter, though I can see that there is a certain barrier to entry in becoming familiar with how it works. For most people I suppose that they will need to use it so infrequently for the purpose of recreating licensed assets that they will just continue to use their work arounds.

    If there was a more user friendly UI which could be used to generate quite specific images on the appropriate licence I think some people would use it. Code will put off a lot of people, even if it’s not that complicated when you break it down. We found in some of our research that a lot of OER use is ‘on the fly’ and for a specific need at the time (inspiration for a lesson, slide for a presentation, etc.). So people Google and search Wikipedia and once they find the thing they need they move onto something else. So, workflow for time pressured people is significant element.

    One approach worth exploring might be to focus on subject areas where there aren’t sufficiently specific materials in the commons. The science examples you provide have great potential, and I can see how for subject specialists (inc. learning design, librarians) they could be a really useful tool once mastered.

    Returning to the original prompt about reproducibility: I think reproducibility itself is conveyed by a wealth of tools for editing and sharing content. I think the more interesting element you’re picking up on might be better described as ‘RECORDED’. The idea of an OER repository with forking, version control, etc. is still relevant but no-one has found a good way of making it happen as far as I can tell. If we can record modifications – especially if they can be connected with a context of use and some evaluation – that would be a really important step in understanding the ‘reproducibility’ of OER.

    1. Hi Rob

      Thanks for the comments. I’m really not getting this across, am I?!

      You said: “We found in some of our research that a lot of OER use is ‘on the fly’ and for a specific need at the time (inspiration for a lesson, slide for a presentation, etc.).”

      That’s exactly what I’m trying to address, because from my own experience, the resources are never really quite right. Eg a diagram is almost right but something about it needs tweaking. Which means its not right without redrawing or taking into an editor and fiddling around with (if you have an appropriate editor and the required skills).

      The ROER approach tries to do several things:

      1) it provides a context where people can create and explore variants of an asset through producing the means of generating the asset in a parameterised way;
      2) it allows third parties to see a published asset and then easily create a variant of it;
      3) it allows learners to explore what it means to create variations of the thing (which in turn may help them understand the thing).

      The web is the best metaphor: in the beginning there were only a few crappy web pages around, but folk “View Source”d, copied, and modified the bits they could recognise. I’m working towards the same sort of idea. (You also need to realise that where there is lots of code in a notebook, with just a few parameters than need to be changed to tweak a diagram, the next iteration of the notebook may hide the boilerplate code behind a single line of magic. Eg there are lots of notebooks with code that show how to embed maps, but you can iterate tools to one line it: A secondary benefit is that as well as generating the map inline with a single line of code, it can be saved as a ready-to-go html file that can be embedded in an iframe in another page.

      As far as folk not getting started with notebooks, because its a new thing to learn: I know, I know… Crazy…

Comments are closed.

%d bloggers like this: