COuRsE Data

One of the things that has never really been clear to me is what it is that universities think they sell and what students think they are “buying”. (OU modules have always(?) had a price tag associated with them, although large amounts of financial support has also traditionally been available). One partial view might focus on one of the more tangible exchanges that are evident when taking a university degree, specifically the modules taken as part of a qualification programme, and the way they are bundled, organised and presented to students. Curriculum innovation works at both the level of keeping these modules up to date, as well as introducing new modules (and potentially new degree programmes, either as new aggregations of, and pathways through, collection of modules).

If we think of universities as organisations in the business of selling, at least in part, structured collections of course modules*, then we might speculate around the processes that are used to come up with new collections that are desirable to fee-paying students (and consequently, employers).

(* I know, I know – we might also think of the cost centre services that go along with course delivery as part of the package, the assessment, the facilities, the pastoral care, the structured academic content; or the “payoff” in terms of improved employability, or higher lifetime earnings. But when I buy a bar of chocolate, I don’t see it as covering the factory automation, raw ingredients, logistics or supply chain costs, nor am I buying in to delight or gluttony. I’m buying a bar of chocolate. I’m also not saying that the courses are necessarily the thing students are buying, it’s just one particular lens we can use to see whether it makes storytelling sense to view the system in that way…)

In part, programmes of study leading to named qualifications in particular subject or topic areas are influenced by the QAA benchmark statements:

Subject benchmark statements set out expectations about standards of degrees in a range of subject areas.

Subject benchmark statements do not represent a national curriculum in a subject area. Rather, they allow for flexibility and innovation in programme design within an overall conceptual framework established by an academic subject community. They are intended to assist those involved in programme design, delivery and review and may also be of interest to prospective students and employers, seeking information about the nature and standards of awards in a subject area.

In terms of curriculum development, there is a chicken-and-egg element to the role QAA statements can play. As the Recognition scheme for subject benchmark statements suggests in its guidance relating to the creation of new benchmark statements:

The proposal will need to demonstrate that a new or revised statement would provide the
benefits of a wider understanding about the scope and nature of the subject and the academic
standards underpinning it. This could be desirable for one or more of the following reasons.
• The subject is growing and more degree programmes are being provided in it
• A degree in the subject may be required for entry into a profession, but there are no explicit
academic standards associated with the subject for this purpose. There may also be a lack of
understanding within the relevant profession of what level of attainment can be expected
of a graduate in the subject, or of its appropriateness for entry into the profession
• The prospective benefits of agreed and explicit standards in the relevant subject have been
highlighted by, for example, external examiners and validating boards, higher education
providers, subject groups, or stakeholder organisations.

(See also Statements in development for examples of statements currently under consideration.)

Unpicking the course module view a little further, modules are typically associated with notional academic credit points, which are awarded “when you have shown, through assessment, that you have successfully completed a module or a programme by meeting the specific set of learning outcomes for that module or programme” (Academic credit in higher education in England – an introduction; see also QAA – Academic Credit). Note that credit points do not reflect how well you passed the assessment, just that you achieved at least the minimum standard required. Credit points themselves relate to two considerations: “[t]he credit value indicates both the amount of learning expected (the number of credits) and its depth, complexity and intellectual demand (the credit level).” The “amount of learning” is captured by the “notional hours of
learning” spent on the subject within the module. The level is based on level descriptors that “are used to help work out the level of learning in individual modules.”

Credit level descriptors are guides that help identify the relative demand, complexity and depth of learning, and learner autonomy expected at each level, and also indicate the differences between the levels.
They are general descriptions of the learning involved at a particular level; they are not specific requirements of what must be covered in a particular module, unit or programme.

So to recap – modules are designed in order to deliver a set of learning outcomes (that include subject or topic specific learning outcomes as well as more general skills) that can be acquired in a notional amount of time and that are assessed at a particular academic level in exchange for a academic credit.

Qualifications are then awarded based on credit awarded in programmes of study, such as undergraduate or postgraduate degrees. Qualifications typically require the demonstration of some sort of progression through credit levels within a subject area, specify a range of qualification level learning outcomes that need to be delivered within the context of the programme as a whole, and may also require students to demonstrate aptitude across a range of assessment styles (or alternatively, offer a range of assessment styles so as not to disadvantage students who struggle with a particular style of assessment).

Whilst “traditional” universities typically offered named degree programmes in specific areas, the Open University originally offered an Open Degree (which is still available), in which students were free to choose whatever modules (then referred to as OU courses) they wanted, subject to certain requirements on the number of courses taken at each credit level (akin to each year of a traditional university degree; for more on credit points and credit equivalence. Whilst course choice was free, many students followed the same common pathways through courses to come out with degrees that were, essentially subject degrees. In recent years, the OU has moved increasingly towards the award of named degrees, where students are required to take particular modules. Indeed, it is increasingly difficult to find the individual modules that students originally “bought” on the OU website – the emphasis now is on selling qualification level credit bundles, rather than module level credit points.

But how do universities decide what modules to offer? And how does curriculum innovation work? A bottom-up approach might be to refresh modules within a qualification, and then create new qualifications by rebundling sets of modules that together define some sort of coherent whole (this is how ‘as-if’ subject degrees were self-assembled by OU students in the Open Degree). A top-down approach might be to come up with an idea for a degree programme, and then commission modules to deliver that programme of study. Alternatively, we might look to mass-dynamics in a free choice system, such as an open degree, and come up with a middle-out(?) approach that suggests programmes of study that formalise the module collections freely chosen by students interested in studying a particular set of topics that make sense to them.

(It is interesting to note that possibly uniquely within UK Higher Education, The Open University had the scale of numbers in undergraduate students to start to say interesting things about the way students selected courses under the Open degree model. Furthermore, as popularity in “Big Data” solutions and recommendations driven by crowd-behaviour becomes commonplace, so the OU is reducing the amount of personalisation possible by pushing hard-coded, predefined pathways. At the same time, institutions such as Southampton University seem to be looking to open up personalisation pathways (for example, Southampton Curriculum innovation, discussed here: Graduates for the 21st Century – Curriculum innovation [audio]) and standalone HE level courses are increasingly available, sans credit, (as marketing warez; but for what exactly?) via the various open online course platforms.

So now we’re at the point where I actually wanted to start this post… How do we go about the process of curriculum innovation (for example, OECD Education Working Papers No. 82 – Bringing About Curriculum Innovations) given that we already have a load of inventory? If we sell credit points in particular subject areas or topics, how do we decide what topics to cover and how do we bundle those points up into qualifications?

One place to start might be mapping out where we are at the current time, which is where course data comes in. For example, what does the interest map based on learning outcomes delivered by your university actually look like? Or if you work in HE, do you know (or can you readily find out):

  • which modules are associated with any particular qualification?
  • which qualifications are associated with any particular module, either as a required or optional component?
  • which modules have path dependencies (eg where one module is the pre-requisite of another, or modules are excluded combinations)?
  • which module are required in which pathways, and which are optional?
  • in free choice modules (that maybe span programmes), which modules tend to be taken together?
  • which modules deliver which qualification level learning outcomes?
  • which modules deliver which sort of assessment types?
  • are there any modules that already offer particular learning outcomes at a particular level?

To provide a little more context, imagine these scenarios:

  • Module X is tired and needs to be replaced – what qualifications or other modules might be affected as a result? For example, does the module uniquely cover a a particular qualification level learning outcome, or assessment type?
  • A new qualification is proposed with a particular set of learning outcomes – what modules are available that already deliver some (or all) of these learning outcomes?
  • The quality folk want to know how your programme demonstrates progression across credit levels with respect to a particular set of subject related learning outcomes. Can you easily map this out?
  • The quality folk also want to know whether a particular course is gameable in terms of assessment types covered by the course modules. Could a student select a set of modules that means they never have to do teamwork, project work/report, a presentation, an exam, etc?
  • You need to generate a set of course transcripts (sets of learning outcomes, by credit level) for a proposed new assemblage of outstanding modules, some of which are core/compulsory modules, some of which are optional. Can you do it?
  • Do you have the scaffolding data available to build course recommenders based on population flows and module selections of previous cohorts of student?
  • Which modules deliver the content that potential students think they want to study, eg when searching your online course prospectus (you do use search logs for situational awareness around what potential students are searching for, don’t you?!)

So – how well do you fare?

[Note: this post is inspired by personal reflections around the University of Lincoln ON Course Course data project, on which I have, via the OU, a small consultancy retainer, and of which: more later.]

Scraperwiki Powered OpenLearn Searches – Learning Outcomes and Glossary Items
Generating OpenLearn Navigation Mindmaps Automagically


  1. Sheila MacNeill (@sheilmcn)

    Hi Tony

    The JISC Curriculum Design Programme looked at many of these issues. However a critical issue was actually getting course documentation/data into a state where these kind of questions could start to be addressed. There was often a gap between what was in course/module descriptions and what actually happened. You might want to look at the synthesis report and also work of MMU in particular. They have introduced a new course database as they are currently re-structuring their whole undergraduate curriculum.


    • Tony Hirst

      Thanks, Sheila – I’ll check out MMU. The Lincoln ONcourse project has also been getting Lincoln’s course data into shape – I hope to get a post out on that next week.

  2. Pingback: Open data in higher education – the ‘next big thing’? | Universities UK blogUniversities UK blog