Fragment – Accessibility Side Effects? Free Training Data for Automated Markers

Another of those “woke up and suddenly started thinking about this” sort of things…

Yesterday, I was in on a call yesterday discussing potential projects around an “intelligent” automated short answer question marking system that could be plugged in to a Jupyter notebook environment (related approach here).

Somewhen towards the end of last year, I did a quick sketch of a simple marker support tool that does quick pairwise similarity comparisons between sentences in a submitted answer and a specimen answer. (The report can report on all pairwise comparisons or just the betst matching sentence in the specimen compared to to the submitted text.)

One issue with this is the need to generate the specimen text. This is also true for developing or training intelligent automated markers that try to match submitted texts against specimen texts.

As part of the developmnet, or training, process, automated marking tools may also require a large number of sample texts to train the system on. (My simple marker support simularity tool doesn’t: it’s there purely to help a marker cross-reference sentences in a text with sentences in the sample text.)

So… this is what I woke up wondering: if we set a question in a data analysis context, asking students to interpret a chart or comment on a set of reported model parameters, can we automatically generate the text from the data, which is to say, from the underyling chart object or model parameters.

I’ve touched on this before, in a slightly different context, specifically creating text desciptions of charts as an accessibility support measure for visually impaired readers (First Thoughts on Automatically Generating Accessible Text Descriptions of ggplot Charts in R), as well as more generally (for example, Data Textualisation – Making Human Readable Sense of Data.

So I wonder, if we have an automated system to mark free text short answers that ask students to comment on some sort of data analysis, could our automated marking system:

• take the chart or model parameters the student generated and from that generate a simple “insightful” text report that could be used as the specimen answer to mark the student’s own free text answer (i.e. does the student report back similar words to words out insights generator “sees” and reports back in text form);
• compare the chart of model parameters generated by the student with own own “correct” analysis / charts / model parameters.

In the first case, we are checking the extent to which the student has correctly interpreted their own chart / model (or one we have provided them with or generated for them) as a free text comparison. In the second case, we are checking if the student’s model/chart is correct comparred to our specimen model / chart etc. based on a comparison of model / chart parameters.

Something else. If we have a system for generating text from data (which could be datatables, could be chart or model parameters etc), we might also be able to generate lots of different texts on the theme, based on the same data (I recently started exploring data2text again using Simple Rule Based Approach in Python for Generating Explanatory Texts from pandas Dataframes via the durable_rules package. One of the approaches I’m looking at is to include randomising effects to generate multiple different possible text fragments from the same rule; still early days on that.) If our automated marked then needed to be trained on 500 sample submitted texts, we could then automatically generate those (perhaps omitting some bits, perhaps adding correct interpretations but of misread parameters (so right-but-wrong or wrong-but-right answers), perhaps adding irrelevant sentences etc., perhaps adding typos etc.).

In passing, I was convinced I had posted previously on the topic of “robot writers” generating texts from data not for human consumption but instead for search engines, the idea being that a search engine could index the text and use that to support discovery of a dataset. It seems I had, but hadn’t. In my draft queue (from summer 2015), I note the presence of two still-in-unfinished-draft posts from the Notes on Robot Churnalism series, left languishing because I got zero response back from the first two posts in the series (even though I thought they were pretty good…;-)

Here’s the relevant quote:

One answer I like to this sort of question is that if the search engine’s are reading the words, then the machine generation of textual statements, interpretation and analyses may well be helping to make certain data points discoverable by turning the data into sentences that then become web searchable? (I think I was first introduced to that idea by this video of a talk from 2012 by Larry Adams of Narrative Science: Using Open Data to Generate Personalized Stories.) If you don’t know how to write a query over a dataset to find a particular fact, if someone has generated a list of facts or insights from the the dataset as textual sentences, then you may be able to discover that fact from a straightforward keyword-based query. Just generating hundreds of millions sentences from data so that they can be indexed just in case someone asks some sort of question about that fact might appear wasteful, at least until someone comes up with a good way of indexing spreadsheets or tabular datasets so that you can make search-engine query like requests of them; which I guess is what things like Wolfram Alpha are trying to do? For example, what is the third largest city in the UK?)

On the other hand, we might perhaps need to be sensitive to the idea that that generated content might place a burden on effective discovery. For example, in Sims, Lee, and Roberta E. Munoz. “The Long Tail of Legal Information-Legal Reference Service in the Age of the’Content Farm’.” Law Library Journal Vol. 104:3 p411-425 (2012-29) [PDF], …???

I wish I’d finished those posts now (Notes on Robot Churnalism, Part III – Robot Gatekeepers and Notes on Robot Churnalism, Part IV – Who Cares?), not least to remind myself of what related thoughts I was having at the time… There’s hundreds of words drafted in each case, but a lot of the notes are still of the “no room in the margin” or “no time to write this idea out fully” kind..