Fragment – Programming Privilege

I saw a quote today somewhere that I didn’t save and don’t properly recall along the lines of most people don’t know what it’s like to be in the privileged position of a programmer who is able to code your own custom apps.

Slightly earlier, I’d idly wondered about the rhetoric of “everyone must learn to programme”, and how many people I know found it useful in their work to “do some programming” yesterday, or last week or over the last month.

I also wonder about how many of the folk preaching “everyone should learn to programme”:

a) actually know how to write code (and if they don’t, do they intend to learn? When?);
b) ever use code to do anything (i.e. why should folk learn to code? What’s it useful for?)

Whatever…

Going back to the privilege thing, what I think folk don’t realise is how little you need to do to get things done, if you know what to do – or look for.

One of the reasons I like the RStudio and Jupyter notebook environments, particularly for visualising data, is that if you have the data looking right, (i.e. when you display it as a table, it has things arranged in the right way in a table made up of  rows and columns) you can create all manner of wonderful, interactive graphical outputs with a single line of code. Other folk have written templates from creating embeddable interactive HTML widgets, and transforming the data in my data table into the data the widget needs to display the chart.

In itself, code can often be just a written version of a menu option you might select or a set a values you have selected in an interactive form. Nothing new, just turning point-and-click choices into word equivalents and then writing them down, so the machine can do whatever it is rather than you.

Things are also – arguably – getting easier.

For example, over the last week Simon Willison has been tinkering with a really useful set of components for querying data. This started with a tool to put CSV files into a simple database without the user having to do anything other than say – put these files into a database. And then a tool that let you publish the database as a service, either on your own computer or the cloud (again with a single line) so you could query it through a browser. Now there’s a UI, and a simple tutorial for building a simple app to search for trees in San Francisco. (Writing the HTML might be tricky, but that page could be based on a template? Or the app could perhaps be accessed via Jupyter notebook using dynamically created ipywidgets and published using appmode.)

Taken together, with a few lines of code, you can create and publish your own interactive data explorer , along with an API you can access from elsewhere.

Perhaps that is a privilege only “coders” can avail themselves of. But why?

With knowledge of a few small tools, the sorts of thing they do (rather than how they do it), and how to wire them together, you can start to create powerful, and useful, applications.

Which makes me wonder again about whether it is really so much harder to teach programming stuff to adult novices using real tools to solve real problems compared to teaching them how to use Scratch to make an animated owl blink its eyes or make a turtle draw a square?

Pedagogy is an oft used word in the OU, which in derivation means “lead a child”. (The adult equivalent, andragogy, is rarely used, self-directed heutagogy even less so.) Can we really do no better at teaching adults programming and what it might be used for than starting them off with months of programming using tools developed for teaching primary school children, some of which was invented getting on for fifty years ago? Are we really that stuck for coming up with teaching ideas that not only teach principles of “computational thinking” and core programming ideas, but also help learners see how they might be able to use the ideas in practice and perhaps even set them up with a chance of using them for real outside of their studies? (SMEs employing three people probably don’t need a full time developer, but if one of them can hack a solution to problem that solves a need, or makes a particular task slightly easier, it doesn’t really matter?)

Or perhaps the practical uses get in the way of teaching the “pure*, abstract coding principles?

I JUST DON’T UNDERSTAND WHAT WE’RE TRYING TO DO?!

Innovation’s End

In that oft referred to work on innovation, The Innovator’s Dilemma, Clayton Christensen suggested that old guard companies struggle to innovate internally because of the value networks they have built up around their current business models. Upstart companies compete around the edges, providing cheaper but lower quality alternative offerings that allow the old guard to retreat to the higher value, higher quality products. As the upstarts improve their offerings, they grow market share and start to compete for the higher value customers. The upstarts also develop their own value networks which may be better adapted to an emerging new economy than the old guard’s network.

I don’t know if this model is still in favour, or whether it has been debunked by a more recent business author with an alternative story to sell, but in its original form it was a compelling tale, easily co-opted and reused, as I have done here. I imagine over the years, the model has evolved and become more refined, perhaps offering ever more upmarket consultancy opportunities to Christensen and his acolytes.

The theory was also one of the things I grasped at this evening to try to help get my head round why the great opportunities for creative play around the technologies being developed by companies such as Google, Amazon and Yahoo five or so years ago don’t seem to be there any more. (See for example this post mourning the loss of a playful web.)

The following screenshots – taken from Data Scraping Wikipedia with Google Spreadsheets – show how the original version of Google spreadsheets used to allow you to generate different file output formats, with their own URL, from a particular sheet in a Google spreadsheet:

publishAsWebpage

publishFormats

publishOPtions

morePublishOptions

In the new Google spreadsheets, this is what you’re now offered from the Publish to Web options:

googleshshare

[A glimmer of hope – there’s still a route to CSV URLs in the new Google spreadsheets. But the big question is – will the Google Query language still work with the new Google spreadsheets?]

embed changes everything

(For some reason, WordPress won’t let me put angle brackets round that phrase. #ffs)

That’s what I said in this video put together for a presentation to a bunch of publishers visiting the OU Library at an event I couldn’t be at in person (back when I used to be invited to give presentations at events…)

I saw embed as a way that the publishers could retain control over content whilst still allowing people to access the content, and make it accessible, in ways that the publishers wouldn’t have thought of.

Where content could be syndicated but remain under control of the publisher, the idea was that new value networks could spring up around legacy content, and the publishers could then find a way to take a cut. (Publishers don’t see it that way of course – they want it all. However big the pie, they want all of it. If someone else finds a way to make the pie bigger, that’s not interesting. My pie. All mine. My value network, not yours, even if yours feeds mine. Because it’s mine. All mine.)

I used to build things around Amazon’s API, and Yahoo’s APIs, and Google APIs, and Twitter’s API. As those companies innovated, they built bare bones services that they let others play with. Against the established value network order of SOAP and enterprise service models let the RESTful upstarts play with their toys. And the upstarts let us play with their toys. And we did, because they were easy to play with.

But they’re not anymore. The upstarts started to build up their services, improve them, entrench them. And now they’re not something you can play with. The toys became enterprise warez and now you need professional tools to play with them. I used to hack around URLs and play with the result using a few lines of Javascript. Now I need credentials and heavyweight libraries, programming frameworks and tooling.

Christensen saw how the old guard, with their entrenched value networks couldn’t compete. The upstarts had value networks with playful edges and low hanging technological fruit we could pick up and play with. The old guard entrenched upwards, the upstarts upped their technology too, their value networks started to get real monetary value baked in, grown up services, ffs stop playing with our edges and bending our branches looking for low hanging fruit, because there isn’t any more. Go away and play somewhere else.

Go away and play somewhere else.

Go somewhere else.

Lock (y)our content in, Google, lock it in. Go play with yourself. Your social network sucked and your search is getting ropey. You want to lock up content, well so does every other content generating site, which means you’re all gonna be faced with the problem of ranking content that all intranets face. And their searches universally suck.

The innovator’s dilemma presented incumbents with the problem of how to generate new products and business models that might threaten their current ones. The upstarts started scruffy and let people play alongside, let people innovate along with them. The upstarts moved upwards and locked out the innovation networks around them. Innovations end. Innovation’s end. Innovation send. Away.

< embed > changes everything. Only this time it’s gone the wrong way. I saw embed as a way for us to get their closed content. Now Google’s gone the other way – open data has become an embedded package.

“God help us.” Withnail.

PS Google – why did my, sorry, your Chrome browser ask for my contacts today? Why? #ffs, why?

Scripting Charts WIth GraphViz – Hierarchies; and a Question of Attitude

A couple of weeks ago, my other was finishing off corrections to her PhD thesis. The layout of one of the diagrams – a simple hierarchy written originally using the Draw tools in an old versioof MS-Word – had gone wrong, so in the final hours before the final printing session, I offered to recreate it.

Not being a draughtsman, of course I decided to script the diagram, using GraphVIz:

The labels are added to the nodes using the GraphViz label command, such as:

n7[label="Trait SE"];

The edges are defined in the normal way:

n4->n8;
n4->n9;

But there was a problem – in the above figure, two nodes are placed by the GraphvViz layout in the wrong place – the requirement was that the high and low nodes were ordered according to their parents, and as, indeed, they had been ordered in the GraphViz dot file.

A bit of digging turned up a fix, though:

graph [ ordering="out" ];

is a switch that forces GraphViz to place the nodes in a left-to-right fashion in the order in which they are declared.

During the digging, I also found the following type of construct

{rank=same;ordering=out;n8;n9;n10;n11;n12;n13;n14;n15

which will force a set of nodes to be positioned along the same horizontal row. Whilst I didn’t need it for the simple graph I was plotting, I can see this being a useful thing to know.

There are a few more things, though, that i want to point out about this whole exercise.

Firstly, I now tend to assume that I probably should be able to script a diagram, rather than have to draw it. (See also, for example, Writing Diagrams, RESTful Image Generation – When Text Just Won’t Do and Progressive Enhancement – Some Examples.)

Secondly, when the layout “went wrong”, I assumed there’d be a fix – and set about searching for it – and indeed found it, (along with another possibly useful trick along the way).

This second point is an attitudinal thing; knowing an amount of programming, I know that most of the things I want to do most of the time are probably possible because they the exactly the sorts of problems are likely to crop up again and again, and as such solutions are likely to have been coded in, or workarounds found. I assume my problem is nothing special, and I look for the answer; and often find it.

This whole attitude thing is getting to be a big bugbear of mine. Take a lot of the mashups that I post here on OUseful.info. They are generally intended not to be one off solutions. This blog is my notebook, so I use it to record “how to” stuff. And a lot of the posts are contrived to demonstrate minimally worked examples of how to do various things.

So for example, in a recent workshop I demonstrated the Last Week’s Football Reports from the Guardian Content Store API (with a little dash of SPARQL).

Now to me, this is a mashup that shows how to :

– construct a relative date limited query on the Guardian content API;
– create a media RSS feed from the result;
– identify a convention in the Guardian copy that essentially let me finesse metadata from a free text field;
– create a SPARQL query over dbpedia and use the result to annotate each result from the Guardian content API;
– create a geoRSS feed from the result that could be plotted directly on a map.

Now I appreciate that no-one in the (techie) workshop had brought a laptop, and so couldn’t really see inside the pipe (the room layout was poor, the projection screen small, my presentation completely unprepared etc etc), but even so, the discounting of the mashup as “but no-one would want to do anything with football match reviews” was…. typical.

So here’s an issue I’ve some to notice more and more. A lot of people see things literally. I look at the football match review pipe and I see it as giving me a worked example of how to create a SPARQL query in a Yahoo pipe, for example (as well as a whole load of other things, even down to how to construct a complex string, and a host of other tiny little building blocks, as well as how to string them together).

Take GraphViz as another example. I see a GraphViz file as a way of rapidly scripting and laying out diagrams using a representation that can accommodate change. It is possible to view source and correct a typo in a node label, whereas it might not be so easy to see how to do that in a jpg or gif.

“Yes but”, now comes the response, “yes, but: an average person won’t be able to use GraphViz to draw a [complicated] diagram”. Which is where my attitude problem comes in again:

1) most people don’t draw complicated diagrams anyway, ever. A hierarchical diagram with maybe 3 layers and 7 or 8 nodes would be as much as they’d ever draw; and if it was more complicated, most people wouldn’t be able to do it in Microsoft Word anyway… I.e. they wouldn’t be able to draw a presentable diagram anyway…

2) even if writing a simple script is too hard, there are already drag and drop drop interfaces that allow the construction of GraphViz drawings that can then be tidied up by the layout engine.

So where am I at? I’m going to have a a big rethink about presenting workshops (good job I got rejected from presenting at the OU’s internal conference, then…) to try to help people to see past the literal and to the deeper truth of mashup recipes, and try to find ways of helping others shift their attitude to see technology as an enabler.

And I also need a response to the retort that “it won’t work for complicated examples” along the lines of: a) you may be right; but b) most people don’t want to do the complicated things anyway…