Via a tweet from @mhawksey in response to a tweet from @sheilmcn, or something like that, I came across a post by Sheila on the topic of Cloud gazing, maps and networks – some thoughts on #oldsmooc so far. The post mentioned a prototyped mindmap style browser for Cloudworks, created in part to test out the Cloudworks API.
Having tinkered with mindmap style presentations using the d3.js library in the browser before (Viewing OpenLearn Mindmaps Using d3.js; the app itself may well have rotted by now) I thought I’d have a go at exploring something similar for Cloudworks. With a promptly delivered API key by Nick Freear, it only took a few minutes to repurpose an old script to cast a test call to the Cloudworks API into a form that could easily be visualised using the d3.js library. The approach I took? To grab JSON data from the API, construct a tree using the Python networkx library, and drop a JSON serialisation of the network into a templated d3.js page. (networkx has a couple of JSON export functions that will create tree based and graph/network based JSON data structures that d3.js can feed from.
Here’s the Python fragment:
#http://cloudworks.ac.uk/api/clouds/{cloud_id}.{format}?api_key={api_key}
import urllib2,json, networkx as nx
from networkx.readwrite import json_graph
id=cloudscapeID #need logic
urlstub="http://cloudworks.ac.uk/api/"
urlcloudscapestub=urlstub+"cloudscapes/"+str(id)
urlsuffix=".json?api_key="+str(key)
ctyp="/clouds"
url=urlcloudscapestub+ctyp+urlsuffix
entities=json.load(urllib2.urlopen(url))
#print entities
#I seem to remember issues with non-ascii before, though maybe that was for XML? Hmmm...
def ascii(s): return "".join(i for i in s.encode('utf-8') if ord(i)<128)
def graphRoot(DG,title,root=1):
DG.add_node(root,name=ascii(title))
return DG,root
def gNodeAdd(DG,root,node,name):
node=node+1
DG.add_node(node,name=ascii(name))
DG.add_edge(root,node)
return DG,node
DG=nx.DiGraph()
DG,root=graphRoot(DG,id)
currnode=root
#This simple example just grabs a list of clouds associated with a cloudscape
for c in entities['items']:
DG,currnode=gNodeAdd(DG,root,currnode,c['title'])
#We're going to use the tree based JSON data format to feed the d3.js mindmap view
jdata = json_graph.tree_data(DG,root=1)
#print json.dumps(jdata)
#The page template is defined elsewhere.
#It loads the JSON from a declaration in the Javascript of the form: jsonData=%(jdata)s
print page_template % vars()
The rendered view is something along the lines of:
You can find the original code here.
Now I know that: a) this isn’t very interesting to look at; and b) doesn’t even work as a navigation surface, but my intention was purely to demonstrate a recipe from getting data out of the Cloudworks API and into a d3.js mindmap view in the browser, and it does that. A couple of obvious next steps: i) add in additional API calls to grow the tree (easy); ii) linkify some of the nodes (I’m not sure I know who to do that at them moment?)
Sheila’s post ended with a brief reflection: “I’m also now wondering if a network diagram of cloudscape (showing the interconnectedness between clouds, cloudscapes and people) would be helpful ? Both in terms of not only visualising and conceptualising networks but also in starting to make more explicit links between people, activities and networks.”
So here’s another recipe, again using networkx but this time dropping the data into a graph based JSON format and using the d3.js force based layout to render it. What the script does is grab the followers of a particular cloudscape, grab each of their followers, and then graph how the followers of a particular cloudscape follow each other.
Because I had some problems getting the data into the template, I also used a slightly different wiring approach:
import urllib2,json,scraperwiki,networkx as nx
from networkx.readwrite import json_graph
id=cloudscapeID #need logic
typ='cloudscape'
urlstub="http://cloudworks.ac.uk/api/"
urlcloudscapestub=urlstub+"cloudscapes/"+str(id)
urlsuffix=".json?api_key="+str(key)
ctyp="/followers"
url=urlcloudscapestub+ctyp+urlsuffix
entities=json.load(urllib2.urlopen(url))
def ascii(s): return "".join(i for i in s.encode('utf-8') if ord(i)<128)
def getUserFollowers(id):
urlstub="http://cloudworks.ac.uk/api/"
urluserstub=urlstub+"users/"+str(id)
urlsuffix=".json?api_key="+str(key)
ctyp="/followers"
url=urluserstub+ctyp+urlsuffix
results=json.load(urllib2.urlopen(url))
#print results
f=[]
for r in results['items']: f.append(r['user_id'])
return f
DG=nx.DiGraph()
followerIDs=[]
#Seed graph with nodes corresponding of followers of a cloudscape
for c in entities['items']:
curruid=c['user_id']
DG.add_node(curruid,name=ascii(c['name']).strip())
followerIDs.append(curruid)
#construct graph of how followers of a cloudscape follow each other
for c in entities['items']:
curruid=c['user_id']
followers=getUserFollowers(curruid)
for followerid in followers:
if followerid in followerIDs:
DG.add_edge(curruid,followerid)
scraperwiki.utils.httpresponseheader("Content-Type", "text/json")
#Print out the json representation of the network/graph as JSON
jdata = json_graph.node_link_data(DG)
print json_graph.dumps(jdata)
In this case, I generate a JSON representation of the network that is then loaded into a separate HTML page that deploys the d3.js force directed layout visualisation, in this case how the followers of a particular cloudscape follow each other.
This hits the Cloudworks API once for the cloudscape, then once for each follower of the cloudscape, in order to construct the graph and then pass the JSON version to the HTML page.
Again, I’m posting it as a minimum viable recipe that could be developed as a way of building out Sheila’s idea (though the graph definition would probably need to be a little more elaborate, eg in terms of node labeling). Some work on the graph rendering probably wouldn’t go amiss either, eg in respect of node sizing, colouring and labeling.
Still, what do you expect in just a couple of hours?!;-)
OUseful.Info, the blog... 
Hi Tony
As usual what you can do in a couple of hours always amazes me. I’ve been thinking about this a bit more too, and your second recipe is getting close to what my initial thoughts. I’m now thinking very much from a learner perspective and wondering if actually what would be more useful would be viewing related cloudscapes and clouds (rather or as well as followers)? So kind of making the cloudscapes ie activies and learning the central nodes. I’ll try and blog about this in more detail and get my thoughts together a bit more coherently.
@Sheila Can clouds be in more than one cloudscape? In which case, could do map of cloudscapes that all clouds in a particular cloud are in, either as a bipartite graph (cloudscape and cloud nodes) or collapsing it to show cloudscape-cloudscape links (where edge represents ‘connected by a cloud’ relation; maps around tags are maybe another possibility?
Hi Tony
Yes they can. Cloudscapes are a collection of clouds basically so you can add a cloud to mulitple cloudscapes. I’m thinking more of cloudscapes in #oldsmooc context too as there are cloudscapes created by the course team for each week of the course. It’s then up to participants to add clouds. Cloudscape -cloudscape links could be useful, as would be maps around tags.
I’ve just been looking at Cloudworks again to think more about this. Think it actually could be quite a powerful PLE tool particularly for moocs as it gives people an online space to share their learning activities content and link to other “stuff” and people, so kind of getting towards Dave Cormiers rhyzomatic learning idea.
@Sheila Okay, just added a couple more view demos. For example:
– graphs of tags associated with clouds in a cloudscape https://views.scraperwiki.com/run/cloudworks_network_d3js_force_directed_view/?cloudscapeID=2451&viewtype=cloudscapecloudtags
– graphs of cloudscapes associated with clouds in a particular cloudscape https://views.scraperwiki.com/run/cloudworks_network_d3js_force_directed_view/?cloudscapeID=2451&viewtype=cloudscapecloudcloudscape
Hmm… the cloudscapes associated with clouds in a cloudscape suggests a possible dynamic navigation scheme? For example, initial display is of clouds in a cloudscape; click on cloud and pop out other cloudscapes it’s in. Click on one of those cloudscapes to pop out other clouds associated with that cloudscape, and so on, building the graph all the while?
Hi Tony – again brilliant. Been in a meeting this afternoon so just seeing this now. Also sent link to initial post to David Sherlock who did the original mindmap. He’s now thinking about something like this http://bl.ocks.org/4063423
first layer being cloudscape
2nd being clouds
3rd being content, tweets, comments etc
WIth hover or something to see what each thing is, you could then easily see the most active spaces.
And yes like your idea for possible dynamic navigation. Think that would be really useful for a learner. Also wondering if in future some recommendations could be built in too, you know the kind of thing, people who followed this cloud followed that cloud type of thing.
Sunburst doesn’t quite work for that because it relies on a tree structure. if clouds are leaves, then because they can be in many cloudscapes, then sunburst won’t work…Arggh- WordPress didnlt show me your full comment… okay, sunburst make sort of work for that, but what would you be trying to show – how active posts were based on the comments they receive?
Another possibility might be a variation on hierarchical edge bundling http://mbostock.github.com/d3/talk/20111116/bundle.html (sticking for now with what’s possible OTS using d3.js. Eg group together the clouds and cloudscapes associated with an individual, which is a one-one relationship (unless a cloud or cloudscape can have more than one owner?) and then draw lines between clouds and cloudscapes they are a part of? To seed this, we’d need a list of users.
Sounds like a very interesting navigation concept, but I would have a “semantic organisation” layer between the cloudscape and the clouds. There are just too many clouds to manage (if there are only a few, we don’t need any fancy navigation).
So, how about:
centre point: cloudscape
2nd layer: tags, arranged by some proximity metric, e.g. tags that co-occur most frequently will be adjacent
3rd layer: clouds
etc.
Doesnlt work on sunburst, which requires a tree; cloudscape has many tags (that’s okay) but each tags apply to many clouds, and different tags may map onto the same cloud (which breaks the sunburst)
OK – it was just an idea.
I misread the comment – see my revised comment…
I think clouds/cloudscapes can only have 1 owner. List of users could be huge – but then again that’s why we have computer programmes :-) It was David’s idea and haven’t had chance to discuss it in detail with him, will try and get him to comment. But i *think* he maybe coming from trying to identify which clouds/cloudscapes are most active.
A cloud has a single owner, but cloudscapes can have multiple.
Hi Tony, Sheila, Martin,
First – really amazing work. I’d like to feed in the user perspective. One thing our participants are finding very challenging is forming teams and study groups. I wonder if we could think up some possible workflows / scenarios, and work out from there?
For example:
– I joined OLDS MOOC on my own (no organic group of reference).
– I do activity 3 in week 1 (http://www.olds.ac.uk/the-course/week-1/get-started) and add my cloud to the “introduce yourself” cloudscape.
– I tag my cloud to indicate my areas of interest, professional field, geographic location, basically anything I want to be found by.
– I click the “explore neighbourhood” button on my cloud (that doesn’t exist yet).
– This button takes me to a map of the cloudscape(s) to which my cloud is associated, and shows me a tag map of these, with my tags as the point(s) or origin.
– I can then find clouds which represent people with similar interests / context, and hook up with them.
A similar scenario would work for team / study circle forming out of the dreambazaar activity:
http://www.olds.ac.uk/the-course/week-1/dreambazaar
Does this make sense?
Yishay
Erm, sort of… though being a bear of very little brain the journey stuff just confuses me – the actual question relevant to the experiment related here in sense of ‘could we do this?’ is “a map of the cloudscape(s) to which my cloud is associated [so: cloud -> cloudscape? Yes, we can do this], and shows me a tag map of these [we can do cloudscape-tag lines], with my tags as the point(s) or origin [erm, so you want myTag-mycloud-cloudscape-cloudscapeTag?].
– I can then find clouds [you didn’t mention anybody else’s clouds? Or do you want to pull out cloudscape-memberClouds (and then from those memberClouds their tags too)] which represent people with similar interests [what represents people? Clouds? so you also want clouds-user?] / context, and hook up with them.”
Does this work?
my cloud -> cloudscapes it’s in (red arrows) & tags it has (green arrows),
tags / cloudscapes -> clouds in said cloudscapes with same tags
clouds -> people
and a fantasy: have a button to expand, i.e. add the same for each cloud in the graph
Yes, can do that, but messy – will do a gephi demo tonight if I get a chance
Hi Yishay
I can see where you’re coming from and think idea of workflows are exactly what’s needed. However as Tony points out what he’s being doing here is more proof of concept. But I hope what it’s showing is a (several) potential development routes for cloudworks, which will take a bit more than even some of Tony’s “half hours”;-). I think this is a really exciting opportunity to show the potential for developing Cloudworks a bit more to make it more useful and user friendly particularly in the context of Moocs – or indeed any online learning course.
Sheila