Searching the UK Parliament API from Slack Slash Commands Using a Python Microservice via Webhooks

Several years ago now, I remember being excited by the idea of webhooks which provided a simple callback mechanism for executing remote microservice commands on the web via an HTTP request. For whatever reason, webhooks never really became part of my everyday toolkit, but with Amazon Lambda functions coming to my attention again recently as Google experimented with a rival service, I’ve started looking at them again.

To get back into the swing of how these things work, I thought I’d tried to put together a Python simple script that could run a search query against a data collection from the UK Parliament API; the request would be triggered from a slash command in Slack.

On the Slack side, you need to define a couple of custom integrations:

  • Slash Command that will define the name of the command that you want to handle and provide a callback URL that should be accessed whenever the slash command is issued;
  • an Incoming Webhook that  provides a callback URL on Slack that can handle a response from the microservice accessed via the slash command callback.


The slash command is declared and the URL of the service to be accessed needs to be specified. To begin with, you may not have this URL, so it can be left blank to start with, though you’ll need to add it in when you get your callback service address. When the callback URL is requested, a token is passed along with any extra text from the slash command string. The callback service can check this token against a local copy of the token to check that the request has come from a known source.


The incoming webhook creates an endpoint that the service called from the slash command can itself callback to, providing a response to the slash command message.


To handle the slash command, I’m going to develop a simple microservice on using Python 2.7. To begin with, I’ll define a couple of hidden variables that I can access as variables from my callback script. These are a copy of the token that will be issued as part of the slash command request (so I can verify that the service request has come from a known source) and the Slack incoming webhook address.


I can access these environment variables in my script as elements in the Hook['env'] dict. The data package from Slack can be accessed via the Hook['params'] dict.

The service definition begins with the declaration of the service name, which will provide a stub for the callback URL. The automatically generated Home URL is the one that needs to be provided as the callback URL in the Slack slash command configuration.


The code for the service can be specified locally, or pulled in from a (public) gist.


In the service I’ve defined, I make a request over the Parliamentary research briefings dataset (others are available) and return a list of keyword matching briefings as well as links to the briefing document homepage.

import json, re
import urllib2
from urlparse import urlparse
from urllib import urlopen, urlencode

class UKParliamentReader():

    Chat to the UK Parliament API

    def __init__(self):
        """ Need to think more about the structure of this... """

    def qpatch(self,query):
        for a in query.split('or'):
            t.append('({})'.format(a.strip().replace(' ',' AND ')))
        return ' OR '.join(t)

    def search_one(self,query, typ='Research Papers',page=0,ps=100):
        url='{}?{}'.format(url, urlencode(urlargs))
        data =json.loads(urlopen(url).read())
        for i in data['result']['items']:
            response.append("{} [{}]".format( i['title'],i['identifier']['_value']))
        return response

    def search_all(self,query, typ='Research Papers',ps=100):

    def responder(self,hook):
        r2="; \n".join(r)
        payload={"channel": "#slashtest", "username": "parlibot",
                 "text":"I know about the following Parliamentary research papers:\n\n {}".format(r2)}
        req = urllib2.Request(ukparl_url)
        req.add_header('Content-Type', 'application/json')
        response = urllib2.urlopen(req, json.dumps(payload))

if Hook['params']['token'] == Hook['env']['ukparl_token']:

With everything now set up, I can make use of the slash command:


Next up, I’ll see if I can work out a similar recipe for using Amazon AWS Lambda functions…

See also: Chatting With ONS Data Via a Simple Slack Bot

NOTE: this recipe was inspired by the following example of using to create a Javascript powered slash command handler: Making custom Slack slash commands with

onFormSubmit – Raising Web Scale Events in Google Spreadsheets

What happens if you want to actually do something with a particular response from a web survey form at the time it is submitted, other than just collect it?

One of the handy things about Google Spreadsheets is the ability to create interactive web survey forms that can collect data that is then posted into a corresponding spreadsheet. Around the time of the Google I/O event, several event related features were released as part of Google Apps script, the javascript scripting framework that supports an increasing number of Google apps. And by “event” I don’t mean something like the upcoming Isle of Wight Festival – I mean computational events, that can be used to trigger other computational actions…

One of the new events is onFormSubmit, which I finally got round to playing with last night. Here’s my “Hello World” example:

So here’s the code:

//Test function for Google Apps Script onFormSubmit
//Two sheets in a single spreadsheet doc
//First sheet corresponds to form
//Second sheet just displays one of the elements from the most recent form submission
// the function testOnSub() has a trigger associated with it: 'From spreadsheet' 'On form submit'
function testOnSub() {
  var ss = SpreadsheetApp.openById(SPREADSHEET_KEY);
  var sheet=ss.getSheets()[1];
  var form=ss.getSheets()[0];
  var lr=form.getLastRow();
  var el=form.getRange(lr,2,1,1).getValue();
  var t=el;

Here’s how to set it…

Google apps script - spreadsheet events

What next? Earlier this week, I watched a compelling presentation from @progrium, based around the following slide deck:

Among the handy tools demonstrated (I loved the idea of clickhooks (src), clickable links with webhook callback actions) was a webhook debugging tool, postbin. What this tool does is just capture and redisplay stuff that is posted to it… which makes it ideal for a quick demo…

So for example, suppose I have a Google form set up, and I want to perform a particular action using a third party webservice on some element contained in the form submission, or maybe only on certain items according to what information was submitted via the form, as soon as the form is submitted. Here’s one way of doing that (code on gisthub):

// Simple spreadsheet, with first sheet containing form submission repsonses
// when the form is submitted:
// 1) grab the latest response,
// 2) post it to a third party service via an HTTP POST
function testWebhook() {
  var ss = SpreadsheetApp.openById(SPREADSHEET_ID);
  var form=ss.getSheets()[0];
  var lr=form.getLastRow();
  var el=form.getRange(lr,2,1,1).getValue();
  var t=el;
  //The following escape palaver is gleaned from a Google help forum...
  var p="val1="+encodeURIComponent(t).replace(/%20/g, "+")+"&val2="+encodeURIComponent(form.getRange(lr,3,1,1).getValue()).replace(/%20/g, "+");

  // Here's where we do the callback...
  var x=UrlFetchApp.fetch('',{method: 'post', payload: p});

Attach the on form submit trigger event to the function, and here’s the response when we submit a form:

Pastebin response from Google spreadsheet onFormSubmit callback

Clever, eh?

So what does this mean? It means that I can set up a Google Survey form and as soon as anyone posts a submission, I can process it, either within the Google Apps environment using Google Apps script, or using third party services that accept an HTTP post input.

As Jeff Lindsay suggests, the evented web is increasingly a reality…

PUSH and the Feed-Based AutoResponder

Several times over the last few years, I’ve posted about “daily RSS feeds” or feeds that point to static content that can be scheduled so that the content is delivered to you in a drip feed fashion over an extended period of time via the feed. One problem with this is what to do if you want the next item delivered ahead of schedule? For example, you read one item, and immediately want the next, but it’s not slated for delivery for another 22 hours?

Ideally, what we want is something like an auto-responder. (You’re probably familiar with auto-responders in the context of email or SMS: send a message to a particular address or number, and immediately get a response back.)

So here’s something I’ve started mulling over, now things like the Google Feed API support PubSubHubbub/PUSH (the ability to literally push content to another service once it is published).

Suppose I subscribe to an RSS feed in Google Reader. The feed has a URL that is unique to me. The feed points to “static” content, such as a report. When I subscribe to the feed, it contains just the first item. Now suppose I read the post, and click a link that corresponds to a “feed management” URL which results in another item being added to the feed, and a ping being sent to a PubSubHubbub listener hub. My feed reader, which is also signed up to the hub, is informed that new content is available and it appears in my feed reader, almost immediately. That is, we have an autoresponding feed :-)

PS This is related… Back last year when I was in Cambridge as part of the Arcadia Fellowship programme, I took the opportunity to grab a coffee with Jim Downing and Peter Murrary Rust. I remember Peter asked what was particulalry interesting me at the time, and it was the notion of webhooks. I don’t think I’ve blogged explicitly about these before, but the idea behind a webhook is simple: it’s an HTTP callback…

(I’m just imagining blank faces…)

Here’s something you might know… there’s a way of getting content into a web page once it’s loaded using the HTML <script> tag. This tag can be used to load Javascript content into a web page dynamically using a javascript object (the object contains the content you want). In order to get hold of the content, you need to wrap it in a Javascript function; then, when the Javascript has loaded, the function is called and does something to the content, like display it in your web page. If you ever hear about JSON, this is what people are talking about: a web page makes a request to a URL for some content or data packaged as a Javascript object, and the webserver on the end of the URL passes that content back. So that the web page can grab hold of the javascript object, it often passes a function name to the webservice. When the webservice passes the object back, it does so as the single argument in the user specified function. This is called a callback. Web page asks URL for data and sets package it in this callback function; webservice gets the data, wraps up in the callback function container, and sends it back to the webpage, which executes the callback function.

So what are webhooks? Webhooks are callbacks too, but rather than wrapping the data in a javascript function call, the callback is made to another webservice, via a URL and HTTP POST, using the requested data. Got that? In a JSON scenario, I ask a web service for data and give it a callback function name; it gets the data, wraps it up in with the function name I provided, and passes it all back to me, at which point I execute the function using the data it was passed back with. In the webhooks scenario, I make a request for data and ask that a service I have identified by a URL is given the data, so that it can do something with it.

And that’s webhooks.

So how do webhooks relate to things like PUSH? Well, if a service offers a PUSH service, don’t you think it would be handy to be able to sign up to that service with a URL of another service that will do something with the data being PUSHed? That’s a webhook…

PPS something I keep meaning to play with: webhooks for WordPress – HookPress.